DETECTION AND LOCATION OF PROBLEMS IN WORK MACHINES USING AN ACOUSTIC SENSORS

Description

1 / 23 DETECTION AND LOCATION OF PROBLEMS IN WORK MACHINES USING AN ACOUSTIC SENSORS FIELD OF TECHNIQUE

[0001] The methods described here generally refer to the operation of a work machine and, more particularly, to the detection and location of problems in a work machine using an acoustic sensor, such as a set of microphones or an acoustic camera. BACKGROUND

[0002] Many problems that occur during the operation of work machines can initially manifest as noises. For example, a medium-sized wheel loader has at least twenty-two large pin joints for load support. The pin in such a pin joint may make a loud noise when there is a problem or when the pin has reached the end of its service life. Intervening as soon as a noise is heard can save costs and prevent bigger problems in the future.

[0003] Normally, a local operator in the cab of a work machine will hear any noise that indicates a problem and trace that noise back to its source to locate the problem. However, remotely operated or autonomous work machines do not have a local operator. Therefore, such noises are likely to go unnoticed. Furthermore, even if there is a local operator, the local operator may not hear the noise (e.g., in a soundproof cab), may not recognize that the noise represents a problem, may not be able to locate the source of the noise, or may not report the noise (e.g., to a supervisor, service technician, etc.).

[0004] International patent publication no. WO / 2011 / 138488 A1 utilizes a microphone installed on the exterior of the cabin of ground-work machinery to capture sounds produced by the machine during operation, compares the captured sounds to a stored fingerprint representing a broken part, and alerts the operator when a broken part is detected. The present Petition 870250070467, dated 11 / 08 / 2025, page 11 / 48 2 / 23 disclosure aims to overcome one or more deficiencies in the state of the art discovered by the inventor. SUMMARY

[0005] In one embodiment, a method comprises using at least one hardware processor in a work machine to, in real time with the operation of the work machine: receive audio data captured by one or more acoustic sensors; apply spatial filtering to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a different location in the work machine than any other of the one or more portions of the audio data; and for each of the one or more portions of the audio data, apply acoustic filtering to the portion of the audio data, determine whether or not a problem exists based on the acoustic filtering and, upon determining that the problem exists, initiate corrective action.

[0006] In one embodiment, a method comprises using at least one hardware processor in a work machine to, in real time with the operation of the work machine: receive audio data captured by a set of microphones mounted in a cabin of a work machine; apply spatial filtering to the audio data to identify a plurality of portions of the audio data, each of the plurality of portions of the audio data being captured from a different location in the work machine than any other of the plurality of portions of the audio data, and each location corresponding to a component of the work machine;For each of the multiple portions of the audio data, apply acoustic filtering to the audio data portion, determine whether or not a problem exists based on the acoustic filtering, and, if a problem is determined to exist, issue a notification to the operator, identifying the component corresponding to the location from which the audio data portion was captured. Petition 870250070467, dated 11 / 08 / 2025, p. 12 / 48 3 / 23

[0007] In one embodiment, a working machine comprises: a machine body; a working implement; one or more acoustic sensors mounted on the working machine;and a controller configured to, in real time with the operation of the work machine, receive audio data captured by one or more acoustic sensors, apply spatial filtering to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a different location on the work machine than any other of the one or more portions of the audio data, and each location corresponding to a component of the work machine and, for each of the one or more portions of the audio data, apply acoustic filtering to the portion of the audio data, determine whether or not a problem exists based on the acoustic filtering and, upon determining that the problem exists, issue a notification that identifies the component corresponding to the location from which the portion of the audio data was captured. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The details of embodiments of the present disclosure, both as to its structure and operation, can be obtained, in part, by studying the accompanying drawings, in which similar reference numerals refer to similar parts and in which:

[0009] Figure 1 illustrates an example of a working machine according to an embodiment;

[0010] Figure 2 illustrates a process for detecting and locating problems in a work machine using an acoustic sensor according to one embodiment; and

[0011] Figure 3 illustrates an exemplary controller for implementing a process to detect and locate problems in a work machine using an acoustic sensor according to a modality. Petition 870250070467, dated 11 / 08 / 2025, p. 13 / 48 4 / 23 DETAILED DESCRIPTION

[0012] The detailed description presented below, in connection with the accompanying drawings, is intended to be a description of various embodiments and is not intended to represent the only embodiments in which the invention can be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the embodiments. However, it will be evident to those skilled in the art that embodiments of the invention can be practiced without these specific details.

[0013] In some cases, well-known structures and components are shown in a simplified form to facilitate description. For clarity and ease of explanation, some surfaces and details may be omitted in the present description and figures. It should also be understood that the various components illustrated in this document are not necessarily drawn to scale. In other words, the features revealed in various embodiments may be implemented using different relative dimensions within and between components than those illustrated in the drawings.

[0014] References in this document to “lateral”, “superior”, “inferior”, “frontal”, “rear”, “above”, “below”, “front”, “back”, “left”, “right” and the like are used for convenience of understanding to convey the relative positions of various components with respect to each other and do not imply any specific orientation of those components in absolute terms (e.g., with respect to the external environment or the ground). Furthermore, the terms “respective” and “respectively” mean an association between members of a group of first components and members of a group of second components. For example, the expression “each component A connected to a respective component B” would mean A1 connected to B1, A2 connected to B2… and AN connected to BN. Also, as used in this document, a reference numeral with an attached letter will be used for… Petition 870250070467, dated 11 / 08 / 2025, page 14 / 48 5 / 23 refers to a specific component, while the same reference numeral without any attached letter will be used to collectively refer to a plurality of the component or to refer to a generic or arbitrary occurrence of the component.

[0015] Figure 1 illustrates an example of a work machine 100 according to an embodiment. The work machine 100 is illustrated as a wheel loader. However, the work machine 100 can be any type of work machine, including a dump truck, asphalt paver, backhoe loader, skid steer loader, track loader, cold milling machine, motor grader, compactor, crawler tractor, electric cable shovel, forestry machine, hydraulic mining shovel, material handler, pipe layer, road reclaimer, telescopic handler, scraper tractor or similar. The work machine 100 can be operated by a local operator, a remote operator (e.g., via a wireless communication network) and / or autonomously (e.g., semi-autonomous with some human supervision or fully autonomous without human supervision).

[0016] As illustrated, the working machine 100 may comprise a machine body 110, a working implement 120, a cabin 130 supported by the machine body 110, and one or more ground interaction members 140. The machine body 110 may comprise an internal combustion engine, an electric motor (e.g., battery-powered), or the like. The machine body 110 may be connected to the working implement 120 by means of a joint 115. The joint 115 may allow the articulation or rotation of the working implement 120, relative to the machine body 110, around a geometric axis A. This joint may allow the working machine 100 to be steered and / or the working implement 120 to be operated at an angle relative to the machine body 110.

[0017] The work implement 120 is illustrated as a bucket. However, the work implement 120 may comprise any apparatus capable of Petition 870250070467, dated 11 / 08 / 2025, page 15 / 48 6 / 23 perform work under the control of an operator or autonomous control system. Examples of other work implements 120 include, without limitation, a bucket arm, a tipper bucket, a grader, a milling machine, a drill press, a crane, a forklift, and the like.

[0018] The cabin 130 may comprise a seat for the operator, one or more input devices (e.g., joysticks, levers, buttons, pedals, etc.) for controlling the work machine 100, a display console (e.g., comprising a touch-sensitive panel display) and / or the like. In an embodiment in which the work machine 100 is operated remotely and / or autonomously and the work machine 100 does not support local operation, the cabin 130 may be omitted. The operation of the work machine 100 may comprise an operator or autonomous control system that accelerates and decelerates (e.g., brakes) the work machine 100, drives the work machine 100, operates the work implement 120 and / or the like.

[0019] The ground interaction member (or members) 140 is configured to move the machine body 110 and / or the work implement 120 relative to the ground. The ground interaction members 140 are illustrated as wheels, but may comprise other types of components for moving the machine body 110 and / or the work implement 120 relative to the ground, such as tracks, rollers, and / or the like. The ground interaction members 140 may be driven by a transmission system, which, in turn, is driven by an internal combustion engine or electric motor within the machine body 110. The work machine 100 may comprise ground interaction members 140A that support the machine body 110 and / or ground interaction members 140B that support the work implement 120.

[0020] The working machine 100 may comprise a controller 150 (for example, within the machine body 110, working implement 120 and / or cabin 130), which may be an electronic control unit (ECU). The controller 115 may be communicatively coupled (for example, by Petition 870250070467, dated 11 / 08 / 2025, page 16 / 48 7 / 23 wired or wireless communications) to input devices, inside the cab 130 or at a remote terminal, and to one or more subsystems of the work machine 110, including one or more actuators (e.g., valves, hydraulic cylinders, etc.) within the work implement 120. The controller 150 may receive control inputs from a local or remote operator. Alternatively or additionally, the controller 150 may comprise or be communicatively coupled to an autonomous control system that generates control inputs automatically. In either case, the controller 150 may control one or more of the actuators on the work machine 100 according to the control inputs received.

[0021] The controller 150 can also be communicatively coupled to one or more sensors 160 within the work machine 100. Although the sensors 160 have been illustrated as sensors 160A, 160B, 160C and 160D, it should be understood that these are given as arbitrary examples. In reality, the sensors 160 may comprise any number of sensors at any of several locations within and around the work machine 100. The sensors 160 may include any type of sensor or set of sensors capable of measuring values ​​of one or more parameters of one or more subsystems of the work machine 100 and / or the external environment of the work machine 100. Examples of such parameters include, without limitation, the position of one or more components of the work machine 100, motor speed, machine speed, pressure of a fluid (e.g., fuel, oil, coolant, hydraulic fluid, etc.).), fluid flow rate, fluid temperature, fluid contamination level, fluid viscosity, electric current, electric voltage, battery state of charge, fluid consumption rates, load level, transmission emission ratio, slippage, inclination, traction, mileage, ambient temperature and / or similar. Of particular relevance for certain embodiments, the sensor (or sensors) 160 may emit one or more operational parameters, which represent a position of each among a few. Petition 870250070467, dated 11 / 08 / 2025, page 17 / 48 8 / 23 or more components 170 of the work machine 100, such as a valve, hydraulic cylinder, pin joint, pin and / or the like. The controller 150 can collect these parameters, including these operating parameters, from the sensors 160 and process them as described elsewhere in this document.

[0022] For illustrative purposes, several components 170 are illustrated, including components 170A, 170B, 170C, 170D, and 170E. In the illustrated example, components 170 comprise a plurality of pins that fasten two or more components together at a joint. For example, component 170A is a pin that joins the machine body 110 and the work implement 120 at the articulation 115, component 170B is a pin that joins a bucket arm to a work implement chassis 120, component 170C is a pin that joins the piston of a hydraulic cylinder to a pivoting component, and so on. However, it should be understood that component 170 may refer to any arbitrary component of interest in the work machine 100, including other types of components besides pins, such as valves, hydraulic cylinders, and / or the like. Specifically, a component 170 could be any component that might manifest a problem through sound.

[0023] The work machine 100 may comprise one or more acoustic sensors 180 that are configured to capture audio data. The acoustic sensors 180 may comprise a set of microphones, an acoustic camera and / or the like. The acoustic sensors 180 may be mounted in the cab 130 of the work machine 100, for example, on the top of the cab 130 of the work machine 100. Alternatively or additionally, the acoustic sensors 180 may be mounted in another position on the work machine 100, such as on the machine body 110, inside an engine or engine compartment of the machine body 110, on the work implement 120, on the front or rear of the cab 130 and / or the like. However, it is generally advantageous that the mounting location of the acoustic sensors 180 be external to the parts of the work machine 100 that are prone to the accumulation of dirt and other Petition 870250070467, dated 11 / 08 / 2025, page 18 / 48 9 / 23 debris, so as to keep the acoustic sensors relatively free of debris that could interfere with the capture of audio data. The acoustic sensors 180 can be positioned and / or oriented to capture noise from a plurality of components 170 of interest in a plurality of locations on the work machine 100.

[0024] Figure 2 illustrates a process 200 for detecting and locating problems in a work machine 100 using an acoustic sensor 180 according to one embodiment. The process 200 can be implemented by the controller 150, in real time with the operation of the work machine 100. As used in this document, the term “real time” should be understood as events that occur simultaneously, as well as events that are temporally spaced due to common latencies in processing, communications, memory access and / or the like. The term “problem” will be used, in general, to refer to an issue with a component 170, but should be broadly understood to include immediate issues, such as a partial or total failure of a component 170, and non-immediate issues, such as a degree of wear of a component 170.

[0025] Although process 200 is illustrated with a certain arrangement and ordering of subprocesses, process 200 can be implemented with fewer, more, or different subprocesses and a different arrangement and / or ordering of subprocesses. Furthermore, it should be understood that any subprocess that does not depend on the completion of another subprocess can be executed before, after, or in parallel with that other independent subprocess, even if the subprocesses are described or illustrated in a specific order.

[0026] Subprocess 210 can determine whether or not process 200 should be terminated. Process 200 can run continuously, in real time, while worker machine 100 is operational (for example, from the moment worker machine 100 is turned on until the moment worker machine 100 is turned off). Alternatively or additionally, the process Petition 870250070467, dated 11 / 08 / 2025, page 19 / 48 10 / 23 200 can be activated and / or deactivated by a local operator (e.g., via an input device inside cabin 130), a remote operator (e.g., via an input device on a remote terminal), an autonomous control system, and / or similar. Thus, process 200 can terminate when work machine 100 is switched off and / or when process 200 is deactivated. When the end of process 200 is determined (i.e., “Yes” in subprocess 210), process 200 can terminate. Otherwise, until the end of process 200 is determined (i.e., “No” in subprocess 210), process 200 can proceed to subprocess 220.

[0027] Subprocess 220 can determine whether or not new audio data has been captured by acoustic sensors 180. In one embodiment, audio data can be transmitted from acoustic sensors 180 in real time according to a sampling rate. In this case, subprocess 220 can determine that new audio data has been captured at the end of each time interval defined by the sampling rate. In an alternative embodiment, audio data can be collected only when the noise captured by acoustic sensors 180 satisfies one or more criteria (e.g., exceeds a predefined volume limit). In this case, subprocess 220 can determine that new audio data has been captured whenever audio data that satisfies these criteria is collected by acoustic sensors 180. In either case, upon determining that new audio data has been captured (i.e., “Yes” in subprocess 220), process 200 can proceed to subprocess 230.Otherwise, if it is not determined that new audio data was captured (i.e., “No” in subprocess 220), process 200 can return to subprocess 210.

[0028] Subprocess 230 can receive audio data captured by acoustic sensors 180 and apply spatial filtering to the audio data to identify one or more portions of the audio data, each representing a different location on the working machine 100. In a preferred embodiment, Petition 870250070467, dated 11 / 08 / 2025, page 20 / 48 11 / 23 Spatial filtering identifies a plurality of portions of the audio data, each representing a different location on the working machine 100 than any other among the plurality of portions of the audio data. Each location, represented by a portion of the audio data, may correspond to a specific component 170 of the working machine 100.

[0029] Spatial filtering can determine from which direction each portion of the audio data was captured. In this case, each direction can be mapped to a location on the work machine 100, based on the position of each component 170, relative to the acoustic sensors 180. In the case of one or more components 170 moving relative to the acoustic sensors 180, or being on another component (e.g., work implement 120) that moves relative to the acoustic sensors 180, the component 170 that corresponds to a direction from which a portion of the audio data was captured can be determined based on the relative positions of the components 170, as determined, for example, from positional parameters emitted by one or more sensors 160. In other words, spatial filtering can track the kinematic location of one or more components 170 to be monitored based on the emission from the sensors 160.Spatial filtering can retain audio data, such as portions of audio data, from directions that intersect with the location of a component being monitored for problems, while filtering (e.g., canceling) sound from any other directions. Thus, after spatial filtering, only portions of the audio data that represent components to be monitored remain for further processing.

[0030] In one embodiment, the acoustic sensors 180 comprise an array of microphones. An array of microphones comprises a plurality of microphones operating in tandem. The plurality of microphones may comprise omnidirectional microphones and / or directional microphones distributed around the perimeter of the array of microphones. Spatial filtering may comprise the processing (e.g., by the controller 150) of the signals. Petition 870250070467, dated 11 / 08 / 2025, page 21 / 48 12 / 23 sound patterns (i.e., audio data) captured by the plurality of microphones in the microphone array to locate the origin (e.g., component 170) of each one or more sounds in the audio data.

[0031] In an alternative or additional embodiment, the acoustic sensors 180 may comprise an acoustic camera. An acoustic camera generally comprises a set of microphones and an optical camera. The microphone array may be used, as described above, to capture audio data and locate the origin of each of one or more sounds in the audio data, and the optical camera may be used to map the origin of each sound to two-dimensional image data captured by the optical camera. In an embodiment using an acoustic camera, this two-dimensional image data may be provided to a local operator (e.g., on a display inside the cabin 130) or to a remote operator (e.g., on a display on a remote terminal) to assist the operator in visually identifying the component 170 to which each portion of the audio data belongs.

[0032] In one embodiment, spatial filtering may comprise varying one or more operating parameters of the working machine 100 and using the resulting variation of the sound in the audio data to determine which component 170 is the source of the sound. For example, the controller 150 may actuate a pump or other device to vary the pressure of a fluid through a valve, according to a predefined pattern, and determine that when the sound in the audio data varies according to a similar or corresponding pattern, the valve is the component 170 that is producing the sound. More generally, the controller 150 may actuate any component 170 to be monitored or actuate a component that impacts component 170, according to a predefined pattern, and determine whether component 170 is the source of a sound in the audio data when the sound in the audio data varies according to a pattern that matches the predefined pattern.In this case, for spatial filtering, the 100-piece machine can be placed in a diagnostic mode, in which... Petition 870250070467, dated 11 / 08 / 2025, page 22 / 48 13 / 23 The operation of the work machine 100 is interrupted, in order to prevent the operator from affecting the operation of components 170 during spatial filtering and vice versa.

[0033] Subprocess 240 can iterate through each portion of the audio data identified in subprocess 230. In other words, subprocesses 240-270 can be executed for each of the one or more portions of audio data emitted by spatial filtering in subprocess 230. When another portion of audio data, representing another location, still needs to be considered (i.e., “Yes” in subprocess 240), process 200 can proceed to subprocess 250. Otherwise, once all portions of audio data have been processed (i.e., “No” in subprocess 240), process 200 can return to subprocess 210.

[0034] Subprocess 250 can apply acoustic filtering to the portion of audio data currently being processed. Acoustic filtering can transform audio data, filter (e.g., cancel or otherwise delete) certain sounds from the audio data, enhance certain sounds in the audio data, and / or otherwise manipulate the audio data to allow a determination to be made as to whether or not a problem exists. For example, acoustic filtering might comprise a bandpass filter that isolates one or more frequency bands in the portion of audio data currently being processed, while canceling all other frequency bands in the portion of audio data currently being processed.As another example, acoustic filtering may involve comparing the portion of audio data currently being processed with a reference acoustic standard to determine whether or not the reference acoustic standard exists in the portion of audio data. As yet another example, acoustic filtering may involve determining one or more operating parameters of the working machine 100 that coincide with a moment (e.g., at the same time or close to it) when the portion of audio data currently being processed is being processed. Petition 870250070467, dated 11 / 08 / 2025, page 23 / 48 14 / 23 processed, captured, and noise filtering of the audio data portion based on operational parameters. Examples of such operational parameters include, but are not limited to, engine speed, pump speed, pump displacement, transmission change, valve actuation, tilt actuation, elevation actuation, third function valve actuation, fourth function valve actuation, steering actuation, braking, ground speed, and the like.

[0035] Subprocess 260 can determine whether or not a problem exists based on the acoustic filtering in subprocess 250. In particular, subprocess 260 can determine whether or not the acoustically filtered portion of the audio data currently being processed represents an issue with a component 170 that corresponds to that portion of the audio data, as determined by the spatial filtering in subprocess 230. In other words, subprocess 260 can detect a problem with the corresponding component 170 based on the portion of the audio data captured from the location of that component 170. If it determines that the problem exists (i.e., “Yes” in subprocess 260), process 200 can proceed to subprocess 270. Otherwise, if it does not determine that the problem exists (i.e., “No” in subprocess 260), process 200 can return to subprocess 240.

[0036] In an embodiment in which the acoustic filtering comprises a bandpass filter, subprocess 260 may comprise determining whether or not an audio feature exists within the frequency bands emitted by the bandpass filter. This audio feature may comprise the presence of any sound within the frequency band(s), the presence of a reference acoustic pattern within the frequency band(s), a sound magnitude within the frequency band(s) exceeding a predefined limit, and / or the like. It should be understood that, in this case, the presence of this audio feature in the frequency band(s) emitted by the bandpass filter indicates that the corresponding component 170 is emitting a sound. Petition 870250070467, dated 11 / 08 / 2025, page 24 / 48 15 / 23 indicates a problem with component 170. Thus, when the audio characteristic exists within the frequency range (or ranges) emitted by the bandpass filter, subprocess 260 can determine that there is a problem (i.e., “Yes” in subprocess 260). Otherwise, subprocess 260 can determine that there is no problem (i.e., “No” in subprocess 260).

[0037] In another embodiment where acoustic filtering comprises comparing the portion of audio data currently being processed with a reference acoustic standard, subprocess 260 may comprise determining whether or not the portion of audio data corresponds to the reference acoustic standard. A reference acoustic standard may comprise a frequency, volume, pitch, and / or any other audio characteristic. When the reference acoustic standard exists within the portion of audio data, as determined by the comparison, subprocess 260 may determine that there is a problem (i.e., “Yes” in subprocess 260). Otherwise, subprocess 260 may determine that there is no problem (i.e., “No” in subprocess 260).

[0038] In one embodiment, subprocess 260 can determine whether or not a problem exists based on both acoustic filtering and whether component 170, corresponding to the origin of the portion of audio data being processed at the moment, was moving at the time the portion of audio data was captured. For example, subprocess 260 can determine the component 170 of worker machine 100 that corresponds to the location on worker machine 100 from which the portion of audio data was captured, determine whether or not component 170 was moving at the time the portion of audio data was captured based on sensor data from one or more sensors 160 on worker machine 100, and determine whether or not the problem exists based on acoustic filtering and the determination of whether or not component 170 was moving at the time the portion of audio data was captured. More generally, in this embodiment, the Petition 870250070467, dated 11 / 08 / 2025, page 25 / 48 16 / 23 subprocess 260 can determine that a problem exists (i.e., “Yes” in subprocess 260) when both the acoustically filtered portion of the audio data indicates that the problem exists (for example, using any acoustic filtering described in this document) and the sensor data indicates that component 170, corresponding to that portion of the audio data, was simultaneously in motion or otherwise impacted by simultaneous motion.

[0039] Subprocess 270 can initiate corrective action when subprocess 260 determines that a problem exists and then return to subprocess 240. The corrective action may comprise issuing a notification of the problem detected in subprocess 260. In the case of a local or remote operator, the notification may be sent to a display (e.g., touch panel display) or other visual indication (e.g., indicator light), tactile device (e.g., within a joystick or other control device), and / or other device within the cabin 130 or on a remote terminal, respectively. Alternatively or additionally, the notification may be sent to a supervisor, service technician, or other recipient to ensure that the problem is not simply ignored by the operator.In the case of autonomous control, the notification may comprise a message between processes that is sent to the autonomous control system (e.g., within controller 150), in order to mitigate or replace the loss of operator function.

[0040] In one embodiment, subprocess 270 comprises determining the component 170 of the working machine 100 that corresponds to the location on the working machine 100 from which the portion of the audio data was captured. In this case, the corrective action notification may identify the determined component 170 from which the portion of the audio data was captured. Notably, component 170 may already have been, at least essentially, identified by spatial filtering in subprocess 230. In particular, each location that corresponds to a portion of the audio data, emitted by the filtering. Petition 870250070467, dated 11 / 08 / 2025, page 26 / 48 17 / 23 spatial, can be pre-mapped and / or mapped based on positional data from sensor (or sensors) 160, to a specific component 170 that is identified in the notification. In an embodiment in which the notification is displayed (e.g., on a display in the cabin 130 or on a remote terminal), the visual representation of the notification may comprise a name or other identifier of component 170, a visual representation (e.g., image, diagram, etc.) of component 170 in isolation or in the context of the working machine 100, for example, as a real-time image or video stream of component 170 captured by an acoustic camera or other camera and / or similar.

[0041] Figure 3 illustrates an exemplary controller 150 for implementing process 200 to detect and locate problems in a work machine 100 using an acoustic sensor 180 according to one embodiment. As mentioned elsewhere in this document, the controller 150 may comprise or consist of an electronic control unit (ECU) within the work machine 100.

[0042] The 150 controller may comprise one or more 310 processors. The 310 processor (or processors) may comprise a central processing unit (CPU). Additional processors may be provided, such as a graphics processing unit (GPU), an auxiliary processor for managing input / output, an auxiliary processor for performing floating-point mathematical operations, a special-purpose microprocessor with an architecture suitable for fast execution of signal processing algorithms (e.g., digital signal processor), a subordinate processor (e.g., back-end processor), an additional microprocessor or controller for dual or multi-processor systems, and / or a coprocessor. These auxiliary processors may be discrete processors or may be integrated into a main 310 processor. Examples of processors that may be used with the 150 controller include, without limitation, any of Petition 870250070467, dated 11 / 08 / 2025, p. 27 / 48 18 / 23 of the processors (e.g., Pentium™, Core i7™, Xeon™, etc.) available from Intel Corporation of Santa Clara, California, any of the processors available from Advanced Micro Devices, Incorporated (AMD) of Santa Clara, California, any of the processors (e.g., A-series, M-series, etc.) available from Apple Inc. of Cupertino, any of the processors (e.g., Exynos™) available from Samsung Electronics Co., Ltd. of Seoul, South Korea, any of the processors available from NXP Semiconductors NV of Eindhoven, Netherlands, and / or similar.

[0043] Processor 310 can be connected to a communication bus 305. Communication bus 305 may include a data channel to facilitate the transfer of information between storage and other peripheral components of controller 150. In addition, communication bus 305 may provide a set of signals used for communication with processor 310, including a data bus, an address bus, and / or a control bus (not shown).The 305 communication bus can comprise any standard or non-standard bus architecture, such as, for example, industry standard architecture (ISA) compliant bus architectures, extended industry standard architecture (EISA), microchannel architecture (MCA), peripheral component interconnect local bus (PCI), standards promulgated by the Institute of Electrical and Electronics Engineers (IEEE), including IEEE 488 general-purpose interface bus (GPIB), IEEE 696 / S-100 and / or similar.

[0044] Controller 150 may comprise main memory 315. Main memory 315 provides storage of instructions and / or other data for software running on processor 310. It should be understood that instructions stored in memory and executed by processor 310 may be written and / or compiled in any suitable language, including, without limitation, C / C++, Java, JavaScript, Perl, Python, Visual Basic, .NET and similar languages. Main memory 315 is typically memory based on Petition 870250070467, dated 11 / 08 / 2025, page 28 / 48 19 / 23 semiconductors, such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other types of semiconductor-based memory include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), and similar types, including read-only memory (ROM).

[0045] Controller 150 may comprise secondary memory 320. Secondary memory 320 is a non-transient, computer-readable medium that has instructions and / or other data for software stored thereon. In this description, the term “computer-readable medium” is used to refer to any non-transient, computer-readable storage medium used to provide computer executable code and / or other data to or within controller 150. Computer software stored in secondary memory 320 is read into main memory 315 for execution by processor 310. Secondary memory 320 may include, for example, semiconductor-based memory such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), and flash memory (block-oriented memory similar to EEPROM).

[0046] The controller 150 may comprise an input / output (I / O) interface 335. The I / O interface 335 provides an interface between one or more components of the controller 150 and one or more input and / or output devices. For example, the I / O interface 335 may receive the output from one or more sensors 160 and / or send control signals to one or more subsystems or other components of the work machine 100.

[0047] The controller 150 may include a communication interface 340. The communication interface 340 allows software to be transferred between the controller 150 and external devices, networks, or other sources and / or destinations of information. For example, instructions and / or other data may be transferred. Petition 870250070467, dated 11 / 08 / 2025, p. 29 / 48 20 / 23 to the 150 controller, through one or more networks, from a network server via a 340 communication interface. Examples of 340 communication interfaces include an integrated network adapter, network interface card (NIC), Personal Computer Memory Card International Association (PCMCIA) network card, bus network adapter, wireless network adapter, Universal Serial Bus (USB) network adapter, modem, wireless data card, communication port, infrared interface, IEEE 1394 firewire, and any other device capable of interfacing the 150 controller with a network or other computing device.The communication interface 340 preferably implements industry-promoted protocol standards, such as Ethernet IEEE 802 standards, Fibre Channel, Digital Subscriber Line (DSL), Asynchronous Digital Subscriber Line (ADSL), Frame Relay, Asynchronous Transfer Mode (ATM), Integrated Digital Services Network (ISDN), Personal Communications Services (PCS), Transmission Control Protocol / Internet Protocol (TCP / IP), Serial Line Internet Protocol / Point-to-Point Protocol (SLIP / PPP), and so on, but may also implement custom or non-standard interface protocols.

[0048] The software transferred by the communication interface 340 is usually in the form of electrical communication signals 355. These signals 355 may be provided to the communication interface 340 via a communication channel 350 between the communication interface 340 and an external system 345.In one embodiment, the 350 communication channel can be a wired or wireless network or any variety of other communication links. The 350 communication channel carries 355 signals and can be implemented using a variety of wired or wireless communication media, including wire or cable, fiber optics, conventional telephone line, cellular telephone link, wireless data communication link, radio frequency (“RF”) link, or infrared link, to name just a few.

[0049] The executable code for the computer is stored in main memory. Petition 870250070467, dated 11 / 08 / 2025, pages 30 / 48 21 / 23 315 and / or in secondary memory 320. The executable computer code may also be received from an external system 345 via the communication interface 340 and stored in main memory 315 and / or secondary memory 320. This executable computer code, when executed by the processor (or processors) 310, may enable the controller 150 to perform the various functions of the disclosed embodiments, including, for example, the process 200. Industrial applicability

[0050] Many problems that occur during the operation of work machines 100 may initially manifest as noises. Normally, a local operator will detect any such noise. However, this is not possible when there is no local operator, for example, in the case of a remotely operated or autonomous work machine 100. Furthermore, even when there is an operator, the operator may not hear the noise, recognize the noise, locate the source of the noise, or report the noise.

[0051] Consequently, the disclosed embodiments utilize one or more acoustic sensors 180, such as a microphone array or acoustic camera, mounted on the work machine 100. The controller 150 of the work machine 100 can automatically monitor the audio data captured by the acoustic sensor (or acoustic sensors) 180, apply spatial filtering to map portions of the audio data to specific components 170, apply acoustic filtering to detect problems with the components 170, and initiate corrective action, such as notifying an operator, supervisor, and / or autonomous control system, upon detecting a problem with a component 170. Thus, the disclosed embodiments correct the situation where the absence of a local operator and / or the failure of a local operator allows an audible problem with a component 170 of a work machine 100 to go unnoticed. Advantageously, detecting, locating, and resolving the problem early can Petition 870250070467, dated 11 / 08 / 2025, page 31 / 48 22 / 23 to prevent larger or more serious issues from occurring in the future.

[0052] As an example, a medium-sized wheel loader has at least twenty-two large pin joints for load support. If a pin in one of these pin joints has a problem, it will usually start making noise. In this case, the acoustic sensor (or acoustic sensors) 180 will capture the noise in the audio data. In subprocess 230, the portion of the audio data that represents this noise can be located at the specific pin that produces the noise, using spatial filtering.Based on the knowledge that this portion of the audio data corresponds to a pin, subprocesses 250 and 260 can determine that the noise, represented by the portion of the audio data, is within a frequency range characteristic of a problem with the pin, correlates to the simultaneous rotation of the pin joint (e.g., determined from positional measurements emitted by sensor(s) 160), follows a rotation of the pin as the pin moves through space relative to the acoustic sensor(s) 180 and / or similar. In this case, subprocess 270 can notify an operator or autonomous control system about the problem, as well as identify the specific pin and / or the location of the specific pin that is the subject of the issue.

[0053] It will be understood that the benefits and advantages described above may relate to one modality or may relate to several modalities. It is intended that aspects described in connection with one modality may be used with the other modalities. Any explanation in connection with one modality applies to similar features of the other modalities, and elements of multiple modalities may be combined to form other modalities. Modalities are not limited to those that solve some or all of the stated problems or to those that have some or all of the stated benefits and advantages.

[0054] The above detailed description is merely illustrative and is not intended to limit the invention or the application and uses of the invention. The embodiments described do not Petition 870250070467, dated 11 / 08 / 2025, pp. 32 / 48 Sections 23 / 23 are limited to use in conjunction with a specific type of work machine. Therefore, although the present embodiments are, for the sake of explanation, represented and described as implemented on a wheel loader, it should be noted that they can be implemented on various other types of work machines where issues or other problems manifest themselves audibly and in various other systems and environments. Furthermore, there is no intention to link to any theory presented in any previous section. It is also understood that the illustrations may include exaggerated dimensions and graphic representations to better illustrate the referenced items shown, and are not considered limiting unless expressly stated as such. Petition 870250070467, dated 11 / 08 / 2025, pages 33 / 48

Claims

1 / 5 CLAIMS 1. A method characterized in that it comprises the use of at least one hardware processor in a work machine to, in real time with the operation of the work machine: receive audio data captured by one or more acoustic sensors; apply spatial filtering to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a different location in the work machine than any other of the one or more portions of the audio data; and to each one or more portions of the audio data, apply acoustic filtering to the portion of the audio data, determine whether or not a problem exists based on the acoustic filtering in the subprocess and, upon determining that the problem exists, initiate a corrective action.

2. A method according to claim 1, characterized in that one or more acoustic sensors comprise an array of microphones.

3. A method according to claim 1, characterized in that one or more acoustic sensors comprise an acoustic chamber.

4. A method according to claim 1, characterized in that one or more acoustic sensors are mounted in a cabin of the working machine.

5. Method, according to claim 1, characterized in that the acoustic filtering comprises a band-pass filter that isolates one or more frequency bands from the audio data portion.

6. Method according to claim 5, characterized in that determining whether the problem exists or not comprises determining whether an audio characteristic exists or not within one or more frequency ranges of the audio data portion. Petition 870250070467, dated 11 / 08 / 2025, pp. 34 / 48 2 / 5 7. A method according to claim 1, characterized in that applying acoustic filtering comprises comparing the portion of the audio data with a reference acoustic standard.

8. A method according to claim 7, characterized in that determining whether the problem exists or not comprises determining whether the portion of the audio data corresponds to the reference acoustic standard or not.

9. A method according to claim 1, characterized in that the acoustic filtering comprises: determining one or more operating parameters of the working machine that coincide with the moment when the portion of the audio data was captured; and filtering noise from the portion of the audio data based on the one or more operating parameters.

10. Method according to claim 9, characterized in that one or more operating parameters comprise one or more of the following: engine speed, pump speed, pump displacement, transmission change, valve action, tilt actuation, lift actuation, third function valve actuation, fourth function valve actuation, steering actuation, braking or ground speed.

11. Method according to claim 1, characterized in that the corrective action comprises issuing a notification of the problem.

12. Method, according to claim 11, characterized in that initiating corrective action comprises determining a component of the work machine that corresponds to the location on the work machine from which the portion of audio data was captured, and in which the notification identifies the determined component.

13. Method according to claim 12, characterized in that the component comprises a pin.

14. Method, according to claim 12, characterized by the fact that the notification comprises an indication in a cabin of the work machine.

15. A method according to claim 1, characterized in that determining whether the problem exists or not comprises: determining a component of the working machine that corresponds to the location on the working machine from which the portion of the audio data was captured; determining whether or not the component was moving at the time the portion of the audio data was captured based on sensor data from one or more sensors on the working machine; and determining whether or not the problem exists based on acoustic filtering and determining whether or not the component was moving at the time.

16. A method characterized by the fact that it comprises the use of at least one hardware processor in a work machine to, in real time with the operation of the work machine: receive audio data captured by a set of microphones mounted in a cabin of the work machine; apply spatial filtering to the audio data to identify a plurality of portions of the audio data, each of which portions of the audio data is captured from a different location in the work machine than any other portion of the audio data, and each location corresponds to a component of the work machine;and to each of the plurality of portions of the audio data, apply acoustic filtering to the portion of the audio data, determine whether or not there is a problem based on the acoustic filtering in the subprocess, and upon determining that the problem exists, send a notification to an operator, in which the notification identifies the component corresponding to the location from which the portion of the audio data was captured.

17. A method according to claim 16, characterized in that the acoustic filtering comprises a bandpass filter that isolates one or more frequency bands from the portion of the audio data, and in that determining whether the problem exists or not comprises determining whether an audio characteristic exists or not within one or more frequency bands of the portion of the audio data.

18. A method according to claim 16, characterized in that the application of acoustic filtering comprises comparing the portion of the audio data to a reference acoustic standard and in that determining whether the problem exists or not comprises determining whether the portion of the audio data corresponds to the reference acoustic standard or not.

19. A method according to claim 16, characterized in that the acoustic filtering comprises: determining one or more operating parameters of the working machine that coincide with the moment when the portion of the audio data was captured; and filtering noise from the portion of the audio data based on one or more operating parameters, wherein the one or more operating parameters comprise one or more of the following: engine speed, pump speed, pump displacement, transmission change, valve actuation, tilt actuation, lift actuation, third function valve actuation, fourth function valve actuation, steering actuation, braking, or ground speed.

20. A working machine characterized by the fact that it comprises: a machine body; a working implement; one or more acoustic sensors mounted on the working machine; and a controller configured to operate in real time with the Petition 870250070467, dated 11 / 08 / 2025, page 1.37 / 48 5 / 5 work machine, receive audio data captured by one or more acoustic sensors; apply spatial filtering to the audio data to identify one or more portions of the audio data, each of which portions of audio data is captured from a different location on the work machine than any other portion of audio data, and each location corresponds to a component of the work machine; and apply acoustic filtering to each portion of audio data, determine whether or not a problem exists based on the acoustic filtering in the subprocess, and if a problem is determined to exist, send a notification identifying the component corresponding to the location from which the portion of audio data was captured. Petition 870250070467, dated 11 / 08 / 2025, p. 38 / 48.

Images