Detection and localization of problems in work machine using acoustic sensors

Abstract

In the case of remote or autonomous operation of a work machine, problems of the work machine that appear as noise may not be noticed. Thus, the disclosed embodiments utilize acoustic sensors mounted on the work machine to monitor noise emanating from different locations on the work machine, which noise corresponds to known components. Spatial filtering may be used to map noise to particular components, while acoustic filtering may be used to identify noise indicative of problems or other problems of those components. When a problem is detected, an action, such as notifying an operator or an autonomous control system, may be initiated to remedy the problem.

Description

Technical Field

[0001] The embodiments described herein generally relate to the operation of work machines, and more specifically to the use of acoustic sensors, such as microphone arrays or acoustic cameras, to detect and locate problems in work machines. Background Technology

[0002] Many problems that arise during the operation of machinery may initially manifest as noise. For example, medium-duty wheel loaders have at least 22 large load-bearing pin joints. In such joints, the pins can generate significant noise when a problem exists or when the pins have reached the end of their lifespan. Intervening as soon as the noise is heard can save costs and completely prevent further problems.

[0003] Typically, a local operator in the cabin of a working machine will hear any noise indicating a problem and use it to pinpoint the root cause. However, remotely operated or autonomous working machines do not have a local operator. Therefore, such noise is likely to go unnoticed. Furthermore, even if a local operator is present, they may not hear the noise (e.g., in a soundproof cabin), may not identify that the noise represents a problem, may not be able to locate the noise source, or may not be able to report the noise (e.g., to a supervisor, service technician, etc.).

[0004] International Patent Publication No. WO / 2011 / 138488A1 utilizes a microphone mounted on the exterior of the cab of earthmoving machinery to capture sounds generated by the machinery during operation, compares the captured sounds with stored fingerprints indicating part breakage, and warns the operator when a part breakage is detected. This invention aims to overcome one or more deficiencies in the prior art discovered by the inventors. Summary of the Invention

[0005] In one embodiment, a method includes using at least one hardware processor in a work machine to, in real time, along with the operation of the work machine: receive audio data captured by one or more acoustic sensors; apply spatial filtering to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a location on the work machine that is different from any other portion of the one or more portions of the audio data; and for each of the one or more portions of the audio data, apply acoustic filtering to the portion of the audio data, determine based on the acoustic filtering whether a problem exists, and, when a problem is determined to exist, initiate remedial action.

[0006] In one embodiment, a method includes using at least one hardware processor in a work machine to, in real time, in conjunction with the operation of the work machine: receive audio data captured by a microphone array mounted on the hull of the work machine; apply spatial filtering to the audio data to identify multiple portions of the audio data, each of the multiple portions of the audio data being captured from a location on the work machine that differs from any other portion of the multiple portions of the audio data, and each location corresponding to a component of the work machine; and for each of the multiple portions of the audio data, apply acoustic filtering to the portion of the audio data, determine, based on the acoustic filtering, whether a problem exists, and, when a problem is determined to exist, output a notification to an operator that identifies the component corresponding to the location from which the portion of the audio data was captured.

[0007] In one embodiment, a working machine includes: a machine body; a working implement; one or more acoustic sensors mounted on the working machine; and a controller configured to receive, in real time, audio data captured by the one or more acoustic sensors in conjunction with the operation of the working machine, apply spatial filtering to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a location on the working machine that is different from any other portion of the one or more portions of the audio data, and each location corresponding to a component of the working machine, and for each of the one or more portions of the audio data, applying acoustic filtering to the portion of the audio data, determining whether a problem exists based on the acoustic filtering, and when a problem is determined to exist, outputting a notification identifying the component corresponding to the location from which the portion of the audio data was captured. Attached Figure Description

[0008] Details of embodiments of the invention, relating to their structure and operation, can be gathered in part by studying the accompanying drawings, wherein like reference numerals indicate like parts, and wherein:

[0009] Figure 1 An example operating machine according to an embodiment is shown;

[0010] Figure 2 The process for detecting and locating problems in a working machine using acoustic sensors, according to an embodiment, is illustrated; and

[0011] Figure 3 An example controller is shown according to an embodiment for implementing a process of detecting and locating problems in a working machine using acoustic sensors. Detailed Implementation

[0012] The specific embodiments described below with reference to the accompanying drawings are intended to illustrate various examples and are not intended to represent only embodiments in which the invention can be practiced. The detailed description includes specific details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments of the invention can be practiced without these specific details.

[0013] In some cases, for the sake of brevity, well-known structures and components are shown in a simplified form. For clarity and ease of explanation, some surfaces and details may be omitted in this specification and the accompanying drawings. It should also be understood that the various components shown herein are not necessarily drawn to scale. In other words, features disclosed in various embodiments may be implemented using relative dimensions within and between components that differ from those shown in the accompanying drawings.

[0014] The use of terms such as “side,” “top,” “bottom,” “front,” “back,” “above,” “below,” “forward,” “backward,” “left,” and “right” in this document is for ease of understanding and to convey the relative position of the various components with respect to each other, without implying any particular orientation of those components in absolute terms (e.g., relative to the external environment or the ground). Furthermore, the terms “correspondingly” and “respectively” indicate the association between members of a first set of components and members of a second set of components. For example, the phrase “each component A connected to the corresponding component B” would mean A1 connected to B1, A2 connected to B2, ..., and AN connected to BN. Additionally, as used herein, reference numerals with additional letters will be used to indicate a specific component, while the same reference numerals without any additional letters will be used to collectively indicate multiple components, or to indicate a general or arbitrary instance of components.

[0015] Figure 1 An example work machine 100 according to an embodiment is shown. The work machine 100 is shown as a wheel loader. However, the work machine 100 can be any type of work machine, including dump trucks, asphalt pavers, backhoe loaders, skid-steer loaders, track loaders, cold planers, automatic graders, compactors, bulldozers, electric rope shovels, forestry machinery, hydraulic mining shovels, material handling machines, pipelaying machines, road reclaimers, telescopic forklifts, towed scrapers, etc. The work machine 100 can be operated by a local operator, a remote operator (e.g., via a wireless communication network), and / or autonomously (e.g., semi-autonomously with some human supervision, or fully autonomously without human supervision).

[0016] As shown in the figure, the work machine 100 may include a machine body 110, a work implement 120, a cabin 130 supported by the machine body 110, and one or more ground engagement members 140. The machine body 110 may include an internal combustion engine, an electric motor (e.g., driven by a battery pack), etc. The machine body 110 may be connected to the work implement 120 via a link 115. The link 115 allows the work implement 120 to be hinged or rotated relative to the machine body 110 about axis A. This hinge allows the work machine 100 to be manipulated and / or allows the work implement 120 to be operated at an angle relative to the machine body 110.

[0017] The work implement 120 is shown as a bucket. However, the work implement 120 may include any equipment capable of performing work under the control of an operator or an autonomous control system. Other examples of work implements 120 include, but are not limited to, bucket arms, bucket beds, graders, planers, drilling rigs, cranes, forklifts, etc.

[0018] Cabin 130 may include an operator's seat, one or more input devices (e.g., joysticks, levers, buttons, pedals, etc.) for controlling the work machine 100, a display console (e.g., including a touch panel display), etc. In embodiments where the work machine 100 is remotely and / or autonomously operated and the work machine 100 does not support local operation, cabin 130 may be omitted. Operation of the work machine 100 may include accelerating and decelerating (e.g., braking) the work machine 100, steering the work machine 100, operating the work implement 120, etc., by an operator or autonomous control system.

[0019] Ground engagement member 140 is configured relative to ground moving machine body 110 and / or work implement 120. Ground engagement member 140 is shown as a wheel, but may include other types of components for use relative to ground moving machine body 110 and / or work implement 120, such as tracks, rollers, etc. Ground engagement member 140 may be driven by a drivetrain, which in turn is driven by an internal combustion engine or electric motor within machine body 110. Work machine 100 may include ground engagement member 140A supporting machine body 110 and / or ground engagement member 140B supporting work implement 120.

[0020] The working machine 100 may include a controller 150 (e.g., within the machine body 110, the working implement 120, and / or the cabin 130), which may be an electronic control unit (ECU). The controller 150 may be communicatively coupled (e.g., via wired or wireless communication) to input devices within the cabin 130 or at a remote terminal, and coupled to one or more subsystems of the working machine 110, including one or more actuators (e.g., valves, hydraulic cylinders, etc.) within the working implement 120. The controller 150 may receive control input from a local or remote operator. Alternatively or additionally, the controller 150 may include an autonomous control system that automatically generates control input, or be communicatively coupled to an autonomous control system that automatically generates control input. In either case, the controller 150 may control one or more actuators in the working machine 100 based on the received control input.

[0021] The controller 150 can also be communicatively coupled to one or more sensors 160 within the work machine 100. While sensors 160 are shown as sensors 160A, 160B, 160C, and 160D, it should be understood that these are provided as arbitrary examples. In practice, sensors 160 can include any number of sensors located at any of a variety of locations within and around the work machine 100. Sensors 160 can include any type of sensor or sensor array capable of measuring the values ​​of one or more parameters of one or more subsystems of the work machine 100 and / or the external environment of the work machine 100. Examples of such parameters include, but are not limited to: the position of one or more components of the work machine 100, engine speed, machine speed, pressure of fluids (e.g., fuel, oil, coolant, hydraulic fluid, etc.), fluid flow rate, fluid temperature, fluid contamination level, fluid viscosity, current, voltage, battery state of charge, fluid consumption rate, load level, transmission output ratio, slip, gradient, traction, mileage, ambient temperature, etc. Of particular relevance to certain embodiments is that sensor 160 may output one or more operating parameters, such as valves, hydraulic cylinders, pin joints, pins, etc., representing the position of each of one or more components 170 of the working machine 100. Controller 150 may collect these parameters (including these operating parameters) from sensor 160 and process them as described elsewhere herein.

[0022] For illustrative purposes, various components 170 are shown, including components 170A, 170B, 170C, 170D, and 170E. In the illustrated example, component 170 includes multiple pins that secure two or more components together in a joint. For example, component 170A is a pin that engages the machine body 110 and the work implement 120 at the link 115; component 170B is a pin that engages the bucket arm to the chassis of the work implement 120; component 170C is a pin that engages the piston of a hydraulic cylinder to a pivotable component, and so on. However, it should be understood that component 170 can indicate any identifiable component in the work machine 100, including other types of components besides pins, such as valves, hydraulic cylinders, etc. In particular, component 170 can be any component whose problem can be indicated by sound.

[0023] The work machine 100 may include one or more acoustic sensors 180 configured to capture audio data. The acoustic sensors 180 may include microphone arrays, acoustic cameras, etc. The acoustic sensors 180 may be mounted on the cabin 130 of the work machine 100, for example, on the top of the cabin 130. Alternatively or additionally, the acoustic sensors 180 may be mounted at other locations on the work machine 100, such as on the machine body 110, inside the engine or motor housing of the machine body 110, on the work implement 120, at the front or rear of the cabin 130, etc. However, it is generally advantageous to mount the acoustic sensors 180 on the exterior of parts of the work machine 100 that are easily exposed to dirt and other debris, so as to keep the acoustic sensors relatively free from debris that could interfere with the capture of audio data. The acoustic sensors 180 may be positioned and / or oriented to capture noise from multiple conspicuous components 170 at multiple locations on the work machine 100.

[0024] Figure 2 A process 200 for detecting and locating problems in a work machine 100 using an acoustic sensor 180, according to an embodiment, is illustrated. Process 200 can be implemented in real-time by a controller 150 in conjunction with the operation of the work machine 100. As used herein, the terms “real-time” or “in real-time” should be understood to mean events occurring simultaneously, as well as events spaced apart in time due to normal latency in processing, communication, memory access, etc. The term “problem” is generally used to indicate a problem with component 170, but should be broadly understood to include both direct problems (such as partial or complete failure of component 170) and indirect problems (such as the degree of wear of component 170).

[0025] Although process 200 is shown as having a specific arrangement and order of subprocesses, process 200 can be implemented with fewer, more, or different subprocesses, as well as different arrangements and / or orders of subprocesses. Furthermore, it should be understood that even if subprocesses are described or shown in a specific order, any subprocess can be executed before, after, or in parallel with other independent subprocesses, independent of the completion of another subprocess.

[0026] Sub-process 210 can determine whether to terminate process 200. Process 200 can be executed continuously and in real-time as long as the machine 100 is operable (e.g., from the time the machine 100 is turned on until the time the machine 100 is turned off). Alternatively or additionally, process 200 can be turned on and / or off by a local operator (e.g., via an input device within cabin 130), a remote operator (e.g., via an input device at a remote terminal), an autonomous control system, etc. Therefore, process 200 can terminate when the machine 100 is turned off and / or when process 200 is turned off. When it is determined that process 200 will terminate (i.e., "yes" in sub-process 210), process 200 may terminate. Otherwise, process 200 can proceed to sub-process 220 until it is determined that process 200 will terminate (i.e., "no" in sub-process 210).

[0027] Subprocess 220 can determine whether new audio data has been captured by acoustic sensor 180. In an embodiment, audio data can be streamed from acoustic sensor 180 in real time according to the sampling rate. In this case, subprocess 220 can determine that new audio data has been captured at the end of each time interval defined by the sampling rate. In an alternative embodiment, audio data can only be collected if the noise captured by acoustic sensor 180 meets one or more criteria (e.g., exceeds a predefined volume threshold). In this case, subprocess 220 can determine that new audio data has been captured whenever acoustic sensor 180 collects audio data that meets these criteria. In any case, when it is determined that new audio data has been captured (i.e., "yes" in subprocess 220), process 200 can proceed to subprocess 230. Otherwise, when it is not determined that new audio data has been captured (i.e., "no" in subprocess 220), process 200 can return to subprocess 210.

[0028] Subprocess 230 may receive audio data captured by acoustic sensor 180 and apply spatial filtering to the audio data to identify one or more portions of the audio data, each portion representing a different location on the work machine 100. In a preferred embodiment, spatial filtering identifies multiple portions of the audio data, each portion representing a location on the work machine 100 that differs from any other portion of the multiple portions of the audio data. Each location represented by a portion of the audio data may correspond to a specific part 170 of the work machine 100.

[0029] Spatial filtering can determine from which direction each portion of audio data is captured. In this case, each direction can be mapped to a position on the work machine 100 based on the position of each component 170 relative to the acoustic sensor 180. In the case where one or more components 170 move relative to the acoustic sensor 180, or on another component (e.g., work implement 120) that moves relative to the acoustic sensor 180, the component 170 corresponding to the direction from which a portion of audio data is captured can be determined based on the relative position of the component 170, for example, according to position parameters output by one or more sensors 160. In other words, spatial filtering can track the kinematic position of one or more components 170 to be monitored based on the output of the sensor 160. Spatial filtering can retain audio data from directions as portions of audio data, where each direction intersects with the position of the monitored component 170 to identify problems, while filtering out (e.g., canceling) sound from any other direction. Therefore, after spatial filtering, only the portion of audio data representing the component to be monitored is retained for further processing.

[0030] In one embodiment, the acoustic sensor 180 includes a microphone array. The microphone array includes a plurality of microphones operating in a front-to-back manner. The plurality of microphones may include omnidirectional and / or directional microphones distributed around the periphery of the microphone array. Spatial filtering may include processing (e.g., by controller 150) the acoustic signals (i.e., audio data) captured by the plurality of microphones in the microphone array to locate the source (e.g., component 170) of each of one or more sounds in the audio data.

[0031] In alternative or additional embodiments, acoustic sensor 180 may include an acoustic camera. An acoustic camera typically includes a microphone array and an optical camera. As described above, a microphone array can be used to capture audio data and locate the source of each of one or more sounds in the audio data, and an optical camera can be used to map the source of each sound to two-dimensional image data captured by the optical camera. In embodiments utilizing an acoustic camera, this two-dimensional image data can be provided to a local operator (e.g., on a display within cabin 130) or a remote operator (e.g., on a display at a remote terminal) to help the operator visually identify the component 170 to which each portion of the audio data belongs.

[0032] In an embodiment, spatial filtering may include altering one or more operating parameters of the work machine 100 and using the resulting changes in sound in the audio data to verify which component 170 is the source of the sound. For example, controller 150 may actuate a pump or other device to change the fluid pressure through a valve according to a predefined pattern and determine that the valve is the component 170 generating the sound when the sound in the audio data changes according to a similar or corresponding pattern. More generally, controller 150 may actuate any component 170 to be monitored or actuate components affecting component 170 according to a predefined pattern and verify that component 170 is the source of the sound in the audio data when the sound in the audio data changes according to a pattern corresponding to the predefined pattern. In this case, for spatial filtering, the work machine 100 may be placed in a diagnostic mode in which operation of the work machine 100 is stopped, thereby preventing the operator from affecting the actuation of component 170 during spatial filtering, and vice versa.

[0033] Subprocess 240 can iterate over each portion of the audio data identified in subprocess 230. In other words, subprocesses 240 through 270 can be executed for each of one or more portions of the audio data output by spatial filtering in subprocess 230. When another portion of the audio data indicating another location is still to be considered (i.e., "yes" in subprocess 240), process 200 can proceed to subprocess 250. Otherwise, once all portions of the audio data have been processed (i.e., "no" in subprocess 240), process 200 can return to subprocess 210.

[0034] Subprocess 250 may apply acoustic filtering to a portion of the audio data currently being processed. Acoustic filtering may transform the audio data, filter out (e.g., eliminate or otherwise exclude) certain sounds from the audio data, enhance certain sounds in the audio data, and / or otherwise manipulate the audio data to make it possible to determine whether a problem exists. For example, acoustic filtering may include a bandpass filter that isolates one or more frequency bands in a portion of the audio data currently being processed while eliminating all other frequency bands in that portion of the audio data. As another example, acoustic filtering may include comparing a portion of the audio data currently being processed with a reference acoustic pattern to determine whether the reference acoustic pattern is present in that portion of the audio data. As yet another example, acoustic filtering may include determining one or more operating parameters of the operating machine 100 that coincide with the time (e.g., at or near the same time) when the portion of the audio data currently being processed was captured, and filtering out noise from the portion of the audio data based on the operating parameters. Examples of such operating parameters include, but are not limited to, engine speed, pump speed, pump displacement, transmission shifting, valve actuation, tilt actuation, lift actuation, third-function valve actuation, fourth-function valve actuation, steering actuation, braking, and ground speed.

[0035] Sub-process 260 can determine whether a problem exists based on the acoustic filtering in sub-process 250. Specifically, sub-process 260 can determine whether the acoustically filtered portion of the currently processed audio data represents a problem with the corresponding part 170, as determined by spatial filtering in sub-process 230. In other words, sub-process 260 can detect a problem with the corresponding part 170 based on a portion of the audio data captured from the location of part 170. When a problem is determined to exist (i.e., "yes" in sub-process 260), process 200 can proceed to sub-process 270. Otherwise, when a problem is not determined to exist (i.e., "no" in sub-process 260), process 200 can return to sub-process 240.

[0036] In embodiments where the acoustic filtering includes a bandpass filter, subprocess 260 may include determining whether an audio feature exists within the frequency band output by the bandpass filter. This audio feature may include the presence of any sound within the frequency band, the presence of a reference sound pattern within the frequency band, or a sound amplitude exceeding a predefined threshold. It should be understood that, in this case, the presence of this audio feature within the frequency band output by the bandpass filter indicates that the corresponding component 170 is emitting a sound, indicating a problem with component 170. Therefore, when the audio feature exists within the frequency band output by the bandpass filter, subprocess 260 can determine that a problem exists (i.e., "yes" in subprocess 260). Otherwise, subprocess 260 can determine that no problem exists (i.e., "no" in subprocess 260).

[0037] In embodiments where acoustic filtering includes comparing a portion of currently processed audio data with a reference acoustic pattern, subprocess 260 may include determining whether the portion of the audio data matches the reference acoustic pattern. The reference acoustic pattern may include patterns of frequency, volume, pitch, and / or any other audio characteristics. When the comparison determines that the reference acoustic pattern exists within a portion of the audio data, subprocess 260 may determine if a problem exists (i.e., "yes" in subprocess 260). Otherwise, subprocess 260 may determine if no problem exists (i.e., "no" in subprocess 260).

[0038] In an embodiment, subprocess 260 may determine whether a problem exists based on both acoustic filtering and whether the component 170 of the source corresponding to the portion of the audio data currently being processed is moving at the time of audio data capture. For example, subprocess 260 may determine the component 170 of the work machine 100 corresponding to the position on the work machine 100 from which the audio data is captured; determine whether the component 170 is moving at the time of audio data capture based on sensor data from one or more sensors 160 on the work machine 100; and determine whether a problem exists based on acoustic filtering and the determination of whether the component 170 is moving at the time of audio data capture. More generally, in this embodiment, subprocess 260 may determine that a problem exists (i.e., "yes" in subprocess 260) when the acoustically filtered portion of the audio data indicates a problem (e.g., using any of the acoustic filters described herein) and sensor data indicates that the component 170 corresponding to the portion of the audio data is simultaneously moving or otherwise affected by simultaneous movement.

[0039] When subprocess 260 determines that a problem exists, subprocess 270 may initiate remedial actions and then return to subprocess 240. Remedial actions may include outputting a notification of the problem detected in subprocess 260. In the case of local or remote operators, the notification may be output to a display (e.g., a touch panel display) or other visual indicator (e.g., an indicator light), a tactile device (e.g., within a joystick or other control device), and / or other devices within cabin 130 or at a remote terminal. Alternatively or additionally, the notification may be sent to a supervisor, service technician, or other recipient to ensure that the problem is not simply ignored by the operator. In the case of autonomous control, the notification may include an inter-process message sent to the autonomous control system (e.g., within controller 150) to thereby mitigate or replace the loss of the operator's role.

[0040] In an embodiment, subprocess 270 includes determining a component 170 of the work machine 100, the component 170 corresponding to the location of a portion of the work machine 100 from which audio data is captured. In this case, a notification of a remedial action can identify the determined component 170 from which the portion of audio data is captured. Notably, the component 170 may have been (at least substantially) identified by spatial filtering in subprocess 230. In particular, each location corresponding to a portion of the audio data output by spatial filtering can be pre-mapped and / or mapped to the specific component 170 identified in the notification based on location data from sensor 160. In embodiments where the notification is displayed (e.g., on a display in cabin 130 or at a remote terminal), the visual representation of the notification may include the name or other identifier of the component 170, a visual representation of the component 170 in isolation or in the environment of the work machine 100 (e.g., an image, a schematic diagram, etc.), for example, as a real-time image or video stream of the component 170 captured by an acoustic camera or other camera.

[0041] Figure 3 An example controller 150 is shown according to an embodiment for implementing a process 200 of detecting and locating problems in a work machine 100 using an acoustic sensor 180. As described elsewhere herein, the controller 150 may include or be composed of an electronic control unit (ECU) within the work machine 100.

[0042] Controller 150 may include one or more processors 310. Processor 310 may include a central processing unit (CPU). Additional processors may be provided, such as a graphics processing unit (GPU), an auxiliary processor for managing input / output, an auxiliary processor for performing floating-point mathematical operations, a dedicated microprocessor (e.g., a digital signal processor) with a fast execution architecture suitable for signal processing algorithms, a slave processor (e.g., a back-end processor), an additional microprocessor or controller and / or coprocessor for dual- or multi-processor systems. Such auxiliary processors may be discrete processors or may be integrated with the main processor 310. Examples of processors that may be used with controller 150 include, but are not limited to, any processor available from Intel Corporation, Santa Clara, California (e.g., Pentium). TM Core i7 TM Xeon TM Any processor available from AMD (Santa Clara, California), any processor available from Apple (Cupitino, South Korea) (e.g., A-series, M-series, etc.), or any processor available from Samsung Electronics (Sengoku, South Korea) (e.g., Exynos). TMAny processors available from Eindhoven NXP Semiconductors in the Netherlands, etc.

[0043] Processor 310 can be connected to communication bus 305. Communication bus 305 may include a data channel to facilitate information transfer between memory and other peripheral components of controller 150. Furthermore, communication bus 305 may provide a set of signals for communicating with processor 310, including a data bus, address bus, and / or control bus (not shown). Communication bus 305 may include any standard or non-standard bus architecture, such as Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Micro Channel Architecture (MCA), Peripheral Component Interconnect (PCI) local bus, or bus architectures based on standards issued by the Institute of Electrical and Electronics Engineers (IEEE), including IEEE 488 Universal Interface Bus (GPIB), IEEE 696 / S-100, etc.

[0044] Controller 150 may include main memory 315. Main memory 315 provides storage for instructions and / or other data for software executed on processor 310. It should be understood that instructions stored in memory and executed by processor 310 may be written and / or compiled in any suitable language, including but not limited to C / C++, Java, JavaScript, Perl, Python, Visual Basic, .NET, etc. Main memory 315 is typically a semiconductor-based memory, such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), etc., including read-only memory (ROM).

[0045] Controller 150 may include auxiliary memory 320. Auxiliary memory 320 is a non-transitory computer-readable medium on which instructions and / or other data for software are stored. In this specification, the term "computer-readable medium" is used to refer to any non-transitory computer-readable storage medium used to provide computer-executable code and / or other data to or within controller 150. Computer software stored on auxiliary memory 320 is read into main memory 315 for execution by processor 310. Auxiliary memory 320 may include, for example, semiconductor-based memory such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), and flash memory (a block-oriented memory similar to EEPROM).

[0046] The controller 150 may include an input / output (I / O) interface 335. The I / O interface 335 provides an interface between one or more components of the controller 150 and one or more input and / or output devices. For example, the I / O interface 335 may receive outputs from one or more sensors 160 and / or output control signals to one or more subsystems or other components of the machine 100.

[0047] Controller 150 may include a communication interface 340. Communication interface 340 allows software to transfer information between controller 150 and external devices, networks, or other sources and / or destinations. For example, instructions and / or other data can be transmitted from a network server to controller 150 via communication interface 340 through one or more networks. Examples of communication interface 340 include a built-in network adapter, a network interface card (NIC), a PCMCIA network card, a card bus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a wireless data card, a communication port, an infrared interface, an IEEE 1394 FireWire, and any other device that enables controller 150 to interface with a network or another computing device. The communication interface 340 preferably implements industry-published protocol standards, such as Ethernet IEEE 802 standard, Fibre Channel, Digital Subscriber Line (DSL), Asynchronous Digital Subscriber Line (ADSL), Frame Relay, Asynchronous Transfer Mode (ATM), Integrated Digital Services Network (ISDN), Personal Communication Services (PCS), Transmission Control Protocol / Internet Protocol (TCP / IP), Serial Line Internet Protocol / Point-to-Point Protocol (SLIP / PPP), etc., but it can also implement customized or non-standard interface protocols.

[0048] Software transmitted via communication interface 340 is typically in the form of electrical communication signals 355. These signals 355 can be provided to communication interface 340 via communication channel 350 between communication interface 340 and external system 345. In embodiments, communication channel 350 can be a wired or wireless network, or any other type of communication link. Communication channel 350 carries signals 355 and can be implemented using various wired or wireless communication methods, including wires or cables, optical fibers, conventional telephone lines, cellular telephone links, wireless data communication links, radio frequency (“RF”) links, or infrared links, to name just a few.

[0049] Computer-executable code is stored in main memory 315 and / or secondary memory 320. The computer-executable code can also be received from an external system 345 via communication interface 340 and stored in main memory 315 and / or secondary memory 320. When executed by processor 310, such computer-executable code enables controller 150 to perform various functions of the disclosed embodiments, including, for example, process 200.

[0050] Industrial applicability

[0051] Many problems that arise during the operation of the working machine 100 initially manifest as noise. Typically, a local operator will detect any such noise. However, this is not possible when there is no local operator, such as in the case of remote operation or autonomous working machine 100. Furthermore, even when an operator is present, the operator may not hear the noise, may not be able to identify the noise, may not be able to locate the noise source, or may not be able to report the noise.

[0052] Therefore, the disclosed embodiments utilize one or more acoustic sensors 180, such as microphone arrays or acoustic cameras, mounted on the work machine 100. The controller 150 of the work machine 100 can automatically monitor the audio data captured by the acoustic sensors 180; apply spatial filtering to map portions of the audio data to specific components 170; apply acoustic filtering to detect problems with component 170; and initiate remedial actions, such as notifying operators, supervisors, and / or autonomous control systems, when a problem with component 170 is detected. Thus, the disclosed embodiments correct situations where the lack of a local operator and / or the failure of a local operator causes acoustic problems with component 170 of the work machine 100 to go unnoticed. Advantageously, early detection, localization, and resolution of problems can completely prevent further or more serious problems from occurring.

[0053] As an example, a medium-sized wheel loader has at least 22 large load-bearing pin joints. If a pin in one of these joints malfunctions, it will typically begin to generate noise. In this case, acoustic sensor 180 will capture the noise in the audio data. In subprocess 230, spatial filtering can be used to locate the portion of the audio data representing the noise to the specific pin generating the noise. Based on the knowledge that this portion of the audio data corresponds to the pin, subprocesses 250 and 260 can determine that the noise represented by the portion of the audio data is related to the simultaneous rotation of the pin joint (e.g., determined by position measurements output from sensor 160) within a frequency range characteristic of the pin problem, following the pin's rotation as the pin moves through space relative to acoustic sensor 180, etc. In this case, subprocess 270 can notify the operator or autonomous control system of the problem and identify the specific pin and / or the location of the specific pin as the subject of the problem.

[0054] It should be understood that the above benefits and advantages may relate to one embodiment or several embodiments. Aspects described in connection with one embodiment are intended to be used in conjunction with other embodiments. Any interpretation in connection with one embodiment applies to similar features of other embodiments, and elements of multiple embodiments may be combined to form other embodiments. The embodiments are not limited to embodiments that solve any or all of the described problems or embodiments that have any or all of the described benefits and advantages.

[0055] The foregoing specific embodiments are merely exemplary in nature and are not intended to limit the invention or its application and use. The described embodiments are not limited to use with a particular type of work machine. Therefore, although this embodiment is depicted and described as being implemented in a wheel loader for ease of illustration, it should be understood that the invention can be implemented in a variety of other types of work machines, where problems or other issues are audibly manifested, and in a variety of other systems and environments. Furthermore, one is not expected to be bound by any theory set forth in any of the foregoing sections. It should also be understood that the illustrations may include exaggerated dimensions and graphic representations to better illustrate the referenced items shown and are not intended to be limiting unless expressly stated otherwise.

Claims

1. A method comprising using at least one hardware processor in a job machine to: [The following is a list of parameters, not part of the main text:] Receives audio data captured by one or more acoustic sensors; Spatial filtering is applied to the audio data to identify one or more portions of the audio data, each of the one or more portions of the audio data being captured from a location on the work machine that is different from any other portion of the one or more portions of the audio data; as well as For each of one or more portions of the audio data: Acoustic filtering is applied to the portion of the audio data. The presence of a problem is determined based on the acoustic filtering, and Once the problem is identified, remedial action is initiated.

2. The method of claim 1, wherein the one or more acoustic sensors comprise a microphone array.

3. The method of claim 1, wherein the one or more acoustic sensors include an acoustic camera.

4. The method of claim 1, wherein the one or more acoustic sensors are mounted on the hull of the operating machine.

5. The method of claim 1, wherein the acoustic filtering comprises a bandpass filter that isolates one or more frequency bands of the portion of the audio data.

6. The method of claim 5, wherein determining whether the problem exists includes determining whether an audio feature exists in one or more frequency bands of the portion of the audio data.

7. The method of claim 1, wherein applying the acoustic filtering includes comparing said portion of the audio data with a reference acoustic pattern.

8. The method of claim 7, wherein determining whether the problem exists includes determining whether the portion of the audio data matches the reference acoustic pattern.

9. The method according to claim 1, wherein the acoustic filtering comprises: Determine one or more operating parameters of the work machine that correspond to the time when the portion of the audio data was captured; as well as Noise is filtered out from the portion of the audio data based on one or more of the operating parameters.

10. The method of claim 9, wherein the one or more operating parameters include one or more of engine speed, pump speed, pump displacement, transmission shift, valve actuation, tilt actuation, lift actuation, third function valve actuation, fourth function valve actuation, steering actuation, braking, or ground speed.

11. The method of claim 1, wherein the remedial action includes outputting a notification of the problem.

12. The method of claim 11, wherein initiating the remedial action includes determining a component of the work machine corresponding to the location of the portion of the audio data captured therefrom on the work machine, and wherein the notification identifies the determined component.

13. The method of claim 12, wherein the component comprises a pin.

14. The method of claim 12, wherein the notification includes instructions in the cabin of the operating machine.

15. The method of claim 1, wherein determining whether the problem exists comprises: Determine the component of the work machine that corresponds to the location of the portion of the work machine from which the audio data is captured; The determination of whether the component is moving at the moment the audio data is captured is based on sensor data from one or more sensors on the machine. as well as The existence of the problem is determined based on the acoustic filtering and whether the component is moving at that moment.

Images