System for analyzing and displaying acoustic data

Abstract

To solve the problem of often requiring users to manually identify various acoustic parameters of interest prior to inspection in order to analyze sounds of interest.SOLUTION: Some systems include: an acoustic sensor array configured to receive acoustic signals; an electromagnetic imaging tool configured to receive electromagnetic radiation; a user interface; a display; and a processor. The processor can receive an annotation input from the user interface and update a display image based on the received annotation input. The processor can be configured to determine one or more acoustic parameters associated with the received acoustic signal and determine criticality associated with the acoustic signal. A user can annotate the display image with determined criticality information or other determined information.SELECTED DRAWING: None

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Patent Application No. 62 / 702,716, filed July 24, 2018, and is a divisional application of Japanese Patent Application No. 2021-504177, filed July 24, 2019, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Currently available acoustic imaging devices include acoustic sensor array configurations with various frequency sensitivity limitations due to a variety of factors. For example, some acoustic imaging devices are designed to respond in the acoustic frequency range between approximately 20 Hz and approximately 20 kHz. Other devices (e.g., ultrasonic devices) are designed to respond in the acoustic frequency range between approximately 38 kHz and approximately 45 kHz.

[0003] However, acoustic imaging devices typically designed to operate within the 20 Hz to 20 kHz frequency range cannot effectively detect or image high frequencies, for example, up to approximately 50 kHz or higher. Similarly, acoustic or ultrasonic devices designed to operate within the 20 kHz to 50 kHz frequency range cannot effectively detect and / or image low frequencies, for example, 20 kHz or lower. This is for a variety of reasons. For example, sensor arrays optimized for low frequencies (e.g., audio frequencies) typically include individual sensors that are much further apart than do sensor arrays optimized for high frequencies (e.g., ultrasonic frequencies).

[0004] In addition to or apart from hardware concerns, different computational algorithms and methods of acoustic imaging are often better suited to acoustic signals with different frequencies and / or different distances to the target, making it difficult, especially for inexperienced users, to determine how to best image a scene acoustically without these.

[0005] This discrepancy in imaging different acoustic frequency ranges is due in part to the physics behind the propagation of sound waves of different frequencies and wavelengths through air. Certain array orientations, array sizes, and computational methods are generally better suited to acoustic signals with different frequency (e.g., audio, ultrasonic, etc.) characteristics.

[0006] Similarly, different array characteristics and / or computational methods may be more suitable for acoustic scenes at different ranges to the target, for example, near-field acoustic holography aimed at targets at very close ranges and various acoustic beamforming methods aimed at targets at greater distances.

[0007] Thus, acoustic inspections using acoustic arrays (e.g., for acoustic imaging) may require, for example, extensive equipment for analyzing acoustic signals having different frequency ranges and expertise in understanding when different hardware and computing techniques are appropriate for performing acoustic analysis, which can make acoustic inspections time- and cost-intensive and require specialists to perform such inspections.

[0008] For example, a user may be forced to manually select various hardware and / or software for an acoustic analysis. However, it may be impossible for an inexperienced analyst to know the optimal combination of hardware and software for a given analysis and / or acoustic scene. Moreover, isolating a target sound from within a scene, especially in a noisy scene, presents its own challenges and may prove a tedious frustration for an inexperienced user. For example, a given acoustic scene may contain acoustic signals containing multiple frequencies, intensities, and other characteristics that obscure the target acoustic signal, especially in a noisy environment.

[0009] Previous systems often required users to manually identify various acoustic parameters of a target prior to inspection in order to analyze the target's sounds, but inexperienced users may not know how to best separate and / or identify the various sounds of the target.

[0010] Furthermore, when multiple imaging techniques (e.g., visible light imaging technique, infrared imaging technique, ultraviolet imaging technique, audio imaging technique, or other imaging techniques) are used in concert to inspect the same object or scene, the physical placement and / or other settings (e.g., focus position) of the tools used to implement the different imaging techniques can strongly affect the analysis. For example, different positions and / or focus positions of each imaging device can result in parallax, and the resulting images can be misaligned. This can result in an inability to properly locate target areas and / or problem areas within the scene, documentation errors, or misdiagnosis of problems. For example, with respect to acoustic image data, it can be difficult to identify the location or source of a target acoustic signal if the acoustic image data is misaligned with respect to the image data from other imaging techniques (e.g., visible light image data and / or infrared image data).

[0011] Existing ultrasonic testing and inspection tools use ultrasonic sensors, with or without a parabolic dish, to help focus sound onto the receiving sensor. When a particular frequency of sound is detected, it is typically displayed on the device's display as a rising or falling number, or on a frequency or decibel level graph. This can be very confusing and unintuitive for many users. No live scene image or sound visualization is available.

[0012] Isolating, locating, and analyzing specific sounds can be a tedious process and confusing for many end users. Complex and non-intuitive interfaces between devices and humans can be barriers to effective use of the devices, and even operating basic functions on the devices can require additional training.

[0013] High-end audio imaging devices have the potential to generate false-color visualizations of sound integrated with still or live visual images of a scene. Even in these devices, selection and adjustment controls are important for proper visualization of sound. However, traditional controls have been developed for use by highly trained audio engineers and professionals. These controls are often non-intuitive for the average user, resulting in some confusion over proper selection and visualization parameter control. Use of these controls by individuals with a low level of training can be cumbersome, leading to incorrect parameter selection and ultimately to poor audio visualization.

[0014] Furthermore, additional contextual information is often required with this method to generate appropriate analytical reporting activities. Typically, a technician who wants to gather additional contextual information about a scene being inspected with a conventional ultrasonic test tool or acoustic imager must take a photograph with a separate camera or device and / or record it with written or recorded annotations on a separate device, such as a PC, tablet, smartphone, or other mobile device. These secondary annotations must then be manually synchronized or matched with the data from the ultrasonic tool or acoustic imager. This can be quite time-consuming and prone to error when matching the appropriate data with the corresponding secondary contextual information. Summary of the Invention

[0015] One aspect of the present disclosure is directed to an acoustic analysis system that may include an acoustic sensor array of a plurality of acoustic sensor elements, each configured to receive an acoustic signal from an acoustic scene and output acoustic data based on the received acoustic signal.

[0016] The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data indicative of the received electromagnetic radiation. The electromagnetic imaging tool may be configured to detect electromagnetic radiation from a range of wavelengths, such as a range including the visible light spectrum and / or the near infrared light spectrum. In some systems, the electromagnetic imaging system may include a visible light camera module and / or an infrared camera module.

[0017] The system may include a user interface, a display, and a processor, the processor being in communication with the acoustic sensor array, the electromagnetic imaging tool, the user interface, and the display.

[0018] In some systems, the processor may be configured to receive electromagnetic data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor may also generate acoustic image data of the scene based on the received acoustic data, generate a display image comprising the combined acoustic and electromagnetic image data, and provide the display image on the display. In some embodiments, the processor may receive annotation input from a user interface and update the display image on the display based on the received annotation input. The annotation input may comprise freestyle annotations received via a touchscreen, selection of icons or predefined shapes, and / or alphanumeric input.

[0019] In some systems, the processor is configured to determine one or more acoustic parameters associated with the received acoustic signal and determine a threshold associated with the acoustic signal, for example, based on a comparison of the one or more acoustic parameters to one or more predetermined thresholds. In some embodiments, the processor can update the display based on the determined threshold. A user can annotate the image with the determined threshold information. A user can also annotate the image with determined information, such as a distance value to a target.

[0020] The details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0021] [Figure 1A] FIG. 1A shows front and rear views of an exemplary acoustic imaging device. [Figure 1B] FIG. 1B shows front and rear views of an exemplary acoustic imaging device.

[0022] [Figure 2] FIG. 2 is a functional block diagram illustrating components of an example acoustic analysis system.

[0023] [Figure 3A] FIG. 3A shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system. [Figure 3B] FIG. 3B shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system. [Figure 3C] FIG. 3C shows a schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system.

[0024] [Figure 4A] FIG. 4A shows a schematic diagram of parallax error in generating frames of visible light image data and acoustic image data. [Figure 4B]FIG. 4B shows a schematic diagram of parallax error in generating frames of visible light image data and acoustic image data.

[0025] [Figure 5A] FIG. 5A shows the parallax correction between the visible light image and the acoustic image. [Figure 5B] FIG. 5B shows the parallax correction between the visible light image and the acoustic image.

[0026] [Figure 5C] FIG. 5C is a colorized version of FIGS. 5A and 5B. [Figure 5D] FIG. 5D is a colorized version of FIGS. 5A and 5B.

[0027] [Figure 6] FIG. 6 is a process flow diagram illustrating an exemplary method for generating a final image that combines acoustic image data and electromagnetic image data.

[0028] [Figure 7] FIG. 7 is a process flow diagram illustrating an exemplary process for generating acoustic image data from a received acoustic signal.

[0029] [Figure 8] FIG. 8 shows an exemplary look-up table for determining the appropriate algorithm and sensor array to use during the acoustic imaging process.

[0030] [Figure 9A] FIG. 9A is an exemplary plot of the frequency content of received image data over time in an acoustic scene.

[0031] [Figure 9B] FIG. 9B shows an example scene including multiple locations emitting acoustic signals.

[0032] [Figure 9C]FIG. 9C shows a plurality of combined acoustic and visible light image data at a plurality of predefined frequency ranges.

[0033] [Figure 10A] FIG. 10A is an exemplary display image including combined visible light image data and acoustic image data. [Figure 10B] FIG. 10B is an exemplary display image including combined visible light image data and acoustic image data.

[0034] [Figure 11A] FIG. 11A shows an exemplary plot of frequency versus time of acoustic data in an acoustic scene. [Figure 11B] FIG. 11B shows an exemplary plot of frequency versus time of acoustic data in an acoustic scene.

[0035] [Figure 12A] FIG. 12A illustrates several exemplary methods for comparing acoustic image data with past acoustic image data stored in a database. [Figure 12B] FIG. 12B illustrates several exemplary methods for comparing acoustic image data with past acoustic image data stored in a database. [Figure 12C] FIG. 12C illustrates several exemplary methods for comparing acoustic image data with past acoustic image data stored in a database.

[0036] [Figure 13] FIG. 13 is a process flow diagram illustrating an exemplary operation for comparing received acoustic image data to a database for object diagnosis.

[0037] [Figure 14] FIG. 14 shows the visualization of acoustic data using a gradient palette scheme.

[0038] [Figure 15] FIG. 15 shows a visualization of acoustic data using multiple shaded concentric circles.

[0039] [Figure 16] Figure 16 shows an example visualization that includes both non-numeric and alphanumeric information.

[0040] [Figure 17] Figure 17 shows another example visualization that includes both non-numeric and alphanumeric information.

[0041] [Figure 18] Figure 18 shows another example visualization that includes both non-numeric and alphanumeric information.

[0042] [Figure 19] FIG. 19 shows an example visualization showing indicators of different sizes and colors representing different acoustic parameter values.

[0043] [Figure 20] FIG. 20 shows an example visualization showing multiple indicators with different colors indicating the severity indicated by the acoustic signal from the corresponding location.

[0044] [Figure 21] FIG. 21 shows a scene including indicators at multiple locations within the scene that distinguish between acoustic signals that meet predetermined conditions.

[0045] [Figure 22] FIG. 22 shows a display image including a number of icons positioned within the display image that indicate sound profiles recognized within the scene.

[0046] [Figure 23] FIG. 23 shows another exemplary display image showing acoustic data via multiple indicators using concentric circles and alphanumeric information representing acoustic intensity associated with each acoustic signal.

[0047] [Figure 24] FIG. 24 shows an exemplary display image with indicators and additional alphanumeric information associated with the displayed acoustic signal.

[0048] [Figure 25A] FIG. 25A shows a system including a display where an indicator within the display image is selected and a laser pointer that emits a laser toward the scene.

[0049] [Figure 25B] FIG. 25B shows the display shown in the system view of FIG.

[0050] [Figure 26] FIG. 26 shows a display image having a gradient palette scheme representing audio image data and including indicators including audio image blending controls.

[0051] [Figure 27] FIG. 27 shows a display image having a concentric palette scheme representing audio image data and including indicators including audio image blending controls.

[0052] [Figure 28] FIG. 28 shows a display image including an indicator with gradient toning that indicates locations within the scene that meet one or more filter conditions.

[0053] [Figure 29] FIG. 29 shows a display image including an indicator with concentric circular tones that indicate locations within the scene that satisfy one or more filter conditions.

[0054] [Figure 30]FIG. 30 shows a display image containing two indicators, each with a gradient color tone indicating the location in the scene where a different filter condition is satisfied.

[0055] [Figure 31] FIG. 31 shows a display interface including a display image and a virtual keyboard.

[0056] [Figure 32] FIG. 32 shows a display integrated into eyewear that can be worn by a user to display a display image.

[0057] [Figure 33A] FIG. 33A shows a dynamic display image including an indicator with dynamic intensity based on the pointing of the acoustic sensor array. [Figure 33B] FIG. 33B shows a dynamic display image including an indicator with dynamic intensity based on the pointing of the acoustic sensor array.

[0058] [Figure 34] FIG. 34 shows an exemplary display image that provides a user or technician with an indication regarding the potential criticality of air leaks identified within a scene and the potential damage costs resulting from the air leaks.

[0059] [Figure 35] FIG. 35 shows an example of a user annotating a display image with freeform annotations on the display.

[0060] [Figure 36] FIG. 36 shows an example of an annotated display image that includes location information associated with instructions.

[0061] [Figure 37] FIG. 37 shows an example of a user annotating a display image with icon annotations on the display.

[0062] [Figure 38] FIG. 38 shows an example of a user annotating a display image with shape annotations on the display.

[0063] [Figure 39] FIG. 39 shows an example of an annotated display image including annotations of shapes and icons on the display.

[0064] [Figure 40] FIG. 40 shows an interface including a display image and a multi-parameter data visualization including multiple frequency ranges on the right hand side of the display image.

[0065] [Figure 41] FIG. 41 shows an interface including a display image and a multi-parameter display of frequency information including multiple frequency ranges located along the bottom edge of the display image.

[0066] [Figure 42] FIG. 42 shows a display image containing frequency information for multiple frequency bands and peak values ​​for multiple frequency bands.

[0067] [Figure 43] FIG. 43 shows a display image including a multi-parameter display showing intensity information for multiple frequencies.

[0068] [Figure 44] FIG. 44 shows a multi-parameter display that includes a toned set of frequency ranges, where the toning represents the decibel range into which each frequency range falls.

[0069] [Figure 45] FIG. 45 shows an example display image including a multi-parameter display showing different frequency ranges and indicators color-toned according to severity.

[0070] [Figure 46] FIG. 46 shows the trends of intensity (in dB) versus time in each of several frequency ranges within a multi-parameter display of the display image. DETAILED DESCRIPTION OF THE INVENTION

[0071] 1A and 1B show front and rear views of an exemplary acoustic imaging device. FIG. 1A shows the front of acoustic imaging device 100 having a housing 102 supporting acoustic sensor array 104 and electromagnetic imaging tool 106. In one embodiment, acoustic sensor array 104 includes a plurality of acoustic sensor elements, each configured to receive an acoustic signal from an acoustic scene and output acoustic data based on the received acoustic signal. Electromagnetic imaging tool 106 may be configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representative of the received electromagnetic radiation. Electromagnetic imaging tool 106 may be configured to detect electromagnetic radiation in one or more ranges of wavelengths, such as visible light, infrared, or ultraviolet.

[0072] In the illustrated example, acoustic imaging device 100 includes an ambient light sensor 108 and a position sensor 116, such as a GPS. Device 100 includes a laser pointer 110, which in some embodiments includes a laser rangefinder. Device 100 includes a torch 112, which may be configured to emit visible light radiation toward the scene, and an infrared illuminator 118, which may be configured to emit infrared radiation toward the scene. In some examples, device 100 may include an illuminator for illuminating the scene over a range of wavelengths. Device 100 further includes a projector 114, such as an image reprojector, which may be configured to project a generated image onto the scene, such as a colored image, and / or a dot projector configured to project a series of dots onto the scene, for example, to determine a depth profile of the scene.

[0073] 1B shows the back side of acoustic imaging device 100. As shown, the device includes a display 120 capable of displaying images or other data. In one example, display 120 includes a touchscreen display. Acoustic imaging device 100 includes a speaker capable of providing audio feedback signals to a user and a wireless interface 124 capable of enabling wireless communication between acoustic imaging device 100 and an external device. The device further includes controls 126, which may include one or more buttons, knobs, dials, switches, or other interface components, to enable a user to interact with acoustic imaging device 100. In one example, controls 126 and the touchscreen display combine to provide a user interface for acoustic imaging device 100.

[0074] In various embodiments, the acoustic imaging device need not include all of the elements shown in the embodiment of Figures 1A and 1B. One or more of the components shown can be excluded from the acoustic imaging device. In some examples, one or more of the components shown in the embodiment of Figures 1A and 1B can be included as part of the acoustic imaging system, but can be included separately from the housing 102. Such components can communicate with other components of the acoustic imaging system via wired or wireless communication techniques, for example, using the wireless interface 124.

[0075] FIG. 2 is a functional block diagram illustrating components of an example acoustic analysis system 200. The example acoustic analysis system 200 of FIG. 2 can include multiple acoustic sensors, such as microphones, MEMS, transducers, etc., arranged in an acoustic sensor array 202. Such an array can be one-dimensional, two-dimensional, or three-dimensional. In various examples, the acoustic sensor array can define any suitable size and shape. In one example, the acoustic sensor array 202 includes multiple acoustic sensors arranged in a grid pattern, such as, for example, an array of sensor elements arranged in vertical columns and horizontal rows. In various examples, the acoustic sensor array 202 can include a horizontal row by vertical column array, such as, for example, an 8×8, 16×16, 32×32, 64×64, 128×128, 256×256, etc. Other examples are possible, and various sensor arrays need not include the same number of rows as columns. In some embodiments, such sensors may be disposed on a substrate, such as, for example, a printed circuit board (PCB) board.

[0076] In the configuration shown in FIG. 2 , a processor 212 in communication with the acoustic sensor array 202 can receive acoustic data from each of the multiple acoustic sensors. During exemplary operation of the acoustic analysis system 200, the processor 212 can be in communication with the acoustic sensor array 202 to generate acoustic image data. For example, the processor 212 can be configured to analyze data received from each of the multiple acoustic sensors arranged in the acoustic sensor array and determine an acoustic scene by “backpropagating” acoustic signals to acoustic sources. In one embodiment, the processor 212 can generate a digital “frame” of acoustic image data by identifying various source locations and intensities of acoustic signals throughout a two-dimensional scene. By generating a frame of acoustic image data, the processor 212 substantially captures an acoustic image of the target scene at a given time. In one example, the frame includes a plurality of pixels that make up the acoustic image, each pixel representing a portion of the source scene from which the acoustic signal was backpropagated.

[0077] Components described as processors in acoustic analysis system 200, including processor 212, may be implemented as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic circuits, etc., alone or in any suitable combination. Processor 212 may also include memory that stores program instructions and associated data that, when executed by processor 212, cause acoustic analysis system 200 and processor 212 to perform the functions attributed thereto in this disclosure. Memory may include fixed or removable magnetic, optical, or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magnetic disks, EEPROM, etc. Memory may also include removable memory portions that may be used to provide memory updates or increased memory capacity. Removable memory also allows acoustic image data to be easily transferred to another computing device or removed before acoustic analysis system 200 is used for another application. The processor 212 may also be implemented as a system-on-chip, integrating some or all components of a computer or other electronic system onto a single chip. The processor 212 (processing circuitry) may be configured to communicate processed data to a display 214 or other output / control device 218.

[0078] In one embodiment, the acoustic sensors in acoustic sensor array 202 generate a series of signals corresponding to the acoustic signals received by each acoustic sensor to represent an acoustic image. A "frame" of acoustic image data is generated as signals from each acoustic sensor are acquired by scanning all of the rows that make up acoustic sensor array 202. In one example, processor 212 can acquire acoustic image frames at a rate sufficient to generate a video representation of the acoustic image data (e.g., 30 Hz or 60 Hz). Independent of the particular circuitry, acoustic analysis system 200 can be configured to manipulate the acoustic data representing the sound profile of the target scene to provide an output that can be displayed, stored, transmitted, or otherwise utilized by a user.

[0079] In some embodiments, "backpropagation" of received acoustic signals to generate acoustic image data involves analyzing the received signals at multiple acoustic sensors in the acoustic sensor array 202, e.g., via a processor. In various examples, performing backpropagation is a function of one or more parameters, including distance to the target, frequency, sound intensity (e.g., dB level), and sensor array dimensions / configuration, including, for example, spacing and placement of individual sensors within one or more arrays. In some embodiments, such parameters may be pre-programmed into the system, e.g., in memory. For example, characteristics of the acoustic sensor array 202 may be stored in memory, such as internal memory or memory specifically associated with the acoustic sensor array 202. Other parameters, such as distance to the target, may be received in various ways. For example, in some examples, the acoustic analysis system 200 includes a distance measurement tool 204 in communication with the processor 212. The distance measurement tool may be configured to provide distance information representing the distance from the distance measurement tool 204 to a particular location within the target scene. Various distance measurement tools may include other known distance measurement devices, such as a laser rangefinder or other optical or acoustic distance measurement devices. Additionally or alternatively, the distance measurement tool may be configured to generate three-dimensional depth data such that each portion of a target scene has an associated distance-to-target value. Thus, in some examples, distance measurements to a target as used herein may correspond to distances to each location within the target scene. Such three-dimensional depth data may be generated, for example, via multiple imaging tools having different views of the target scene or via other known distance scanning tools. Generally, in various embodiments, the distance measurement tool may be used to perform one or more distance measurement functions, including, but not limited to, laser distance measurement, active sonic distance measurement, passive ultrasonic distance measurement, LIDAR distance measurement, RADAR distance measurement, millimeter wave distance measurement, etc.

[0080] Distance information from distance measurement tool 204 can be used in backpropagation calculations. Additionally or alternatively, system 200 may include a user interface 216 that allows a user to manually input a distance-to-target parameter. For example, a user may input a distance-to-target value into system 200 if the distance to the component suspected of generating the acoustic signal is known or is difficult to measure with distance measurement tool 204.

[0081] In the illustrated embodiment, acoustic analysis system 200 includes an electromagnetic imaging tool 203 for generating image data representative of a target scene. The exemplary electromagnetic imaging tool may be configured to receive electromagnetic radiation from the target scene and generate electromagnetic image data representative of the received electromagnetic radiation. In one example, electromagnetic imaging tool 203 may be configured to generate electromagnetic image data representative of a particular range of wavelengths within the electromagnetic spectrum, such as infrared radiation, visible light radiation, and ultraviolet radiation. For example, in one embodiment, electromagnetic imaging tool 203 may include one or more camera modules, such as visible light camera module 206, configured to generate image data representative of a particular range of wavelengths within the electromagnetic spectrum.

[0082] Visible light camera modules are generally well known. For example, smartphones and many other devices include various visible light camera modules. In one embodiment, visible light camera module 206 may be configured to receive visible light energy from a target scene and focus the visible light energy onto a visible light sensor to generate visible light energy data that may be displayed, for example, in the form of a visible light image on display 214 and / or stored in memory. Visible light camera module 206 may include any suitable components for performing the functions attributed to the module herein. In the example of FIG. 2 , visible light camera module 206 is shown as including a visible light lens assembly 208 and a visible light sensor 210. In one such embodiment, visible light lens assembly 208 includes at least one lens that receives visible light energy emitted by the target scene and focuses the visible light energy onto visible light sensor 210. Visible light sensor 210 may include multiple visible light sensor elements, such as, for example, a CMOS detector, a CCD detector, a PIN diode, an avalanche photodiode, or the like. The visible light sensor 210 responds to the focused energy by generating an electrical signal that can be converted and displayed as a visible light image on the display 214. In some examples, the visible light module 206 is user configurable, for example, to provide output to the display 214 in a variety of formats. The visible light camera module 206 may include compensation features for changes in illumination or other operating conditions or user preferences. The visible light camera module may provide a digital output that includes image data, which may include data in a variety of formats (e.g., RGB, CYMK, YCbCr, etc.).

[0083] In operation of an example visible light camera module 206, light energy received from a target scene may pass through the visible light lens assembly 208 and be focused onto the visible light sensor 210. When the light energy strikes the visible light sensor elements of the visible light sensor 210, photons within the photodetector may be emitted and converted into a detected current. The processor 212 may process this detected current to form a visible light image of the target scene.

[0084] During use of the acoustic analysis system 200, the processor 212 can control the visible light camera module 206 to generate visible light data from a captured target scene to create a visible light image. The visible light data may include color(s) associated with different portions of the captured target scene and / or luminosity data indicating the magnitude of light associated with different portions of the captured target scene. The processor 212 can generate a “frame” of visible light image data by measuring the response of each visible light sensor element of the acoustic analysis system 200 once. By generating a frame of visible light data, the processor 212 captures a visible light image of the target scene at a given time. The processor 212 can also repeatedly measure the response of each visible light sensor element of the acoustic analysis system 200 to generate a dynamic visible light image (e.g., a moving representation) of the target scene. In some examples, the visible light camera module 206 may include its own dedicated processor or other circuitry (e.g., an ASIC) capable of operating the visible light camera module 206. In one such embodiment, the dedicated processor communicates with processor 212 to provide visible light image data (e.g., RGB image data) to processor 212. In an alternative embodiment, a dedicated processor for visible light camera module 206 may be integrated with processor 212.

[0085] Each sensor element of the visible light camera module 206 functions as a sensor pixel, allowing the processor 212 to generate a two-dimensional image or picture representation of the visible light from the target scene by converting the electrical response of each sensor element into a time-multiplexed electrical signal that can be processed, for example, for visualization on a display 214 and / or storage in memory.

[0086] Processor 212 may control display 214 to display at least a portion of the captured visible light image of the target scene. In one example, processor 212 controls display 214 so that the electrical response of each sensor element of visible light camera module 206 is associated with a single pixel on display 214. In other examples, processor 212 may increase or decrease the resolution of the visible light image so that more or fewer pixels are displayed on display 214 than there are sensor elements in visible light camera module 206. Processor 212 may control display 214 to display the entire visible light image (e.g., all of the portion of the target scene captured by acoustic analysis system 200) or less than the entire visible light image (e.g., less than the entire target scene captured by acoustic analysis system 200).

[0087] In certain embodiments, processor 212 may control display 214 to simultaneously display at least a portion of the visible light image captured by acoustic analysis system 200 and at least a portion of the acoustic image generated via acoustic sensor array 202. Such simultaneous display may be useful in that an operator may reference features displayed in the visible light image to help view the source of an acoustic signal simultaneously displayed in the acoustic image. In various examples, processor 212 may control display 214 to display the visible light image and the acoustic image side-by-side in a picture-in-picture arrangement, where one image surrounds the other, or in any other suitable arrangement in which the visible light and acoustic images are displayed simultaneously.

[0088] For example, processor 212 can control display 214 to display the visible light image and the acoustic image in a combined arrangement. In such an arrangement, for each pixel or set of pixels in the visible light image that represents a portion of the target scene, there is a corresponding pixel or set of pixels in the acoustic image that represents substantially the same portion of the target scene. In various embodiments, the size and / or resolution of the acoustic image and the visible light image need not be the same. Thus, a single pixel in either the acoustic image or the visible light image may correspond to a set of pixels in the other image, or sets of pixels of different sizes. Similarly, there may be pixels in either the visible light image or the acoustic image that correspond to a set of pixels in the other image. Thus, as used herein, correspondence need not be a one-to-one pixel relationship, but may include pixels or groups of pixels of mismatched sizes. Various combination techniques for regions of mismatched sizes in the images may be performed, such as upsampling or downsampling one of the images or combining a pixel with the average value of the corresponding set of pixels. Other examples are known and are within the scope of this disclosure.

[0089] Thus, corresponding pixels need not have a direct one-to-one relationship. Rather, in some embodiments, a single acoustic pixel has multiple corresponding visible-light pixels, or a single visible-light pixel has multiple corresponding acoustic pixels. Additionally or alternatively, in some embodiments, not every visible-light pixel has a corresponding acoustic pixel, or vice versa. Such an embodiment may, for example, represent a picture-in-picture type display, as described above. Thus, a visible-light pixel does not necessarily have the same pixel coordinates in the visible-light image as does a corresponding acoustic pixel. Thus, as used herein, corresponding pixels generally refer to pixels from any image (e.g., a visible-light image, an acoustic image, a combined image, a display image, etc.) that contains information from substantially the same portion of the target scene. Such pixels need not have a one-to-one relationship between images, nor need they have similar coordinate locations within each image.

[0090] Similarly, images with corresponding pixels (i.e., pixels representing the same portion of the target scene) may be referred to as corresponding images. Thus, in one such arrangement, corresponding visible-light and acoustic images may be overlaid on one another at corresponding pixels. An operator may interface with user interface 216 to control the transparency or opacity of one or both of the images displayed on display 214. For example, an operator may interface with user interface 216 to adjust the acoustic image between fully transparent and fully opaque, and the visible-light image between fully transparent and fully opaque. Such an exemplary combination arrangement, which may be referred to as an alpha-blended arrangement, may allow the operator to adjust display 214 to display an acoustic-only image, a visible-light-only image, or any overlapping combination of the two images between the extremes of an acoustic-only image and a visible-light-only image. Processor 212 may also combine scene information with other data, such as alarm data. In general, an alpha-blended combination of visible-light and acoustic images can include anywhere from 100% acoustic and 0% visible light to 0% acoustic and 100% visible light. In some embodiments, the amount of blending can be adjusted by the camera user, so in some embodiments the blended image can be adjusted between 100% visible light and 100% acoustic.

[0091] Additionally, in some embodiments, processor 212 may interpret and execute commands from user interface 216 and / or output / controller 218. Additionally, input signals may be used to modify the processing of visible light and / or acoustic image data generated by processor 212.

[0092] An operator may interact with acoustic analysis system 200 via user interface 216, which may include buttons, keys, or another mechanism for receiving input from a user. The operator may receive output from acoustic analysis system 200 via display 214. Display 214 may be configured to display acoustic images and / or visible light images in any acceptable color tone or color scheme, and the color tone may change, for example, in response to user control. In some embodiments, acoustic image data may be presented in a color tone to represent various magnitudes of acoustic data from different locations within a scene. For example, in some examples, display 214 is configured to display acoustic images in a single color tone, such as grayscale. In other examples, display 214 is configured to display acoustic images in a color palette, such as amber, ironbow, blue-red, or other high-contrast color scheme. Combinations of grayscale and color palette displays are also contemplated. In some examples, displays configured to display such information may include processing capabilities for generating and presenting such image data. In other examples, being configured to display such information may include the ability to receive image data from other components, such as processor 212. For example, processor 212 may generate a value (e.g., an RGB value, a grayscale value, or other display option) for each pixel to be displayed. Display 214 may receive such information and map each pixel to a visual representation.

[0093] The processor 212 can control the display 214 to simultaneously display at least a portion of the acoustic image and at least a portion of the visible light image in any suitable arrangement, although a picture-in-picture arrangement helps an operator to easily focus and / or interpret the acoustic image by displaying a corresponding visible image of the same scene in adjacent alignment.

[0094] A power source (not shown) provides operating power to the various components of the acoustic analysis system 200. In various examples, the power source may include a rechargeable or non-rechargeable battery and power generation circuit, an AC power source, an inductive power pickup, a photovoltaic power source, or any other suitable power supply configuration. Combinations of power supply components, such as a rechargeable battery and another component configured to provide power to operate the device and / or to charge the rechargeable battery, are also possible.

[0095] During operation of acoustic analysis system 200, processor 212, with the aid of instructions associated with program information stored in memory, controls acoustic sensor array 202 and visible light camera module 206 to generate visible light and acoustic images of a target scene. Processor 212 also controls display 214 to display the visible light and / or acoustic images generated by acoustic analysis system 200.

[0096] As mentioned above, in some situations, it may be difficult to identify and distinguish actual (visible) features of a target scene in an acoustic image. In addition to supplementing the acoustic image with visible light information, in some embodiments, it may be useful to highlight visible contours within the target scene. In some embodiments, known contour detection methods may be performed on the visible light image of the target scene. Because of the correspondence between the acoustic image and the visible light image, visible light pixels determined to represent a visible contour of the target scene correspond to acoustic pixels that also represent a visible contour in the acoustic image. It will be understood that "edge," as used herein, need not refer to a physical boundary of an object, but may refer to a sufficiently sharp gradation in the visible light image. Examples may include a physical boundary of an object, a color change within an object, shading across the scene, etc.

[0097] While generally described with respect to FIG. 2 as including a visible light camera module 206, in certain examples, the electromagnetic imaging tool 203 of the acoustic analysis system 200 can additionally or alternatively include imaging tools capable of generating image data representing various spectra. For example, in various examples, the electromagnetic imaging tool 203 can include one or more tools capable of generating infrared image data, visible light image data, ultraviolet image data, or any other useful wavelength, or combinations thereof. In certain embodiments, the acoustic imaging system can include an infrared camera module having an infrared lens assembly and an infrared sensor array. For example, additional components for interfacing with the infrared camera module can be included, such as those described in U.S. Patent Application No. 14 / 837,757, filed August 27, 2015, and entitled "EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES AND CAMERAS," which is assigned to the assignee of the present application and incorporated by reference in its entirety.

[0098] In some examples, two or more data streams may be blended for display. For example, an exemplary system including a visible light camera module 206, an acoustic sensor array 202, and an infrared camera module (not shown in FIG. 2) may be configured to generate an output image including a blend of visible light (VL) image data, infrared (IR) image data, and acoustic image data. In an exemplary blending technique, the display image may be represented by the formula α × IR + β × VL + γ × Acoustic, where α + β + γ = 1. In general, any number of data streams may be combined for display. In various embodiments, blending ratios such as α, β, and γ may be set by a user. Additionally or alternatively, the configured display program can be configured to include different image data streams based on an alarm condition (e.g., one or more values ​​in one or more data streams meet a predetermined threshold) or other condition, such as described in U.S. Pat. No. 7,538,326, entitled "VISIBLE LIGHT AND IR COMBINED IMAGE CAMERA WITH A LASER POINTER," which is assigned to the assignee of the present application and incorporated by reference in its entirety.

[0099] One or more components of the acoustic analysis system 200 described with respect to FIG. 2 can be included in a portable (e.g., handheld) acoustic analysis tool. For example, in some embodiments, the portable acoustic analysis tool can include a housing 230 configured to house the components within the acoustic analysis tool. In some examples, one or more components of the system 200 can be located outside the housing 230 of the acoustic analysis tool. For example, in some embodiments, the processor 212, the display 214, the user interface 216, and / or the output controller 218 can be located outside the housing of the acoustic analysis tool and can communicate with various other system components via, for example, wireless communication (e.g., Bluetooth communication, Wi-Fi, etc.). Such components external to the acoustic analysis tool can be provided via an external device, such as, for example, a computer, a smartphone, a tablet, a wearable device, etc. Additionally or alternatively, other test and measurement or data collection tools configured to function as master or slave devices with respect to the acoustic analysis tool can similarly provide various components of the acoustic analysis system external to the acoustic analysis tool. The external device can communicate with the portable acoustic analysis tool via wired and / or wireless connections and can be used to perform various processing, display, and / or interface steps.

[0100] In some embodiments, such external devices may provide redundant functionality as components housed in the portable acoustic analysis tool. For example, in some embodiments, the acoustic analysis tool may include a display for displaying acoustic image data and may be further configured to communicate the image data to an external device for storage and / or display. Similarly, in some embodiments, a user may interface with the acoustic analysis tool via an application (“app”) running on a smartphone, tablet, computer, etc., to perform one or more functions that could also be performed by the acoustic analysis tool itself.

[0101] 3A is a schematic diagram of an example configuration of an acoustic sensor array in an acoustic analysis system. In the illustrated example, acoustic sensor array 302 includes a plurality of first acoustic sensors (shown in white) and a plurality of second acoustic sensors (shaded). The first acoustic sensors are arranged in a first array 320, and the second acoustic sensors are arranged in a second array 322. In some examples, first array 320 and second array 322 can be selectively used to receive acoustic signals to generate acoustic image data. For example, in some configurations, the sensitivity of a particular acoustic sensor array to a particular acoustic frequency is a function of the distance between the acoustic sensor elements.

[0102] In some configurations, more closely spaced sensor elements (e.g., second array 322) may be better able to resolve high-frequency acoustic signals (e.g., sounds with frequencies above 20 kHz, such as ultrasonic signals between 20 kHz and 100 kHz) than more closely spaced sensor elements (e.g., first array 320). Similarly, more widely spaced sensor elements (e.g., first array 320) may be better suited to detecting lower-frequency acoustic signals (e.g., below 20 kHz) than more closely spaced sensor elements (e.g., second array 322). Various acoustic sensor arrays with sensor elements spaced farther apart may be provided to detect acoustic signals in various frequency ranges, such as infrasonic frequencies (less than 20 Hz), audio frequencies (approximately 20 Hz to 20 kHz), and ultrasonic frequencies (20 kHz to 100 kHz). In some embodiments, a partial array (e.g., all acoustic sensor elements other than array 320) may be used to optimize detection of a particular frequency band.

[0103] Furthermore, in some examples, certain acoustic sensor elements may be better suited to detecting acoustic signals having different frequency characteristics, such as low or high frequencies. Thus, in some embodiments, an array configured to detect low-frequency acoustic signals, such as first array 320 having further-spaced sensor elements, may include first acoustic sensor elements that are better suited to detecting low-frequency acoustic signals. Similarly, an array configured to detect higher-frequency acoustic signals, such as second array 322, may include second acoustic sensor elements that are better suited to detecting high-frequency acoustic signals. Thus, in some examples, first array 320 and second array 322 of acoustic sensor elements may include different types of acoustic sensor elements. Alternatively, in some embodiments, first array 320 and second array 322 may include the same type of acoustic sensor elements.

[0104] Thus, in an exemplary embodiment, acoustic sensor array 302 may include multiple arrays of acoustic sensor elements, such as first array 320 and second array 322. In some embodiments, the arrays may be used individually or in combination. For example, in some instances, a user may select whether to use first array 320, second array 322, or both first array 320 and second array 322 simultaneously to perform an acoustic imaging procedure. In some instances, a user may select which array(s) to use via a user interface. Additionally or alternatively, in some embodiments, the acoustic analysis system may automatically select the array(s) to use based on an analysis of the received acoustic signals or other input data, such as an expected frequency range. While the configuration shown in FIG. 3A generally includes two arrays (first array 320 and second array 322) arranged generally in a rectangular grid, it will be understood that multiple acoustic sensor elements may be grouped into any number of individual arrays of any shape. Additionally, in some embodiments, one or more acoustic sensor elements may be included in multiple separate arrays that may be selected for operation. As described elsewhere herein, in various embodiments, the process of backpropagating acoustic signals to establish acoustic image data from a scene is performed based on the placement of the acoustic sensor elements. Thus, the placement of the acoustic sensors may be known or otherwise accessible by the processor to perform the acoustic image generation techniques.

[0105] The acoustic analysis system of FIG. 3A further includes a distance measurement tool 304 and a camera module 306 disposed within the acoustic sensor array 302. The camera module 306 may represent a camera module of an electromagnetic imaging tool (e.g., 203) and may include a visible light camera module, an infrared camera module, an ultraviolet camera module, etc. Additionally, although not shown in FIG. 3A , the acoustic analysis system may include one or more additional camera modules of the same or different type as the camera module 306. In the illustrated example, the distance measurement tool 304 and the camera module 306 are disposed within a grid of acoustic sensor elements of the first array 320 and the second array 322. While shown disposed between grid sites within the first array 320 and the second array 322, in some embodiments, one or more components (e.g., the camera module 306 and / or the distance measurement tool 304) may be disposed at one or more corresponding grid sites within the first array 320 and / or the second array 322. In certain such embodiments, the component(s) may be located at grid sites in place of acoustic sensor elements that would typically be at such locations according to a grid arrangement.

[0106] As described elsewhere herein, an acoustic sensor array can include acoustic sensor elements arranged in any of a variety of configurations. Figures 3B and 3C are schematic diagrams illustrating exemplary acoustic sensor array configurations. Figure 3B shows an acoustic sensor array 390 including multiple acoustic sensor elements evenly spaced in a substantially square grid. A distance measurement tool 314 and a camera array 316 are disposed within the acoustic sensor array 390. In the illustrated example, the acoustic sensor elements in the acoustic sensor array 390 are the same type of sensor, although in some embodiments, different types of acoustic sensor elements can be used within the array 390.

[0107] FIG. 3C illustrates multiple acoustic sensor arrays. Acoustic sensor arrays 392, 394, and 396 each include multiple acoustic sensor elements arranged in arrays of different shapes. In the example of FIG. 3C, acoustic sensor arrays 392, 394, and 396 can be used separately or together in any combination to create sensor arrays of various sizes. In the illustrated embodiment, the sensor elements of array 396 are spaced closer together than the sensor elements of array 392. In one example, array 396 is designed to sense high-frequency acoustic data, and array 392 is designed to sense low-frequency acoustic data.

[0108] In various embodiments, arrays 392, 394, and 396 can include the same or different types of acoustic sensor elements. For example, acoustic sensor array 392 can include sensor elements having a lower frequency operating range than the frequency operating range of the sensor elements of acoustic sensor array 396.

[0109] As described elsewhere herein, in some examples, different acoustic sensor arrays (e.g., 392, 394, 396) can be selectively turned off and on during various modes of operation (e.g., different desired frequency spectrums to be imaged). Additionally or alternatively, various acoustic sensor elements (e.g., some or all of the acoustic sensor elements in one or more sensor arrays) can be enabled or disabled according to desired system operation. For example, in some acoustic imaging processes, data from a large number of sensor elements (e.g., densely spaced sensor elements such as in sensor array 396) may slightly improve the resolution of the acoustic image data, but at the expense of the processing required to extract the acoustic image data from the data received at each sensor element. That is, in some examples, the increased processing burden (e.g., in cost, processing time, power consumption, etc.) required to process a large number of input signals (e.g., from a large number of acoustic sensor elements) is negatively compared to any additional signal resolution provided by the additional data streams. Thus, in some embodiments, it may be worthwhile to disable or ignore data from one or more acoustic sensor elements depending on the desired acoustic imaging operation.

[0110] Similar to the systems of FIGS. 3A and 3B, the system of FIG. 3C includes a distance measurement tool 314 and a camera array 316 disposed within acoustic sensor arrays 392, 394, and 396. In some examples, additional components, such as additional camera arrays (e.g., used to image a different portion of the electromagnetic spectrum from camera array 316), may be similarly disposed within acoustic sensor arrays 392, 394, and 396. While shown in FIGS. 3A-3C as being disposed within one or more acoustic sensor arrays, the distance measurement tool and / or one or more imaging tools (e.g., visible light camera modules, infrared camera modules, ultraviolet sensors, etc.) may be disposed outside the acoustic sensor array(s). In some such examples, the distance measurement tool and / or one or more imaging tools disposed outside the acoustic sensor array(s) may be supported by the acoustic imaging tool, e.g., by a housing that houses the acoustic sensor array, or may be disposed outside the housing of the acoustic imaging tool.

[0111] In some instances, general misalignment between an acoustic sensor array and an imaging tool, such as a camera module, can lead to misregistration of corresponding image data generated by the acoustic sensor array and the imaging tool. Figure 4A shows a schematic diagram of parallax error in generating frames of visible light image data and acoustic image data. Generally, the parallax error can be vertical, horizontal, or both. In the illustrated embodiment, the imaging tool includes an acoustic sensor array 420 and a visible light camera module 406. Visible light image frame 440 is shown captured according to field of view 441 of visible light camera module 406, while acoustic image frame 450 is shown captured according to field of view 451 of acoustic sensor array 420.

[0112] As shown, the visible light image frame 440 and the acoustic imaging frame 450 are not aligned with one another. In some embodiments, a processor (e.g., processor 212 of FIG. 2 ) is configured to manipulate one or both of the visible light image frame 440 and the acoustic image frame 450 to align the visible light image data and the acoustic image data. Such manipulation may include shifting one image frame relative to the other. The amount by which the image frames are shifted with respect to one another may be determined based on various factors, including, for example, the distance of the target from the visible light camera module 406 and / or the acoustic sensor array 420. Such distance data may be determined, for example, using the distance measurement tool 404 or by receiving a distance value via a user interface (e.g., 216).

[0113] FIG. 4B is a schematic diagram similar to that of FIG. 4A but includes a visible light image of the scene. In the example of FIG. 4B, visible light image 442 shows a scene of multiple power lines and support towers. Acoustic image 452 includes multiple locations 454, 456, and 458, which show high-magnitude acoustic data coming from such locations. As shown, both visible light image 442 and acoustic image 452 are displayed simultaneously. However, observation of both images reveals at least one acoustic image maximum at location 458 that does not appear to coincide with a particular structure in visible light image 442. Therefore, a person observing both images can conclude that there is a registration error (e.g., parallax error) between acoustic image 452 and visible light image 442.

[0114] 5A and 5B illustrate parallax correction between a visible light image and an acoustic image. FIG. 5A shows a visible light image 542 and an acoustic image 552, similar to FIG. 4B. Acoustic image 552 includes local maxima at locations 554, 556, and 558. As can be seen, the maxima at locations 554 and 558 do not appear to correspond to any structure in the visible light image. In the example of FIG. 5B, visible light image 542 and acoustic image 552 are overlaid with respect to each other. The local maxima at locations 554, 556, and 558 in the acoustic image appear to correspond to various locations in visible light image 542.

[0115] In use, an operator may view the representation of FIG. 5B (e.g., via display 214) to determine approximate locations within the visible scene 542 that are likely sources of received acoustic signals. Such signals may be further processed to determine information regarding the acoustic signatures of various components within the scene. In various embodiments, acoustic parameters such as frequency content, periodicity, amplitude, etc. may be analyzed for various locations within the acoustic image. When overlaid on the visible light data so that such parameters can be associated with various system components, the acoustic image data may be used to analyze various properties (e.g., performance characteristics) of objects within the visible light image.

[0116] Figures 5C and 5D are colored versions of Figures 5A and 5B. As shown in Figures 5A and 5B and more easily seen in the colored representations of Figures 5C and 5D, locations 554, 556, and 558 exhibit a circular gradation of color. As described elsewhere herein, the acoustic image data can be visually represented according to a color-matching technique, in which each pixel of the acoustic image data is colored based on the acoustic intensity at the corresponding location. Thus, in the exemplary representations of Figures 5A-5D, the circular gradation at locations 554, 556, and 558 generally represents a gradation of acoustic intensity in the imaging plane based on the backpropagated received acoustic signal.

[0117] 4A, 4B, and 5A-5D are described with respect to acoustic image data and visible light image data, it will be understood that such processes may similarly be performed using a variety of electromagnetic image data. For example, as described elsewhere herein, in various embodiments, various such processes may be performed using a combination of acoustic image data and one or more of visible light image data, infrared image data, ultraviolet image data, etc.

[0118] As described elsewhere herein, in some embodiments, backpropagation of acoustic signals to form an acoustic image can be based on distance values ​​to a target. That is, in some examples, the backpropagation calculations can be based on distance and can include determining a two-dimensional acoustic scene located at that distance from the acoustic sensor array. Given a two-dimensional imaging plane, spherical acoustic waves emanating from a source within the plane generally appear circular in cross section, with intensity decaying radially, as shown in FIGS. 5A-5B.

[0119] In one such example, portions of the acoustic scene representing data that are not located at the distance-to-target used in the backpropagation calculations introduce errors into the acoustic image data, such as inaccuracies in the location of one or more sounds in the scene. Such errors can lead to disparity errors between the acoustic image data and other image data when the acoustic image is displayed simultaneously (e.g., blended, combined, etc.) with other image data (e.g., electromagnetic image data such as visible, infrared, or ultraviolet image data). Thus, in one embodiment, one technique for correcting disparity errors (e.g., as shown in FIGS. 5A and 5B ) involves adjusting the distance-to-target values ​​used in the backpropagation calculations to generate the acoustic image data.

[0120] In some cases, the system may be configured to perform a backpropagation process using a first distance-to-target value and display a display image such as that shown in FIG. 5A, where the acoustic image data and another data stream may not be aligned. The acoustic analysis system may then adjust the distance-to-target value used for backpropagation, perform backpropagation again, and update the display image with the new acoustic image data. This process may be repeated, with the acoustic analysis system iterating through multiple distance-to-target values ​​while the user observes the resulting display image on the display. As the distance-to-target value changes, the user may observe a gradual transition from the display image shown in FIG. 5A to the display image shown in FIG. 5B. In some such cases, the user may visually observe when the acoustic image data appears properly overlaid on another data stream, such as electromagnetic image data. The user may notify the acoustic analysis system that the acoustic image data appears properly overlaid, indicating to the system that the distance-to-target value used to perform the most recent backpropagation was approximately correct, and may save that distance value in memory as the correct distance to the target. Similarly, the user may manually adjust the distance to the target value as the updated backpropagation process updates the display image with the new distance value until the user is satisfied that the acoustic image data is properly registered. The user may choose to save the current distance to the target in the acoustic analysis system as the current distance to the target.

[0121] In one example, correcting for parallax error may include adjusting the position of the acoustic image data relative to other image data (e.g., electromagnetic image data) by a predetermined amount and in a predetermined direction based on range data to the target, in one embodiment, such adjustment is independent of the generation of the acoustic image data by backpropagating the acoustic signal to the identified range to the target.

[0122] In some embodiments, in addition to being used to generate acoustic image data and reduce parallax error between the acoustic image data and other image data, the distance-to-target value may be used for other decisions. For example, in one example, a processor (e.g., 212) may use the distance-to-target value to focus or assist a user in focusing an image, such as an infrared image, as described in U.S. Patent No. 7,538,326, which is incorporated by reference. As described therein, this may similarly be used to correct parallax error between visible light image data and infrared image data. Thus, in one example, the distance value may be used to overlay acoustic image data with electromagnetic image data, such as infrared image data or visible light image data.

[0123] As described elsewhere herein, in some examples, a distance measurement tool (e.g., 204) is configured to provide distance information that can be used by a processor (e.g., 212) to generate and overlay acoustic image data. In some embodiments, the distance measurement tool includes a laser rangefinder configured to emit light into the target scene at a location where the distance is to be measured. In some such examples, the laser rangefinder can emit light in the visible spectrum, allowing a user to view the laser spot on the physical scene and confirm that the rangefinder is measuring the distance to the desired portion of the scene. Additionally or alternatively, the laser rangefinder is configured to emit light in a spectrum to which one or more imaging components (e.g., camera modules) are sensitive. Thus, a user viewing the target scene via an analysis tool (e.g., via display 214) can observe the laser spot in the scene to confirm that the laser is measuring the distance to the correct location in the target scene. In one example, the processor (e.g., 212) can be configured to generate a reference mark in the display image representing the location where the laser spot will be placed in the acoustic scene based on the current distance value (e.g., based on a known distance-based parallax relationship between the laser rangefinder and the acoustic sensor array). The location of the reference mark can be compared to the location of the actual laser mark (e.g., graphically on the display and / or physically in the target scene), and the scene can be adjusted until the reference mark and the laser are aligned. Such a process can be performed similar to the infrared superposition and focus technique described in U.S. Patent No. 7,538,326, which is incorporated by reference.

[0124] 6 is a process flow diagram illustrating an exemplary method for generating a final image that combines acoustic image data and electromagnetic image data. The method includes receiving (680) acoustic signals via an acoustic sensor array and receiving (682) distance information. The distance information can be received via a distance measurement device and / or user interface, for example, via manual input or as a result of a distance calibration process in which distance is determined based on an observed registration.

[0125] The method further includes backpropagating the received acoustic signals to determine acoustic image data representative of the acoustic scene 684. As described elsewhere herein, backpropagation can include analyzing multiple acoustic signals received at multiple sensor elements in the acoustic sensor array in combination with received distance information to determine a source pattern of the received acoustic signals.

[0126] 6 further includes capturing electromagnetic image data (686) and overlaying the electromagnetic image data on the acoustic image data (688). In some embodiments, overlaying the electromagnetic image data on the acoustic image data occurs as part of the backpropagation step for generating the acoustic image data (684). In other examples, overlaying the electromagnetic image data on the acoustic image data occurs separately from generating the acoustic image data.

[0127] The method of FIG. 6 includes combining (690) the acoustic image data with the electromagnetic image data to generate a display image. As described elsewhere herein, combining the electromagnetic image data and the acoustic image data may include alpha-blending the electromagnetic image data and the acoustic image data. Combining the image data may include overlaying one image dataset over another, such as in a picture-in-picture mode, or overlaying an image dataset at a location where a particular condition (e.g., an alarm condition) is met. The display image may be presented to a user, for example, via a display supported by a housing that supports the acoustic sensor array and / or via a display separate from the sensor array, such as the display of an external device (e.g., a smartphone, tablet, computer, etc.).

[0128] Additionally or alternatively, the display image may be saved to local (e.g., on-board) and / or remote memory for future viewing. In some embodiments, the saved display image may include metadata that allows for future adjustment of display image characteristics, such as blending ratios, backpropagation distances, or other parameters used to generate the image. In some examples, raw acoustic signal data and / or electromagnetic image data may be saved along with the display image for subsequent processing or analysis.

[0129] Although shown as a method of combining acoustic image data and electromagnetic image data to generate a final image, it will be appreciated that the method of Figure 6 may be used to combine acoustic image data with one or more sets of image data spanning any portion of the electromagnetic spectrum, such as visible light image data, infrared image data, ultraviolet image data, etc. In one such example, multiple sets of image data, such as visible light image data and infrared image data, may both be combined with acoustic image data to generate a display image via a method similar to that described with respect to Figure 6.

[0130] In some examples, receiving 680 the acoustic signals via the sensor arrays can include selecting an acoustic sensor array for receiving the acoustic signals. For example, as described with respect to FIGS. 3A-C, an acoustic analysis system can include multiple acoustic sensor arrays that can be adapted to analyze acoustic signals of varying frequencies. Additionally or alternatively, in some examples, different acoustic sensor arrays can be useful for analyzing acoustic signals propagating from different distances. In some embodiments, different arrays can be nested within one another. Additionally or alternatively, subarrays can be selectively used to receive acoustic image signals.

[0131] 3A shows a first array 320 and a second array 322 nested within the first array. In an exemplary embodiment, the first array 320 may include a sensor array configured (e.g., spaced apart) to receive acoustic signals and generate acoustic image data at frequencies in a first frequency range. The second array 322 may include, for example, a second sensor array configured to be used alone or in combination with all or a portion of the first array 320 to generate acoustic image data at frequencies in a second frequency range.

[0132] 3C illustrates a first array 392, a second array 394 at least partially nested within the first array 392, and a third array 396 at least partially nested within the first array 392 and the second array 394. In some embodiments, the first array 392 may be configured to receive acoustic signals and generate acoustic image data at frequencies in a first frequency range. The second array 394 may be used in conjunction with all or a portion of the first array 392 to receive acoustic signals and generate acoustic image data at frequencies in a second frequency range. The third array 396 may be used alone, in conjunction with all or a portion of the second array 394, and / or in conjunction with all or a portion of the first array 392 to receive acoustic signals and generate acoustic image data at frequencies in a third frequency range.

[0133] In some embodiments, in a nested array configuration, acoustic sensor elements from one array may be positioned between acoustic sensor elements, such as elements of a third array 396 generally between elements of the first array 392. In one such example, acoustic sensor elements in a nested array (e.g., third array 396) may be positioned in the same plane as, in front of, or behind the acoustic sensor elements in the array in which it is nested (e.g., first array 392).

[0134] In various implementations, arrays used to sense higher frequency acoustic signals generally require shorter distances between individual sensors. Thus, with reference to FIG. 3C , for example, third array 396 may be more suitable for performing acoustic imaging processes involving high frequency acoustic signals. Other sensor arrays (e.g., first array 392) may be sufficient for performing acoustic imaging processes involving lower frequency signals and may be used to reduce the computational demands of processing signals from fewer acoustic sensor elements when compared to array 396. Thus, in some examples, a high frequency sensor array may be nested within a low frequency sensor array. As described elsewhere herein, such arrays may generally operate individually (e.g., via switching between active arrays) or together.

[0135] In addition to, or instead of, selecting an appropriate sensor array based on an expected / desired frequency spectrum for analysis, in some instances, different sensor arrays may be better suited to performing acoustic imaging processes at different distances to the target scene. For example, in some embodiments, when the distance between the acoustic sensor array and the target scene is small, outer sensor elements in the acoustic sensor array may receive significantly less useful acoustic information from the target scene than more centrally located sensor elements.

[0136] On the other hand, when the distance between the acoustic sensor array and the target scene is large, closely spaced acoustic sensor elements may not provide useful information by themselves. That is, when the first and second acoustic sensor elements are close to each other and the target scene is generally far away, the second acoustic sensor element may not provide information that is significantly different from the first acoustic sensor element. Therefore, the data streams from such first and second sensor elements may be redundant and unnecessarily consume processing time and resources for analysis.

[0137] As described elsewhere herein, in addition to influencing which sensor array is best suited to perform acoustic imaging, the distance to the target can also be used in performing backpropagation to determine acoustic image data from received acoustic signals. However, in addition to being an input to the backpropagation algorithm, the distance to the target can also be used to select an appropriate backpropagation algorithm to use. For example, in some instances, at long distances, spherically propagating acoustic waves can be approximated as being substantially planar compared to the size of the acoustic sensor array. Thus, in some embodiments, when the distance to the target is large, the backpropagation of the received acoustic signals can include acoustic beamforming calculations. However, closer to the source of the acoustic waves, the planar approximation of the acoustic waves may not be appropriate. Therefore, various backpropagation algorithms, such as near-field acoustic holography, can be used.

[0138] As described, the distance metric to the target can be used in various ways in the acoustic imaging process, such as determining the active sensor array(s), determining the backpropagation algorithm, executing the backpropagation algorithm, and / or overlaying the resulting acoustic image with electromagnetic image data (e.g., visible light, infrared, etc.). Figure 7 is a process flow diagram illustrating an exemplary process for generating acoustic image data from received acoustic signals.

[0139] 7 includes receiving distance information (780), for example, from a distance measurement device or receiving input distance information, such as via a user interface. The method further includes selecting (782) one or more acoustic sensor array(s) for performing acoustic imaging based on the received distance information. As discussed, in various examples, the selected array(s) can include a single array, a combination of multiple arrays, or a portion of one or more arrays.

[0140] 7 further includes selecting 784 a processing technique for performing acoustic imaging based on the received distance information. In one example, selecting the processing technique can include selecting a backpropagation algorithm for generating acoustic image data from the acoustic signals.

[0141] After selecting 782 an acoustic sensor array to perform acoustic imaging and selecting 784 a processing technique, the method includes receiving 786 acoustic signals via the selected acoustic sensor array. The received acoustic signals are then backpropagated using the distance and the selected processing technique to determine 788 acoustic image data.

[0142] 7 can be performed by a user, an acoustic analysis system (e.g., via processor 212), or a combination thereof. For example, in some embodiments, the processor may be configured to receive (780) distance information via a distance measurement tool and / or user input. In some examples, for example, when the distance to the object is known and / or difficult to analyze via a distance measurement tool (e.g., small object size and / or long distance to the target), a user may input a value that overrides the measured distance to use as the distance information. The processor may further be configured to automatically select an appropriate acoustic sensor array to perform acoustic imaging based on the received distance information, for example, using a lookup table or other database. In some embodiments, selecting an acoustic sensor array includes enabling and / or disabling one or more acoustic sensor elements to utilize the desired acoustic sensor array.

[0143] Similarly, in some examples, the processor may be configured to automatically select a processing technique (e.g., a backpropagation algorithm) for performing acoustic imaging based on the received distance information. In some such examples, this may include selecting one from a number of known processing techniques stored in memory. Additionally or alternatively, selecting a processing technique may involve adjusting portions of a single algorithm to arrive at a desired processing technique. For example, in some embodiments, a single backpropagation algorithm may include multiple terms and variables (e.g., based on the distance information). In some such examples, selecting a processing technique (784) may involve defining one or more values ​​in a single algorithm, such as adjusting the coefficients of one or more terms (e.g., setting various coefficients to zero or one).

[0144] Thus, in certain embodiments, the acoustic imaging system may automate some steps in the acoustic imaging process by suggesting and / or automatically implementing selected acoustic sensor arrays and / or processing techniques (e.g., backpropagation algorithms) based on received distance data. This may speed up, improve, and simplify the acoustic imaging process, eliminating the need for acoustic imaging experts to perform the acoustic imaging process. Thus, in various examples, the acoustic imaging system may automatically implement such parameters, notify the user that such parameters are about to be implemented, ask the user for permission to implement such parameters, and suggest such parameters for manual input by the user.

[0145] Automatic selection and / or suggestion of such parameters (e.g., processing techniques, sensor arrays) can be useful for optimizing the localization of acoustic image data with respect to other forms of image data, processing speed, and analysis of acoustic image data. For example, as described elsewhere herein, accurate backpropagation determination (e.g., using appropriate algorithms and / or accurate distance metrics) can reduce parallax errors between acoustic image data and other (e.g., electromagnetic, such as visible light, infrared, etc.) image data. Furthermore, utilizing appropriate algorithms and / or sensor arrays, as may be automatically selected or suggested by the acoustic analysis system, can optimize the accuracy of thermal image data to enable analysis of received acoustic data.

[0146] As described, in some examples, the acoustic analysis system may be configured to automatically select an algorithm and / or a sensor array for performing the acoustic imaging process based on the received distance information. In some such embodiments, the system includes a lookup table, stored, for example, in memory, for determining which of a plurality of backpropagation algorithms and acoustic sensor arrays to use for determining the acoustic image data. Figure 8 shows an exemplary lookup table for determining the appropriate algorithm and sensor array to use during the acoustic imaging process.

[0147] In the illustrated example, the lookup table of FIG. 8 includes N columns, each representing a different array: Array 1, Array 2, ..., Array N. In various examples, each array includes a unique set of arranged acoustic sensor elements. Different arrays may include sensor elements arranged in grids (e.g., array 392 and array 396 in FIG. 3C). Arrays in the lookup table may also include combinations of sensor elements from one or more such grids. Generally, in some embodiments, Array 1, Array 2, ..., Array N each correspond to a unique combination of acoustic sensor elements. Some of such combinations may include the entire set of sensor elements arranged in a particular grid, or may include a subset of sensor elements arranged in a particular grid. Any of the various combinations of acoustic sensor elements are possible choices for use as sensor arrays in the lookup table.

[0148] 8 further includes M rows, each representing a different algorithm: Algorithm 1, Algorithm 2, ..., Algorithm M. In some examples, the different algorithms may include different processes for performing backpropagation analysis of the received acoustic signal. As described elsewhere herein, in some examples, the different algorithms may be similar to each other while having different coefficients and / or terms for modifying the backpropagation results.

[0149] The example lookup table of Figure 8 includes M x N entries. In one embodiment, an acoustic analysis system utilizing such a lookup table is configured to analyze received distance information and categorize the distance information into one of M x N bins, with each bin corresponding to an entry in the lookup table of Figure 8. In such an example, when the acoustic analysis system receives distance information, the system can locate entry (i, j) in the lookup table that corresponds to the bin in which the distance information resides and determine the appropriate algorithm and sensor array to use during the acoustic imaging process. For example, if the received distance information corresponds to the bin associated with entry (i, j), the acoustic analysis system can automatically utilize or suggest the use of algorithm i and array j for the acoustic imaging process.

[0150] In various such examples, the distance information bins can correspond to uniformly sized distance ranges, e.g., a first bin corresponds to a distance of within 1 foot, a second bin corresponds to a distance of 1-2 feet, etc. In other examples, the bins need not correspond to uniformly sized distance spans. Furthermore, in some embodiments, fewer than M×N bins can be used. For example, in some embodiments, there may be an algorithm (e.g., algorithm x) that has never been used with a particular array (e.g., array y). Thus, in such examples, there is no corresponding distance information bin for entry (x, y) in the M×N lookup table.

[0151] In some embodiments, statistical analysis of the populated range bins can be used to identify the most common distances or ranges within the target scene. In some such embodiments, the range bin with the greatest number of corresponding locations (e.g., the greatest number of locations for acoustic signals) may be used as the distance information in the process of FIG. 7. That is, in some embodiments, the utilized acoustic sensor array and / or processing technique can be implemented and / or recommended based on a statistical analysis of the distance distribution of various objects within the target scene. This can increase the likelihood that the sensor array and / or processing technique used for acoustic imaging of the scene will be appropriate for the greatest number of locations within the acoustic scene.

[0152] Additionally or alternatively, parameters other than distance information may be used to select an appropriate sensor array and / or processing technique to use in generating the acoustic image data. As described elsewhere herein, various sensor arrays may be configured to be sensitive to particular frequencies and / or frequency bands. In some examples, similar but different backpropagation calculations may be used according to different acoustic signal frequency content. Thus, in some examples, one or more parameters may be used to determine the processing technique and / or acoustic sensor array.

[0153] In some embodiments, an acoustic analysis system may be used to initially analyze various parameters for received acoustic signal processing / analysis. Referring back to FIG. 7, a method for generating acoustic image data may include, after receiving (786) an acoustic signal, analyzing (790) the frequency content of the received signal. In one such example, if acoustic sensor array(s) and / or processing techniques are selected (e.g., via steps 782 and / or 784, respectively), the method may include updating (792) the selected array(s) and / or the selected processing technique based on the analyzed frequency content.

[0154] After updating the sensor array(s) and / or processing technique, the method can perform various actions using the updated parameters. For example, if the selected sensor array is updated (792) based on the analyzed frequency content (790), new acoustic signals can be received from the (newly) selected acoustic sensor array (786) and then backpropagated (788) to determine acoustic image data. Alternatively, if the processing technique is updated at 792, already captured acoustic signals can be backpropagated according to the updated processing technique to determine updated acoustic image data. If both the processing technique and the sensor array are updated, new acoustic signals can be received using the updated sensor array and backpropagated according to the updated processing technique.

[0155] In some embodiments, the acoustic analysis system can receive 778 frequency information without analyzing 790 the frequency content of the received acoustic signal. For example, in some instances, the acoustic analysis system can receive information regarding a desired or expected frequency range for future acoustic analysis. In some such instances, the desired or expected frequency information can be used to select one or more sensor arrays and / or processing techniques that best match the frequency information. In some such instances, selecting 782 the acoustic sensor array(s) and / or selecting 784 the processing technique can be based on the received frequency information in addition to, or instead of, the received distance information.

[0156] In some examples, received acoustic signals (e.g., received via acoustic sensor elements) may be analyzed, for example, via a processor (e.g., 210) of the acoustic analysis system. Such analysis may be used to determine one or more characteristics of the acoustic signals, such as frequency, intensity, periodicity, apparent proximity (e.g., distance estimated based on received acoustic signals), measured proximity, or any combination thereof. In some examples, acoustic image data may be filtered, for example, to display only acoustic image data representative of acoustic signals having particular frequency content, periodicity, etc. In some examples, any number of such filters may be applied simultaneously.

[0157] As described elsewhere herein, in some embodiments, a series of frames of acoustic image data may be captured over time, similar to acoustic video data. Additionally or alternatively, even if acoustic image data is not repeatedly generated, in some instances, the acoustic signal is repeatedly sampled and analyzed. Thus, parameters of the acoustic data, such as frequency, may be monitored over time with or without repeated acoustic image data generation (e.g., video).

[0158] 9A is an exemplary plot of the frequency content of received image data over time in an acoustic scene. As shown, the acoustic scene represented by the plot of FIG. 9A generally includes four sustained frequencies over time, labeled as Frequency 1, Frequency 2, Frequency 3, and Frequency 4. Frequency data, such as the frequency content of the target scene, can be determined by processing the received acoustic signal using, for example, a Fast Fourier Transform (FFT) or other known frequency analysis methods.

[0159] 9B illustrates an exemplary scene including multiple locations emitting acoustic signals. In the illustrated image, acoustic image data is combined with visible light image data to illustrate acoustic signals present at locations 910, 920, 930, and 940. In certain embodiments, the acoustic analysis system is configured to display acoustic image data for any detected frequency range. For example, in the exemplary embodiment, location 910 includes acoustic image data including frequency 1, location 920 includes acoustic image data including frequency 2, location 930 includes acoustic image data including frequency 3, and location 940 includes acoustic image data including frequency 4.

[0160] In some such examples, displaying a representative frequency range of the acoustic image data is a selectable mode of operation. Similarly, in some embodiments, the acoustic analysis system is configured to display acoustic image data representing only frequencies within a predetermined frequency band. In some such examples, displaying acoustic image data representing a predetermined frequency range includes selecting one or more acoustic sensor arrays to receive acoustic signals from which to generate the acoustic image data. Such arrays may be configured to receive selective frequency ranges. Similarly, in some examples, one or more filters may be used to limit the frequency content used to generate the acoustic image data. Additionally or alternatively, in some embodiments, acoustic image data containing information representing a wide range of frequencies may be analyzed and displayed on a display only if the acoustic image data meets a predetermined condition (e.g., falls within a predetermined frequency range).

[0161] 9C shows multiple combined acoustic and visible light image data at multiple predefined frequency ranges. A first image includes acoustic image data at a first location 910 that includes frequency content at frequency 1. A second image includes acoustic image data at a second location 920 that includes frequency content at frequency 2. A third image includes acoustic image data at a third location 930 that includes frequency content at frequency 3. A fourth image includes acoustic image data at a fourth location 940 that includes frequency content at frequency 4.

[0162] In an exemplary embodiment, a user may select various frequency ranges, such as ranges including frequency 1, frequency 2, frequency 3, or frequency 4, to filter acoustic image data representing frequency content outside the selected frequency range. Thus, in such an example, either a first, second, third, or fourth image may be displayed as a result of the desired frequency range selected by the user.

[0163] Additionally or alternatively, in some examples, the sound analysis system may cycle through multiple display images, each having different frequency content. For example, with reference to Figure 9C, in an exemplary embodiment, the sound analysis system may display first, second, third, and fourth images, as indicated by the arrows in Figure 9C.

[0164] In some examples, the display image can include text or other indications representing the frequency content displayed in the image, allowing a user to observe which locations within the image contain acoustic image data representing particular frequency content. For example, with respect to FIG. 9C , each image can show a textual representation of the frequencies represented in the acoustic image data. With respect to FIG. 9B , an image showing multiple frequency ranges can include an indication of the frequency content at each location that contains acoustic image data. In one such example, a user can, for example, select a location within the image via a user interface and view the frequency content present at that location in the acoustic scene. For example, a user may select a first location 910, and the acoustic analysis system may present the frequency content of the first location (e.g., Frequency 1). Thus, in various examples, a user can use the acoustic analysis system to analyze the frequency content of an acoustic scene, such as by seeing where in the scene corresponds to particular frequency content and / or by seeing what frequency content is present at various locations.

[0165] During an exemplary acoustic imaging operation, filtering acoustic image data by frequency can be useful, for example, to reduce image clutter from background or other insignificant sounds. In an exemplary acoustic imaging procedure, a user may wish to remove background sounds, such as floor noise in an industrial environment. In one such example, the background noise may include primarily low-frequency noise. Thus, the user may select to display acoustic image data representing acoustic signals greater than a predetermined frequency (e.g., 10 kHz). In another example, a user may wish to analyze a particular object that typically emits acoustic signals within a particular range, such as corona discharges from transmission lines (e.g., as shown in FIGS. 5A-5D). In such an example, the user may select a particular frequency range for acoustic imaging (e.g., 11 kHz to 14 kHz for corona discharges).

[0166] In some examples, the acoustic analysis system may be used to analyze and / or present information related to the intensity of a received acoustic signal. For example, in some embodiments, backpropagating the received acoustic signal may include determining acoustic intensity values ​​at multiple locations within the acoustic scene. In some examples, acoustic image data is included in a display image only if the intensity of the acoustic signal meets one or more predetermined requirements, similar to the frequency described above.

[0167] In various such embodiments, the display image may include acoustic image data representing acoustic signals above a predetermined threshold (e.g., 15 dB), below a predetermined threshold (e.g., 100 dB), or within a predetermined intensity range (e.g., 15 dB to 40 dB). In some embodiments, the threshold may be based on a statistical analysis of the acoustic scene, such as above or below a standard deviation from a mean acoustic intensity.

[0168] Similar to the frequency information described above, in some embodiments, limiting the acoustic image data to represent acoustic signals that meet one or more intensity requirements may include filtering the received acoustic signals such that only received signals that meet predetermined conditions are used to generate the acoustic image data. In other examples, the acoustic image data is filtered to adjust which acoustic image data is displayed.

[0169] Additionally or alternatively, in some embodiments, the acoustic intensity at a location within the acoustic scene may be monitored over time (e.g., in combination with a video acoustic image representation or via background analysis without necessarily updating the display image). In one such example, the predetermined requirements for displaying the acoustic image data may include the amount or rate of change of the acoustic intensity at a location within the image.

[0170] 10A and 10B are exemplary display images including combined visible light image data and acoustic image data. FIG. 10A illustrates a display image including acoustic image data indicated at multiple locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. In some examples, the intensity values ​​can be toned, e.g., acoustic intensity values ​​are assigned colors based on a predetermined toning technique. In exemplary embodiments, the intensity values ​​can be categorized according to intensity ranges (e.g., 10 dB to 20 dB, 20 dB to 30 dB, etc.). Each intensity range can be associated with a particular color according to the toning technique. The acoustic image data can include multiple pixels, each colored with a color associated with the intensity range within which the intensity represented by the pixel of the acoustic image data falls. In addition to, or instead of, being distinguished by color, different intensities can be distinguished according to other characteristics, such as transparency (e.g., in an image overlay in which acoustic image data is overlaid on other image data).

[0171] Other parameters may also be color-tuned, such as the rate of change of sound intensity. Similar to intensity, changes in the rate of change of sound intensity can be color-tuned such that parts of the scene exhibiting different rates and / or amounts of sound intensity change are displayed in different colors.

[0172] In the illustrated example, the acoustic image data is toned according to an intensity palette, such that acoustic image data representing different acoustic signal intensities are shown in different colors and / or shades. For example, acoustic image data at locations 1010 and 1030 show a toned representation of a first intensity, locations 1040, 1060, and 1080 show a toned representation of a second intensity, and locations 1020, 1050, 1070, and 1090 show a toned representation of a third intensity. As shown in the exemplary representation of FIG. 10A , each location showing a toned representation of the acoustic image data shows a circular pattern with a color gradation extending from the center outward. This may be due to attenuation of acoustic intensity as the signal propagates from the source of the acoustic signal.

[0173] In the example of Figure 10A, the acoustic image data is combined with the visible light image data to generate a display image, which can be presented to a user, for example, via a display. A user can view the display image of Figure 10A to see which locations within the visible scene are generating acoustic signals, as well as the strength of such signals. Thus, a user can quickly and easily observe which locations are emitting sounds and compare the strength of sounds coming from various locations within the scene.

[0174] Similar to the discussion of frequency elsewhere herein, in some embodiments, acoustic image data may be presented only if the corresponding acoustic signal meets a predetermined intensity condition. Figure 10B shows an exemplary display image similar to that of Figure 10A, including visible light image data and an acoustic image representing an acoustic signal above a predetermined threshold. As shown, of locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 in Figure 10A that contain acoustic image data, only locations 1020, 1050, 1070, and 1090 contain acoustic image data representing an acoustic signal that meets the predetermined condition.

[0175] In an exemplary scenario, Figure 10A may include all acoustic image data above the noise floor threshold at each of locations 1010-1090, while Figure 10B shows the same scene as Figure 10A, but only acoustic image data with intensities above 40 dB. This can help a user identify which sound sources within an environment (e.g., within the target scene of Figures 10A and 10B) are contributing to a particular sound (e.g., the loudest sound in the scene).

[0176] As described elsewhere herein, in addition to or instead of being directly compared to an intensity threshold (e.g., 40 dB), in some such examples, the predetermined requirement for displaying acoustic image data may include an amount or rate of change of acoustic intensity at a certain location within the image. In some such examples, acoustic image data may be presented only if the rate or amount of change of acoustic intensity at a given location meets a predetermined condition (e.g., greater than a threshold, less than a threshold, within a predetermined range, etc.). In some embodiments, the amount or rate of change of acoustic intensity may be displayed as intensity acoustic image data or in combination with the intensity acoustic image data, with a toned intensity change metric. For example, in an exemplary embodiment, if rate of change is used as a threshold for determining locations that include acoustic image data, the acoustic image data may include a toned intensity change rate metric for display.

[0177] In some examples, a user may manually set intensity requirements (e.g., minimum value, maximum value, range, rate of change, amount of change, etc.) for the displayed acoustic image data. As discussed elsewhere herein, including only acoustic image data that meets this intensity requirement can be achieved during acoustic image data generation (e.g., by filtering the received acoustic signal) and / or can be performed by not displaying generated acoustic image data that represent acoustic signals that do not meet the set requirement(s). In one such example, filtering the display image according to intensity values ​​can be performed after the acoustic image data and visible light image data have been captured and stored in memory. That is, the data stored in memory can be used to generate a display image that includes any number of filtering parameters, such as, for example, displaying only acoustic image data that meets predefined intensity conditions.

[0178] In some examples, setting a lower limit on the intensity of the acoustic image (e.g., displaying only acoustic image data representing acoustic signals above a predetermined intensity) can eliminate the inclusion of undesirable background or ambient sounds and / or sound reflections from the acoustic image data. In other examples, setting an upper limit on the intensity of the acoustic image (e.g., displaying only acoustic image data representing acoustic signals below a predetermined intensity) can eliminate the inclusion of anticipated loud sounds in the acoustic image data in order to observe acoustic signals that are normally obscured by such loud sounds.

[0179] Several display functions are possible. For example, similar to the frequency analysis / display described with respect to FIG. 9C, in one example, the acoustic analysis system can cycle through multiple display images showing acoustic image data, each of which meets a different intensity requirement. Similarly, in one example, a user can scroll through a series of acoustic intensity ranges to view locations within the acoustic image data having a given range of acoustic intensities.

[0180] Another parameter that can be used to analyze acoustic data is the periodicity value of the acoustic signal. Figures 11A and 11B show exemplary plots of frequency versus time for acoustic data in an acoustic scene. As shown in the plot of Figure 11A, the acoustic data includes a signal at frequency X having a first periodicity, a signal at frequency Y having a second periodicity, and a signal at frequency Z having a third periodicity. In the illustrated example, acoustic signals with different frequencies may also include different periodicities in the acoustic signal.

[0181] In some such instances, the acoustic signal may be filtered based on periodicity in addition to or instead of frequency content. For example, in some instances, multiple sources of acoustic signals within an acoustic scene may generate acoustic signals at particular frequencies. If a user wishes to isolate one such source for acoustic imaging, the user may select whether to include or exclude acoustic image data in the final display image based on the periodicity associated with the acoustic data.

[0182] FIG. 11B shows a plot of frequency versus time for an acoustic signal. As shown, the frequency increases approximately linearly with time. However, as shown, the signal contains a periodicity that is approximately constant over time. Therefore, such signals may or may not appear in the acoustic image depending on the display parameters selected. For example, a signal may meet the frequency criteria to be displayed at one time, but fall outside the displayed frequency range at other times. However, a user can choose to include or exclude such signals from the acoustic image data based on the signal's periodicity, regardless of frequency content.

[0183] In some examples, extracting acoustic signals with a particular periodicity can be useful for analyzing a particular portion of a target scene (e.g., a particular device or type of device that typically operates with a particular periodicity). For example, if a target object operates with a particular periodicity (e.g., once per second), filtering out signals with a different periodicity can improve the acoustic analysis of the target object. For example, referring to FIG. 11B , if a target object operates with a periodicity of 4, isolating signals with a periodicity of 4 for analysis may result in an improved analysis of the target object. For example, a target object may emit a sound with a periodicity of 4 but with increasing frequency, as shown in FIG. 11B . This may mean that the object's characteristics may be changing (e.g., increased torque or load) and should be investigated.

[0184] In an exemplary acoustic imaging process, background noise (e.g., floor noise in an industrial environment, wind in an outdoor environment, etc.) is generally not periodic, but certain target objects in the scene emit periodic acoustic signals (e.g., machinery operating at regular intervals). Therefore, a user may choose to exclude non-periodic acoustic signals from the acoustic image to remove the background signal and more clearly present the target acoustic data. In another example, a user may be attempting to find the source of a constant tone and therefore may choose to exclude periodic signals from the acoustic image data that may obscure the display of the constant tone. In general, a user may choose to include acoustic signals above a particular periodicity, below a particular periodicity, or within a desired range of periodicity in the acoustic image data. In various examples, periodicity may be identified either by the length of time between periodic signals or by the frequency of occurrence of the periodic signals. Similar to the frequency shown in FIG. 11B, analysis of intensity at a given periodicity (e.g., due to a target object operating at that periodicity) may be used to track how the acoustic signal from the object changes over time. In general, in some embodiments, the periodicity may be used to perform rate of change analysis of various parameters such as frequency, intensity, etc.

[0185] As described elsewhere herein, in some examples, various portions of a target scene can be associated with different distances from the acoustic imaging sensor. For example, in some embodiments, the distance information can include three-dimensional depth information for various portions within the scene. Additionally or alternatively, a user may be able to measure (e.g., with a laser distance tool) or manually enter distance values ​​associated with multiple locations within the scene. In some examples, such different distance values ​​for various portions of the scene can be used to adjust backpropagation calculations at such locations to correspond to the particular distance values ​​at those locations.

[0186] Additionally or alternatively, if different portions of a scene are associated with different distance values, proximity (e.g., measured proximity and / or apparent proximity) from the acoustic sensor array can be another distinguishing parameter between such portions. For example, with reference to FIG. 10B , locations 1020, 1050, 1070, and 1090 are each associated with a different distance value. In one example, similar to the frequency or periodicity described elsewhere herein, a user can select a particular distance range for the acoustic image data to be included in the display. For example, a user may select to display only acoustic image data representing acoustic signals closer than a predetermined distance, farther than a predetermined distance, or within a predetermined distance range.

[0187] Furthermore, in some embodiments, similar to the frequency aspects of FIG. 9C , the acoustic analysis system may be configured to cycle through multiple distance ranges and display only acoustic image data representing acoustic signals emitted from locations within the target scene that satisfy the current distance range. Such cycling through various displays can help a user visually distinguish information between different acoustic signals. For example, in some cases, objects may appear close to each other from the line of sight of an associated electromagnetic imaging tool (e.g., a visible light camera module), and therefore acoustic image data for such objects combined with electromagnetic image data may be difficult to distinguish. However, if the objects are separated by depth differences, cycling through different depth ranges of acoustic image data can be used to separate each source of acoustic data from other sources.

[0188] In general, the acoustic analysis system may be configured to apply various settings to include and / or exclude acoustic image data representing acoustic signals that satisfy one or more predefined parameters. In one example, the acoustic analysis system may be used to select multiple conditions that must be satisfied by an acoustic signal for the acoustic image data to indicate that the acoustic signal is being displayed, for example, in a display image.

[0189] For example, with respect to Figures 10A and 10B, only acoustic signals in the scene of Figure 10A that exceed a threshold intensity are shown in Figure 10B. However, additional or alternative limitations are possible. For example, in some embodiments, a user may further filter the acoustic image data so that acoustic image data is shown only for acoustic signals that have frequency content within a predetermined frequency range and / or have a predetermined periodicity. In an exemplary embodiment, limited to a predetermined frequency and / or target periodicity, acoustic image data may be excluded from additional locations, such as 1020 and 1090.

[0190] In general, a user can apply any number of acoustic data requirements to include or exclude acoustic image data from a display image, including parameters such as intensity, frequency, periodicity, apparent proximity, measured proximity, sound pressure, particle velocity, particle displacement, acoustic power, sound energy, sound energy density, sound exposure, pitch, amplitude, brightness, harmonics, rate of change of such parameters, etc. Furthermore, in some embodiments, a user may combine requirements using any suitable logical combination, such as AND, OR, XOR, etc. For example, a user may want to display only acoustic signals having (intensity above a predetermined threshold) AND (frequency within a predetermined range).

[0191] Additionally or alternatively, the acoustic analysis system may be configured to cycle through one or more parameter ranges to show different portions of the target scene, as shown with respect to cycling through multiple frequencies in Figure 9C. Typically, one or more parameters may be cycled in this manner. For example, a parameter (e.g., intensity) may be divided into multiple ranges (e.g., 10 dB to 20 dB and 20 dB to 30 dB), and the acoustic analysis system may cycle through such ranges, displaying all acoustic image data that falls within a first range, then all acoustic image data that falls within a second range.

[0192] Similarly, in some embodiments, the acoustic analysis system may be configured to combine parameter requirements by cycling through nested ranges. For example, in an exemplary embodiment, acoustic image data satisfying a first intensity range and a first frequency range may be displayed. The displayed frequency range may be cycled while limiting the displayed acoustic image data to acoustic signals satisfying the first intensity range. After cycling through the frequency range, the intensity range may be updated to a second intensity range so that the displayed acoustic image data satisfies both the second intensity range and the first frequency range. Similar to the process incorporating the first intensity range, the frequency range may be similarly cycled while maintaining the second intensity range. This process may continue until all combinations of frequency and intensity ranges are satisfied. Similar such processes may be performed for any of a number of parameters.

[0193] Additionally or alternatively, in some embodiments, the acoustic analysis system may be configured to identify and distinguish between multiple sounds within an acoustic scene. For example, with reference to FIG. 9B , the acoustic analysis system may be configured to identify four distinct sounds at locations 910, 920, 930, and 940. The system may be configured to cycle through multiple displays showing acoustic image data at a single distinct location, similar to that shown in FIG. 9C , each of which does not necessarily depend on parameter values. Similarly, such cycling between acoustic image data at distinct locations may be performed after one or more parameter requirements have constrained the displayed acoustic image data.

[0194] For example, with respect to Figures 10A and 10B, before an intensity threshold is applied, the acoustic analysis system may cycle through multiple acoustic image scenes (e.g., as display images including acoustic image scenes accompanied by visible light image data), each containing acoustic image data at a single location. In one embodiment, the illustrated example of Figure 10A cycles through 10 separate images, each containing image data at a different one of locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. However, according to one embodiment, after an intensity filter is applied so that only locations with intensities above the threshold are displayed (e.g., as in Figure 10B), the acoustic analysis system may update the cycling process to cycle through only images corresponding to locations that satisfy the filtering threshold. That is, with respect to Figure 10B, the cycling process may be updated to cycle through only four images, each showing separate acoustic image data at locations 1020, 1050, 1070, and 1090.

[0195] Thus, in various embodiments, each location within the target scene containing acoustic image data is shown in one of a plurality of cycling display images, either before or after applying one or more filters to limit what acoustic image data is displayed. Such cycling of individual acoustic source locations can assist a user viewing the images in identifying a particular acoustic source. In some embodiments, each image in the cycling contains only a single source of acoustic data, and in some such embodiments, further includes one or more parameters of the acoustic data, such as frequency content, intensity, periodicity, apparent proximity, etc.

[0196] In addition to, or instead of, cycling through images showing acoustic image data that meet certain criteria, in some examples, the location of acoustic signal sources can be detected in the acoustic image data and displayed separately from other acoustic signals. For example, with reference to FIG. 10A , in some embodiments, acoustic image data representing acoustic signals emanating from each of locations 1010-1090 can be identified and cycled through. For example, in an exemplary operating process, display images containing acoustic image data at one of locations 1010-1090 can be cycled through automatically or at the user's direction for individual analysis of each source of acoustic signals. In various embodiments, the order in which different locations of acoustic image data are displayed during cycling can depend on various parameters, such as location, proximity, intensity, frequency content, etc.

[0197] Additionally or alternatively, in some examples, acoustic image data from individual locations may be cycled through after applying one or more filters to isolate only acoustic image data that meet one or more predetermined conditions. For example, with reference to FIG. 10B , locations 1020, 1050, 1070, and 1090 are shown as including acoustic image data representing acoustic signals that meet predetermined intensity requirements. In some embodiments, such display requirements may be applied to individual cycled displays of source locations of the acoustic signals. For example, with further reference to FIG. 10B , a display image including image data from only one of locations 1020, 1050, 1070, and 1090 that meets an acoustic intensity condition may be cycled through for individual analysis at each location.

[0198] 10A and 10B, acoustic image data collected from a scene may generally be shown in FIG. 10A at locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. Such locations may contain acoustic image data representing acoustic signals having various acoustic parameters such as different intensities, frequency content, periodicity, etc.

[0199] As described elsewhere herein, a user may wish to isolate acoustic signals having one or more particular acoustic parameters, such as acoustic signals having a minimum acoustic intensity. Acoustic image data representing acoustic signals that do not meet such conditions can be filtered out of the image, e.g., the acoustic image data for locations 1020, 1050, 1070, and 1090 can be discarded as shown in FIG. 10B . However, a user may wish to further identify particular sound sources that meet a display condition (e.g., having an intensity above a threshold). Thus, a user may select to display the acoustic image data associated with locations 1020, 1050, 1070, and 1090 one at a time to view and individually analyze the location of each sound source. In various embodiments, a user may choose to manually cycle through such locations, or a processor may automatically update the display image to sequentially display the acoustic image data for each location. This may help a user further filter out and ignore acoustic signals that are not targeted but that happen to meet one or more filtering parameters applied to the image.

[0200] Although described with respect to intensity and Figures 10A and 10B, in general, display images containing acoustic image data from a single location selected from a plurality of locations can be cycled through one at a time for individual analysis. The plurality of locations containing typical acoustic image data can be the entire set of locations corresponding to sources of acoustic signals within an acoustic scene, or a subset of such locations, e.g., including only locations having acoustic signals that satisfy one or more conditions. Such conditions can depend on any one or more acoustic parameters, such as intensity, frequency content, periodicity, proximity, etc., and can be met based on various parameters being below a predetermined value, above a predetermined value, or within a predetermined range of values.

[0201] In various examples, modifying a display image to selectively include acoustic image data therein can be accomplished in various ways. In some embodiments, a display image (e.g., including electromagnetic image data and acoustic image data) can be a real-time image, where the electromagnetic image data and acoustic image data are continuously updated to reflect changes in a scene. In some examples, where specific conditions are used to determine whether acoustic image data is included in a display image, received acoustic signals are analyzed to determine whether they include acoustic image data at various locations in the updated real-time image. That is, when a new display image is generated based on newly received acoustic signals and electromagnetic emissions, construction of the display image can depend on analysis of the acoustic signals to determine which acoustic signals meet specific conditions (e.g., intensity thresholds, etc.) allowed for in the display image. The display image can then be generated including acoustic image data only when appropriate in accordance with such conditions.

[0202] In other examples, a display image may be generated from data stored in memory, such as previously captured acoustic data and electromagnetic image data. In one such example, the previously acquired acoustic data is analyzed for various conditions observed in the acoustic image data, and the previously captured acoustic data is combined with the electromagnetic image data at locations where the acoustic data meets such conditions. In such an embodiment, a single scene may be viewed in multiple ways, for example, by analyzing different acoustic parameters. A display image representing previously captured acoustic image data may be updated based on updated conditions observed in the display image regarding whether to include acoustic image data at various locations in the display image.

[0203] In some embodiments, one or more acoustic parameters used to selectively include acoustic image data in a display image may be used to modify the display image and / or image capture technique. For example, in a real-time imaging example, various conditions for determining whether to include acoustic image data in a display may include distance to a target (e.g., apparent distance or measured distance) and / or frequency content. As described elsewhere herein, certain such parameters may be used in selecting an acoustic sensor array and / or a processing technique for generating acoustic image data. Thus, in certain such examples, if acoustic image data is represented only based on such parameters that satisfy one or more predetermined conditions, the acoustic sensor array and / or a processing technique for generating acoustic image data may be selected based on such conditions.

[0204] For example, in an exemplary embodiment, if acoustic image data is to be included in real-time images only at locations where the corresponding acoustic signals contain frequency content within a first frequency range, one or more acoustic sensor arrays may be selected to acquire acoustic signals optimal for the first frequency range. Similarly, if acoustic image data is to be included in real-time images only at locations where the source of the acoustic signal is within a first distance range, one or more acoustic sensor arrays may be selected to acquire acoustic signals optimal for acoustic imaging in the first distance range. Additionally or alternatively, processing techniques for generating acoustic image data may be selected based on desired frequency or distance requirements, e.g., as described with respect to FIG. 6 . Such selected acoustic imaging sensor arrays and processing techniques may then be used to receive acoustic signals and generate acoustic image data for updated real-time display images to optimize the included acoustic image data.

[0205] Similarly, in some embodiments in which a display image is generated from historical data previously stored in memory, various conditions that determine the locations for inclusion of acoustic image data in the display image may be used to update the acoustic image data representing the acoustic scene. For example, in some embodiments, the data stored in memory includes raw acoustic data received by the acoustic sensor array from the time the acoustic signal was received. Based on the conditions (e.g., desired distance and / or frequency range) for determining whether acoustic image data is included at various locations in the display image, a processing technique (e.g., a backpropagation algorithm) may be selected for use with the raw data stored in memory to generate acoustic image data optimized for the desired parameters for display.

[0206] While generally described and illustrated using visible light image data and acoustic image data, it will be understood that the processes described with respect to Figures 9A-9C, 10A, and 10B may be used with any of a variety of electromagnetic image data. For example, in various embodiments, similar processes may be performed using infrared image data or ultraviolet image data instead of visible light image data. Additionally or alternatively, combinations of the electromagnetic spectrum may be used in such processes, such as blended infrared image data and visible light image data. In general, in various examples, acoustic image data may be selectively displayed (e.g., included when the corresponding acoustic signal meets one or more predetermined parameters) in combination with any combination of electromagnetic image data.

[0207] In some embodiments, the acoustic analysis system is configured to store one or more acoustic signals and / or acoustic image data, for example, in a database in local memory and / or accessible from an external or remote device. Such acoustic signals may include acoustic image data representative of an acoustic scene during normal operation and / or other parameters associated with the acoustic scene, such as frequency data, intensity data, periodicity data, etc. In various examples, the database scenes may include acoustic image data and / or other acoustic parameters (e.g., intensity, frequency, periodicity, etc.) representative of a broad scene (e.g., a factory) and / or a more specific scene (e.g., a particular object).

[0208] In some embodiments, a database scene may be generic to a particular type of device, such as a particular model of device. Additionally or alternatively, a database scene may be specific to an individual object, even if the different objects are different instances of the same object (e.g., two separate machines of the same model). Similarly, a database scene may be more specific, for example, by including a particular operational state of the object. For example, if a particular object has multiple modes of operation, the database may include multiple scenes of such an object, one for each operational mode.

[0209] In various embodiments, the database scene can be a single acoustic image and / or associated acoustic parameters. In other examples, the database scene can include composite data formed from multiple previously captured acoustic images and / or associated parameters. Generally, the database scene (e.g., acoustic image and / or parameters) can include an acoustic representation of a scene during normal operation. In some examples, the database can also include other elements associated with the scene, such as corresponding visible light images, infrared images, ultraviolet images, or combinations thereof. In some embodiments, the database generation and / or comparison can be performed similarly to the database generation and comparison of infrared image data described in U.S. Patent Application No. 15 / 190,792, filed June 23, 2016, entitled "THERMAL ANOMALY DETECTION," which is assigned to the assignee of the present application and incorporated herein by reference in its entirety. In some embodiments, the database can be generated by capturing acoustic image data and / or one or more associated acoustic parameters (e.g., frequency, intensity, periodicity, etc.) of a scene while objects in the scene are operating normally. In one such example, a user may tag a captured database image to associate the image with one or more objects, locations, scenes, etc., so that the captured acoustic image and / or associated parameter(s) can be identified in future database analysis and comparison.

[0210] The newly generated acoustic image data can be compared to acoustic image data stored in a database to determine whether the sound profile of the acoustic scene is within typical operating criteria. Additionally or alternatively, acoustic parameters such as intensity, frequency, periodicity, etc. from the live acoustic scene and / or the newly generated acoustic image may be compared to similar parameters in the database.

[0211] Comparing the current acoustic image data to historical acoustic image data stored in a database (e.g., a previously captured image, a composite image generated from multiple previously captured images, a factory-provided predicted image, etc.) can be performed in a number of ways. FIGS. 12A-12C illustrate several exemplary methods for comparing acoustic image data to historical acoustic image data stored in a database. FIG. 12A illustrates an acoustic imaging tool 1200 including an acoustic sensor array 1202 having an acoustic field of view 1212 and an electromagnetic imaging tool 1204 having an electromagnetic field of view 1214. As shown, the electromagnetic field of view 1214 and the acoustic field of view 1212 include a target scene 1220 including a target object 1222. In some embodiments, the acoustic imaging tool 1200 is permanently fixed in a position such that the target object 1222 is within the electromagnetic field of view 1214 and the acoustic field of view 1212. In some embodiments, the acoustic imaging tool 1200 may be powered via inductive or parasitic power, may be hardwired to the AC mains power in a building, or may be configured to continuously monitor the object 1222.

[0212] The fixed acoustic imaging tool 1200 may be configured to periodically capture acoustic and / or electromagnetic image data of the object 1222. Because the acoustic imaging tool 1200 is fixed in a substantially fixed position, images captured at different times are from substantially the same point. In some examples, acoustic image data captured via the acoustic imaging tool 1200 may be compared to a database of acoustic image data representing substantially the same scene, for example, to detect anomalies in the acoustic scene. This may be done, for example, as described in U.S. Patent Application No. 15 / 190,792, which is incorporated by reference.

[0213] 12B shows an exemplary display on a handheld acoustic imaging tool, for example. Display 1230 includes two sections, 1232 and 1234. In the illustrated example, section 1234 shows a database image 1244 of the target object, while section 1232 includes a live image 1242 of real-time acoustic image data of the object. With such a side-by-side display, a user can compare live image 1242 with database image 1244 to see differences between a typical acoustic signal (e.g., as shown in database image 1244) and the current real-time image 1242. Similarly, a user can compare whether live image 1242 closely matches database image 1244. If so, the user can capture the live acoustic image for further analysis and / or comparison with database image 1244.

[0214] 12C shows another exemplary display, for example, on a handheld acoustic imaging tool. Display 1250 of FIG. 12C shows a database image 1254 and a live image 1256 on the same display 1252. In the example of FIG. 12C, a user can similarly compare the acoustic image data in live image 1256 with the acoustic image data in database image 1254 to show differences. Additionally, the user can adjust the alignment of the acoustic imaging tool to align objects in live image 1256 with objects in database image 1254 for further analysis and comparison.

[0215] 12A-12C, live and / or recently captured acoustic images may be compared to previous acoustic image data from a database, etc. In one example, such a process is used to overlay live and / or recently captured acoustic images with database images for automated comparison. Other processes that can be used to "recapture" acoustic image data from similar locations as the database image are described in U.S. patent application Ser. No. 13 / 331,633, filed December 20, 2011, entitled "THERMAL IMAGING CAMERA FOR INFRARED REPHOTOGRAPHY," U.S. patent application Ser. No. 13 / 331,644, filed December 20, 2011, entitled "THERMAL IMAGING CAMERA FOR INFRARED REPHOTOGRAPHY," and U.S. patent application Ser. No. 13 / 336,607, filed December 23, 2011, entitled "THERMAL IMAGING CAMERA FOR INFRARED REPHOTOGRAPHY," each of which is assigned to the assignee of the present application and incorporated by reference in its entirety.

[0216] Comparing real-time acoustic image data and / or acoustic signatures with corresponding acoustic images and / or acoustic signatures of comparable scenes / objects may be used to provide a quick and simplified analysis of the operating state of a scene / object. For example, the comparison may indicate that a particular location within the acoustic scene is emitting an acoustic signal with a different intensity or frequency spectrum than during typical operation, which may indicate a problem. Similarly, a location within a scene may be emitting an acoustic signal that is typically silent. Additionally or alternatively, comparison of the overall acoustic signatures of a live scene and a historical scene from a database may generally indicate changes in acoustic parameters within the scene, such as frequency content, acoustic intensity, etc.

[0217] In some examples, the acoustic analysis system is configured to compare a recent / real-time acoustic scene to a database. In some embodiments, the acoustic analysis system is configured to characterize differences between the recent / real-time scene and the database scene and diagnose one or more possible problems with the current scene based on the comparison. For example, in some embodiments, a user may pre-select a target object or scene for comparison with the acoustic database. Based on the selected object / scene, the acoustic analysis system may compare the database image and / or other parameters with the recent / current image and / or other parameters to analyze the scene. Based on the object / scene selected from the database, the acoustic analysis system may be able to identify one or more differences between the database image / parameters and the recent / current image / parameter and associate the identified difference(s) with a possible cause of the one or more differences.

[0218] In some examples, the acoustic analysis system can be pre-programmed with multiple diagnostic information, e.g., associating various discrepancies between database images / parameters and recent / current images / parameters with possible causes and / or solutions to the causes. Additionally or alternatively, a user may load such diagnostic information, e.g., from a repository of diagnostic data. Such data may be provided, e.g., by the acoustic analysis system manufacturer, the target object manufacturer, etc. In yet another example, the acoustic analysis system may additionally or alternatively learn the diagnostic information, e.g., via one or more machine learning processes. In one such example, a user may diagnose one or more problems with a target scene after observing acoustic deviations of the scene from typical scenes and input data representing one or more problems and / or one or more solutions into the acoustic analysis system. Over time, the system can be configured to learn, through multiple data entries, to associate various discrepancies between recent / current images and / or parameters and those stored in the database with specific problems and / or solutions. Upon diagnosing a problem and / or determining a proposed solution, the acoustic analysis system may be configured to output the suspected problem and / or proposed solution to the user, e.g., via a display. Such a display can be located on a handheld acoustic inspection tool or on a remote device (e.g., a user's smartphone, tablet, computer, etc.) Additionally or alternatively, such a display indicating potential problems and / or solutions can be communicated, for example, over a network, to a remote site, such as an off-site operator / system monitor.

[0219] In one example diagnostic characteristic, an acoustic analysis system may observe a particular periodic creaking sound that indicates that a running machine needs additional oiling. Similarly, a constant high-pitched signal may indicate a gas or air leak in the target scene. Other problems may similarly have a recognizable acoustic signature, such as a broken bearing in the object under analysis, and viewing the acoustic signature via an acoustic imaging system (e.g., a handheld acoustic imaging tool) may help diagnose an anomaly in the system or object.

[0220] An acoustic analysis system that can compare received acoustic signals with criteria (e.g., acoustic image data and / or parameters from a database) and perform diagnostic information and / or suggest corrective actions can eliminate the need for an experienced expert to analyze the acoustic data of a scene. Rather, acoustic inspection and analysis can be performed by a system operator with limited or no experience analyzing acoustic data.

[0221] 13 is a process flow diagram illustrating exemplary operations for comparing received acoustic image data to a database for object diagnosis. The method includes receiving 1380 a selection of a target and retrieving 1382 a reference acoustic image and / or acoustic parameters of the target from the database. For example, a user may wish to perform an acoustic analysis of a particular object and may select such object from a predefined list of objects having available reference acoustic images and / or parameters available in the database.

[0222] The method further includes capturing 1384 acoustic image data and associated parameters representative of the target, for example, using a handheld acoustic imaging tool. After capturing 1384 the acoustic image data and associated parameters, the method includes comparing 1386 the captured acoustic image data and / or associated parameters to the acquired reference image and / or parameters.

[0223] 13 further includes diagnosing 1390 an operational problem of the target of interest based on the comparison if the captured acoustic image data and / or parameters deviate 1388 sufficiently from the baseline. The method may further include displaying 1392 an indication of the potential problem and / or corrective action to the user. In some embodiments, a comparison indication, e.g., a difference image showing the difference between the current acoustic image data and the baseline acoustic image data, may also or separately be displayed to the user.

[0224] In one such example, determining whether there is a deviation from the reference (1388) includes comparing one or more acoustic parameters of the captured data to similar parameters in the reference data and determining whether a difference between the captured parameters and the reference parameters exceeds a predetermined threshold. In various examples, different parameters may include different thresholds, and such thresholds may be absolute thresholds, statistical thresholds, etc. In some embodiments, the comparison may be performed on a location-by-location basis, or for a subset of locations in the scene.

[0225] For example, with reference to FIG. 9B , only locations (e.g., locations 910 and 940) that contain acoustic image data and appear on the object can be analyzed for object motion. In such an example, different acoustic parameters at each compared location (e.g., 910 and 940) are compared individually between the captured image and the database image. For example, comparing the captured data and / or associated parameters with those from the database, with reference to FIG. 9B , can include comparing the frequency, intensity, and periodicity of location 910 in the captured image with the frequency, intensity, and periodicity of location 910 in the database image, respectively. A similar comparison can be performed at location 940 between the captured image and the database image. As described, each comparison can include a different metric for determining (1388) whether there is sufficient deviation from a norm.

[0226] Diagnosing 1390 operational problems and displaying 1392 indications of potential problems and / or corrective actions can be performed based on a combination of comparisons between the captured image data and reference image data and / or parameters. In some examples, such diagnosis can include multidimensional analysis, such as combining comparisons of multiple parameters at a given location. For example, in an exemplary embodiment, a particular condition can be indicated by both a frequency deviation from baseline greater than a first threshold and an intensity deviation from baseline greater than a second threshold.

[0227] In some examples, even after displaying 1392 the possible problems and / or corrective action indications, the process may include capturing 1384 new acoustic image data and associated parameters and repeating the comparison and diagnosis process, so that the user can observe whether the corrective actions taken effectively alter the object's acoustic signature to correct the identified problems and / or bring the object's acoustic signature into conformance with the criteria.

[0228] In one embodiment, after comparing 1386 the captured data to the reference data, if there is no sufficient deviation 1388 from the reference, the process may end 1394 with a conclusion that the object is operating normally based on the object's current acoustic signature. Additionally or alternatively, new acoustic image data and associated parameters of the target may be captured 1384, and the comparison and diagnosis process may be repeated. In one example, the continuous, repeated analysis may be performed using a fixed acoustic analysis system, including, for example, acoustic imaging tool 1200 of FIG. 12A.

[0229] Comparison of acoustic data (e.g., image data and / or other acoustic parameters) allows a user to more easily identify whether an object is functioning properly and, if not, helps diagnose problems with the object. In some instances, comparison to a baseline helps a user ignore "normal" sounds in a scene, such as expected operating sounds or floor / background sounds, that may be unrelated to operational problems with the object.

[0230] During operation, observation of the acoustic image data and / or associated acoustic parameters, or the results of a comparison between the current acoustic scene and a database acoustic scene, may indicate target locations to the user for further inspection. For example, a comparison acoustic image showing deviations from the database image may indicate one or more locations within the scene that are behaving abnormally. Similarly, viewing an acoustic image having an acoustic signature at one or more unexpected locations can indicate target locations to the user. For example, referring to FIG. 10B, a user observing FIG. 10B on the display of an acoustic imaging system may notice that a particular location (e.g., 1020) is emitting an unexpected acoustic signal, or similarly, comparison with a reference image indicates unexpected parameters (e.g., unexpected frequency, intensity, etc.) of the acoustic signal at that location.

[0231] In one such example, a user may approach a location to more closely inspect such a location for anomalies. As the user approaches an object, the distance-to-target value may be updated to reflect the new distance between the acoustic array and the target location. The acoustic sensor array and / or backpropagation algorithm may be updated based on the updated distance-to-target. Additionally or alternatively, the updated acoustic analysis from a closer location may result in a different analysis of the acoustic signal from the target. For example, high-frequency acoustic signals (e.g., ultrasonic signals) tend to attenuate over a relatively short distance from the source of the acoustic signal. Thus, as the user approaches the target for further inspection, additional signals (e.g., high-frequency signals) may be visible to the acoustic sensor array. Such apparent changes in the observable scene may also result in adjustments to the acoustic sensor array and / or backpropagation algorithm used for acoustic imaging.

[0232] Thus, the sensor array and / or backpropagation algorithm used for acoustic imaging may be updated one or more times as the user approaches the target object or region. Each update may use a different sensor array and / or backpropagation algorithm to provide additional detail about the target object or region that may not have been observable from a distance. For example, based on an initial observation of the wider scene, approaching the target object or region may increase the acoustic intensity of the target acoustic signal relative to background sounds in the environment.

[0233] In some embodiments, an acoustic analysis system (e.g., a handheld acoustic imaging tool) may prompt a user to move closer to a target object or region within a scene. For example, upon comparing a current acoustic image with a reference database image, the acoustic analysis system may identify one or more locations within the scene that deviate from the reference. The acoustic analysis system may highlight such one or more locations to the user, e.g., via a display, and suggest that the user approach the identified locations for further analysis. In some examples, the acoustic analysis system may classify identified locations, such as subcomponents of objects or specific objects within an environment, as having a unique reference profile stored in a database. The system may be configured to suggest and / or implement such profiles of classified locations to facilitate further analysis of the identified locations as the user approaches for additional inspection.

[0234] The systems and processes described herein can be used to improve the speed, efficiency, accuracy, and thoroughness of acoustic inspections. Various automated actions and / or suggestions (e.g., sensor arrays, backpropagation algorithms, etc.) can improve the ease of inspection, in that even inexperienced users can perform thorough acoustic inspections of an acoustic scene. Furthermore, such processes can be used to analyze broad scenes, such as entire systems, individual objects, and subcomponents of individual objects. Predefined and / or user-generated profiles of reference acoustic data for an acoustic scene can help even inexperienced users identify anomalies in the captured acoustic data.

[0235] Overlaying acoustic image data with other data streams, such as visible light, infrared, and / or ultraviolet image data, can provide additional context and detail to objects emitting acoustic signals represented in the acoustic image data. Combining an acoustic sensor array with a distance measurement tool (e.g., a laser rangefinder) can aid a user in quickly and easily determining appropriate target distance values ​​to use in the acoustic imaging process. In various examples, the acoustic sensor array, distance measurement tool, processor, memory, and one or more additional imaging tools (e.g., a visible light camera module, an infrared camera module, etc.) can be supported by a single housing in a handheld acoustic imaging tool that can provide efficient acoustic analysis of multiple scenes. Such a handheld acoustic imaging tool can be moved from scene to scene to rapidly analyze multiple target objects. Similarly, the handheld tool can enable a user to approach specific locations within a scene for further inspection or analysis.

[0236] Described herein are various systems and methods for performing acoustic imaging and generating and displaying acoustic image data. An exemplary system can include an acoustic sensor array including a plurality of acoustic sensor elements configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals.

[0237] The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representative of the received electromagnetic radiation. Such imaging tools may include infrared imaging tools, visible light imaging tools, ultraviolet imaging tools, etc., or combinations thereof.

[0238] The system can include a processor in communication with the acoustic sensor array and the electromagnetic imaging tool. The processor can be configured to receive electromagnetic image data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor can be configured to generate acoustic image data of the scene based on the received acoustic data and received range information representing the distance to the target, for example, via a backpropagation calculation. The acoustic image data can include a visual representation of the acoustic data, such as with color toning or color schemes as described elsewhere herein.

[0239] The processor may be configured to combine the generated acoustic image data with the received electromagnetic image data to generate a display image including both the acoustic image data and the electromagnetic image data and to communicate the display image to the display. Combining the acoustic image data with the electromagnetic image data may include, for example, correcting for parallax error between the acoustic image data and the electromagnetic image data based on the received distance information.

[0240] In one example, the distance information can be received from a distance measurement tool in communication with the processor. The distance measurement tool can include, for example, an optical distance measurement device, such as a laser distance measurement device, and / or an acoustic distance measurement device. Additionally or alternatively, a user can manually input the distance information, for example, via a user interface.

[0241] The system may include a laser pointer to help locate a point of interest, such as a sound or sound profile, based on selected parameters such as frequency, decibel level, periodicity, distance, etc., or a combination thereof. Such a laser pointer may be used to precisely align and match the field of view of the scene with the appropriate sound visualization displayed on the display. This is useful in environments where the object under inspection is distant relative to the acoustic imaging device, or when it is not clear where the sound visualization on the display is relative to the actual scene.

[0242] In some examples, the laser pointer may be visualized on a display. Such visualization may include generating (e.g., via a processor) a laser pointer spot on a display representation of the laser pointer in the actual scene. In some examples, the location of the laser pointer may be highlighted on the display, for example, with an icon or another aligned display marker representing the laser pointer in the scene, to better determine its location on the display relative to the actual scene.

[0243] As described elsewhere herein, the thermal imaging system may be configured to create false color (e.g., toned), symbols, or other non-numerical visual representations of the acoustic data generated by one or more acoustic sensors, such as by creating acoustic image data. Additionally or alternatively, the system may provide audio feedback to the user via speakers, headphones, a wired or remote communication headset, etc. Such audio or heterodyned audio transmissions may be synchronized with the visual representation of the detected and displayed sounds.

[0244] In various examples, acoustic data can be visualized in various ways, for example, to facilitate understanding of such data and to prevent viewers from making incorrect assumptions about the nature of the visualized sounds. In some examples, different types of visualizations can provide an intuitive understanding of the visualized sounds.

[0245] In some embodiments, the generated display includes non-numeric visual representations accompanied by numeric and / or alphanumeric contextual data to provide a complete presentation of information about the sound being visualized, which can assist the user in determining and / or implementing one or more appropriate courses of action.

[0246] Various display features, including various non-numeric graphic representations (e.g., symbols, color tones, etc.) and alphanumeric information, can be combined. In some embodiments, the display features present in a given scene representation can be customized by a user, for example, from multiple selectable settings. Additionally or alternatively, pre-set combinations of display features can be selected by a user to automatically include a desired combination of information in a display image. In various embodiments, aspects of the display image can be adjusted by a user, for example, via a virtual interface (e.g., provided via a touchscreen) and / or physical controls.

[0247] FIG. 14 illustrates a visualization of acoustic data using a gradient palette. As shown, an acoustic parameter (e.g., intensity) is depicted via a gradient palette. In an exemplary gradient palette, each parameter value has a unique color associated with it according to the palette. A change in the parameter value at a given pixel typically results in a change in the color associated with that pixel to represent the new parameter value. As shown in the example of FIG. 14, the acoustic parameter values ​​appear to vary radially from a central location of the acoustic signal at locations 1432, 1434, and 1436. Other examples of gradient palettes are described in U.S. Patent Application No. 15 / 802,153, filed November 2, 2017, and assigned to the assignee of the present application.

[0248] FIG. 15 illustrates a visualization of acoustic data using multiple shaded concentric circles. In one such example, as opposed to a gradient palette scheme having a color associated with a parameter value, each identical color in the concentric circles shown in FIG. 15 can represent pixels having an acoustic parameter value within the range of values ​​associated with that color. In the illustrated example, the acoustic parameters (e.g., intensity) associated with the acoustic signal at locations 1532, 1534, and 1536 vary radially from the center of the acoustic signal. In the exemplary palette scheme, pixels shown in red represent acoustic parameter values ​​within a first range of parameter values, pixels shown in yellow represent acoustic parameter values ​​within a second range of parameter values, and pixels shown in green represent acoustic parameter values ​​within a third range of parameter values, although other display techniques, including additional or alternative colors, patterns, etc., are possible. In various embodiments, the range of values ​​can correspond to an absolute range, such as intensity values ​​between 10 dB and 20 dB, or can be a relative range, such as intensity values ​​between 90% and 100% of maximum intensity.

[0249] As described elsewhere herein, in some embodiments, a display image including electromagnetic image data and acoustic image data can include both a visual representation of the acoustic signal and an alphanumeric representation of one or more parameters associated with the acoustic signal. FIG. 16 shows an exemplary visualization showing both non-numeric information (e.g., toning through parameter value ranges) and alphanumeric information. In the illustrated example, an alphanumeric sound intensity value label is associated with each of three locations having toned acoustic image data (e.g., intensity data). As shown, the acoustic signals have corresponding visual indicators representing their associated acoustic parameters (1602, 1604, 1606) and alphanumeric information (1612, 1614, 1616, respectively). In an exemplary embodiment, the alphanumeric information can provide a numerical value, such as a maximum intensity value, associated with the location where the toned acoustic data is displayed. In some examples, a user can select one or more locations for displaying the toning and / or alphanumeric data. For example, a user can select to annotate the display image with alphanumeric representations of acoustic parameters associated with one or more acoustic signals in the scene.

[0250] In some examples, the alphanumeric information can represent multiple parameters (e.g., acoustic parameters) associated with an acoustic signal at a given location within a scene. FIG. 17 shows an example of a visualization including both non-numeric information (e.g., toning via parameter value ranges) and alphanumeric information. In the illustrated example, sound intensity values ​​and corresponding frequency values ​​(e.g., average frequency or peak frequency) are shown in alphanumeric information 1712, 1714, and 1716 associated with each of three locations with toned acoustic data (e.g., intensity data) indicated via indicators 1702, 1704, and 1706, respectively. Similar to the example described with respect to FIG. 16, the inclusion of various such data at various locations can be initiated by the user. For example, a user may select to annotate a display image with alphanumeric representations of one or more acoustic parameters associated with one or more acoustic signals within the scene.

[0251] 18 shows another exemplary visualization showing both non-numeric information (e.g., toning via parameter value ranges) and alphanumeric information. In the illustrated example, distance measurements are included along with alphanumeric information 1812, 1814, 1816 associated with each of three locations having toned acoustic data (e.g., intensity data), indicated via indicators 1802, 1804, 1806, respectively. Similar to that described with respect to FIG. 16, the inclusion of different such data at different locations may be selected by a user, for example, as part of a display image annotation.

[0252] In some examples, non-numeric representations may be used to convey information related to multiple acoustic parameters. For example, FIG. 19 shows an exemplary visualization showing indicators 1902, 1904, 1906 (circles in this case) of different sizes and colors representing different acoustic parameter values. In an exemplary embodiment, the size of the indicator corresponds to the intensity of the acoustic signal at a given location, and the color of the indicator corresponds to the peak or average frequency. In an exemplary embodiment, the indicator size may indicate a relative value, such that comparing the size of one indicator to the size of another indicator indicates the relative difference between the acoustic parameter values ​​represented at the locations associated with the indicators. Additionally or alternatively, alphanumeric information may be included to provide absolute or relative acoustic parameter values.

[0253] In some embodiments, colored indicators may be used to represent the severity of one or more detected acoustic signals and / or associated acoustic parameters, such as the amount of deviation from a reference parameter. FIG. 20 shows an exemplary visualization illustrating multiple indicators 2002, 2004, 2006 with different colors indicating the severity indicated by the acoustic signals from corresponding locations. For example, in an exemplary embodiment, a red indicator indicates a significant severity based on one or more acoustic parameters (e.g., when compared to a reference, such as a reference for typical operating conditions), a yellow indicator indicates a moderate severity, and a green indicator indicates a minor severity. In other examples, other color schemes or appearance characteristics (e.g., indicator transparency, indicator size, etc.) may be used to visually distinguish the severity of the acoustic signals. In the illustrated example, indicator 2004 represents the highest level of severity, indicator 2006 represents the next highest level, and indicator 2002 represents the least severe acoustic signal.

[0254] As described elsewhere herein, in various embodiments, one or more acoustic parameters may be displayed in a visual representation of an acoustic scene, e.g., via a toned color display or a grayscale display. In some embodiments, a system may be configured to identify one or more locations within a scene that satisfy one or more acoustic conditions, such as a specified frequency range, intensity range, distance range, etc. In some examples, various locations corresponding to sound profiles (e.g., satisfying a particular set of conditions or parameters) may be identified. Such identified locations may be presented in a manner distinct from acoustic image data toning techniques otherwise used in creating display images. For example, FIG. 21 illustrates a scene including indicators at multiple locations within the scene. Indicators 2101, 2102, 2103, 2104, and 2105 are positioned within the scene. Indicators 2103, 2104, and 2105 include toned acoustic image data representing values ​​of one or more acoustic parameters corresponding to, e.g., a scale 2110. Indicators 2101 and 2102 are shown to have a unique presentation technique that is distinct from the toning techniques that appear at locations 2103, 2104, and 2105. In such embodiments, a user may quickly and easily identify those locations within an image that meet one or more desired conditions. In one such example, a user may select one or more desired conditions for display in a distinct manner based, for example, on the selection of a range of values ​​from a scale such as 2110.

[0255] Additionally or alternatively, locations that meet the conditions of a particular sound profile may be presented with an icon representing the met condition, such as the corresponding sound profile. For example, FIG. 22 shows multiple icons 2202, 2204, 2206, and 2208 arranged in a display image illustrating recognized sound profiles within a scene. Example profiles shown in FIG. 22 include bearing wear, air leaks, and electrical arcing. Such profiles may be identified by an acoustic signal meeting a set of one or more parameters associated with such profile in order to be classified as such a profile.

[0256] 23 shows another example display image showing acoustic data via a plurality of indicators 2302, 2304, and 2306 using concentric circles and alphanumeric information representing acoustic intensity associated with each acoustic signal. As described elsewhere herein, in some examples, the size of the indicators can represent one or more acoustic parameters present at the corresponding location. In some embodiments, the indicators can be a single color, and the acoustic parameters can be indicated in one or more other ways, such as by indicator size, line thickness, line type (e.g., solid, dashed, etc.), etc.

[0257] In some examples, the display may include alphanumeric information based on a selection made by a user. For example, in some embodiments, the system (e.g., via a processor) may include information representing one or more acoustic parameters of an acoustic signal located at a particular location in response to user selection of an indicator on the display at such location (e.g., via a user interface). FIG. 24 shows an exemplary display image with an indicator and additional alphanumeric information associated with a displayed acoustic signal. In one example, an indicator 2402 on the display may be selected for further analysis (e.g., represented via crosshairs, which may indicate selection, such as via touchscreen input). The display shows alphanumeric information 2404 including a list of data associated with the location corresponding to the indicator, including peak intensity and corresponding frequency, frequency range, measured distance to the location, and critical level indicated by the acoustic signal from that location.

[0258] In some examples, the display image may include multiple indicators representing corresponding multiple acoustic signals in the scene. In some embodiments, in such cases, a user may select one or more of the indicators (e.g., via a touchscreen or other user interface), and in response to detecting the selection, the processor may present additional information about the acoustic signals. Such additional information may include alphanumeric values ​​of one or more acoustic parameters. In some examples, such additional information may be displayed for multiple acoustic signals simultaneously. In other examples, such additional information about a given acoustic signal is hidden when another acoustic signal is selected.

[0259] As described elsewhere herein, in some examples, the system may include a laser pointer. In some examples, the laser pointer may have a fixed direction or may have adjustable pointing controllable, for example, via a processor. In some examples, the system may be configured to point the laser pointer at a location in a target scene associated with a selected location in the image. FIG. 25A shows a system (embodied in one example as a handheld tool) including a display, such as the display shown in FIG. 24, in which an indicator 2502 has been selected. The laser pointer 2504 emits a laser beam 2506 toward the scene, and the laser creates a laser spot 2508 in the scene corresponding to the location of the selected indicator 2502 in the image. This may help a user visualize the location of the selected and / or analyzed acoustic signal in the environment. In some embodiments, the laser spot 2508 is detectable by an electromagnetic imaging tool and is visible on the display along with the displayed indicator 2502 and alphanumeric information 2512, including acoustic parameter information. In one example, the acoustic imaging system is configured to detect or estimate the position of the laser within the scene and provide a visual indication of the laser position 2510. Figure 25B shows a display image such as that shown in the system view of Figure 25A.

[0260] In embodiments where the laser pointer has a fixed orientation, the user may view a display image visually indicating the laser position as feedback, allowing the user to adjust the aim of the laser to coincide with the selected acoustic signal.

[0261] As described elsewhere herein, in some embodiments, acoustic image data can be combined with electromagnetic image data and presented in a display image. In some examples, the acoustic image data can include adjustable transparency so that various aspects of the electromagnetic image data are not completely obscured; Figure 26 shows acoustic image data represented by an indicator 2602 at a location in a scene, the indicator 2602 including a palette colorization scheme. The system can include a display device, either integral with or separate from the acoustic imaging tool, configured to present display data including the electromagnetic image data and the acoustic image data.

[0262] In some embodiments, a device (e.g., a handheld acoustic imaging tool) may include physical blending controls 2614 (e.g., one or more buttons, knobs, sliders, etc., that may be included as part of a user interface) and / or virtual blending controls 2604, such as via a touchscreen or other virtually implemented interface. In some embodiments, such functionality may be provided by an external display device, such as a smartphone, tablet, computer, etc.

[0263] Figure 27 illustrates virtual and / or physical blending control tools for a display image, including a partially transparent concentric circle toning technique. Similar to the description for Figure 26, indicator 2702 can represent an acoustic signal in a scene. The acoustic imaging system can include physical blending controls 2714 and / or virtual blending controls 2704 that can be used to adjust the transparency of the acoustic image data (e.g., indicator 2702) in the display image.

[0264] Additionally or alternatively, physical and / or virtual interfaces may be used to adjust one or more display parameters. For example, in some embodiments, one or more filters may be applied to selectively display acoustic image data that meet one or more conditions, as described elsewhere herein. FIG. 28 illustrates a scene including an indicator 2802 having a gradient color tone that indicates locations within the scene that meet one or more filters (e.g., having one or more acoustic or other parameters that meet one or more corresponding thresholds or predetermined conditions). In various examples, the filtering may be selected and / or adjusted via physical controls 2814 (e.g., via one or more buttons, knobs, switches, etc.) and / or virtual controls 2804 (e.g., a touchscreen). In the illustrated example, the filtering includes displaying acoustic image data only for acoustic signals having acoustic parameters (e.g., frequencies) that fall within a predefined range 2806 of the acoustic parameters. As illustrated, the predefined range 2806 is a subset of the possible filter range 2816. In one example, a user may adjust the limits of the predefined range 2806 to adjust the filter effect, for example, via virtual 2804 or physical 2814 controls.

[0265] 29 illustrates virtual and / or physical filter adjustment of a display image, including a partially transparent concentric circle toning technique. As shown, indicators 2902 are shown in the scene based on acoustic parameters that fall within a predetermined range 2906 of values ​​based on the filter. The filters may be adjustable within a range 2916 of values, for example, via virtual 2904 and / or physical 2914 controls.

[0266] In some embodiments, multiple filters may be utilized to customize a display image that includes the toned acoustic image data. FIG. 30 illustrates a display image showing a first indicator and a second indicator. As described elsewhere herein, one or more filters may be applied to the display image (e.g., via physical and / or virtual filter controls) to customize the displayed data. In the illustrated example of FIG. 30, the filtering includes establishing a first filter range 3006 and a second filter range 3008. In some examples, the filter range may be adjustable within a range of values ​​3016, e.g., via virtual 3004 and / or physical 3014 controls.

[0267] Such filter ranges may represent any of a variety of parameters, such as frequency, amplitude, proximity, etc. As shown, the first and second filter ranges are each associated with a color (which, in some examples, is user-adjustable), and indicators 3002, 3012 are placed at locations within the image where the corresponding acoustic signals satisfy one or more filter conditions associated with each filter range. As shown, the first indicator 3002 represents acoustic signals that satisfy the first filter range 3006 (shown in dark shading), and the second indicator 3012 represents acoustic signals that satisfy the second filter range 3008 (shown in light shading). Thus, a user may be able to quickly identify locations within a scene having acoustic data that satisfy various conditions while simultaneously identifying which locations satisfy which conditions.

[0268] In certain examples, a display device, such as an acoustic imaging tool or an external display device, can include a virtual keyboard as an input device, as shown in FIG. 31. FIG. 31 shows a display interface including indicators 3102 that represent one or more acoustic parameters of an acoustic signal in a scene. A virtual keyboard 3110 is included on the display and can be used to add alphanumeric information 3112 to the display image. Utilizing such a virtual keyboard allows a user to enter customized annotations, such as various inspection notes, labels, date / timestamps, or other data that can be saved with the image. In various examples, the virtual keyboard can be used to add text that is included in the image data and / or added to the image data, such as by being stored in metadata associated with the display image.

[0269] Various devices may be used to present display images that include various combinations of acoustic image data and other data, such as alphanumeric data, image data from one or more electromagnetic spectrums, symbols, etc. In some examples, a handheld acoustic imaging tool may include a built-in display for presenting the display image. In other examples, the information to be displayed or the data processed to generate the display (e.g., raw sensor data) may be communicated to an external device for display. Such external devices may include, for example, smartphones, tablets, computers, wearable devices, etc. In some embodiments, the display image is presented in combination with real-time electromagnetic image data (e.g., visible light image data) in an augmented reality type display.

[0270] FIG. 32 illustrates a display integrated into eyewear 3210 that may be worn by a user. In one example, the eyewear may include one or more integrated imaging tools, such as those described in commonly assigned U.S. Patent Publication No. 20160076937, entitled "DISPLAY OF IMAGES FROM AN IMAGING TOOL EMBEDDED OR ATTACHED TO A TEST AND MEASUREMENT TOOL," relevant portions of which are incorporated herein by reference. In one such example, the integrated display may display a real-time display image 3220. For example, the display may display electromagnetic image data (e.g., visible light image data) representing a scene toward which the user is facing, and simultaneously (e.g., via blending, overlay, etc.) display one or more additional data streams, such as acoustic image data (e.g., including indicator 3202), to provide additional information to the user. In one embodiment, eyewear such as that shown in Figure 32 includes a transparent display screen so that when a display image is not provided on the display, the user can view the scene directly with their eyes through the eyewear rather than being presented with real-time visible light image data. In one such example, additional data, such as audio image data, alphanumeric data, etc., is displayed on another transparent display within the user's field of view so that the user can view such data in addition to their view of the scene through the display.

[0271] As described elsewhere herein, in various examples, the various data presented in the display image can be combined in various ways, including blending with other data streams (e.g., blending acoustic image data with visible light image data). In some examples, the strength of the blending can vary between different locations within a single display image. In some embodiments, a user can manually adjust the blending ratio for each of multiple locations (e.g., each of multiple indicators of a detected acoustic signal). Additionally or alternatively, the blending can be a function of one or more parameters, such as frequency, amplitude, proximity, etc.

[0272] In some embodiments, the acoustic imaging tool may be configured to determine the extent to which the sensor array is pointing to each of a plurality of locations emitting detected acoustic signals and blend the corresponding acoustic image data with, for example, visible light image data. FIG. 33A shows an exemplary display including a first indicator 3302 and a second indicator 3304 representing acoustic signals in an acoustic scene. In FIG. 33A, the acoustic sensor more directly points to pipe 1, which corresponds to the location of first indicator 3302, compared to pipe 2, which corresponds to the location of second indicator 3304. Thus, in the display technique of FIG. 33A, first indicator 3302 is displayed more prominently (e.g., with a higher blending coefficient or lower transparency) than second indicator 3304. Conversely, in FIG. 33B, the acoustic sensor more directly points to pipe 2, which corresponds to the location of second indicator 3304, compared to pipe 1, which corresponds to the location of first indicator 3302. 33B , the second indicator 3304 is displayed more prominently (e.g., has a higher blending coefficient or less transparency) than the first indicator 3302. In general, in one embodiment, the acoustic imaging system can determine a metric indicative of the degree to which the sensor is aimed at a given location (e.g., corresponding to an indicator in the acoustic image data) and adjust the blending ratio corresponding to the location accordingly (e.g., the greater the degree of aiming, the higher the corresponding blending ratio).

[0273] In some examples, the processor may store sound profiles detected within a scene. For example, in an illustrated embodiment, a user may store detected acoustic data (e.g., displayed as acoustic image data) as sound profiles corresponding to one or more parameters. In some such examples, such sound profiles may be labeled according to and / or associated with one or more characteristics of the scene, such as the presence of an air leak. Additionally or alternatively, predefined sound profiles may be loaded into the system memory during factory assembly of the acoustic imaging system and / or downloaded to or otherwise communicated to the acoustic imaging system.

[0274] In some examples, a sound profile may include one or more sounds present in a scene. Various sound profiles may be defined by one or more acoustic parameters, such as frequency, decibel level, periodicity, or distance (e.g., an example sound profile may include frequency values ​​within a predetermined range and a periodicity within a predetermined range associated with the profile). A sound profile may be defined by one or more sounds in a scene, and in some examples, multiple sounds may each be defined by one or more acoustic parameters, such as frequency, decibel level, periodicity, or distance. Multiple sounds in a given profile may be identified by similar parameters (e.g., two sounds each have a frequency range and a periodicity range) or by different parameters (e.g., one sound has a corresponding frequency range, while the other sound has a corresponding decibel level range and maximum distance value).

[0275] In some examples, the system may be configured to provide a notification regarding a sound profile, such as when one or more sounds in an acoustic scene correspond to a known sound profile. For example, the notification may alert a user or technician to a recognized sound profile. Additionally or alternatively, the system may be configured to annotate acoustic image data, electromagnetic image data, and / or a display image based on the recognized sound profile. The notification may include an audible sound, a visualization on a display, an LED light, etc.

[0276] In some examples, the system processor may be configured to analyze the criticality of the acoustic signature, e.g., considering one or more sound profiles. Correspondence between acoustic data (e.g., related to sound profiles) and system criticality may be learned, e.g., based on machine learning input and / or user input.

[0277] In some examples, the processor may be configured to compare data within an acoustic scene to one or more known sound profile patterns to analyze the scene, for example, with respect to the criticality of a detected signature. In various examples, the processor may be configured to notify a user based on potential criticality. This may be done, for example, based on a comparison of the identified acoustic signature to one or more stored criteria, a relative value compared to a user-defined threshold, and / or a value determined automatically via machine learning algorithms and / or artificial intelligence programming (e.g., based on past performance, errors, and corresponding historical acoustic signatures).

[0278] In various such examples, the processor may be configured to analyze an acoustic scene with respect to one or more sound profiles to estimate the impact of the discovered sound profiles on the scene or of objects within the scene. Estimating the impact may include the potential criticality of the sound profiles and / or the potential cost or profit loss associated with the acoustic signature. In some examples, the system (e.g., via the processor) may be configured to identify one or more air leaks within the scene (e.g., by comparing the acoustic signature to one or more sound profiles associated with air leaks within the acoustic scene). In some such examples, the system may automatically calculate and / or report various data, such as the number of detected air leaks, the severity of one or more air leaks, and / or the estimated cost savings associated with repairing such one or more leaks. In some examples, cost estimates may be based on pre-programmed and / or user-entered values ​​associated with the detected leaks. In an exemplary application, the system may be configured to determine the cost impact of a compressed air leak per unit of time (e.g., hour) if not properly repaired.

[0279] In some examples, the acoustic imaging system may be configured to determine various information regarding air leaks in an acoustic scene, such as pressure, hole diameter, or leak volume. In some examples, one or more such values ​​may be input by a user, and the remaining values ​​may be generated. For example, in some examples, a user may input a pressure value associated with a particular air line, and using the input pressure information, the system may be configured to determine the hole diameter or leak volume based on acoustic data from the scene. Such determinations may be made, for example, using lookup tables and / or formulas stored in memory.

[0280] In an exemplary scenario, a detected sound profile may be associated with a 100 PSIG air leak through a ¼ inch diameter hole. In an embodiment, an acoustic imaging system with access to such a stored sound profile may be programmed to recognize such a profile within an acoustic scene and estimate, e.g., based on a look-up table, an hourly cost associated with such a leak. In a similar example, a detected sound profile may be associated with a ¼ inch diameter hole based on an input pressure of 100 PSIG (e.g., via manual input). An acoustic imaging system with access to such a stored sound profile may be programmed to recognize such a profile within an acoustic scene and estimate, e.g., based on a look-up table, an hourly cost associated with such a leak.

[0281] In some examples, the acoustic imaging system may be configured to perform a cost savings analysis to identify multiple false negatives based on a formula and / or a lookup table. In some examples, such formula and / or lookup table can be stored in a memory of the acoustic imaging device or in another location accessible by it to perform a cost analysis of the identified leaks.

[0282] In one example implementation, the acoustic imaging system may be configured to characterize one or more leaks detected in an environment during a scene or facility inspection. Such characterization may be performed, for example, based on stored sound profiles corresponding to such leaks. Detected leaks (e.g., of a determined leak volume) may be used to calculate cost savings when such leaks are identified. For example, the acoustic imaging system may be used to determine the number of leaks present and the leak volumes associated with such leaks to calculate cost savings associated with such leaks.

[0283] In one example, cost savings can be calculated by multiplying the number of leaks, the leak rate (cfm), the amount of energy associated with the leak (e.g., kW / cfm), the number of operating hours, and the cost per energy, as shown in Equation (1) below.

[0284] Cost savings ($) = number of leaks × leak rate (cfm) × kW / cfm × number of hours × $ / kWh…(1)

[0285] An acoustic imaging system may be used to determine the number of leaks and the leak volume (e.g., in cfm) associated with such leaks. Other parameters may be programmed into the system (e.g., energy per air generation in kW / cfm), accessed via a database (e.g., current energy cost in $ / kWh), or estimated by the system (e.g., average number of operating hours). The system may be programmed to calculate cost savings associated with identifying such leaks.

[0286] In one example, a system has 100 1 / 32 inch leaks at 90 PSIG, 50 1 / 16 inch leaks at 90 PSIG, and 10 1 / 4 inch leaks at 100 PSIG. Assuming 7000 annual operating hours, a total energy bill of 0.05$ / kWh, and a compressed air generation requirement of approximately 18kW / 100cfm, the cost savings associated with each leak according to equation (1) are as follows:

[0287] Cost savings from a 1 / 32 inch leak = 100 x 1.5 x 0.61 x 0.18 x 7000 x 0.05 = $5,765

[0288] Cost savings from a 1 / 16 inch leak = 50 x 5.9 x 0.61 x 0.18 x 7000 x 0.05 = $11,337

[0289] Cost savings from a 1 / 4 inch leak = 10 x 104 x 0.61 x 0.18 x 7000 x 0.05 = $39,967

[0290] As shown in this example, the savings from removing just ten 1 / 4 inch leaks account for nearly 70% of the total savings. With the leaks identified, in some examples, the acoustic imaging system may be configured to analyze the leaks to identify which leaks, if any, will result in higher cost savings. In some examples, the system may rank or prioritize the leaks that have higher cost savings and provide such ranking or priority to the user. Additionally or alternatively, the system may be configured to provide a notification to the user regarding the cost savings of one or more identified leaks.

[0291] 34 shows an exemplary display image that provides a user or technician with instructions related to the potential criticality of an identified air leak within a scene and the potential damage costs resulting from the air leak. In one exemplary embodiment, the acoustic imaging system (e.g., via a processor) detects the presence of a leak within an acoustic scene, for example, through recognition of an acoustic signal that resembles a known sound profile associated with a leak present within the acoustic scene. The system may be configured to confirm the criticality of the identified leak by analyzing the detected acoustic signal, for example, by determining one or more sound profiles corresponding to the detected acoustic signal.

[0292] As described elsewhere herein, in some instances, the acoustic image data may be toned according to a determined threshold. In some embodiments, the threshold for the acoustic data may be determined according to one or more sound profiles. For example, a user may select a leak detection mode of operation, and the acoustic imaging system may analyze the acoustic data against one or more sound profiles associated with a leak to determine the threshold for the acoustic data in the scene.

[0293] As described above, the system may be configured to calculate a numerical cost associated with one or more leaks, such as via one or more equations and / or lookup tables. In certain exemplary embodiments, the system may, for example, identify a leak size and a pressure associated with the leak based on a sound profile associated with such leak, and then calculate a cost per unit time associated with such leak via a lookup table and / or equation.

[0294] In the illustrated example of Figure 34, a criticality and cost per unit time ($ / year) are associated with each of a plurality of locations within the acoustic scene. In the illustrated embodiment, such locations are indicated via acoustic image data that includes a plurality of indicators 3410, 3412, 3414 that are color-toned according to the criticality and that combine with visible light image data to form a display image that provides leakage criticality information to a user for each location. Such criticality may be determined by analysis of the acoustic scene in conjunction with one or more corresponding sound profiles, as described herein.

[0295] In various embodiments, leaks of varying degrees of criticality and / or potential loss costs may be indicated by acoustic image data including various colors, shapes, sizes, opacities, etc. Similarly, one or more icons may be used to represent a particular leak (e.g., corresponding to a particular sound profile) or the cost or range of criticality of that leak. Additionally or alternatively, alphanumeric information, such as cost / year, may be included proximate the corresponding leak. The display image of FIG. 34 further includes a notification 3420 indicating the approximate cost associated with each criticality level. Such values ​​may be calculated based on acoustic parameters for each acoustic signal detected in the scene, taking into account, for example, profiles stored in memory representing particular air leaks.

[0296] In some instances, additional contextual information is useful or necessary for proper analysis and scene reporting activities. For example, while acoustically imaging a scene during an operation, a user or technician may wish to record contextual information about the scene they are inspecting. In previous systems, to accomplish this task, the user or technician must take photographs with a separate camera or device, take written notes, or record notes using a separate device. Such notes must be manually synchronized with the data from the acoustic imaging device, potentially leading to errors in acquisition, reacquisition, and data mismatch, and potentially leading to errors in analysis and reporting.

[0297] In some embodiments, an acoustic imaging system of the present disclosure can capture acoustic data of a target scene and associate it with information about the target scene. Such information about the target scene can include details about one or more objects in the scene, the surroundings of the scene, and / or the surroundings of a location in the scene. In some embodiments, the associated information can be captured in the form of an image, audio recording, or video recording and associated with the acoustic data representing the scene (e.g., the acoustic data itself, a display image including corresponding acoustic image data, etc.). In some examples, the associated information can be associated with the acoustic data to provide a better understanding of what the information represents. For example, the associated information can include details about the target scene or objects within the target scene.

[0298] In some examples, the system may include one or more devices for gathering information about objects in a scene or the scene in general. One or more such devices may include a camera, a positioning device, a clock, a timer, and / or various sensors, such as temperature, electromagnetic, or humidity sensors. In some examples, the acoustic imaging system may include a camera (e.g., embedded in the housing of the acoustic imaging device) that may be configured to gather annotation information for image and / or video footage capture. Such an embedded camera may generate photographic or video annotations that may be added to or otherwise stored with the acoustic image data or other data captured by the acoustic imaging device.

[0299] In some examples, such embedded cameras may be configured to generate electromagnetic image data that can be combined with acoustic image data for display, as described elsewhere herein. In some embodiments, the acoustic imaging system may be configured to store various information, such as acoustic data, acoustic image data, electromagnetic image data, and annotation data (e.g., sensor data, image / video annotation data, etc.), along with a timestamp. In some examples, the acoustic imaging system may be configured to display and / or record associated information both on the device and later in software. The associated information may be displayed on a display and / or stored as metadata, for example, along with a saved display image or acoustic image file.

[0300] According to certain embodiments, through the use of embedded cameras, image capture devices or video capture devices within an acoustic imaging system, such devices may be utilized to generate photographic or video annotations that are appended to and / or stored with the primary acoustic data, combined electromagnetic and acoustic image data, and / or audio recordings.

[0301] In some examples, a user or technician may annotate acoustic image data shown on a display or other data collected by the acoustic imaging system, for example, via a user interface (e.g., a touch screen, one or more buttons, etc.) For example, in some examples, a user or technician may use on-display annotation to annotate data while it is being recorded or during playback of previously collected data.

[0302] In some examples, the acoustic imaging system allows for annotating a display image (e.g., including acoustic image data and / or electromagnetic image data) via on-screen interaction from a system user. In some such examples, the device stores all relevant annotation information along with the primary acoustic image, either prior to the need for synchronization or subsequent data matching. Such implementations can reduce or eliminate human memory errors that result in inaccurate pairing of primary and secondary data.

[0303] Various on-display annotations that may be added by a user (e.g., via a user interface) may include, but are not limited to, on-display drawings (e.g., freehand), on-display text and writing, on-display shape creation, on-display movement of an object, on-display placement of pre-configured markers, on-display placement of pre-configured text, instruments, or notes, on-display placement of pre-configured shapes, on-display placement of pre-configured drawings or illustrations, on-display placement of pre-configured or programmed icons, or on-display visualization of one or more acoustic parameters.

[0304] In some examples, an acoustic imaging device may include a display that can show collected (e.g., live or pre-collected) acoustic and / or electromagnetic image data, and a user can annotate such images via controls integrated into the display or a touch interface.

[0305] FIG. 35 shows an example of a user annotating a display image with annotations on the display. In the example of FIG. 35, three audio signals are shown at corresponding locations on the display image via color-toned indicators representing audio data associated with such audio signals. As shown, in some examples, a user can annotate the display image by drawing a freeform shape 3540 to highlight or emphasize portions of the display image. In this example, the user annotates the display image by surrounding a near-center sound with the freeform drawing 3540. The illustrated example also shows other annotation information in the form of text 3542 that can be used to identify or describe one or more components of the display image, such as components associated with the circled sound. For example, in the illustrated example, a label of the apparent source of the circled sound ("Main Line 2B-28") has been added to the display image. Such text can be handwritten (e.g., via a touchscreen interface) or typed (e.g., via a virtual keyboard or a physical keyboard in communication with the system).

[0306] Figure 36 shows an example of an annotated display image containing instructions and associated location information. In this figure, two sounds are shown on the display in addition to textual information and graphical indications instructing the user on tasks to be performed. A user can annotate the image as shown to further instruct or encourage the user to perform one or more tasks while conducting an acoustic check of such location.

[0307] 36, a user can annotate the display image by inserting pre-configured markers (e.g., arrow 3640) and / or handwritten text 3642. For example, in the illustrated example, the user has annotated the display image by inserting an arrow 3640 pointing at a valve switch in the scene and the textual instruction "Close the main valve first" 3642.

[0308] FIG. 37 shows an example of a user annotating a display image with annotations on the display. In this example, a single acoustic signal is shown via a corresponding indicator on the display image. The illustrated example shows a user annotating the display image by inserting a pre-programmed icon 3740, such as an air leak icon, into the display image. Such icons may be selected by the user and / or suggested by the system based on one or more recognized characteristic sounds (e.g., compared to historical sound profiles) and placed at identified sounds. The user may also annotate the image to identify or label possible sources of observed acoustic signals with free-form labels, predefined shapes, etc.

[0309] FIG. 38 shows an example of a user annotating a display image with annotations on the display. In this example, a single audio signal is shown via a corresponding indicator on the display image. As shown, the user can annotate the display image by inserting a shape 3840 into the image. In the example shown, the user has added a box 3840 around a component on the left hand side of the display image to indicate a possible source of audio from the audio image data. Such a box 3840 can be selected as a predetermined shape (e.g., a positionable and adjustable size rectangle). Other shapes can be used, sized, and positioned by the user to annotate the display image as desired.

[0310] In some examples, different types of labels may be combined on a display. For example, FIG. 39 shows an icon label 3942 placed on a display image near an indicator of acoustic image data representing acoustic signals in the scene, and a rectangle 3940 surrounding a component in the scene. Various combinations of annotations are possible. In some embodiments, annotations are automatically added to a display image, for example, when a particular sound profile is recognized in a scene. In some examples, the system may prompt the user to annotate the display image taking into account the recognized sound profile.

[0311] Additionally or alternatively, in some instances, a user may choose to annotate an image by including visual or textual information representing acoustic parameters associated with the acoustic scene. For example, a user may select a particular type of display (e.g., via a touchscreen or physical controls) to show one or more acoustic parameters associated with the scene. A user may also annotate an image to include information regarding cost or criticality indications associated with portions of the scene, such as detected leaks.

[0312] Annotations may include display features included within a live view of a display image and / or included in a separate captured display image that is saved in memory, for example.

[0313] As described elsewhere herein, in various embodiments, an acoustic imaging device can employ significantly different methods for displaying, locating, illustrating, and analyzing detected sounds. Visualization methods can include various colored shapes, icons, and various levels of transparency adjustment to adjust the visual background against which they are displayed. By simplifying parameter control for sound visualization on the device or making such control more intuitive, users can achieve better visualization results more easily, in less time, and with less training. Various methods of visualization parameter control can be implemented according to the application and the needs of the user. Many of these methods can be adapted for use with specific individual levels of education and training in sound visualization and sound localization, providing a device more suited to various types of applications and organizations. As described elsewhere herein, in some examples, a user can choose to annotate a display image by including specific data visualization techniques in the display image.

[0314] FIG. 40 illustrates an interface including a display image 4002 and a multi-parameter data visualization 4040 including multiple frequency ranges on the right-hand side of the display image. In some examples, a user can select one or more frequency ranges from the multiple displayed frequency ranges (e.g., via a touchscreen and / or other interface), for example, to filter the acoustic image data to display an acoustic image having frequency content associated with the selected frequency ranges (e.g., frequency content above a certain magnitude). In the illustrated example, two frequency ranges 4042 and 4044 have been selected. Corresponding indicators 4010 and 4012 in the display image indicate locations within the scene having acoustic signals that satisfy the corresponding frequency ranges among the multiple frequency ranges displayed on the screen. In the illustrated example, frequency range 4042 and corresponding indicator 4010 are shown with light shading, while frequency range 4044 and corresponding indicator 4012 are shown with dark shading. In general, in some embodiments, acoustic signals within a scene that satisfy a certain frequency range can be represented via an indicator having a corresponding visualization for that frequency range.

[0315] In general, frequency ranges can be displayed in various ways, such as a right-hand axis, a left-hand axis, a bottom-end axis, a top-end axis, or a center axis. In various embodiments, frequency ranges may be divided into resolutions, including increments of 1 kHz (e.g., 1 kHz to 2 kHz, 2 kHz to 3 kHz, etc.). Such frequency ranges do not necessarily have to be the same size or span the same frequency range. In some examples, multi-parameter display The physical size (e.g., width) of the frequency range displayed in 4040 corresponds to one or more parameters such as the relative amount of frequency content, the amplitude of such frequencies within the audio scene, or the proximity of such frequencies.

[0316] In various embodiments, frequency ranges are selected by virtual control through touchscreen interaction and / or physical control mechanisms such as a directional pad, where the user scrolls up, down, left, right, and holds a button until the desired range bar is highlighted and then selected. In some instances, multiple ranges are selected or deselected by the user.

[0317] In one embodiment, frequency bars (e.g., 4042, 4044) on display image 4002 associated with various frequency ranges rise and fall with the decibel level of the range in the real-time display image. In various examples, the decibel level of the range may be determined in a number of ways, such as a peak decibel level, an average decibel level, a minimum decibel level, a time-based average decibel level, etc. In one embodiment, the decibel level associated with each frequency range may be tracked over time. Tracking over time may include storing frequency information at each of a number of times, such as at given intervals. Additionally or alternatively, tracking frequency data over time may include tracking the peak decibel level observed in each frequency range over time (e.g., within a particular time period, within an operating session). The peak level may be calculated in a variety of ways.

[0318] The display of frequency information can include peak frequency data in addition to current / present frequency information. FIG. 41 illustrates an interface including a display image 4102 and a multi-parameter display 4140 of frequency information, including multiple frequency ranges located along the bottom edge of the display image. As shown in FIG. 41, the frequency information displayed at the bottom of the display image includes amplitude information (e.g., 4150) for multiple frequency ranges (measured in decibels in the example of FIG. 41). In some examples, maximum decibel levels are displayed for one or more frequency ranges. In some cases, the frequency ranges showing maximum decibel levels can be selected by the user. A peak marker 4152 can remain on the multi-parameter display, indicating the maximum decibel level and can be represented in any number of possible ways, such as a contrasting color, a bar cap, an arrow, a symbol, a number, or the like. In some examples, the maximum decibel levels shown in the data visualization include the maximum detected over a period of time, such as within the past 5 seconds, the past 30 minutes, etc., from the start of the selected measurement. In some examples, the user can reset the maximum value display to ignore previous maximum values.

[0319] In various embodiments, the frequency bands included in the display image can be adjusted by a user or can be automatically determined by the device with a programmed algorithm or machine learning. In various examples, the frequency bands can be evenly sized and distributed, unevenly sized, histogram equalized, or determined by any combination of methods.

[0320] Figure 42 shows a display image including frequency information for multiple frequency bands and peak values ​​for multiple frequency bands similar to that shown in Figure 41. In the example of Figure 42, a multi-parameter display 4240 including intensity and frequency information is shown in the right-hand portion of the screen, with amplitude information (e.g., 4250) displayed horizontally with the axis to the right. As in Figure 41, the multi-parameter display 4240 includes peak markers 4252 that indicate the peak amplitudes for one or more frequency ranges over a period of time.

[0321] In some instances, the decibel level may increase, for example, to the right and left of the central axis in the mirror image. Similarly, decibel level peaks may also appear in the mirror image.

[0322] Such decibel information mirrored about a central axis is shown in Figure 43. The display image of Figure 43 includes a multi-parameter display 4340 showing intensity information for multiple frequencies. Such a mirrored axis display allows the user to better identify small or low decibel level changes that are important in frequency range selection.

[0323] Similar to Figures 41 and 42, the multi-parameter display 4340 of Figure 43 includes peak markers 4352 that indicate the peak amplitudes of one or more frequency ranges within a time period.

[0324] In some embodiments, one or more frequency ranges may be color-toned to indicate additional information about such frequency ranges, such as the decibel level of such frequency ranges. FIG. 44 shows a multi-parameter display 4440 including a set of color-toned frequency ranges, where the color toning indicates the decibel range into which each frequency range falls. For example, in the illustrated embodiment, frequency ranges depicted in white (e.g., 4450) fall between 0 and 20 dB, frequency ranges depicted in light gray (e.g., 4450) fall between 21 and 40 dB, and frequency ranges depicted in dark gray fall between 41 and 100 dB. In some examples, the color toning corresponds to relative rather than absolute values, such that, for example, frequency ranges depicted in white are considered to have a "low" intensity, frequency ranges depicted in light gray are considered to have a "normal" intensity, and frequency ranges depicted in dark gray are considered to have a "high" intensity. In an exemplary embodiment, the bottom three frequency ranges in terms of intensity are labeled white, the middle three frequency ranges in terms of intensity are labeled light gray, and the top three frequency ranges in terms of intensity are labeled dark gray. In another example, frequency ranges having intensities up to one-third of maximum intensity are shown in white, frequency ranges having intensities between one-third and two-thirds of maximum intensity are shown in light gray, and frequency ranges having intensities between two-thirds of maximum intensity and maximum intensity are shown in dark gray. In some examples, the coloring techniques shown in the displayed frequency information may also be used for one or more indicators present in the acoustic image data. In general, either colorization techniques or other visualization techniques (e.g., via various patterns, transparency, etc.) may be used.

[0325] In some examples, frequency ranges may be color-toned relative to the severity of the detected acoustic data in each frequency range. For example, in some examples, frequency ranges depicted in dark gray may be considered to be extremely severe, frequency ranges depicted in light gray may be considered to be moderately severe, and frequency ranges depicted in white may be considered to indicate minor severity. Similarly to the above, in some examples, one or more indicators present in the acoustic image data may include similar color-toned severity indications at one or more locations within the acoustic scene.

[0326] 45 shows an example display image including a multi-parameter display 4540 showing different frequency ranges and indicators 4510, 4512, 4514 that are color-tuned according to severity. In various examples, the colors corresponding to different severity levels may be set automatically by the device or manually by the user.

[0327] As described elsewhere herein, in some examples, frequency intensity data may be stored or tracked over time. In some embodiments, intensity versus time information may be displayed in each of one or more frequency ranges of a multi-parameter display. FIG. 46 illustrates intensity (in dB) versus time trends in each of multiple frequency ranges (e.g., 4650) in a multi-parameter display 4640. For example, in some examples, in addition to intensity versus time, peak intensity information may be displayed in the multi-parameter display 4640, such as peak markers (e.g., 4652), in one or more of the one or more frequency ranges. As shown in FIG. 46, the multi-parameter display 4640 includes a time axis and displays intensity information corresponding to the acoustic frequency or acoustic frequency range over time in each of multiple acoustic frequencies or acoustic frequency ranges (e.g., 4650).

[0328] Various processes described herein may be embodied as a non-transitory computer-readable medium containing executable instructions for causing one or more processors to execute such processes. A system may include one or more processors configured to execute such processes based on instructions stored in memory, for example, integral with the processor or external to the processor. In some cases, various components may be distributed throughout the system. For example, a system may include multiple distributed processors, each configured processor executing at least a portion of the overall processes executed by the system. Furthermore, it will be appreciated that various features and functionality described herein may be combined into a single acoustic imaging system embodied, for example, as a handheld acoustic imaging tool or a distributed system having various distinct and / or separable components.

[0329] Various features of the components described herein may be combined. In some embodiments, features described in this application may be combined with features described in PCT application entitled "SYSTEMS AND METHODS FOR PROJECTING AND DISPLAYING ACOUSTIC DATA," filed July 24, 2019, having Attorney Docket No. 56581.178.2, which is assigned to the assignee of the present application and incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in PCT application entitled "SYSTEMS AND METHODS FOR TAGGING AND LINKING ACOUSTIC IMAGES," filed July 24, 2019, having Attorney Docket No. 56581.179.2, which is assigned to the assignee of the present application and incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in PCT application entitled "SYSTEMS AND METHODS FOR DETACHABLE AND ATTACHABLE ACOUSTIC IMAGING SENSORS," having Attorney Docket No. 56581.180.2, filed July 24, 2019, which is assigned to the assignee of the present application and incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in PCT application entitled "SYSTEMS AND METHODS FOR REPRESENTING ACOUSTIC SIGNATURES FROM A TARGET SCENE," having Attorney Docket No. 56581.182.2, filed July 24, 2019, which is assigned to the assignee of the present application and incorporated herein by reference.

[0330] Various embodiments have been described. Such examples are non-limiting and are not intended to limit or restrict the scope of the invention in any way.

Claims

1. an acoustic sensor array comprising a plurality of acoustic sensor elements configured to receive acoustic signals from a scene and output acoustic data based on the acoustic signals; an electromagnetic imaging tool configured to output electromagnetic image data indicative of electromagnetic radiation from the scene; a processor in communication with the acoustic sensor array and the electromagnetic imaging tool, generating an image based on the acoustic data and the electromagnetic image data; determining a location within the image based on a location of an acoustic signal in the scene; an indicator generating a visual characteristic representative of an acoustic parameter of the acoustic signal is positioned at the location within the image; a processor configured to cause a multi-parameter display separate from the indicator to be displayed on the image, the multi-parameter display including multiple graphs, each graph displaying information representing a parameter for each acoustic frequency of a plurality of acoustic frequencies or each acoustic frequency range of a plurality of acoustic frequency ranges; a user interface in communication with the processor, the processor comprising: receiving, via the user interface, a selection of one or more acoustic frequencies of the plurality of acoustic frequencies or one or more acoustic frequency ranges of the plurality of acoustic frequency ranges; determining one or more locations within the scene having respective acoustic signals containing frequency content within the one or more selected acoustic frequencies or one or more acoustic frequency ranges; an acoustic analysis system configured to cause the indicator to be positioned on the image at each of the one or more locations;

2. 2. The acoustic analysis system of claim 1, wherein each graph of the multi-parameter display includes intensity information representing acoustic intensity for each acoustic frequency of the plurality of acoustic frequencies or each acoustic frequency range of the plurality of acoustic frequency ranges.

3. 3. The acoustic analysis system of claim 2, wherein the multi-parameter display includes a time axis and represents the intensity information over time for each acoustic frequency of the plurality of acoustic frequencies or each acoustic frequency range of the plurality of acoustic frequency ranges.

4. 2. The acoustic analysis system of claim 1, wherein the multi-parameter display includes a display of a current acoustic intensity value and a display of a maximum acoustic intensity value for each of the plurality of acoustic frequencies or the plurality of acoustic frequency ranges.

5. The acoustic analysis system of claim 4 , wherein the maximum acoustic intensity value represents the maximum acoustic intensity observed over a predetermined time period for each of the plurality of acoustic frequencies or ranges of acoustic frequencies.

6. 5. The acoustic analysis system of claim 4, wherein the maximum acoustic intensity value comprises the largest acoustic intensity value detected during a predetermined period of time, or since the start of a selected measurement, or since resetting the maximum acoustic intensity value.

7. 7. The acoustic analysis system of claim 1, wherein the indicator or the multi-parameter display includes criticality information representing criticality of acoustic data at a location within the scene or at a predetermined acoustic frequency or range of acoustic frequencies within the scene.

8. 7. The sound analysis system of claim 1, wherein the processor is configured to tint one or more frequency ranges of the plurality of sound frequency ranges to indicate additional information about the one or more frequency ranges, and wherein the one or more frequency ranges are tinted to indicate a decibel range within which each frequency range falls.

9. 7. The acoustic analysis system of claim 1, wherein the processor is configured to tint one or more frequency ranges of the plurality of acoustic frequency ranges to indicate additional information about the one or more frequency ranges, and wherein the one or more frequency ranges are tinted to indicate the severity of the acoustic data in frequency ranges of the plurality of acoustic frequency ranges.

10. An acoustic analysis system as described in any one of claims 1 to 6, wherein the visual characteristics include a color-tuned display of the intensity of the acoustic signal.

11. 11. The audio analysis system of claim 10, wherein the visual characteristics include a gradient color tone that indicates locations within a scene where acoustic parameters of the audio signal satisfy one or more filter conditions.

12. 11. The acoustic analysis system of claim 10, wherein the processor is configured to determine acoustic parameters of the acoustic signal within a plurality of ranges and to display different ranges within the plurality of ranges using respective colors in the visual characteristics.

13. 13. The acoustic analysis system of claim 10, wherein the processor is configured to use one or more filters to identify one or more locations within the scene where acoustic parameters of the acoustic signal satisfy one or more acoustic conditions including a user-selected frequency range, intensity range, or distance range, and to position the indicator at one or more locations within the image based on the one or more locations within the scene.

14. 13. The acoustic analysis system of claim 10, wherein the processor is configured to use one or more filters to identify one or more locations in the scene where acoustic parameters of the acoustic signal correspond to stored sound profiles, and to position the indicator at one or more locations in the image based on the one or more locations in the scene.