Systems and methods for projecting and displaying acoustic data

Abstract

A system can include an acoustic sensor array configured to receive acoustic signals, an illuminator configured to emit electromagnetic radiation, an electromagnetic imaging tool configured to receive electromagnetic radiation, a distance measurement tool, and a processor. The processor can illuminate a target scene via the illuminator, receive electromagnetic image data representative of the illuminated scene from the electromagnetic imaging tool, receive acoustic data from the acoustic sensor array, and receive distance information from the distance measurement tool. The processor can be further configured to generate acoustic image data of the scene based on the received acoustic data and the received distance information, and generate a display image including the combined acoustic image data and electromagnetic image data. The processor can determine depths of various sound signals within the scene, and generate a representation of the scene showing the determined depths, including a plan view and a volumetric representation.

Description

[0001] Related matters

[0002] This application claims priority to U.S. Patent Application No. 62 / 702,716, filed July 24, 2018, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Currently available acoustic imaging devices include acoustic sensor array configurations with various frequency sensitivity limitations due to a wide range of factors. For example, some acoustic imaging devices are designed to respond to an acoustic frequency range between approximately 20 Hz and approximately 20 kHz. Other devices (e.g., ultrasound devices) are designed to respond to an acoustic frequency range between approximately 38 kHz and approximately 45 kHz.

[0004] However, acoustic imaging devices typically designed to operate in the 20 Hz to 20 kHz frequency range are ineffective at detecting or imaging higher frequencies, such as those up to or above approximately 50 kHz. Similarly, acoustic or ultrasonic devices designed to operate in the 20 kHz to 50 kHz frequency range are ineffective at detecting and / or imaging lower frequencies, such as those at or below 20 kHz. This can be for a variety of reasons. For example, sensor arrays optimized for lower (e.g., audible) frequencies often contain individual sensors that are more spaced apart than sensor arrays optimized for higher (e.g., ultrasonic) frequencies.

[0005] Aside from hardware considerations or as alternatives, different computational algorithms and methods for acoustic imaging are often better suited to acoustic signals with different frequencies and / or different distances to the target, making it difficult to determine how best to acoustically image a scene, especially in the absence of inexperienced users.

[0006] Such differences in imaging different frequency ranges are partly due to the physical phenomena underlying the propagation of sound waves of different frequencies and wavelengths through the air. Certain array orientations, array sizes, and computational methods are often better suited to sound signals with different frequency characteristics (e.g., audible frequencies, ultrasonic frequencies, etc.).

[0007] Similarly, different array properties and / or computation methods can be better suited to acoustic scenarios at different distances from the target. For example, near-field acoustic holography for targets at very close range, and various acoustic beamforming methods for targets at greater distances.

[0008] Therefore, acoustic inspections using acoustic arrays (e.g., for acoustic imaging) may require a wide range of equipment, such as those for analyzing acoustic signals with different frequency ranges, and expertise in understanding when different hardware and computing technologies are suitable for performing acoustic analysis. This can make acoustic inspections both time-consuming and expensive, and may require specialists to perform such inspections.

[0009] For example, users may be forced to manually select various hardware and / or software for performing acoustic analysis. However, inexperienced analysts may not know the optimal combination of hardware and software for a given acoustic analysis and / or acoustic scenario. Additionally, isolating sounds of interest from within the scenario can present its own challenges, especially in noisy environments, and can be tedious and frustrating for inexperienced users. For instance, a given acoustic scenario (especially in a noisy environment) can include acoustic signals comprising any number of frequencies, intensities, or other characteristics that might obscure the acoustic signals of interest.

[0010] Traditional systems often require users to manually identify various acoustic parameters of interest before analysis. However, inexperienced users may not know how best to isolate and / or identify the various sounds of interest.

[0011] Additionally, when multiple imaging techniques (e.g., visible light, infrared, ultraviolet, acoustic, or other imaging techniques) are used in tandem while examining the same object or scene, the physical layout and / or other settings (e.g., focus orientation) of the tools used to perform the different imaging techniques can affect the analysis. For example, different positions and / or focus orientations of each imaging device can lead to parallax errors, where the resulting images may be misaligned. This can result in the inability to correctly locate areas of interest and / or problem areas within the scene, document errors, and misdiagnosis of problems. For instance, if acoustic image data is misaligned relative to image data from other imaging techniques (e.g., visible light and / or infrared image data), it may be difficult to identify the location or source of the acoustic signal of interest.

[0012] Existing ultrasonic testing and inspection tools employ one or more ultrasonic sensors, with or without a parabolic dish, to help focus sound toward one or more receiving sensors. When a sound of a specific frequency is detected, it is typically displayed as a rising or falling numerical value, or on a frequency or decibel level graph on the device's display. This can be very confusing and unintuitive for many users. Visualizations of the actual scene, either visually or aurally, are not readily available.

[0013] Isolating, locating, and analyzing specific sounds can be a tedious process and can be confusing for many end users. Advanced acoustic imaging equipment employs non-digital visualization techniques, where one or more sounds can be visualized using a live video background or a static visual background. These rely on the performance of the video or still camera to produce high-quality images. If the environment is dark, or the target being examined is some distance away from the equipment, image quality may be poor, and / or the field of view of the displayed image may be incorrect relative to the actual target of interest. This can cause the equipment to produce and store images and data with questionable quality and usability.

[0014] Additionally, in some cases, sound can travel long distances through open spaces and may be reflected in various objects. This further complicates locating and identifying the source of the sound. Summary of the Invention

[0015] Some aspects of this disclosure relate to acoustic imaging systems. The system may include an acoustic sensor array comprising a plurality of acoustic sensor elements, each of which may be configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals.

[0016] The system may include one or more illuminators configured to emit electromagnetic radiation toward a target scene. These illuminators may be configured to emit electromagnetic radiation encompassing one or more wavelength ranges.

[0017] The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representing the received electromagnetic radiation. The electromagnetic imaging tool may be configured to detect electromagnetic radiation from a wavelength range, such as a range including the visible and / or near-infrared spectrum. In some systems, the electromagnetic imaging system may include a visible light camera module and / or an infrared camera module.

[0018] The system may include a distance measuring tool. This tool can be configured to provide distance information representing the distance to a target. In some systems, the distance measuring tool may include a laser designator.

[0019] The system may include a display and a processor. The processor can communicate with an acoustic sensor array, an illuminator, an electromagnetic imaging tool, a distance measuring tool, and the display.

[0020] In some systems, the processor can be configured to illuminate a target scene via a illuminator, receive electromagnetic image data representing the illuminated scene from an electromagnetic imaging tool, receive acoustic data from an acoustic sensor array, and receive distance information representing the distance to the target from a distance measurement tool. In some embodiments, the illuminator can emit electromagnetic radiation of one or more wavelengths that at least partially overlap with the wavelength range received by the electromagnetic imaging tool.

[0021] In some systems, the processor can be configured to generate acoustic image data of a scene based on received acoustic data and received distance information, and to generate a display image that includes the combined acoustic and electromagnetic image data. The processor can then transmit the display image to a monitor.

[0022] The system may include an ambient light sensor that communicates with a processor. In some systems, the processor may be configured to receive data representing the amount of ambient light from the ambient light sensor. In some systems, the processor may illuminate a target scene with a illuminator in response to the amount of ambient light falling below a predetermined threshold, and / or suggest to a user that the target scene be illuminated in response to the amount of ambient light falling below a predetermined threshold.

[0023] In some examples, the processor can be configured to determine the depth of various acoustic signals within the scene and generate a representation of the scene showing the determined depth. Such a representation may include acoustic image data on a two-dimensional planar view (e.g., a top view of the scene) or within a three-dimensional volumetric representation of the scene.

[0024] Details of one or more examples are set forth below in the accompanying drawings and description. Other features, objects, and advantages will become apparent from this description and the accompanying drawings, as well as from the claims. Attached Figure Description

[0025] Figure 1A and 1B A front view and a rear view of an exemplary acoustic imaging device are shown.

[0026] Figure 2 This is a functional block diagram illustrating the components of an example acoustic analysis system.

[0027] Figure 3A , 3B The diagram above and 3C show an exemplary configuration of an acoustic sensor array within an acoustic analysis system.

[0028] Figure 4A and 4B A schematic diagram of parallax error in the generation of frames of visible light image data and acoustic image data is shown.

[0029] Figure 5A and5B Parallax correction between visible light and acoustic images is shown.

[0030] Figure 5C and 5D yes Figure 5A and 5B The color version.

[0031] Figure 6 This is a flowchart illustrating an exemplary method for generating a final image that combines acoustic image data and electromagnetic image data.

[0032] Figure 7 This is a flowchart illustrating an exemplary process for generating acoustic image data from a received acoustic signal.

[0033] Figure 8 An exemplary lookup table is shown for determining the appropriate algorithm and sensor array to be used during the acoustic imaging process.

[0034] Figure 9A This is an exemplary plot of the frequency content of image data received in an acoustic scene over time.

[0035] Figure 9B An exemplary scenario is shown, including multiple locations where acoustic signals are emitted.

[0036] Figure 9C The diagram shows multiple combined acoustic and visible light image data at several predefined frequency ranges.

[0037] Figure 10A and 10B It is an exemplary display image that includes the combined visible light image data and acoustic image data.

[0038] Figure 11A and 11B An exemplary plot of frequency-to-time comparison of acoustic data in an acoustic scene is shown.

[0039] Figure 12A , 12B Figures 12C and 12C illustrate several exemplary methods for comparing acoustic image data with historical acoustic image data stored in a database.

[0040] Figure 13 This is a flowchart illustrating an exemplary operation of comparing received acoustic image data with a database used for object diagnosis.

[0041] Figure 14 A visualization of acoustic data using a palettization scheme is shown.

[0042] Figure 15 A visualization of acoustic data using multiple concentric shaded circles is shown.

[0043] Figure 16 An exemplary visualization including both non-numeric and alphanumeric information is shown.

[0044] Figure 17 Another example visualization showing both non-numeric and alphanumeric information is presented.

[0045] Figure 18 Another example visualization showing both non-numeric and alphanumeric information is presented.

[0046] Figure 19 An exemplary visualization is shown, which displays indicators of different sizes and colors representing different acoustic parameter values.

[0047] Figure 20 An exemplary visualization is shown, which illustrates multiple indicators with different colors, each color indicating the severity indicated by an acoustic signal from a corresponding location.

[0048] Figure 21 A scene is shown that includes indicators at multiple locations within the scene, which distinguish sound signals that meet predetermined conditions.

[0049] Figure 22 A display image is shown, including multiple icons positioned within the display image, which indicate identified acoustic contours within the scene.

[0050] Figure 23 Another exemplary display image is shown, which illustrates acoustic data via multiple indicators, using concentric circles and alphanumeric information representing the sound intensity associated with each sound signal.

[0051] Figure 24 An example display image is shown, which has indicators and additional alphanumeric information associated with the represented sound signal.

[0052] Figure 25A A system including a display on which an indicator within a display image is selected, and a laser pointer emitting a laser towards the scene, is shown.

[0053] Figure 25B The display is shown in the system view of Figure 25.

[0054] Figure 26 A display image is shown that includes an indicator with a gradient color scheme representing acoustic image data and includes acoustic image blending controls.

[0055] Figure 27 A display image is shown that includes an indicator with a concentric circle color scheme representing acoustic image data and includes acoustic image blending controls.

[0056] Figure 28 A display image is shown that includes an indicator with gradient color tone, which indicates the location in the scene that meets one or more filtering conditions.

[0057] Figure 29 A display image is shown that includes indicators with concentric circle color tones, indicating locations in the scene that satisfy one or more filtering conditions.

[0058] Figure 30 The image shown includes two indicators, each with a gradient color tone, indicating the location in the scene that meets different filtering conditions.

[0059] Figure 31 The display interface includes an image and a virtual keyboard.

[0060] Figure 32 A display embedded in glasses is shown, which can be worn by a user and display images.

[0061] Figure 33A and 33B A dynamic display image is shown, which includes an indicator with dynamic intensity based on the direction of an array of acoustic sensors.

[0062] Figure 34A An acoustic imaging device that emits lasers toward a scene is shown.

[0063] Figure 34B An example display image is shown, which includes visible light image data and acoustic image data, as well as a representative visualization of laser points in the scene.

[0064] Figure 35A The image shows acoustic and electromagnetic image data representing a scene without lighting.

[0065] Figure 35B An image of the scene captured using a visible light illuminator is shown.

[0066] Figure 35C An image of a scene captured using a near-infrared illuminator is shown.

[0067] Figure 36 An example implementation is shown where the user can select a location within the scene, and the system can point a laser pointer at that location.

[0068] Figure 37 A non-limiting embodiment is shown, in which one or more laser pointers are used to identify three sounds.

[0069] Figure 38 A projector system is shown that projects acoustic image data toward a location in a displayed image.

[0070] Figure 39 An acoustic imaging device with a projector system is shown, which projects acoustic image data representing multiple sounds toward the corresponding origins of such sounds.

[0071] Figure 40 An exemplary embodiment is shown, wherein acoustic image data is mapped to a representative area map.

[0072] Figure 41 This shows multiple sounds that are mapped onto the XY region and superimposed on the planar layout diagram.

[0073] Figure 42 An exemplary embodiment is shown, in which acoustic image data is mapped to a volumetric space.

[0074] Figure 43 An acoustic imaging device is shown, which includes a display showing a volumetric representation of acoustic image data including the three-dimensional orientation of multiple detected sounds, indicated by a plurality of corresponding indicators.

[0075] Figure 44 An acoustic imaging system is shown, including a dot projection system for projecting dot patterns onto a scene.

[0076] Figure 45 An exemplary display image is shown, which includes visualization of acoustic image data mapped onto a three-dimensional representation of the scene. Detailed Implementation

[0077] Figure 1A and 1B The front and rear views of an example acoustic imaging device are shown. Figure 1AThe front side of an acoustic imaging device 100 is shown, which has a housing 102 supporting an acoustic sensor array 104 and an electromagnetic imaging tool 106. In some embodiments, the acoustic sensor array 104 includes a plurality of acoustic sensor elements, each configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals. The electromagnetic imaging tool 106 may be configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representing the received electromagnetic radiation. The electromagnetic imaging tool 106 may be configured to detect electromagnetic radiation in one or more wavelength ranges, such as visible light, infrared, ultraviolet light, etc.

[0078] In the illustrated example, acoustic imaging device 100 includes: an ambient light sensor 108 and a position sensor 116, such as a GPS. Device 100 includes: a laser designator 110, which in some embodiments includes a laser rangefinder. Device 100 includes: a torch 112, which can be configured to emit visible light radiation toward a scene; and an infrared illuminator 118, which can be configured to emit infrared radiation toward a scene. In some examples, device 100 may include an illuminator for illuminating a scene in any wavelength range. Device 100 further includes: a projector 114, such as an image reprojector, which can be configured to project a generated image onto a scene such as a color image; and / or a dot projector, which is configured to project a series of points onto the scene, for example, to determine the depth contour of the scene.

[0079] Figure 1B The back of the acoustic imaging device 100 is shown. As shown, the device includes a display 120 that can present images or other data. In some examples, the display 120 includes a touchscreen display. The acoustic imaging device 100 includes a speaker that can provide audio feedback signals to a user, and a wireless interface 124 that enables wireless communication between the acoustic imaging device 100 and external devices. The device further includes controls 126, which may include one or more buttons, knobs, dials, switches, or other docking components to enable a user to interact with the acoustic imaging device 100. In some examples, controls 126 and a touchscreen display are combined to provide a user interface for the acoustic imaging device 100.

[0080] In various embodiments, the acoustic imaging device need not include Figure 1A and 1B Each element shown in the embodiments. One or more of the illustrated components can be excluded from the acoustic imaging device. In some examples, in Figure 1A and 1BOne or more components shown in the embodiments may be included as part of the acoustic imaging system, but separately from the housing 102. Such components may communicate with other components of the acoustic imaging system, for example, using the wireless interface 124, via wired or wireless communication technologies.

[0081] Figure 2 This is a functional block diagram illustrating the components of an example acoustic analysis system 200. Figure 2 An exemplary acoustic analysis system 200 may include a plurality of acoustic sensors, such as microphones, MEMS, transducers, etc., arranged in an acoustic sensor array 202. Such an array may be one-dimensional, two-dimensional, or three-dimensional. In various examples, the acoustic sensor array may define any suitable size and shape. In some examples, the acoustic sensor array 202 includes a plurality of acoustic sensors arranged in a grid pattern, such as, for example, an array of sensor elements arranged in vertical columns and horizontal rows. In various examples, the acoustic sensor array 202 may include an array of vertical columns arranged in, for example, horizontal rows of 8×8, 16×16, 32×32, 64×64, 128×128, 256×256, etc. Other examples are possible, and various sensor arrays need not necessarily include the same number of rows as columns. In some embodiments, such sensors may be positioned on a substrate, such as a printed circuit board (PCB) substrate.

[0082] exist Figure 2 In the illustrated configuration, processor 212, communicating with acoustic sensor array 202, can receive acoustic data from each of the plurality of acoustic sensors. During exemplary operation of acoustic analysis system 200, processor 212 can communicate with acoustic sensor array 202 to generate acoustic image data. For example, processor 212 can be configured to analyze data received from each of the plurality of acoustic sensors arranged in an acoustic sensor array and determine the acoustic scene by “backpropagating” the acoustic signal back to the source of the acoustic signal. In some embodiments, processor 212 can generate digital “frames” of acoustic image data by identifying various source locations and intensities of acoustic signals across a two-dimensional scene. By generating frames of acoustic image data, processor 212 captures an acoustic image of the target scene at a substantially given point in time. In some examples, a frame includes a plurality of pixels constituting the acoustic image, wherein each pixel represents a portion of the source scene to which the acoustic signal has been backpropagated.

[0083] The components described as processors within the acoustic analysis system 200 (including processor 212) can be implemented as one or more processors, individually or in any suitable combination, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic circuits, and so on. Processor 212 may also include memory storing program instructions and associated data that, when executed by processor 212, enable the acoustic analysis system 200 and processor 212 to perform the functions attributed to them in this disclosure. The memory may include any fixed or removable magnetic, optical, or electrical medium, such as RAM, ROM, CD-ROM, hard disk or floppy disk, EEPROM, and so on. The memory may also include removable memory portions that can be used to provide memory updates or increase memory capacity. Removable memory may also allow easy transfer of acoustic image data to another computing device or removal of acoustic image data before the acoustic analysis system 200 is used in another application. Processor 212 may also be implemented as a system-on-a-chip, integrating some or all components of a computer or other electronic system onto a single chip. The processor 212 (processing circuitry) can be configured to transmit processed data to the display 214 or other output / control device 218.

[0084] In some embodiments, the acoustic sensors in the acoustic sensor array 202 generate a series of signals corresponding to the acoustic signals received by each acoustic sensor to represent an acoustic image. "Frames" of acoustic image data are generated when signals from each acoustic sensor are acquired by scanning all rows constituting the acoustic sensor array 202. In some examples, the processor 212 may acquire acoustic image frames at a rate sufficient to generate a video representation of the acoustic image data (e.g., 30 Hz or 60 Hz). Independent of specific circuitry, the acoustic analysis system 200 may be configured to manipulate acoustic data representing the acoustic contours of a target scene to provide an output that can be displayed, stored, transmitted, or otherwise utilized by a user.

[0085] In some embodiments, the acoustic signals received during "backpropagation" to generate acoustic image data include, for example, analysis by a processor of signals received at multiple acoustic sensors in the acoustic sensor array 202. In various examples, backpropagation is performed as a function of one or more parameters, including distance to the target, frequency, sound intensity (e.g., decibel level), and the size / configuration of the sensor array, such as the spacing and arrangement of individual sensors within the array. In some embodiments, such parameters may be pre-programmed into the system, for example, in memory. For example, properties of the acoustic sensor array 202 may be stored in memory, such as internal memory or memory specifically associated with the acoustic sensor array 202. Other parameters, such as distance to the target, may be received in a variety of ways. For example, in some examples, the acoustic analysis system 200 includes a distance measurement tool 204 communicating with the processor 212. This distance measurement tool may be configured to provide distance information representing the distance from the distance measurement tool 204 to a specific location in the target scene. Various distance measurement tools may include laser rangefinders or other known distance measurement devices, such as other optical or audio distance measurement devices. Additionally or alternatively, the distance measurement tool can be configured to generate three-dimensional depth data such that each part of the target scene has an associated distance value to the target. Thus, in some examples, as used herein, the measured distance to the target can correspond to the distance to each location within the target scene. Such three-dimensional depth data can be generated, for example, via multiple imaging tools having different views of the target scene, or via other known distance scanning tools. Generally, in various embodiments, the distance measurement tool can be used to perform one or more distance measurement functions, including but not limited to: laser distance measurement, active acoustic distance measurement, passive ultrasonic distance measurement, LiDAR distance measurement, RADAR distance measurement, millimeter-wave distance measurement, and so on.

[0086] Distance information from distance measuring tool 204 can be used in backpropagation calculations. Additionally or alternatively, system 200 may include a user interface 216 into which a user can manually input parameters regarding the distance to a target. For example, if the distance to a component suspected of generating an acoustic signal is known or difficult to measure using distance measuring tool 204, the user can input the value of the distance to the target into system 200.

[0087] In the illustrated embodiment, the acoustic analysis system 200 includes an electromagnetic imaging tool 203 for generating image data representing a target scene. The exemplary electromagnetic imaging tool can be configured to receive electromagnetic radiation from the target scene and generate electromagnetic image data representing the received electromagnetic radiation. In some examples, the electromagnetic imaging tool 203 can be configured to generate electromagnetic image data representing a specific wavelength range within the electromagnetic spectrum, such as infrared radiation, visible light radiation, and ultraviolet radiation. For example, in some embodiments, the electromagnetic timing tool 203 may include one or more camera modules, such as, for example, a visible light camera module 206, configured to generate image data representing a specific wavelength range in the electromagnetic spectrum.

[0088] Visible light camera modules are generally known. For example, various visible light camera modules are included in smartphones and many other devices. In some embodiments, the visible light camera module 206 may be configured to receive visible light energy from a target scene and focus that energy onto a visible light sensor to generate visible light energy data, which may be displayed on a display 214 and / or stored in memory, for example, as a visible light image. The visible light camera module 206 may include any suitable components for implementing the functions of the modules pertaining to herein. Figure 2 In one example, the visible light camera module 206 is illustrated as including a visible light lens assembly 208 and a visible light sensor 210. In some such embodiments, the visible light lens assembly 208 includes at least one lens that captures the visible light energy emitted by a target scene and focuses the visible light energy onto the visible light sensor 210. The visible light sensor 210 may include multiple visible light sensor elements, such as, for example, a CMOS detector, a CCD detector, a PIN diode, an avalanche photodiode, etc. The visible light sensor 210 responds to the focused energy by generating an electrical signal, which can be converted and displayed as a visible light image on the display 214. In some examples, the visible light module 206 is user-configurable and can provide output to the display 214 in various formats, for example. The visible light camera module 206 may include compensation functions for various lighting or other operating conditions or user preferences. The visible light camera module can provide a digital output including image data, which may include data in various formats (e.g., RGB, CYMK, YCbCr, etc.).

[0089] In the operation of an exemplary visible light camera module 206, light energy received from the target scene can pass through the visible light lens assembly 208 and be focused onto the visible light sensor 210. When the light energy strikes the visible light sensor element of the visible light sensor 210, photons within the photodetector can be released and converted into a detection current. The processor 212 can process this detection current to form a visible light image of the target scene.

[0090] During use of the acoustic analysis system 200, the processor 212 may control the visible light camera module 206 to generate visible light data from the captured target scene for creating a visible light image. The visible light data may include photometric data indicating one or more colors associated with different portions of the captured target scene and / or magnitudes of light associated with those different portions. The processor 212 may generate “frames” of visible light image data by measuring the response of each visible light sensor element of the acoustic analysis system 200 at a single time. By generating frames of visible light data, the processor 212 captures a visible light image of the target scene at a given point in time. The processor 212 may also repeatedly measure the response of each visible light sensor element of the acoustic analysis system 200 to generate a dynamic visible light image (e.g., a video representation) of the target scene. In some examples, the visible light camera module 206 may include its own dedicated processor or other circuitry (e.g., an ASIC) capable of operating the visible light camera module 206. In some such embodiments, the dedicated processor communicates with the processor 212 to provide visible light image data (e.g., RGB image data) to the processor 212. In an alternative embodiment, a dedicated processor for the visible light camera module 206 may be integrated into the processor 212.

[0091] With each sensor element of the visible light camera module 206 acting as a sensor pixel, the processor 212 can generate a two-dimensional image or picture representation of the target scene in visible light by converting the electrical response of each sensor element into a time-multiplexed electrical signal. This time-multiplexed electrical signal can be processed, for example, for visualization on the display 214 and / or for storage in memory.

[0092] Processor 212 can control display 214 to display at least a portion of the captured visible light image of the target scene. In some examples, processor 212 controls display 214 such that the electrical response of each sensor element of visible light camera module 206 is associated with a single pixel on display 214. In other examples, processor 212 can increase or decrease the resolution of the visible light image such that more or fewer pixels are displayed on display 214 than are present in the sensor elements of visible light camera module 206. Processor 212 can control display 214 to display the entire visible light image (e.g., all of the target scene captured by acoustic analysis system 200) or less than the entire visible light image (e.g., a smaller portion of the entire target scene captured by acoustic analysis system 200).

[0093] In some embodiments, processor 212 may control display 214 to simultaneously display at least a portion of a visible light image captured by acoustic analysis system 200 and at least a portion of an acoustic image generated via acoustic sensor array 202. Such concurrent display can be useful because an operator can refer to features displayed in the visible light image to help identify the source of the acoustic signal simultaneously displayed in the acoustic image. In various examples, processor 212 may control display 214 to display the visible light image and acoustic image in a side-by-side arrangement, a figure-by-figure arrangement (where one image surrounds another image), or any other suitable arrangement in which the visible light image and acoustic image are simultaneously displayed.

[0094] For example, processor 212 can control display 214 to display a visible light image and an acoustic image in a combined arrangement. In such an arrangement, for a pixel or set of pixels representing a portion of a target scene in the visible light image, there exists a corresponding pixel or set of pixels representing substantially the same portion of the target scene in the acoustic image. In various embodiments, the size and / or resolution of the acoustic image and the visible light image do not need to be the same. Accordingly, one of the acoustic or visible light images may contain a set of pixels that corresponds to a single pixel in the other, or a set of pixels of a different size. Similarly, one of the visible light or acoustic images may contain an image that corresponds to a set of pixels in the other image. Thus, as used herein, correspondence does not require a one-to-one pixel relationship, but may include mismatched sizes of pixels or groups of pixels. Various combining techniques for mismatched-size image regions can be implemented, such as upsampling or downsampling one of the images, or combining a pixel with the average of a corresponding set of pixels. Other examples are known and are also within the scope of this disclosure.

[0095] Therefore, corresponding pixels do not need to have a direct one-to-one relationship. Instead, in some embodiments, a single acoustic pixel has multiple corresponding visible light pixels, or a visible light pixel has multiple corresponding acoustic pixels. Additionally or alternatively, in some embodiments, not all visible light pixels have corresponding acoustic pixels, or vice versa. Such embodiments can indicate, for example, a figure-and-image type display as previously discussed. Therefore, a visible light pixel will not necessarily have the same pixel coordinates within the visible light image as its corresponding acoustic pixel. Accordingly, as used herein, a corresponding pixel generally refers to a pixel from any image (e.g., a visible light image, an acoustic image, a combined image, a display image, etc.) that includes substantially the same portion of information from the target scene. Such pixels do not need to have a one-to-one relationship between images and do not need to have similar coordinate orientations within their respective images.

[0096] Similarly, images with corresponding pixels (i.e., pixels representing the same portion of the target scene) can be referred to as corresponding images. Therefore, in some such arrangements, corresponding visible light images and acoustic images can be superimposed on each other at corresponding pixels. An operator can interact with user interface 216 to control the transparency or opacity of one or both images displayed on display 214. For example, an operator can interact with user interface 216 to adjust the acoustic image between completely transparent and completely opaque, and also adjust the visible light image between completely transparent and completely opaque. Such exemplary combined arrangements (which may be referred to as alpha-mixing arrangements) can allow an operator to adjust display 214 to display an acoustic-only image, a visible light-only image, or any overlapping combination of the two images between the extremes of acoustic-only and visible light-only images. Processor 212 can also combine scene information with other data, such as alarm data, etc. Generally, alpha-mixing combinations of visible light and acoustic images can include any combination from 100% acoustic and 0% visible light to 0% acoustic and 100% visible light. In some embodiments, the amount of mixing can be adjusted by the user of the camera. Therefore, in some embodiments, the mixed image can be adjusted between 100% visible light and 100% acoustic.

[0097] Additionally, in some embodiments, processor 212 may interpret and execute commands from user interface 216 and / or output / control device 218. Furthermore, input signals may be used to modify the processing of visible light and / or acoustic image data occurring within processor 212.

[0098] An operator can interact with the acoustic analysis system 200 via a user interface 208, which may include buttons, keypads, or another mechanism for receiving input from the user. The operator can receive output from the acoustic analysis system 200 via a display 214. The display 214 can be configured to display acoustic and / or visible light images in any acceptable color palette or scheme, and this palette may change, for example, in response to user control. In some embodiments, acoustic image data may be presented in a color palette to represent the magnitude of change in acoustic data from different locations in a scene. For example, in some examples, the display 214 is configured to display acoustic images in a monochrome color palette (such as grayscale). In other examples, the display 214 is configured to display acoustic images in a color palette such as, for example, amber, ironbow, blue-red, or other high-contrast color schemes. Combinations of grayscale and color palette displays are also contemplated. In some examples, a display configured to display such information may include processing capabilities for generating and presenting such image data. In other examples, being configured to display such information may include the ability to receive image data from other components, such as processor 212. For example, processor 212 may generate values ​​(e.g., RGB values, grayscale values, or other display options) for each pixel to be displayed. Display 214 may receive such information and map each pixel onto the visual display.

[0099] Although the processor 212 can control the display 214 to simultaneously display at least a portion of the acoustic image and at least a portion of the visible light image in any suitable arrangement, the in-figure arrangement can help the operator easily focus and / or interpret the acoustic image by displaying corresponding visible images of the same scene in an adjacent alignment.

[0100] A power source (not shown) delivers operating power to the various components of the acoustic analysis system 200. In various examples, the power source may include a rechargeable or non-rechargeable battery and power generation circuitry, an AC power source, an inductive power pick-up, a photovoltaic power source, or any other suitable power supply component. Combinations of power supply components are also possible, such as a rechargeable battery and another component configured to provide power to operate the equipment and / or charge the rechargeable battery.

[0101] During the operation of the acoustic analysis system 200, the processor 212, with the aid of instructions associated with program information stored in memory, controls the acoustic sensor array 202 and the visible light camera module 206 to generate visible light and acoustic images of the target scene. The processor 212 further controls the display 214 to display the visible light and / or acoustic images generated by the acoustic analysis system 200.

[0102] As mentioned above, in some cases, it can be difficult to identify and distinguish real-world (visible) features of a target scene in an acoustic image. In some embodiments, it may be useful to highlight visible edges within the target scene, in addition to supplementing the acoustic image with visible light information. In some embodiments, known edge detection methods can be implemented on the visible light image of the target scene. Due to the correspondence between the acoustic image and the visible light image, a visible light pixel identified as representing a visible edge in the target scene corresponds to an acoustic pixel that also represents a visible edge in the acoustic image. It will be understood that, as used herein, "edge" does not necessarily refer to the physical boundary of an object, but can refer to any sufficiently sharp gradient in the visible light image. Examples may include the physical boundary of an object, color variations within an object, shadows across the scene, and so on.

[0103] Although usually referred to Figure 2 Described as including a visible light camera module 206, but in some examples, the electromagnetic imaging tool 203 of the acoustic analysis system 200 may additionally or alternatively include an imaging tool capable of generating image data representing a wide variety of spectra. For example, in various examples, the electromagnetic imaging tool 203 may include one or more tools capable of generating infrared image data, visible light image data, ultraviolet image data, or any other useful wavelength or combination thereof. In some embodiments, the acoustic imaging system may include an infrared camera module having an infrared lens assembly and an infrared sensor array. Additional components for interfacing with, for example, the infrared camera module may be included, such as those described in U.S. Patent Application No. 14 / 837,757, filed August 27, 2015, entitled “EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES AND CAMERAS,” which is assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0104] In some examples, two or more data streams can be mixed for display. Examples include a visible light camera module 206, an acoustic sensor array 202, and an infrared camera module (…). Figure 2An exemplary system (not shown) can be configured to produce an output image comprising a mixture of visible light (VL) image data, infrared (IR) image data, and acoustic image data. In an exemplary mixing scheme, the displayed image can be represented as: α×IR+β×VL+γ×Acoustic, where α+β+γ=1. Generally, any number of data streams can be combined for display. In various embodiments, the mixing ratios, such as α, β, and γ, can be set by the user. Additionally or alternatively, for example, as described in U.S. Patent No. 7,538,326 entitled “VISIBLELIGHT ANDIR COMBINED IMAGE CAMERA WITH A LASER POINTER,” the set display program can be configured to include different image data streams based on alarm conditions (e.g., one or more values ​​in one or more data streams satisfy a predetermined threshold) or other conditions, which is assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0105] Compared to Figure 2 One or more components of the described acoustic analysis system 200 may be included in a portable (e.g., handheld) acoustic analysis tool. For example, in some embodiments, the portable acoustic analysis tool may include a housing 230 configured to house the components within the acoustic analysis tool. In some examples, one or more components of system 200 may be located outside the housing 230 of the acoustic analysis tool. For example, in some embodiments, processor 212, display 214, user interface 216, and / or output control device 218 may be located outside the housing of the acoustic analysis tool and may communicate with various other system components, for example, via wireless communication (e.g., Bluetooth communication, Wi-Fi, etc.). Such components outside the acoustic analysis tool may be provided, for example, via external devices such as computers, smartphones, tablet devices, wearable devices, etc. Additionally or alternatively, other test and measurement or data acquisition tools configured to act as master or slave devices relative to the acoustic analysis tool may similarly provide various components of the acoustic analysis system outside the acoustic analysis tool. External devices can communicate with portable acoustic analysis tools via wired and / or wireless connections and can be used to perform various processing, display, and / or interface steps.

[0106] In some embodiments, such an external device can provide redundant functionality as a component housed within a portable acoustic analysis tool. For example, in some embodiments, the acoustic analysis tool may include a display for showing acoustic image data and may be further configured to transmit image data to an external device for storage and / or display. Similarly, in some embodiments, a user may interface with the acoustic analysis tool via an application (“app”) running on a smartphone, tablet, computer, etc., to perform one or more functions that can also be performed with the acoustic analysis tool itself.

[0107] Figure 3A This is a schematic diagram of an exemplary configuration of an acoustic sensor array within an acoustic analysis system. In the illustrated example, the acoustic sensor array 302 includes a plurality of first acoustic sensors (shown in white) and a plurality of second acoustic sensors (shaded). The first acoustic sensors are arranged in a first array 320, and the second acoustic sensors are arranged in a second array 322. In some examples, the first array 320 and the second array 322 can be selectively used to receive acoustic signals used to generate acoustic image data. For example, in some configurations, the sensitivity of a particular acoustic sensor array to a particular acoustic frequency is a function of the distance between the acoustic sensor elements.

[0108] In some configurations, sensor elements that are more closely spaced together (e.g., a second array 322) are better able to resolve high-frequency acoustic signals (e.g., sounds with frequencies greater than 20 kHz, such as ultrasonic signals between 20 kHz and 100 kHz) than sensor elements that are more widely spaced (e.g., a first array 320). Similarly, sensor elements that are more widely spaced (e.g., a first array 320) may be better suited for detecting low-frequency acoustic signals (e.g., <20 kHz) than sensor elements that are more closely spaced (e.g., a second array 322). Various acoustic sensor arrays with sensor elements spaced apart from each other can be provided for detecting acoustic signals with a variety of frequency ranges, such as infrasound frequencies (<20 Hz), audible frequencies (approximately between 20 Hz and 20 kHz), and ultrasonic frequencies (between 20 kHz and 100 kHz). In some embodiments, a portion of the array (e.g., each of the other acoustic sensor elements from array 320) can be used to optimize detection of a specific frequency band.

[0109] Additionally, in some examples, some acoustic sensor elements may be better suited to detecting acoustic signals with different frequency characteristics (such as low-frequency or high-frequency). Therefore, in some embodiments, an array configured to detect low-frequency acoustic signals (such as a first array 320 with more widely spaced sensor elements) may include first acoustic sensor elements better suited to detecting low-frequency acoustic signals. Similarly, an array configured to detect higher-frequency acoustic signals (such as a second array 322) may include second acoustic sensor elements better suited to detecting high-frequency acoustic signals. Therefore, in some examples, the first array 320 and the second array 322 of acoustic sensor elements may include different types of acoustic sensor elements. Alternatively, in some embodiments, the first array 320 and the second array 322 may include the same type of acoustic sensor elements.

[0110] Therefore, in an exemplary embodiment, the acoustic sensor array 302 may include multiple arrays of acoustic sensor elements, such as a first array 320 and a second array 322. In some embodiments, the arrays may be used individually or in combination. For example, in some examples, a user may choose to use the first array 320, the second array 322, or both the first array 320 and the second array 322 simultaneously to perform an acoustic imaging process. In some examples, a user may select which array(s)(s) to use via a user interface. Additionally or alternatively, in some embodiments, the acoustic analysis system may automatically select which array(s)(s) to use based on the analysis of received acoustic signals or other input data (such as expected frequency ranges, etc.). Although Figure 3A The configuration shown typically includes two arrays (first array 320 and second array 322) generally arranged in a rectangular grid, but it will be appreciated that multiple acoustic sensor elements can be grouped into any number of discrete arrays of any shape. Furthermore, in some embodiments, one or more acoustic sensor elements can be included in multiple distinct arrays that can be selected for operation. As described elsewhere herein, in various embodiments, the process for backpropagating acoustic signals to establish acoustic image data from a scene is implemented based on the arrangement of the acoustic sensor elements. Therefore, the arrangement of the acoustic sensors can be known or otherwise accessible to a processor in order to implement acoustic image generation techniques.

[0111] Figure 3A The acoustic analysis system further includes: a distance measuring tool 304 and a camera module 306 positioned within an acoustic sensor array 302. The camera module 306 may represent the camera module of an electromagnetic imaging tool (e.g., 203) and may include a visible light camera module, an infrared camera module, an ultraviolet camera module, etc. Additionally, although in Figure 3ANot shown, but the acoustic analysis system may include one or more additional camera modules of the same or different type as camera module 306. In the illustrated example, distance measuring tool 304 and camera module 306 are positioned within a grid of acoustic sensor elements in the first array 320 and the second array 322. Although shown as being positioned between grid sites within the first array 320 and the second array 322, in some embodiments, one or more components (e.g., camera module 306 and / or distance measuring tool 304) may be positioned at corresponding grid sites in the first array 320 and / or the second array 322. In some such embodiments, components (one or more) may be positioned at grid sites instead of acoustic sensor elements that would typically be located in such positions according to the grid arrangement.

[0112] As described elsewhere in this document, an acoustic sensor array may include acoustic sensor elements arranged in any of a wide variety of configurations. Figure 3B and Figure 3C This is a schematic diagram illustrating an exemplary acoustic sensor array configuration. Figure 3B An acoustic sensor array 390 is shown, comprising a plurality of acoustic sensor elements evenly spaced in a nearly square grid. A distance measuring tool 314 and a camera array 316 are positioned within the acoustic sensor array 390. In the illustrated example, the acoustic sensor elements in the acoustic sensor array 390 are of the same type, although in some embodiments, different types of acoustic sensor elements may be used in the array 390.

[0113] Figure 3C Multiple acoustic sensor arrays are shown. Acoustic sensor arrays 392, 394, and 396 each include multiple acoustic sensor elements arranged in arrays of different shapes. Figure 3C In the examples, acoustic sensor arrays 392, 394, and 396 can be used individually or in any combination to create sensor arrays of various sizes. In the illustrated embodiment, the sensor elements of array 396 are spaced more closely together than the sensor elements of array 392. In some examples, array 396 is designed for sensing high-frequency acoustic data, while array 392 is designed for sensing low-frequency acoustic data.

[0114] In various embodiments, arrays 392, 394, and 396 may include the same or different types of acoustic sensor elements. For example, acoustic sensor array 392 may include sensor elements whose frequency operating range is lower than that of the sensor elements in acoustic sensor array 396.

[0115] As described elsewhere herein, in some examples, different acoustic sensor arrays (e.g., 392, 394, 396) can be selectively turned off and on during various operating modes (e.g., different desired spectra to be imaged). Additionally or alternatively, various acoustic sensor elements (e.g., some or all of the acoustic sensor elements in one or more sensor arrays) can be enabled or disabled depending on the desired system operation. For example, in some acoustic imaging processes, while data from a large number of sensor elements (e.g., sensor elements arranged in a high density, such as in sensor array 396) slightly improves the resolution of the acoustic image data, this comes at the cost of the processing required to extract the acoustic image data from the data received at each sensor element. That is, in some examples, the increased processing requirements (e.g., cost, processing time, power consumption, etc.) required to process a large number of input signals (e.g., signals from a large number of acoustic sensor elements) are negative compared to any additional signal resolution provided by the additional data stream. Therefore, in some embodiments, it may be worthwhile to disable or ignore data from one or more acoustic sensor elements depending on the desired acoustic imaging operation.

[0116] and Figure 3A and Figure 3B The system is similar to that of the system. Figure 3C The system includes a distance measuring tool 314 and a camera array 316 positioned within acoustic sensor arrays 392, 394, and 396. In some examples, additional components, such as an additional camera array (e.g., used to image different portions of the electromagnetic spectrum from camera array 316), can be similarly positioned within acoustic sensor arrays 392, 394, and 396. It will be understood that, although in Figures 3A-3C The device is shown positioned within one or more acoustic sensor arrays, but the distance measuring tool and / or one or more imaging tools (e.g., a visible light camera module, an infrared camera module, an ultraviolet sensor, etc.) may be located outside the acoustic sensor array(s). In some such examples, the distance measuring tool and / or one or more imaging tools located outside the acoustic sensor array(s) may be supported by an acoustic imaging tool, for example, by a housing that houses the acoustic sensor array(s), or may be located outside the housing of the acoustic imaging tool.

[0117] In some examples, general misalignment of acoustic sensor arrays and imaging tools (such as camera modules) can lead to misalignment in the corresponding image data generated by the acoustic sensor arrays and imaging tools. Figure 4AA schematic diagram of parallax error in the generation of frames of visible light image data and acoustic image data is shown. Generally, parallax error can be vertical, horizontal, or both. In the illustrated embodiment, the acoustic sensor array 420 and the imaging tool include a visible light camera module 406. Visible light image frame 440 is shown as being captured based on the field of view 441 of the visible light camera module 406, while acoustic image frame 450 is shown as being captured based on the field of view 451 of the acoustic sensor array 420.

[0118] As shown, the visible light image frame 440 and the acoustic imaging frame 450 are not aligned with each other. In some embodiments, the processor (e.g., Figure 2 The processor 212 is configured to manipulate one or both of the visible light image frame 440 and the acoustic image frame 450 to align the visible light image data and the acoustic image data. Such manipulation may include shifting one image frame relative to another. The amount of shifting of the image frames relative to each other can be determined based on a variety of factors, including, for example, the distance from the visible light camera module 406 and / or the acoustic sensor array 420 to the target. Such distance data can be determined, for example, using a distance measurement tool 404 or by receiving distance values ​​via a user interface (e.g., 216).

[0119] Figure 4B It is similar to Figure 4A A schematic diagram, but including a visible light image of the scene. Figure 4B In the example, visible light image 442 shows a scene with multiple power lines and supporting towers. Acoustic image 452 includes multiple locations 454, 456, 458, which indicate high-value acoustic data from such locations. As shown, both visible light image 442 and acoustic image 452 are displayed simultaneously. However, observation of both images reveals at least one local maximum in the acoustic image at location 458, which appears inconsistent with any particular structure in visible light image 442. Therefore, an observer of both images could conclude that there is a misalignment (e.g., parallax error) between acoustic image 452 and visible light image 442.

[0120] Figure 5A and Figure 5B This illustrates parallax correction between visible light and acoustic images. Similar to... Figure 4B , Figure 5A Visible light image 542 and acoustic image 552 are shown. Acoustic image 552 includes local maxima at positions 554, 556, and 558. As can be seen, the maxima at positions 554 and 558 appear inconsistent with any structure in the visible light image. Figure 5BIn the example, the visible light image 542 and the acoustic image 552 are registered relative to each other. Now, the local maxima at positions 554, 556, and 558 in the acoustic image appear to coincide with the respective positions in the visible light image 542.

[0121] During use, the operator can (e.g., via display 214) view Figure 5B The acoustic image data is represented in the image and the approximate locations of possible sources of the received acoustic signals within the visible scene 542 are determined. Such signals can be further processed to determine information about the acoustic signature of various components within the scene. In various embodiments, acoustic parameters, such as frequency content, periodicity, amplitude, etc., can be analyzed relative to various locations in the acoustic image. When overlaid on visible light data so that such parameters can be associated with various system components, the acoustic image data can be used to analyze various properties (e.g., performance characteristics) of objects in the visible light image.

[0122] Figure 5C and Figure 5D yes Figure 5A and Figure 5B The color version. For example... Figure 5A and Figure 5B As shown, and in Figure 5C and Figure 5D More readily apparent in the color representation, positions 554, 556, and 558 exhibit a circular gradient in color. As described elsewhere in this document, acoustic image data can be visually represented according to a color scheme in which each pixel of the acoustic image data is colored based on the sound intensity at its corresponding location. Therefore, in Figures 5A-5D In the exemplary representation, the circular gradients at positions 554, 556, and 558 typically represent the gradients in the acoustic intensity in the imaging plane of the received acoustic signal based on backpropagation.

[0123] What will be understood is that, although described relative to acoustic image data and visible light image data... Figure 4A , 4B The exemplary illustrations in 5A-5D are provided, but such a process can be similarly performed using a wide variety of electromagnetic image data. For example, as described elsewhere herein, in various embodiments, various such processes can be performed using combinations of acoustic image data with one or more of visible light image data, infrared image data, ultraviolet image data, etc.

[0124] As described elsewhere in this document, in some embodiments, the backpropagation of the acoustic signal to form an acoustic image can be based on a value of distance to the target. That is, in some examples, the backpropagation calculation can be based on distance and may include determining a two-dimensional acoustic scene located at that distance from the acoustic sensor array. Given a two-dimensional imaging plane, the cross-section of a spherical sound wave emitted from a source in the plane typically exhibits a circular shape, where the intensity attenuates radially, such as... Figures 5A-5B As shown in the image.

[0125] In some such examples, portions of the acoustic scene representing data on distances to the target not used in backpropagation calculations will cause errors in the acoustic image data, such as inaccurate positioning of one or more sounds in the scene. Such errors can lead to parallax errors between the acoustic image data and other image data when the acoustic image is displayed simultaneously (e.g., mixed, combined, etc.) with other image data (e.g., electromagnetic image data, such as visible light, infrared, or ultraviolet image data). Therefore, in some embodiments, techniques for correcting parallax errors (e.g., such as...) Figure 5A and Figure 5B (As shown) This includes: adjusting the value of the distance to the target used in the backpropagation calculation for generating acoustic image data.

[0126] In some cases, the system can be configured to use the first distance to the target to perform the backpropagation process and display, for example, the following: Figure 5A The displayed image may show misalignment between acoustic image data and another data stream. The acoustic analysis system can then adjust the distance-to-target value used for backpropagation, perform backpropagation again, and update the displayed image with the new acoustic image data. This process can be repeated, where the acoustic analysis system cycles through multiple distance-to-target values ​​while the user observes the resulting displayed image on the monitor. As the distance-to-target value changes, the user can observe the changes from... Figure 5A The image shown is displayed to Figure 5BThe gradual transition of the displayed image is shown. In some such cases, the user can visually observe when the acoustic image data appears to be correctly registered with another data stream, such as electromagnetic image data. The user can send a signal to the acoustic analysis system indicating that the acoustic image data appears to be correctly registered, thereby instructing the system that the value of the distance to the target used to perform the most recent backpropagation is approximately correct, and that this distance value can be saved to memory as the correct distance to the target. Similarly, when updating the displayed image using a new distance value during an updated backpropagation, the user can manually adjust the value of the distance to the target until the user observes that the acoustic image data is correctly registered. The user can choose to save the current distance to the target as the current distance to the target in the acoustic analysis system.

[0127] In some examples, correcting for parallax error may include adjusting the orientation of acoustic image data relative to other image data (e.g., electromagnetic image data) by a predetermined amount and in a predetermined direction based on data about the distance to the target. In some embodiments, such adjustment is performed independently of the generation of the acoustic image data by backpropagating the acoustic signal to the identified distance to the target.

[0128] In some embodiments, the distance value to the target can be used for other determinations besides generating acoustic image data and reducing parallax error between acoustic image data and other image data. For example, in some examples, as described in U.S. Patent No. 7,538,326, a processor (e.g., 212) can use the distance value to the target to focus or assist a user in focusing an image such as an infrared image, which is incorporated herein by reference. As described therein, this can be similarly used to correct parallax error between visible light image data and infrared image data. Thus, in some examples, the distance value can be used to register acoustic image data with electromagnetic imaging data, such as infrared image data and visible light image data.

[0129] As described elsewhere herein, in some examples, a distance measurement tool (e.g., 204) is configured to provide distance information that can be used by a processor (e.g., 212) to generate and register acoustic image data. In some embodiments, the distance measurement tool includes a laser rangefinder configured to emit light onto a target scene at the location where the distance is being measured. In some such examples, the laser rangefinder may emit light in the visible spectrum, allowing a user to view a laser point in the physical scene to ensure that the rangefinder is measuring the distance to the desired portion of the scene. Additionally or alternatively, the laser rangefinder is configured to emit light in the spectrum to which one or more imaging components (e.g., a camera module) are sensitive. Thus, a user viewing the target scene via an analysis tool (e.g., via display 214) can observe a laser point in the scene to ensure that the laser is measuring the distance to the correct location in the target scene. In some examples, the processor (e.g., 212) may be configured to generate, in the displayed image, a reference marker representing the location where the laser point will be in the acoustic scene, based on the current distance value (e.g., based on a known distance-based parallax relationship between the laser rangefinder and the acoustic sensor array). The position of the reference marker can be compared with the position of the actual laser marker (e.g., graphically on a display and / or physically in the target scene), and the scene can be adjusted until the reference marker and the laser are aligned. Such a process can be performed using infrared registration and focusing techniques similar to those described in U.S. Patent No. 7,538,326, which is incorporated herein by reference.

[0130] Figure 6 This is a flowchart illustrating an exemplary method for generating a final image that combines acoustic image data and electromagnetic image data. The method includes the steps of: receiving an acoustic signal (680) via an acoustic sensor array; and receiving distance information (682). The distance information may be received, for example, via a distance measuring device and / or a user interface, such as via manual input, or as a result of a distance adjustment process in which the distance is determined based on observed registration.

[0131] The method further includes: backpropagating the received acoustic signals to determine acoustic image data representing the acoustic scene (684). As described elsewhere herein, backpropagation may include combining the received distance information to analyze multiple acoustic signals received at multiple sensor elements in an acoustic sensor array to determine the source mode of the received acoustic signals.

[0132] Figure 6The method further includes the steps of: capturing electromagnetic image data (686), and registering acoustic image data with electromagnetic image data (688). In some embodiments, registering acoustic image data with electromagnetic image data is performed as part of a backpropagation step for generating acoustic image data (684). In other examples, registering acoustic image data with electromagnetic image data is performed separately from the generation of acoustic image data.

[0133] Figure 6 The method includes the following steps: combining acoustic image data with electromagnetic image data to generate a display image (690). As described elsewhere herein, combining the electromagnetic and acoustic image data may include: alpha mixing the electromagnetic and acoustic image data. Combining the image data may include, for example, overlaying one image dataset onto another image dataset in a picture-in-picture mode or at a location where certain conditions are met (e.g., alarm conditions). The display image may be presented to the user, for example, via a display supported by a housing (which supports the acoustic sensor array), and / or via a display separate from the sensor array (such as the display of an external device, e.g., a smartphone, tablet, computer, etc.).

[0134] Additionally or alternatively, the display image may be stored in local (e.g., onboard) memory and / or remote memory for future viewing. In some embodiments, the stored display image may include metadata that allows future adjustment of display image properties, such as blending ratio, backpropagation distance, or other parameters used to generate the image. In some examples, raw acoustic signal data and / or electromagnetic image data may be stored along with the display image for subsequent processing or analysis.

[0135] Although shown as a method for generating a final image that combines acoustic and electromagnetic image data, it will be understood that... Figure 6 This method can be used to combine acoustic image data with one or more sets of image data spanning any part of the electromagnetic spectrum, such as visible light image data, infrared image data, ultraviolet image data, and so on. In some such examples, it can be achieved via... Figure 6 The described method is similar to other methods that combine multiple sets of image data, such as visible light image data and infrared image data, with acoustic image data to generate a display image.

[0136] In some examples, receiving acoustic signals via a sensor array (680) may include the following steps: selecting an acoustic sensor array for receiving acoustic signals. For example, regarding... Figure 3AAs described in -C, an acoustic analysis system may include multiple acoustic sensor arrays adapted to analyze acoustic signals with varying frequencies. Additionally or alternatively, in some examples, different acoustic sensor arrays may be used to analyze acoustic signals propagating from different distances. In some embodiments, different arrays may be nested within each other. Additionally or alternatively, portions of the array may be selectively used to receive acoustic image signals.

[0137] For example, Figure 3A A first array 320 and a second array 322 nested within the first array are shown. In an exemplary embodiment, the first array 320 may include a sensor array configured (e.g., spaced apart) to receive acoustic signals and generate acoustic image data for frequencies within a first frequency range. The second array 322 may include, for example, a second sensor array configured to be used alone or in combination with all or part of the first array 320 to generate acoustic image data for frequencies within a second frequency range.

[0138] Similarly, Figure 3C A first array 392, a second array 394 at least partially nested within the first array 392, and a third array 396 at least partially nested within the first array 392 and the second array 394 are illustrated. In some embodiments, the first array 392 may be configured to receive acoustic signals and generate acoustic image data for frequencies within a first frequency range. The second array 394 may be used in conjunction with all or part of the first array 392 to receive acoustic signals and generate acoustic image data for frequencies within a second frequency range. The third array 396 may be used alone, in conjunction with all or part of the second array 394, and / or in conjunction with all or part of the first array 392 to receive acoustic signals and generate acoustic image data for frequencies within a third frequency range.

[0139] In some embodiments, in a nested array configuration, acoustic sensor elements from one array can be positioned between acoustic sensor elements, such as elements in a third array 396 typically positioned between elements in a first array 392. In some such examples, acoustic sensor elements in a nested array (e.g., a third array 396) can be positioned in the same plane as, in front of, or behind, acoustic sensor elements in the array in which they are nested (e.g., a first array 392).

[0140] In various implementations, arrays used to sense higher-frequency acoustic signals typically require smaller distances between the sensors. Therefore, regarding Figure 3CFor example, the third array 396 may be better suited for performing acoustic imaging processes involving high-frequency acoustic signals. Other sensor arrays (e.g., the first array 392) may be sufficient for performing acoustic imaging processes involving lower-frequency signals, and other sensor arrays can be used compared to array 396 to reduce the computational requirements for processing signals from a smaller number of acoustic sensor elements. Thus, in some examples, a high-frequency sensor array may be nested within a low-frequency sensor array. As described elsewhere herein, such arrays can typically operate individually (e.g., via switching between active arrays) or together.

[0141] In addition to selecting an appropriate sensor array based on the expected / desired spectrum used for analysis, or as an alternative, in some examples, different sensor arrays may be more suitable for performing acoustic imaging processes at different distances from the target scene. For example, in some embodiments, if the distance between the acoustic sensor array and the target scene is small, external sensor elements in the acoustic sensor array may receive significantly less useful acoustic information from the target scene compared to sensor elements located more centrally.

[0142] On the other hand, if the distance between the acoustic sensor array and the target scene is large, closely spaced acoustic sensor elements may not individually provide useful information. That is, if the first and second acoustic sensor elements are close together and the target scene is typically far away, the second acoustic sensor element may not provide any information that is different in meaning from that of the first acoustic sensor element. Therefore, the data stream from such first and second sensor elements may be redundant and unnecessarily consume processing time and resources used for analysis.

[0143] As described elsewhere in this document, in addition to influencing which sensor arrays might be best suited for acoustic imaging, the distance to the target can also be used to perform backpropagation to determine acoustic image data based on the received acoustic signals. However, besides being an input to the backpropagation algorithm, the distance to the target can be used to select the appropriate backpropagation algorithm to use. For example, in some examples, at long distances, spherically propagating sound waves can be approximated as substantially planar compared to the size of the acoustic sensor array. Therefore, in some embodiments, when the distance to the target is large, the backpropagation of the received acoustic signals may include acoustic beamforming calculations. However, when closer to the source of the sound waves, the planar approximation may not be suitable. Therefore, different backpropagation algorithms, such as near-field acoustic holography, can be used.

[0144] As described, the distance to the target can be measured in a variety of ways during an acoustic imaging process, such as determining one or more active sensor arrays, determining a backpropagation algorithm, implementing the backpropagation algorithm, and / or registering the resulting acoustic image with electromagnetic image data (e.g., visible light, infrared, etc.). Figure 7 This is a flowchart illustrating an exemplary process for generating acoustic image data from a received acoustic signal.

[0145] Figure 7 The process includes receiving distance information, such as via a user interface, from a distance measuring device, or input distance information (780). The method further includes the step of selecting one or more acoustic sensor arrays for acoustic imaging based on the received distance information (782). As described, in various examples, the selected array(s) may include a single array, a combination of multiple arrays, or portions of one or more arrays.

[0146] Figure 7 The method further includes the step of selecting a processing scheme for performing acoustic imaging based on the received distance information (784). In some examples, selecting the processing scheme may include selecting a backpropagation algorithm for generating acoustic image data from the acoustic signal.

[0147] After selecting an acoustic sensor array (782) and a processing scheme (784) for performing acoustic imaging, the method includes the following steps: receiving an acoustic signal via the selected acoustic sensor array (786). The received acoustic signal is then backpropagated using distance and the selected processing scheme to determine acoustic image data (788).

[0148] In various embodiments, Figure 7 The steps can be performed by a user, an acoustic analysis system (e.g., via processor 212), or a combination thereof. For example, in some embodiments, the processor can be configured to receive distance information (780) via a distance measurement tool and / or user input. In some examples, for instance, if the distance to the object is known and / or difficult to analyze via a distance measurement tool (e.g., the object is small and / or the distance to the target is large), the user can input a value to override the measured distance for use as distance information. The processor can be further configured to automatically select an appropriate acoustic sensor array for performing acoustic imaging based on the received distance information, for example, using a lookup table or other database. In some embodiments, selecting the acoustic sensor array includes enabling and / or disabling one or more acoustic sensor elements to obtain a desired acoustic sensor array.

[0149] Similarly, in some examples, the processor can be configured to automatically select a processing scheme (e.g., a backpropagation algorithm) for performing acoustic imaging based on the received distance information. In some such examples, this may include selecting one from a plurality of known processing schemes stored in memory. Additionally or alternatively, selecting a processing scheme may be equivalent to adjusting parts of a single algorithm to obtain the desired processing scheme. For example, in some embodiments, a single backpropagation algorithm may include multiple terms and variables (e.g., based on distance information). In some such examples, selecting a processing scheme (784) may include defining one or more values ​​in a single algorithm, such as adjusting the coefficients of one or more terms (e.g., setting various coefficients to zero or one, etc.).

[0150] Therefore, in some embodiments, the acoustic imaging system can automate several steps of the acoustic imaging process by suggesting and / or automatically implementing selected acoustic sensor arrays and / or processing schemes (e.g., backpropagation algorithms) based on received distance data. This can accelerate, improve, and simplify the acoustic imaging process, thereby eliminating the need for an acoustic imaging expert to perform the process. Thus, in various examples, the acoustic imaging system can automatically implement parameters, notify the user that such parameters are about to be implemented, request permission from the user to implement such parameters, suggest such parameters for the user to manually input, and so on.

[0151] The automatic selection and / or suggestion of such parameters (e.g., processing scheme, sensor array) can be useful for optimizing the localization of acoustic image data relative to the analysis of other forms of image data, processing speed, and acoustic image data. For example, as described elsewhere in this document, accurate backpropagation determination (e.g., using appropriate algorithms and / or accurate distance metrics) can reduce parallax errors between acoustic image data and other image data (e.g., electromagnetic waves, such as visible light, infrared, etc.). Additionally, utilizing appropriate algorithms and / or sensor arrays, such as those that can be automatically selected or suggested by the acoustic analysis system, can optimize the accuracy of thermal image data, thereby allowing for the analysis of the received acoustic data.

[0152] As described, in some examples, the acoustic analysis system can be configured to automatically select the algorithm and / or sensor array used to perform the acoustic imaging process based on received distance information. In some such embodiments, the system includes, for example, a lookup table stored in memory for determining which of a plurality of backpropagation algorithms and acoustic sensor arrays to use to determine the acoustic image data. Figure 8 An exemplary lookup table is shown for determining the appropriate algorithm and sensor array to be used during the acoustic imaging process.

[0153] In the illustrated example, Figure 8The lookup table consists of N columns, each representing a different array: array 1, array 2, ..., array N. In various examples, each array comprises a unique set of acoustic sensor elements arranged in a grid. Different arrays may include sensor elements arranged in a lattice (e.g., ...). Figure 3C (Arrays 392 and 396 in the lookup table). Arrays within the lookup table may also include combinations of sensor elements from one or more such grids. Generally, in some embodiments, each of the arrays: array 1, array 2, ..., array N corresponds to a unique combination of acoustic sensor elements. Some of these combinations may include the entire set of sensor elements arranged in a particular grid, or may include a subset of sensor elements arranged in a particular grid. Any of the various combinations of acoustic sensor elements is a possible option for use as a sensor array in the lookup table.

[0154] Figure 8 The lookup table further comprises M rows, each representing a different algorithm: Algorithm 1, Algorithm 2, ..., Algorithm M. In some examples, the different algorithms may include different procedures for performing backpropagation analysis on the received audio signal. As described elsewhere in this document, in some examples, some different algorithms may be similar to each other while having different coefficients and / or terms for modifying the backpropagation results.

[0155] Figure 8 An exemplary lookup table includes M×N entries. In some embodiments, an acoustic analysis system utilizing such a lookup table is configured to analyze received distance information and classify the distance information into one of M×N bins, wherein each bin corresponds to Figure 8 The lookup table contains entries. In such an example, when the acoustic analysis system receives distance information, it can find the entry (i, j) corresponding to the bin containing the distance information in the lookup table and determine the appropriate algorithm and sensor array for use during the acoustic imaging process. For example, if the received distance information corresponds to the bin associated with entry (i, j), the acoustic analysis system can automatically utilize or suggest the use of algorithm i and array j for the acoustic imaging process.

[0156] In various such examples, distance information bins may correspond to a uniform distance range; for example, the first bin corresponds to a distance within one foot, the second bin to a distance between one and two feet, and so on. In other examples, the bins do not need to correspond to a uniform distance span. Additionally, in some embodiments, fewer than M×N bins may be used. For example, in some embodiments, there may be algorithms (e.g., Algorithm x) that are never used by a particular array (e.g., Array y). Therefore, in such examples, there will be no corresponding distance information bin in the M×N lookup table for the entry (x, y).

[0157] In some embodiments, statistical analysis of the filled distance bins can be used to identify the most common distances or distance ranges within the target scene. In some such embodiments, the distance bin with the highest number of corresponding locations (e.g., the location with the highest number of acoustic signals) can be used as... Figure 7 The distance information during the process. That is, in some embodiments, the acoustic sensor array and / or processing scheme to be used can be determined and / or recommended based on statistical analysis of the distance distribution of various objects in the target scene. This can increase the likelihood that the sensor array and / or processing scheme used for acoustic imaging of the scene is suitable for the maximum number of locations within the acoustic scene.

[0158] Additionally or alternatively, parameters other than distance information can be used to select appropriate sensor arrays and / or processing schemes for use in generating acoustic image data. As described elsewhere in this document, various sensor arrays can be configured to be sensitive to certain frequencies and / or frequency bands. In some examples, different backpropagation calculations can be similarly used depending on the different acoustic signal frequency content. Therefore, in some examples, one or more parameters can be used to determine the processing scheme and / or acoustic sensor array.

[0159] In some embodiments, an acoustic analysis system can be used to initially analyze various parameters of the received acoustic signal processing / analysis. (Refer to previous step) Figure 7 A method for generating acoustic image data may include the following steps: after receiving an acoustic signal (786), analyzing the frequency content of the received signal (790). In some such examples, if one or more acoustic sensor arrays and / or processing schemes have been selected (e.g., via steps 782 and / or 784, respectively), the method may include the following steps: for example, updating the selected arrays and / or processing schemes based on the analyzed frequency content (792).

[0160] After updating one or more sensor arrays and / or processing schemes, the method can perform various actions using the updated parameters. For example, if one or more selected sensor arrays (792) are updated based on the analyzed frequency content (790), a new acoustic signal (786) can be received from the (most recently) selected acoustic sensor array, and this acoustic signal can then be backpropagated to determine acoustic image data (788). Alternatively, if the processing scheme is updated at 792, the already captured acoustic signal can be backpropagated according to the updated processing scheme to determine the updated acoustic image data. If both the processing scheme and one or more sensor arrays (or the same array) are updated, the updated sensor array can be used to receive new acoustic signals, and the new acoustic signals can be backpropagated according to the updated processing scheme.

[0161] In some embodiments, the acoustic analysis system may receive frequency information (778) without analyzing the frequency content (790) of the received acoustic signal. For example, in some examples, the acoustic analysis system may receive information about a desired or anticipated frequency range for future acoustic analysis. In some such examples, the desired or anticipated frequency information may be used to select one or more sensor arrays and / or the processing scheme best suited to that frequency information. In some such examples, the steps of selecting one or more acoustic sensor arrays (782) and / or selecting one or more processing schemes (784) may be based on the received frequency information in addition to or as an alternative to the received distance information.

[0162] In some examples, the received acoustic signals (e.g., acoustic signals received via acoustic sensor elements) can be analyzed, for example, via a processor (e.g., 210) of an acoustic analysis system. Such analysis can be used to determine one or more properties of the acoustic signals, such as frequency, intensity, periodicity, apparent proximity (e.g., distance estimated based on the received acoustic signals), measured proximity, or any combination thereof. In some examples, acoustic image data can be filtered, for example, to show only acoustic image data representing acoustic signals with specific frequency content, periodicity, etc. In some examples, any number of such filters can be applied simultaneously.

[0163] As described elsewhere in this document, in some embodiments, a series of acoustic image data frames can be captured over time, similar to acoustic video data. Additionally or alternatively, even without the repeated generation of acoustic image data, in some examples, the acoustic signal is repeatedly sampled and analyzed. Therefore, with or without the generation of repeated acoustic image data (e.g., video), parameters of the acoustic data (such as frequency) can be monitored over time.

[0164] Figure 9A This is an exemplary plot of the frequency content of received image data over time in an acoustic scene. As shown, Figure 9A The acoustic scene represented by the plot typically includes four frequencies that remain constant over time, labeled as frequency 1, frequency 2, frequency 3, and frequency 4. Frequency data, such as the frequency content of the target scene, can be determined by processing the received sound signal, for example, using Fast Fourier Transform (FFT) or other known frequency analysis methods.

[0165] Figure 9B An exemplary scenario including multiple locations where acoustic signals are emitted is shown. In the illustrated image, acoustic image data is combined with visible light image data, and acoustic signals present at locations 910, 920, 930, and 940 are shown. In some embodiments, the acoustic analysis system is configured to display acoustic image data belonging to any detected frequency range. For example, in an exemplary embodiment, location 910 includes acoustic image data containing frequency 1, location 920 includes acoustic image data containing frequency 2, location 930 includes acoustic image data containing frequency 3, and location 940 includes acoustic image data containing frequency 4.

[0166] In some such examples, displaying acoustic image data representing a frequency range is an optional mode of operation. Similarly, in some embodiments, the acoustic analysis system is configured to display acoustic image data representing frequencies only within a predetermined frequency band. In some such examples, displaying acoustic image data representing a predetermined frequency range includes selecting one or more acoustic sensor arrays for receiving acoustic signals from which the acoustic image data is generated. Such arrays can be configured to receive a selective frequency range. Similarly, in some examples, one or more filters can be employed to limit the frequency content used to generate the acoustic image data. Additionally or alternatively, in some embodiments, acoustic image data including information representing a broad frequency range can only be analyzed and displayed on a screen if the acoustic image data meets predetermined conditions (e.g., falls within a predetermined frequency range).

[0167] Figure 9CThis diagram illustrates multiple combined acoustic and visible light image data at various predefined frequency ranges. The first image includes acoustic image data at a first location 910, which includes frequency content at frequency 1. The second image includes acoustic image data at a second location 920, which includes frequency content at frequency 2. The third image includes acoustic image data at a third location 930, which includes frequency content at frequency 3. The fourth image includes acoustic image data at a fourth location 940, which includes frequency content at frequency 4.

[0168] In an exemplary embodiment, the user can select various frequency ranges, such as a range including frequency 1, frequency 2, frequency 3, or frequency 4, for filtering acoustic image data representing frequency content outside the selected frequency range. Therefore, in such an example, since the user has selected the desired frequency range, any of the first, second, third, or fourth images can be displayed.

[0169] Additionally or alternatively, in some examples, the acoustic analysis system may cycle between multiple display images, each with different frequency content. For example, regarding Figure 9C In an exemplary embodiment, the acoustic analysis system may display first, second, third, and fourth images in sequence, such as... Figure 9C As shown by the arrow in the image.

[0170] In some examples, displaying an image may include text or other displays representing frequency content shown in the image, allowing the user to observe which locations in the image contain acoustic image data representing certain frequency content. For example, regarding Figure 9C Each image can be shown as a textual representation of the frequencies represented in the acoustic image data. Regarding Figure 9B An image showing multiple frequency ranges may include indications of the frequency content at each location, including acoustic image data. In some such examples, a user may select a location in the image, for example via a user interface, for which they wish to view the frequency content present in the acoustic scene. For example, a user may select a first location 910, and the acoustic analysis system may display the frequency content at the first location (e.g., frequency 1). Thus, in various examples, a user may use the acoustic analysis system to analyze the frequency content of the acoustic scene, such as by seeing where in the scene corresponds to a specific frequency content, and / or by seeing what frequency content is present at various locations.

[0171] During an exemplary acoustic imaging operation, filtering acoustic image data by frequency can help reduce image clutter, such as from background or other unimportant sounds. In an exemplary acoustic imaging process, the user may want to eliminate background noise, such as noise floor in an industrial environment. In some such cases, background noise may primarily consist of low-frequency noise. Therefore, the user can choose to display acoustic image data representing acoustic signals above a predetermined frequency (e.g., 10 kHz). In another example, the user may want to analyze specific objects that typically emit acoustic signals within a certain range, such as corona discharge from a transmission line (e.g., as...). Figure 5A -D5 (as shown). In such an example, the user can select a specific frequency range (e.g., between 11 kHz and 14 kHz for corona discharge) for acoustic imaging.

[0172] In some examples, acoustic analysis systems can be used to analyze and / or present information associated with the intensity of received acoustic signals. For example, in some embodiments, backpropagation of the received acoustic signals may include determining sound intensity values ​​at multiple locations within an acoustic scene. In some examples, similar to the frequencies described above, acoustic image data is included in the displayed image only if the intensity of the acoustic signal meets one or more predetermined requirements.

[0173] In various such embodiments, the displayed image may include acoustic image data representing acoustic signals above a predetermined threshold (e.g., 15 dB), below a predetermined threshold (e.g., 100 dB), or within a predetermined intensity range (e.g., between 15 dB and 40 dB). In some embodiments, the threshold may be based on statistical analysis of the acoustic scene, such as being above or below the standard deviation from the average sound intensity.

[0174] Similar to the description above regarding frequency information, in some embodiments, limiting the acoustic image data representation to acoustic signals that satisfy one or more intensity requirements may include filtering the received acoustic signals such that only received signals that satisfy predetermined conditions are used to generate the acoustic image data. In other examples, the acoustic image data is filtered to adjust which acoustic image data is displayed.

[0175] Additionally or alternatively, in some embodiments, the sound intensity at a location within the acoustic scene can be monitored over time (e.g., in conjunction with a video acoustic image representation or via background analysis, without necessarily updating the displayed image). In some such examples, predetermined requirements for displaying acoustic image data may include the amount or rate of change of sound intensity at a certain location in the image.

[0176] Figure 10A and 10BIt is an exemplary display image that includes the combined visible light image data and acoustic image data. Figure 10A A display image including acoustic image data is shown, illustrated at multiple locations 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. In some examples, intensity values ​​may be color-coded, for example, where a color is assigned to the sound intensity values ​​based on a predetermined color scheme. In an exemplary embodiment, intensity values ​​may be categorized according to intensity ranges (e.g., 10 dB–20 dB, 20 dB–30 dB, etc.). Each intensity range may be associated with a specific color according to the color scheme. The acoustic image data may include multiple pixels, wherein each pixel is colored with a color associated with the intensity range to which the intensity represented by the pixel of the acoustic image data falls. In addition to or as an alternative to color differentiation, different intensities may be distinguished according to other attributes, such as transparency (e.g., in image overlays where acoustic image data is superimposed on other image data), etc.

[0177] Additional parameters, such as the rate of change of sound intensity, can also be color-tuned. Similar to intensity, the rate of change of changing sound intensity can be color-tuned so that scene sections exhibiting different rates and / or amounts of sound intensity change are displayed in different colors.

[0178] In the illustrated example, the acoustic image data is color-coded according to an intensity palette, so that acoustic image data representing different acoustic signal intensities are displayed with different colors and / or shading. For example, the acoustic image data at positions 1010 and 1030 shows a trayed representation of the first intensity, positions 1040, 1060, and 1080 show a trayed representation of the second intensity, and positions 1020, 1050, 1070, and 1090 show a trayed representation of the third intensity. Figure 10A The exemplary representation shown illustrates a circular pattern with a color gradient extending outwards from the center at each location of the color representation of the acoustic image data. This is likely due to the attenuation of sound intensity as the signal propagates from the sound source.

[0179] exist Figure 10A In the example, acoustic image data is combined with visible light image data to generate a display image, which can be presented to the user, for example, via a display screen. The user can view... Figure 10A The system displays an image to show which locations in the visible scene are generating sound signals, and the intensity of these signals. Therefore, users can quickly and easily observe the locations where sound is being generated and compare the intensity of sound from various locations in the scene.

[0180] Similar to the frequency descriptions elsewhere in this document, in some embodiments, acoustic image data can only be presented if the corresponding acoustic signal meets a predetermined intensity condition. Figure 10B It shows something similar to Figure 10A An exemplary display image is shown, comprising visible light image data and an acoustic image representing an acoustic signal above a predetermined threshold. As illustrated, the display image includes acoustic image data. Figure 10A Of positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080 and 1090, only positions 1020, 1050, 1070 and 1090 include acoustic image data representing sound signals that meet predetermined conditions.

[0181] In the exemplary scenario, Figure 10A This can include all acoustic image data above the noise floor threshold at each of the locations 1010-990, and Figure 10B It shows the relationship with Figure 10A The same scene, but it only shows acoustic image data with an intensity greater than 40dB. This can help users identify the environment (e.g., in...). Figure 10A and Figure 10B Which sound sources in the target scene are contributing certain sounds (e.g., the loudest sound in the scene).

[0182] As described elsewhere herein, in addition to or as an alternative to direct comparison with an intensity threshold (e.g., 40 dB), in some such examples, predetermined requirements for displaying acoustic image data may include the amount or rate of change of sound intensity at a certain location in the image. In some such examples, acoustic image data may be presented only if the rate or amount of change of sound intensity at a given location meets predetermined conditions (e.g., greater than a threshold, less than a threshold, within a predetermined range, etc.). In some embodiments, the amount or rate of change of sound intensity may be color-corrected and displayed as intensity acoustic image data, or displayed in combination with intensity acoustic image data. For example, in an exemplary embodiment, when the rate of change is used to determine which locations include a threshold of acoustic image data, the acoustic image data may include a color-corrected measure of the rate of change of intensity for display.

[0183] In some examples, the user can manually set intensity requirements (e.g., minimum, maximum, range, rate of change, amount of change, etc.) for the acoustic image data to be displayed. As discussed elsewhere in this document, this can be achieved during acoustic image data generation (e.g., via filtering the received acoustic signal), and / or by not displaying generated acoustic image data representing acoustic signals that do not meet (one or more) the set requirements. In some such examples, filtering of the displayed image based on intensity values ​​can be performed after the acoustic image data and visible light image data have been captured and stored in memory. That is, the data stored in memory can be used to generate a displayed image that includes any number of filtering parameters, such as showing only acoustic image data that meets predefined intensity conditions.

[0184] In some examples, setting a lower intensity limit in an acoustic image (e.g., displaying only acoustic image data representing acoustic signals above a predetermined intensity) can eliminate the inclusion of unwanted background or ambient sounds and / or sound reflections from the acoustic image data. In other cases, setting an upper intensity limit in an acoustic image (e.g., displaying only acoustic image data representing acoustic signals below a predetermined intensity) can eliminate the inclusion of expected loud sounds in the acoustic image data, allowing observation of acoustic signals that would normally be masked by such loud sounds.

[0185] Several display functions are possible. For example, similar to "About" Figure 9C In some examples of the frequency analysis / display discussed, the acoustic analysis system can cycle through multiple display images, each showing acoustic image data that meets different intensity requirements. Similarly, in some examples, the user can scroll through a range of sound intensity to view the locations within the acoustic image data that have sound intensities within a given range.

[0186] Another parameter that can be used to analyze acoustic data is the periodicity of the acoustic signal. Figure 11A and 11B An exemplary plot of frequency comparison time for acoustic data in an acoustic scene is shown. Figure 11A As shown in the diagram, the acoustic data includes: a signal with a frequency of X having a first periodicity, a signal with a frequency of Y having a second periodicity, and a signal with a frequency of Z having a third periodicity. In the example shown, the acoustic signals with different frequencies may also include different periods in the acoustic signals.

[0187] In some such examples, acoustic signals can be filtered based on periodicity, in addition to or as an alternative to frequency content. For instance, in some examples, multiple acoustic signal sources in an acoustic scene can generate acoustic signals at specific frequencies. If a user wishes to isolate such a sound source for acoustic imaging, the user can choose to include or exclude acoustic image data from the final displayed image based on the periodicity associated with the acoustic data.

[0188] Figure 11B A plot of the frequency of an acoustic signal against time is shown. As shown, the frequency increases approximately linearly over time. However, as also shown, the signal exhibits an approximately constant periodicity over time. Therefore, depending on the selected display parameters, such a signal may or may not appear in the acoustic image. For example, the signal may meet the displayed frequency criteria at some points in time, but be outside the displayed frequency range at other points in time. However, the user can choose to include or exclude such a signal from the acoustic image data based on its periodicity, regardless of the frequency content.

[0189] In some examples, extracting acoustic signals with a specific periodicity can help analyze specific parts of a target scene (e.g., specific devices or device types that typically operate with a particular periodicity). For instance, if the object of interest operates with a certain periodicity (e.g., once per second), excluding signals with a different periodicity can improve the acoustic analysis of the object of interest. For example, refer to... Figure 11B If the object of interest operates with a periodicity of 4, isolating signals with a periodicity of 4 for analysis can yield an improved analysis of the object of interest. For example, the object of interest could emit sounds with a periodicity of 4 but increasing frequency, such as... Figure 11B As shown. This could mean that the object's properties are changing (e.g., increased torque or load), and should be checked.

[0190] In exemplary acoustic imaging processes, background noise (e.g., noise floor in an industrial environment, wind in an outdoor environment, etc.) is typically not periodic, while certain objects of interest in the scene emit periodic acoustic signals (e.g., machinery operating at regular intervals). Therefore, a user can choose to exclude non-periodic acoustic signals from the acoustic image to remove background signals and present the acoustic data of interest more clearly. In other examples, a user may be looking for a constant pitch source, and therefore can choose to exclude periodic signals from the acoustic image data that might obscure the view of the constant pitch. Generally, a user can choose to include acoustic signals above a certain periodicity, below a certain periodicity, or within a desired periodicity range in the acoustic image data. In various examples, periodicity can be identified by the duration between periodic signals or the frequency of occurrence of the periodic signals. Similar to... Figure 11B The frequencies shown, for example, the analysis of intensity within a given periodicity (e.g., since the object of interest operates at that periodicity) can be similarly used to track how the acoustic signal from the object changes over time. Generally, in some embodiments, periodicity can be used to perform rate-of-change analysis on a wide variety of parameters such as frequency, intensity, etc.

[0191] As described elsewhere in this document, in some examples, different parts of the target scene may be associated with different distances to the acoustic imaging sensor. For example, in some embodiments, the distance information may include three-dimensional depth information about different parts of the scene. Additionally or alternatively, a user may be able to measure (e.g., using a laser distance tool) or manually enter distance values ​​associated with multiple locations in the scene. In some examples, these different distance values ​​for different parts of the scene may be used to adjust backpropagation calculations at such locations to accommodate a specific distance value at that location.

[0192] Additionally or alternatively, if different parts of the scene are associated with different distance values, then proximity to the acoustic sensor array (e.g., measured proximity and / or apparent proximity) can be another distinguishing parameter between such parts. For example, regarding Figure 10B Positions 1020, 1050, 1070, and 1090 are all associated with different distance values. In some examples, similar to the frequencies or periodicities discussed elsewhere in this document, the user can select a specific range of distances to include acoustic image data on the display based on that specific range. For example, the user can choose to display only acoustic image data representing sound signals that are closer than a predetermined distance, farther than a predetermined distance, or within a predetermined distance range.

[0193] Furthermore, in some embodiments, similar to regarding Figure 9CAs described in the frequency description, an acoustic analysis system can be configured to cycle through multiple distance ranges, thereby displaying only acoustic image data representing acoustic signals emitted from locations within the target scene that satisfy the current distance range. Such cycling through various displays can help users visually distinguish information between different acoustic signals. For example, in some cases, an object may appear close to the line of sight from an associated electromagnetic imaging tool (e.g., a visible light camera module), and therefore the acoustic image data combined with the electromagnetic image data of such an object may be difficult to distinguish. However, if the objects are separated by depth differences, cycling through different depth ranges of acoustic image data can be used to isolate each acoustic data source from one another.

[0194] Generally, acoustic analysis systems can be configured to apply various settings to include and / or exclude acoustic image data representing acoustic signals that satisfy one or more predefined parameters. In some examples, an acoustic analysis system can be used to select multiple conditions that an acoustic signal must satisfy in order to display acoustic image data representing such signals, for example, in a display image.

[0195] For example, regarding Figure 10A and 10B Only Figure 10B It shows Figure 10A In scenarios where acoustic signals exceed a threshold intensity, additional or alternative limitations are possible. For example, in some embodiments, the user can additionally filter the acoustic image data such that the acoustic image data is displayed only for acoustic signals having frequency content within a predetermined frequency range and / or having a predetermined periodicity. In exemplary embodiments, limited to the predetermined frequencies and / or periodicity of interest, the acoustic image data can be eliminated from additional locations such as 1020 and 1090.

[0196] Generally, users can apply any number of acoustic data requests to include or exclude acoustic image data from the displayed image, including parameters such as intensity, frequency, periodicity, apparent proximity, measured proximity, sound pressure, particle velocity, particle displacement, sound power, sound energy, sound energy density, sound exposure, pitch, amplitude, brilliance, harmonics, and the rate of change of any such parameters. Additionally, in some embodiments, users can combine requests using any suitable logical combination such as AND, OR, XOR, etc. For example, a user might want to display only sound signals with (intensity above a predetermined threshold) AND (frequency within a predetermined range).

[0197] Additionally or alternatively, the acoustic analysis system can be configured to loop through one or more parameter ranges to visualize different parts of the target scene, such as regarding... Figure 9C The acoustic image data is shown by cycling through multiple frequencies. Generally, one or more parameters can be cycled through in such a way. For example, a parameter (e.g., intensity) can be divided into multiple ranges (e.g., 10 dB–20 dB and 20 dB–30 dB), and the acoustic analysis system can cycle through such ranges, displaying all acoustic image data falling into the first range, then all acoustic image data falling into the second range, and so on.

[0198] Similarly, in some embodiments, the acoustic analysis system can be configured to combine parameter requirements by iteratively traversing nested ranges. For example, in an exemplary embodiment, acoustic image data satisfying a first intensity range AND a first frequency range can be displayed. The displayed frequency ranges can be traversed while limiting the displayed acoustic image data to acoustic signals satisfying the first intensity range. After traversing the frequency ranges, the intensity range can be updated to a second intensity range such that the displayed acoustic image data satisfies both the second intensity range and the first frequency range. Similar to the process of incorporating the first intensity range, the frequency ranges can be traversed similarly while maintaining the second intensity range. This process can continue until all combinations of frequency and intensity ranges have been satisfied. A similar process can be performed for any of a plurality of parameters.

[0199] Additionally or alternatively, in some embodiments, the acoustic analysis system can be configured to identify and distinguish multiple sounds in an acoustic scene. For example, regarding Figure 9B The acoustic analysis system can be configured to identify four discrete sounds at locations 910, 920, 930, and 940. The system can be configured to cycle through multiple displays, each showing acoustic image data at a single discrete location, similar to... Figure 9C As shown, although it does not necessarily depend on any parameter value. Similarly, this loop between acoustic image data at various locations can be implemented after one or more parameter requirements limit the displayed acoustic image data.

[0200] For example, regarding Figure 10A and Figure 10B Before applying an intensity threshold, the acoustic analysis system can cycle through multiple acoustic image scenes (e.g., as a display image including acoustic image scenes with visible light image data), where each scene includes acoustic image data at a single location. In some embodiments, according to Figure 10AThe illustrated example has a loop of 10 individual images, each image including image data at different locations among positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. However, according to some embodiments, after applying an intensity filter such that only locations with an intensity greater than a threshold are displayed (e.g., as shown in the example), Figure 10B As in [the previous example], the acoustic analysis system can update the loop process to only cycle through the image corresponding to positions that meet the filtering threshold. That is, regarding [the specific details]... Figure 10B The looping process can be updated to cycle only between four images, each showing discrete acoustic image data at positions 1020, 1050, 1070, and 1090, respectively.

[0201] Therefore, in various embodiments, each location in the target scene, including acoustic image data, is shown in one of a plurality of cyclically displayed images, before or after applying one or more filters to limit which acoustic image data is shown. This cyclical display of the individual sound source locations can help a user view the images to identify the source of a particular sound. In some embodiments, each image in the cycle includes only a single acoustic data source, and in some such embodiments, it further includes one or more parameters of the acoustic data, such as frequency content, intensity, periodicity, apparent proximity, etc.

[0202] In addition to looping between images showing acoustic image data that meet certain conditions, or as an alternative, in some examples, the location of an acoustic signal source can be detected in the acoustic image data, and the location of the acoustic signal source can be displayed in the acoustic image data in isolation from other acoustic signals. For example, regarding Figure 10A In some embodiments, acoustic image data representing acoustic signals emitted from each of locations 1010-990 can be identified and cyclically passed through. For example, during exemplary operation, a display image including acoustic image data at one of locations 1010-990 can be cyclically passed through automatically or under user guidance for individual analysis of each acoustic signal source. In various embodiments, the order in which the different locations of the acoustic image data are displayed cyclically can depend on a wide variety of parameters, such as by location, proximity, intensity, frequency content, etc.

[0203] Additionally or alternatively, in some examples, acoustic image data from various locations may be looped through after applying one or more filters to isolate only acoustic image data that meets one or more predetermined conditions. For example, regarding Figure 10BPositions 1020, 1050, 1070, and 1090 are shown as acoustic image data including acoustic signals representing acoustic signals that meet predetermined intensity requirements. In some embodiments, such display requirements may be applied to various cycles through the source locations of the acoustic signals. For example, further reference... Figure 10B The displayed image includes image data from only one of the locations 1020, 1050, 1070, and 1090 that meet the sound intensity conditions. This displayed image can be cycled through for individual analysis at each location.

[0204] In reference Figure 10A and Figure 10B In the exemplary process, acoustic image data collected from the scene can typically be in Figure 10A Positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 are shown in the image. Such positions can include acoustic image data representing acoustic signals with a wide variety of acoustic parameters, such as various intensities, frequency content, periodicity, etc.

[0205] As described elsewhere in this document, users may wish to isolate acoustic signals with one or more specific acoustic parameters, such as acoustic signals with a minimum sound intensity. Acoustic image data representing acoustic signals that do not meet such conditions can be excluded from the image, for example, such as... Figure 10B As shown, acoustic image data at positions 1020, 1050, 1070, and 1090 are displayed. However, the user may wish to further identify the sources of specific sounds that meet display conditions (e.g., having an intensity above a threshold). Therefore, the user can select to display the acoustic image data associated with positions 1020, 1050, 1070, and 1090 one after another to view the source location of each sound and analyze each sound individually. In various embodiments, the user can choose to manually cycle through such positions, or the processor can automatically update the display image to sequentially display the acoustic image data for each position. This can help the user further eliminate and ignore acoustic signals that are not of interest but happen to meet one or more filter parameters applied to the image.

[0206] Despite regarding strength and Figure 10A and 10BAs described, generally speaking, display images comprising acoustic image data from individual locations selected from multiple locations can be looped through one after another for individual analysis. The multiple locations comprising representative acoustic image data can be an entire set of locations corresponding to acoustic signal sources in an acoustic scene, or it can be a subset of such locations, for example, only locations containing acoustic signals that satisfy one or more conditions. Such conditions can depend on any one or more acoustic parameters, such as intensity, frequency content, periodicity, proximity, etc., and can be satisfied based on various parameters that are below predetermined values, above predetermined values, or within predetermined ranges of values.

[0207] In various examples, modifying a display image to selectively include acoustic image data can be accomplished in a variety of ways. In some embodiments, the display image (e.g., including electromagnetic and acoustic image data) can be a real-time image, wherein the electromagnetic and acoustic image data are continuously updated to reflect changes in the scene. In some examples, when using certain conditions to determine whether acoustic image data should be included in the display image, the received acoustic signals are analyzed to determine whether acoustic image data should be included at various locations in the updated real-time image. That is, as a new display image is generated based on recently received acoustic signals and electromagnetic radiation, the construction of the display image can depend on the analysis of the acoustic signals to determine which acoustic signals satisfy any specific conditions (e.g., intensity thresholds, etc.) placed on the display image. The display image including acoustic image data can then be generated based on such conditions, only where appropriate.

[0208] In other examples, the display image can be generated from data stored in memory, such as previously captured acoustic data and electromagnetic image data. In some such examples, the previously acquired acoustic data is analyzed regarding various conditions to be applied to the acoustic image data, and it is combined with the electromagnetic image data at locations where the previously captured acoustic data satisfies such conditions. In such embodiments, a single scene can be viewed in many ways, for example, by analyzing different acoustic parameters. The display image representing previously captured acoustic image data can be updated based on any updated conditions applied to the display image regarding whether or not acoustic image data is included at various locations in the display image.

[0209] In some embodiments, one or more acoustic parameters used to selectively include acoustic image data in a displayed image can be used to modify the displayed image and / or image capture techniques. For example, in a real-time imaging example, various conditions for determining whether acoustic image data is included in the display may include distance to the target (e.g., apparent distance or measured distance) and / or frequency content. As described elsewhere herein, some of these parameters can be used to select an acoustic sensor array and / or a processing scheme for generating the acoustic image data. Thus, in some such examples, when acoustic image data is represented solely based on such parameters that satisfy one or more predetermined conditions, such conditions can be used to select an acoustic sensor array and / or a processing scheme for generating the acoustic image data.

[0210] For example, in an exemplary embodiment, if the acoustic image data is included only in real-time images at locations where the corresponding acoustic signal comprises frequency content within a first frequency range, one or more acoustic sensor arrays can be selected to acquire the acoustic signal best suited for the first frequency range. Similarly, if the acoustic image data is included only in real-time images at locations where the acoustic signal source is within a first distance range, one or more acoustic sensor arrays can be selected to acquire the acoustic signal best suited for acoustic imaging within the first distance range. Additionally or alternatively, as shown, for example, with reference to... Figure 6 As described, the processing scheme for generating acoustic image data can be selected based on desired frequency or distance conditions. Such selected acoustic imaging sensor array(s) and processing scheme(s) can then be used to receive acoustic signals and generate acoustic image data for updated real-time display images, in order to optimize the included acoustic image data.

[0211] Similarly, in some embodiments where a display image is generated from historical data previously stored in memory, various conditions for determining which locations in the display image include acoustic image data can be used to update the acoustic image data representing the acoustic scene. For example, in some embodiments, the data stored in memory includes raw acoustic data received by one or more acoustic sensor arrays from the time the acoustic signal was received. Based on the conditions used to determine whether acoustic image data is included at various locations in the display image (e.g., desired distances and / or frequency ranges), a processing scheme (e.g., a backpropagation algorithm) can be selected to use the raw data stored in memory to generate acoustic image data optimized for desired parameters for display.

[0212] It will be understood that, although visible light image data and acoustic image data are commonly used for description and illustration, reference... Figure 9AThe processes described in -C, 10A, and 10B can be used, encompassing any image data from a wide variety of electromagnetic image data. For example, in various embodiments, infrared or ultraviolet image data can be used instead of visible light image data to perform a similar process. Additionally or alternatively, combinations of electromagnetic spectra, such as mixing infrared and visible light image data, can be used in such processes. Generally, in various examples, acoustic image data (e.g., acoustic image data included when the corresponding acoustic signal satisfies one or more predetermined parameters) can be selectively represented in combination with any combination of electromagnetic image data.

[0213] In some embodiments, the acoustic analysis system is configured to store one or more acoustic signals and / or acoustic image data in a database (e.g., in local memory) and / or be accessible from external or remote devices. Such acoustic signals may include acoustic image data representing an acoustic scene during normal operation, and / or other parameters associated with the acoustic scene, such as frequency data, intensity data, periodicity data, etc. In various examples, the database scene may include acoustic image data and / or other acoustic parameters (e.g., intensity, frequency, periodicity, etc.) representing a broad scene (e.g., a factory) and / or a more specific scene (e.g., a particular object).

[0214] In some embodiments, a database scenario can be generalized to a specific type of device (such as a specific device model). Additionally or alternatively, even if different such objects are different instances of the same object (e.g., two separate machines with the same model), a database scenario can be specific to a single object. Similarly, a database scenario can be more specific, for example, including a specific operational state of an object. For instance, if a particular object has multiple operational modes, the database can include multiple scenarios with such objects, one scenario for each operational mode.

[0215] In various embodiments, a database scene may be a single acoustic image and / or associated acoustic parameters. In other examples, a database scene may include synthetic data formed from multiple previously captured acoustic images and / or associated parameters. Generally, a database scene (e.g., acoustic images and / or parameters) may include an acoustic representation of the scene during normal operation. In some examples, the database may include other elements associated with the scene, such as corresponding visible light images, infrared images, ultraviolet images, or combinations thereof. In some embodiments, database generation and / or comparison may be performed similarly to the database generation and comparison of infrared image data described in U.S. Patent Application No. 15 / 190,792, filed June 23, 2016, entitled “THERMAL ANOMALY DETECTION,” which is assigned to the assignee of this application and is incorporated herein by reference in its entirety. In some embodiments, the database may be generated by capturing acoustic image data of the scene and / or one or more associated acoustic parameters (e.g., frequency, intensity, periodicity, etc.) while objects within the scene are operating correctly. In some of these examples, users can tag captured database images to associate the images with one or more objects, locations, scenes, etc., so that the captured acoustic images and / or (one or more) associated parameters can be identified in the future for database analysis and comparison.

[0216] Recently generated acoustic image data can be compared with acoustic image data stored in a database to determine whether the acoustic profile of the acoustic scene is within typical operating standards. Additionally or alternatively, acoustic parameters (such as intensity, frequency, periodicity, etc.) from the live acoustic scene and / or recently generated acoustic images can be compared with similar parameters in the database.

[0217] The current acoustic image data can be compared with historical acoustic image data stored in the database in a variety of ways (e.g., previously captured images, composite images generated from multiple previously captured images, expected images provided by the factory, etc.). Figures 12A-12C Several exemplary methods for comparing acoustic image data with historical acoustic image data stored in a database are shown. Figure 12AAn acoustic imaging tool 1200 is shown, comprising an acoustic sensor array 1202 having an acoustic field of view 1212 and an electromagnetic imaging tool 1204 having an electromagnetic field of view 1214. As shown, the electromagnetic field of view 1214 and the acoustic field of view 1212 include a target scene 1220, which includes an object of interest 1222. In some embodiments, the acoustic imaging tool 1200 is permanently fixed in a position such that the object of interest 1222 is within the electromagnetic field of view 1214 and the acoustic field of view 1212. In some embodiments, the acoustic imaging tool 1200 may be powered via inductive or parasitic power, may be wired to the AC mains power supply in a building, or may be configured to continuously monitor the object 1222.

[0218] The fixed acoustic imaging tool 1200 can be configured to periodically capture acoustic and / or electromagnetic image data of the object 1222. Because the acoustic imaging tool 1200 is typically fixed in place, images captured at different times will approximate the same point of interest. In some examples, the acoustic image data captured via the acoustic imaging tool 1200 can be compared with a database of acoustic image data representing approximately the same scene, for example, to detect anomalies or anomalies in the acoustic scene. This can be implemented, for example, as described in U.S. Patent Application No. 15 / 190,792, which is incorporated herein by reference.

[0219] Figure 12B An exemplary display, for example, on a handheld acoustic imaging tool, is shown. The display 1230 comprises two parts—1232 and 1234. In the illustrated example, part 1234 shows a database image 1244 of the object of interest, while part 1232 includes a live display 1242 of the object's real-time acoustic image data. In such a side-by-side view, a user can compare the live image 1242 with the database image 1244 to see any differences between typical acoustic signals (e.g., as shown in the database image 1244) and the current live image 1242. Similarly, a user can compare whether the live image 1242 approximately matches the database image 1244. If so, the user can capture live acoustic images for further analysis and / or comparison with the database image 1244.

[0220] Figure 12C Another exemplary display is shown, for example, on a handheld acoustic imaging tool. Figure 12C Display 1250 shows database image 1254 and live image 1256 on the same display 1252. Figure 12CIn the example, the user can similarly compare the acoustic image data in live image 1256 with the acoustic image data in database image 1254 to view the differences. Additionally, the user can adjust the alignment of the acoustic imaging tool to align the objects in live image 1256 with the objects in database image 1254 for further analysis and comparison.

[0221] As Figures 12A-12C The result of this process can compare live and / or recently captured acoustic images with previous acoustic image data, such as from a database. In some examples, such a process can be used to register live and / or recently captured acoustic images with database images for automated comparison. Other processes that can be used to “recapture” acoustic image data as database images from similar points of interest are described in U.S. Patent Application No. 13 / 331,633, filed December 20, 2011, entitled “THERMAL IMAGING CAMERA FOR INFRARED REPHOTOGRAPHY”, U.S. Patent Application No. 13 / 331,644, filed December 20, 2011, entitled “THERMAL IMAGING CAMERA FOR INFRARED REPHOTOGRAPHY”, and U.S. Patent Application No. 13 / 336,607, filed December 23, 2011, each of which is assigned to the assignee of this application and incorporated herein by reference in its entirety.

[0222] Comparing real-time acoustic image data and / or acoustic features with corresponding acoustic images and / or acoustic features of comparable scenes / objects can be used to provide a quick and simplified analysis of the operational state of a scene / object. For example, a comparison can indicate that certain locations within an acoustic scene are emitting acoustic signals with different intensities or spectra than during typical operation, which can indicate a problem. Similarly, locations in a scene may be emitting acoustic signals that are normally silent. Additionally or alternatively, a comparison of the overall acoustic features of live and historical scenes from a database can generally indicate variations in acoustic parameters within the scene, such as frequency content, sound intensity, etc.

[0223] In some examples, the acoustic analysis system is configured to compare recent / real-time acoustic scenes with a database. In some embodiments, the acoustic analysis system is configured to characterize differences between recent / real-time scenes and database scenes, and based on this comparison, diagnose one or more potential problems in the current scene. For example, in some embodiments, a user can pre-select objects or target scenes of interest for comparison with an acoustic database. The acoustic analysis system can analyze the scene by comparing database images and / or other parameters with recent / current images and / or other parameters based on the selected object / scene. Based on the object / scene selected from the database, the acoustic analysis system may be able to identify one or more differences between the database images / parameters and recent / current images / parameters, and associate the identified differences(s) with possible causes of one or more differences.

[0224] In some examples, the sound analysis system can be pre-programmed with multiple diagnostic information entries, for example, associating various differences between database images / parameters and recent / current images / parameters with possible causes and / or solutions to those causes. Additionally or alternatively, the user can load such diagnostic information, for example, from a repository of diagnostic data. Such data can be provided, for example, by the manufacturer of the acoustic analysis system, the manufacturer of the object of interest, etc. In still other examples, the acoustic analysis system can additionally or alternatively learn the diagnostic information, for example, via one or more machine learning processes. In some such examples, the user can diagnose one or more problems in a target scene after observing acoustic deviations from typical scenes, and data representing those one or more problems and / or one or more solutions can be input into the acoustic analysis system. The system can be configured to learn, over time and via multiple data entries, to associate different differences between recent / current images and / or parameters and those stored in the database with certain problems and / or solutions. When diagnosing problems and / or determining proposed solutions, the acoustic analysis system can be configured to, for example, output the suspected problems and / or proposed solutions to the user via a display. Such displays can be on handheld acoustic inspection tools or remote devices (e.g., a user's smartphone, tablet, computer, etc.). Alternatively or additionally, such displays indicating potential problems and / or solutions can be transmitted, for example, via a network to a remote site, such as a field operator / system monitor.

[0225] In some example diagnostic characterizations, acoustic analysis systems may observe specific periodic squeaking sounds, indicating the need for additional lubrication on operating machinery. Similarly, a constant high-pitched signal can indicate a gas or air leak in a target scene. Other problems may similarly possess identifiable acoustic features, such as bearing damage within the object being analyzed, making it possible to diagnose any anomalies in the system or object by examining acoustic features via an acoustic imaging system (e.g., a handheld acoustic imaging tool).

[0226] Acoustic analysis systems capable of comparing received acoustic signals with baselines (e.g., acoustic image data and / or parameters from a database) and providing diagnostic information and / or suggesting corrective actions can eliminate the need for experienced experts to analyze the acoustic data of a scene. Instead, acoustic inspection and analysis can be performed by system operators with limited or no experience in analyzing acoustic data.

[0227] Figure 13 This is a flowchart illustrating an exemplary operation of comparing received acoustic image data with a database used for object diagnosis. The method includes: receiving a selection of a target of interest (1380); and retrieving a baseline acoustic image and / or acoustic parameters of the target of interest from the database (1382). For example, a user may wish to perform acoustic analysis on a specific object of interest and may select such an object from a predefined list of objects having available baseline acoustic images and / or parameters available in the database.

[0228] The method further includes the following steps: for example, capturing acoustic image data and associated parameters representing the target of interest using a handheld acoustic imaging tool (1384). After capturing the acoustic image data and associated parameters (1384), the method includes comparing the captured acoustic image data and / or associated parameters with a retrieved baseline image and / or parameters (1386).

[0229] Figure 13 The method further includes diagnosing operational problems with the target of interest based on comparison if the captured acoustic image data and / or parameters deviate sufficiently from the baseline (1388). The method may further include the step of displaying instructions to the user regarding potential problems and / or corrective actions (1392). In some embodiments, a comparison display may be additionally or alternatively displayed to the user, for example, a difference image showing the difference between the current acoustic image data and the baseline acoustic image data.

[0230] In some such examples, determining whether there is a deviation from the baseline (1388) includes comparing one or more acoustic parameters of the captured data with similar parameters in the baseline data, and determining whether the difference between the captured and baseline parameters exceeds a predetermined threshold. In various examples, different parameters may include different thresholds, and such thresholds may be absolute thresholds, statistical thresholds, etc. In some embodiments, comparisons may be made on a location-by-location basis, and comparisons may be performed on a subset of locations within the scene.

[0231] For example, refer to Figure 9B It is possible to analyze the acoustic image data and its locations on the object (e.g., locations 910 and 940) solely based on operations on the object. In such an example, different acoustic parameters at each location to be compared (e.g., 910 and 940) are compared individually between the captured image and the database image. For example, refer to... Figure 9B Comparing the captured data and / or associated parameters with those from the database may include comparing the frequency, intensity, and periodicity of position 910 in the captured image with the frequency, intensity, and periodicity of position 910 in the database image, respectively. A similar comparison may be performed between the captured image and the database image at position 940. As described, each comparison may include different measures (1388) for determining whether there is a sufficient deviation from the baseline.

[0232] Diagnostic problems (1390) can be performed based on a combination of comparisons between the captured image data and baseline image data and / or parameters, and indications for possible problems and / or corrective actions can be displayed (1392). In some examples, such diagnosis may include multidimensional analysis, such as a combined comparison of multiple parameters at a given location. For example, in an exemplary embodiment, a condition may be indicated by both a frequency deviation from the baseline greater than a first threshold and an intensity deviation from the baseline greater than a second threshold.

[0233] In some examples, even after instructions for potential problems and / or corrective actions are displayed (1392), the process may include capturing new acoustic image data and associated parameters (1384), and repeating the comparison and diagnostic process. Thus, the user can observe whether any corrective actions taken effectively alter the acoustic characteristics of the object to correct identified problems and / or bring the object's acoustic characteristics to baseline.

[0234] In some embodiments, if, after comparing the captured data with baseline data (1386), there is insufficient deviation from the baseline (1388), the process can terminate (1394), concluding that the object is functioning normally based on its current acoustic characteristics. Additionally or alternatively, new acoustic image data and associated parameters of the target of interest can be captured (1384), and the comparison and diagnostic process can be repeated. In some examples, a fixed acoustic analysis system can be used to perform continuous, repeatable analysis, such as including… Figure 12A Acoustic imaging tool 1200.

[0235] Comparison of acoustic data (e.g., image data and / or other acoustic parameters) can help users more easily identify whether an object is functioning correctly, and if not, diagnose the object's problem. In some examples, comparison with a baseline can help users ignore "normal" sounds in the scene, such as expected operational sounds or noise / background sounds, which may be unrelated to the object's operational problems.

[0236] During operation, observation of acoustic image data and / or associated acoustic parameters, or comparison of the current acoustic scene with a database acoustic scene, can indicate locations of interest for further investigation. For example, comparing acoustic images to database images can indicate one or more locations in the scene where abnormal operation is occurring. Similarly, viewing acoustic images with acoustic features at one or more unexpected locations can indicate locations of interest for the user. For example, referring to… Figure 10B Users observe on the display of the acoustic imaging system. Figure 10B It can recognize that a particular location (e.g., 1020) is unexpectedly emitting an acoustic signal, or similarly, a comparison with a baseline image indicates unexpected parameters of the acoustic signal at that location (e.g., unexpected frequency, intensity, etc.).

[0237] In some such examples, a user can move closer to a location to examine it more closely for anomalies. As the user moves closer to the object, the distance to the target can be updated to reflect the new distance between the acoustic array and the target location. The acoustic sensor array and / or backpropagation algorithm can be updated based on the updated distance to the target. Additionally or alternatively, the updated acoustic analysis from a closer location can produce different analyses of the acoustic signal from the target. For example, high-frequency acoustic signals (e.g., ultrasonic signals) tend to attenuate over relatively short distances to the source. Therefore, when the user moves closer to the target for further examination, the acoustic sensor array may see additional signals (e.g., high-frequency signals). Such significant changes in the observable scene may also lead to adjustments to the acoustic sensor array and / or the backpropagation algorithm used for acoustic imaging.

[0238] Therefore, as the user moves closer to the object or area of ​​interest, the sensor array and / or backpropagation algorithm used for acoustic imaging can be updated once or multiple times. Each update can provide additional details about the object or area of ​​interest that might not be observable from a greater distance using different sensor arrays and / or backpropagation algorithms. For example, moving closer to the object or area of ​​interest, based on an initial observation of a broader scene, can also increase the intensity of the acoustic signal of interest relative to background sounds in the environment.

[0239] In some embodiments, an acoustic analysis system (e.g., a handheld acoustic imaging tool) can prompt a user to move closer to an object or area of ​​interest within a scene. For example, when comparing a current acoustic image to a baseline database image, the acoustic analysis system can identify one or more locations in the scene that deviate from the baseline. The acoustic analysis system can highlight such one or more locations to the user, for example via a display, and suggest that the user move closer to the identified location(s) for further analysis. In some examples, the acoustic analysis system can classify the identified locations (such as objects within the environment or sub-components of a specific object) as having their own baseline profile stored in a database. The system can be configured to suggest and / or implement such a profile for the classified locations to facilitate further analysis of the identified locations as the user moves closer for additional examination.

[0240] The systems and processes described herein can be used to improve the speed, efficiency, accuracy, and thoroughness of acoustic inspections. Various automated actions and / or suggestions (e.g., sensor arrays, backpropagation algorithms, etc.) can enhance the ease of inspection to the point where even inexperienced users can perform thorough acoustic inspections of acoustic scenes. Furthermore, such processes can be used to analyze a wide range of scenes, such as entire systems, individual objects, and sub-components of individual objects. Predefined and / or user-generated profiles of baseline acoustic data for an acoustic scene can even help inexperienced users identify anomalies in the captured acoustic data.

[0241] Registration of acoustic image data with other data streams, such as visible light, infrared, and / or ultraviolet image data, can provide additional background and detail about which objects are emitting the acoustic signals represented in the acoustic image data. Combining an acoustic sensor array with a distance measurement tool (e.g., a laser rangefinder) can help users quickly and easily determine the appropriate distance value to the target for use during the acoustic imaging process. In various examples, the acoustic sensor array, distance measurement tool, processor, memory, and one or more additional imaging tools (e.g., visible light camera module, infrared camera module, etc.) can be supported by a single housing in a handheld acoustic imaging tool that can provide efficient acoustic analysis of multiple scenes. Such a handheld acoustic imaging tool can be moved from one scene to another for rapid analysis of multiple objects of interest. Similarly, using a handheld tool, a user can move closer to the location of interest within the scene for further examination or analysis.

[0242] This document describes various systems and methods for performing acoustic imaging and generating and displaying acoustic image data. Exemplary systems may include an acoustic sensor array comprising a plurality of acoustic sensor elements configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals.

[0243] The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representing the received electromagnetic radiation. Such an imaging tool may include infrared imaging tools, visible light imaging tools, ultraviolet imaging tools, and combinations thereof.

[0244] The system may include a processor that communicates with an acoustic sensor array and an electromagnetic imaging tool. The processor may be configured to receive electromagnetic image data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor may be configured to generate acoustic image data of a scene, for example, via backpropagation calculation, based on the received acoustic data and received distance information representing the distance to the target. The acoustic image data may include a visual representation of the acoustic data, such as through a color palette or color scheme, as described elsewhere herein.

[0245] The processor can be configured to combine the generated acoustic image data and the received electromagnetic image data to generate a display image that includes both the acoustic image data and the electromagnetic image data, and to transmit the display image to a display. Combining the acoustic image data and the electromagnetic image data may include, for example, correcting for parallax errors between the acoustic image data and the electromagnetic image data based on received distance information.

[0246] In some examples, distance information can be received from a distance measurement tool that communicates with the processor. The distance measurement tool may include, for example, optical distance measurement devices (such as laser distance measurement devices) and / or acoustic distance measurement devices. Additionally or alternatively, the user may manually enter distance information, for example, via a user interface.

[0247] The system may include a laser pointer to help identify the location of a point of interest, such as a sound or sound profile, based on selected parameters (such as frequency, decibel level, periodicity, distance, etc., or combinations thereof). Such a laser pointer can be used to accurately pinpoint and align the field of view of a scene using appropriate acoustic visualization displayed on a monitor. This can be useful in environments where the object being examined is at a distance relative to the acoustic imaging device, or when the location of the acoustic visualization on the monitor relative to the actual scene is unclear.

[0248] In some examples, the laser pointer can be visualized on a display. Such visualization may include generating a laser pointer dot on the display (e.g., via a processor) that represents the laser pointer in the actual scene. In some examples, the orientation of the laser pointer on the display may be enhanced, for example, by using an icon representing the laser pointer in the scene or another aligned display marker, to better determine the position of the laser pointer on the display relative to the actual scene.

[0249] As described elsewhere in this document, thermal imaging systems can be configured to create false-color (e.g., color-corrected), symbolic, or other non-digital visual representations of acoustic data generated by one or more acoustic sensors, such as by creating acoustic image data. Additionally or alternatively, the system can provide audio feedback to the user, such as via a speaker, headphones, wired or remote communication headset, etc. The transmission of such audio or heterodyne audio can be synchronized with the visual representation of the detected and displayed sound.

[0250] Acoustic data can be visualized in a variety of ways, for example, to facilitate understanding of such data and to prevent viewers from making incorrect assumptions about the nature of the sound being visualized. In some examples, different types of visualization can provide an intuitive understanding of the visualized sound.

[0251] In some embodiments, the generated display includes a non-numerical visual representation with contextual numeric and / or alphanumeric data to provide a thorough demonstration of information about the sound being visualized, which can help the user determine and / or implement one or more appropriate action guidelines.

[0252] Various display features can be combined, including various non-numerical graphic representations (e.g., symbols, color tones, etc.) and alphanumeric information. In some embodiments, the display features present in a given representation of a scene can be customized by the user, for example, from a plurality of optional settings. Additionally or alternatively, preset combinations of display features can be selected by the user to automatically include the desired combination of information in the displayed image. In various embodiments, aspects of the displayed image can be adjusted by the user, for example, via a virtual interface (e.g., provided via a touchscreen) and / or via physical controls.

[0253] Figure 14 A visualization of acoustic data using a gradient color scheme is shown. As illustrated, acoustic parameters (e.g., intensity) are displayed via the gradient color scheme. In an exemplary gradient color scheme, a parameter value will have a unique color associated with it, depending on the color scheme. A change in the parameter value at a given pixel will typically result in a change in the color associated with that pixel to represent the new parameter value. Figure 14 As shown in the example, the acoustic parameter values ​​appear to change radially from the center azimuth of the acoustic signal at azimuths 232, 234, and 236. Other examples of gradient color gradation are described in U.S. Patent Application No. 15 / 802,153, filed November 2, 2017, which has been assigned to the assignee of this application.

[0254] Figure 15A visualization of acoustic data using multiple concentric circles of shadow is shown. In some such examples, in contrast to gradient color schemes with colors associated with parameter values, Figure 15 Each concentric circle of solid color shown can represent a pixel having an acoustic parameter value within a range associated with that color. In the illustrated example, the acoustic parameter (e.g., intensity) associated with the acoustic signal at positions 332, 334, and 336 changes radially from the center of the acoustic signal. In an exemplary color scheme, pixels shown in red represent acoustic parameter values ​​within a first parameter value range, pixels shown in yellow represent acoustic parameter values ​​within a second parameter value range, and pixels shown in green represent acoustic parameter values ​​within a third parameter value range; however, other display techniques are possible, including additional or alternative colors, patterns, etc. In various embodiments, the range of values ​​can correspond to an absolute range, such as an intensity value between 10 dB and 20 dB, or it can be a relative range, such as an intensity value between 90% and 100% of the maximum intensity.

[0255] As described elsewhere herein, in some embodiments, the display image that includes electromagnetic image data and acoustic image data may include a visual indication of the acoustic signal and an alphanumeric representation of one or more parameters associated with the acoustic signal. Figure 16 An exemplary visualization is shown, illustrating both non-numeric information (e.g., color toning via parameter value ranges) and alphanumeric information. In the illustrated example, alphanumeric sound intensity value labels are associated with each of three locations having color-toned acoustic image data (e.g., intensity data). As shown, the sound signals have corresponding visual indicators representing the acoustic parameters (1602, 1604, 1606) and alphanumeric information (1612, 1614, 1616, respectively) associated with them. In an exemplary embodiment, the alphanumeric information can provide a numerical value, such as a maximum intensity value, associated with the location where color-toned acoustic data is displayed. In some examples, a user can select one or more locations to display color-toned and / or alphanumeric data. For example, a user can choose to annotate the displayed image using alphanumeric representations of acoustic parameters associated with one or more sound signals within the scene.

[0256] In some examples, alphanumeric information can represent multiple parameters (e.g., acoustic parameters) associated with an acoustic signal at a given location in the scene. Figure 17An exemplary visualization is shown, which includes non-numeric information (e.g., color-coding via a range of parameter values) and alphanumeric information. In the illustrated example, sound intensity values ​​and corresponding frequency values ​​(e.g., average frequency or peak frequency) are shown in alphanumeric information 1712, 1714, and 1716 associated with each of three locations having color-coded acoustic data (e.g., intensity data) (shown via indicators 1702, 1704, and 1706, respectively). Similar to the description of... Figure 16 As discussed, users can initiate the inclusion of various types of such data at various locations. For example, a user can choose to annotate the displayed image using alphanumeric representations of one or more acoustic parameters associated with one or more sound signals within the scene.

[0257] Figure 18 Another exemplary visualization is shown, illustrating both non-numeric information (e.g., color tuning via a range of parameter values) and alphanumeric information. In the illustrated example, distance measurements are included with alphanumeric information 1812, 1814, and 1816 associated with each of three locations having color-tuned acoustic data (e.g., intensity data) (shown via indicators 1802, 1804, and 1806, respectively). Similar to the description of... Figure 16 As discussed, for example, as part of displaying image annotations, users can choose to include various types of such data at various locations.

[0258] In some examples, non-numeric representations can be used to convey information related to multiple acoustic parameters. For example, Figure 19 An exemplary visualization is shown, illustrating indicators 1902, 1904, and 1906 (circles in this case) of different sizes and colors representing different acoustic parameter values. In this exemplary embodiment, the size of the indicator corresponds to the intensity of the acoustic signal at a given location, while the color of the indicator corresponds to the peak or average frequency. In this exemplary embodiment, the indicator size may indicate a relative value, such that comparing the size of one indicator to the size of another indicates a relative difference between the acoustic parameter values ​​represented at the location associated with the indicator. Additionally or alternatively, alphanumeric information may be included to provide absolute or relative acoustic parameter values.

[0259] In some embodiments, color indicators may be used to indicate the severity of one or more detected acoustic signals and / or associated acoustic parameters, such as the amount of deviation from baseline parameters. Figure 20An exemplary visualization is shown, illustrating multiple indicators 2002, 2004, and 2006 of different colors, indicating the severity of an acoustic signal originating from a corresponding location. For example, in an exemplary embodiment, a red indicator indicates critical severity based on one or more acoustic parameters (e.g., when compared to a baseline, such as a baseline of typical operating conditions), a yellow indicator indicates moderate severity, and a green indicator indicates minor severity. In other examples, other color schemes or appearance characteristics (e.g., indicator transparency, indicator size, etc.) may be used to visually differentiate the severity of acoustic signals. In the illustrated example, indicator 2004 represents the highest level of severity, indicator 2006 represents the second most severe, and indicator 2002 represents the least severe acoustic signal.

[0260] As described elsewhere herein, in various embodiments, one or more acoustic parameters may be displayed on a visual representation of the acoustic scene, for example, through color-corrected colors or grayscale. In some embodiments, the system may be configured to identify one or more locations in the scene that satisfy one or more acoustic conditions, such as identified frequency ranges, intensity ranges, distance ranges, etc. In some examples, various locations corresponding to acoustic profiles (e.g., satisfying a specific set of conditions or parameters) may be identified. Such identified locations may be presented in a manner distinct from the color-correction scheme of acoustic image data otherwise used when creating the displayed image. For example, Figure 21 A scene is illustrated with indicators at multiple locations within the scene. Indicators 2101, 2102, 2103, 2104, and 2105 are positioned within the scene. Indicators 2103, 2104, and 2105 include color-corrected acoustic image data, for example, representing values ​​of one or more acoustic parameters corresponding to scale 2110. Indicators 2101 and 2102 are shown to have a distinct presentation scheme that is distinguishable from the color-correction schemes appearing at locations 2103, 2104, and 2105. In such embodiments, a user can quickly and easily identify those locations in the image that satisfy one or more desired conditions. In some such examples, a user can select one or more desired conditions for differentiated display based on a range of values, for example, from a scale such as 2110.

[0261] Additionally or alternatively, the location of conditions satisfying a specific sound profile, such as the corresponding acoustic profile, can be represented using icons that symbolize the satisfied conditions. For example, Figure 22 Several icons 2202, 2204, 2206, and 2208, positioned within the displayed image, are shown; these icons indicate identified acoustic contours within the scene. For example... Figure 22The exemplary profiles shown include bearing wear, air leakage, and electric arcing. Such profiles can be identified by acoustic signals that satisfy a set of one or more parameters associated with them in order to be classified as such profiles.

[0262] Figure 23 Another exemplary display is shown, which illustrates acoustic data via multiple indicators 2302, 2304, and 2306, using concentric circles and alphanumeric information representing the sound intensity associated with each acoustic signal. As described elsewhere herein, in some examples, the size of the indicators may indicate one or more acoustic parameters present at the corresponding location. In some embodiments, the indicators may be monochromatic and indicate acoustic parameters in one or more other ways, such as by indicator size, line width, line type (e.g., solid line, dashed line, etc.).

[0263] In some examples, the display may include alphanumeric information based on a user's selection. For instance, in some embodiments, the system (e.g., via a processor) may include information representing one or more acoustic parameters of an acoustic signal located at a particular location in response to a user's selection of an indicator on the display (e.g., via a user interface) at that location. Figure 24 An example display image is shown, featuring indicators and additional alphanumeric information associated with the represented acoustic signal. In the example, an indicator 2402 on the display (e.g., indicated by crosshairs, which can indicate a selection, such as a selection entered via a touchscreen) can be selected for further analysis. The display shows alphanumeric information 2404, which includes a list of data associated with the location corresponding to the indicator, including peak intensity and corresponding frequency, frequency range, distance to the location measurement, and the criticality level indicated by the acoustic signal from that location.

[0264] In some examples, the displayed image may include multiple indicators representing corresponding sound signals in the scene. In some embodiments, in such cases, a user may select one or more indicators (e.g., via a touchscreen or other user interface), and in response to detecting the selection, the processor may present additional information about the sound signal. Such additional information may include alphanumeric representations of one or more acoustic parameters. In some examples, such additional information may be displayed simultaneously for multiple sound signals. In other examples, such additional information for a given sound signal is hidden when another sound signal is selected.

[0265] As described elsewhere in this document, in some examples, the system may include a laser pointer. In some examples, the laser pointer may have a fixed orientation or may have an adjustable pointing, for example, controllable via a processor. In some examples, the system may be configured to align the laser pointer with a location in the target scene associated with a selected location in the image. Figure 25A A system including a display (in some examples, embodied as a handheld tool) is shown, the display being such as... Figure 24 The display shown allows selection of an indicator 2502. A laser pointer 2504 emits a laser beam 2506 toward the scene, creating a laser point 2508 in the scene corresponding to the position of the selected indicator 2502 in the image. This helps the user visualize the location of the selected and / or analyzed acoustic signal in the environment. In some embodiments, the laser point 2508 may be detected by an electromagnetic imaging tool and is visible on the display along with the displayed indicator 2502 and alphanumeric information 2512 including acoustic parameter information. In some examples, the acoustic imaging system is configured to detect or predict the position of a laser in the scene and provide a visual indication 2510 of the laser position. Figure 25B It shows things like Figure 25A The image shown in the system view.

[0266] In embodiments where the laser pointer has a fixed orientation, the user can view a display image with a visual indication of the laser position as feedback, allowing the user to adjust the direction of the laser to match the selected acoustic signal.

[0267] As described elsewhere herein, in some embodiments, acoustic image data may be combined with electromagnetic image data for presentation in a displayed image. In some examples, the acoustic image data may include adjustable transparency so that various aspects of the electromagnetic image data are not completely obscured. Figure 26 The diagram illustrates acoustic image data at a location in a scene, indicated by indicator 2602, where indicator 2602 includes a gradient color scheme. The system may include a display device, which may be integrated with or separate from the acoustic imaging tool, configured to present display data including electromagnetic image data and acoustic image data.

[0268] In some embodiments, the device (e.g., a handheld acoustic imaging tool) may include physical hybrid controls 2614 (e.g., one or more buttons, knobs, sliders, etc., which may be included as part of the user interface) and / or virtual hybrid controls 2604 (such as an interface implemented via a touchscreen or other virtual means). In some embodiments, such functionality may be provided by an external display device such as a smartphone, tablet, computer, etc.

[0269] Figure 27 This demonstrates a virtual and / or physical blending control tool for displaying images, which includes a partially transparent concentric circle color scheme. (Similar to...) Figure 26 As described, indicator 2702 can represent acoustic signals within a scene. The acoustic imaging system may include physical blending control 2714 and / or virtual blending control 2704, which can be used to adjust the transparency of acoustic image data (e.g., indicator 2702) within the displayed image.

[0270] Additionally or alternatively, physical and / or virtual interfaces can be used to adjust one or more display parameters. For example, in some embodiments, one or more filters can be applied to selectively display acoustic image data that meets one or more conditions, as described elsewhere herein. Figure 28 A scene is illustrated including indicator 2802 with gradient color adjustment, indicating the location in the scene that satisfies one or more filters (e.g., one or more acoustic or other parameters satisfying one or more corresponding thresholds or predetermined conditions). In various examples, the filters can be selected and / or adjusted via physical controls 2814 (e.g., via one or more buttons, knobs, switches, etc.) and / or virtual controls 2804 (e.g., a touchscreen). In the illustrated example, the filter includes: displaying acoustic image data only for those acoustic signals having acoustic parameters (e.g., frequencies) falling within a predefined range 2806 of acoustic parameters. As shown, the predefined range 2806 is a subset of the possible filter range 2816. In some examples, the user can adjust the limits of the predefined range 2806, for example, via virtual or physical controls 2804, to adjust the effect of the filter.

[0271] Figure 29 Virtual and / or physical filter adjustments for a displayed image are illustrated, including a partially transparent concentric circle color scheme. As shown, an indicator 2902 is displayed within the scene based on acoustic parameters that fall within a predetermined range 2906 of filter-based values. The filter can be adjusted, for example, via virtual controls 2904 and / or physical controls 2914 within a range 2916 of values.

[0272] In some embodiments, multiple filters can be used to customize the display image, which includes color-tuned acoustic image data. Figure 30 A display image is shown, showing a first indicator and a second indicator. As described elsewhere in this document, one or more filters can be applied to the display image (e.g., via physical filter controls and / or virtual filter controls) to customize the displayed data. Figure 30In the illustrated example, filtering includes establishing a first filter range 3006 and a second filter range 3008. In some examples, the filter range can be adjusted within the range of values ​​3016, for example, via virtual controls 3004 and / or physical controls 3014.

[0273] Such filter ranges can represent any parameter from a wide variety of parameters, such as frequency, amplitude, proximity, etc. As shown, both the first and second filter ranges are associated with color (in some examples, this color can be adjusted by the user), and indicators 3002 and 3012 are positioned in the image at locations where the corresponding acoustic signals satisfy one or more filter conditions associated with each filter range. As shown, the first indicator 3002 represents an acoustic signal that satisfies the first filter range 3006 (shown in dark shading), while the second indicator 3012 represents an acoustic signal that satisfies the second filter range 3008 (shown in lighter shading). Therefore, users can quickly identify locations in a scene with acoustic data that simultaneously satisfy a wide variety of conditions, while also identifying which locations satisfy which conditions.

[0274] In some examples, display devices such as acoustic imaging tools or external display devices may include a virtual keyboard as an input device, such as Figure 31 shown. Figure 31 A display interface including indicator 3102 is shown, which represents one or more acoustic parameters of sound signals in the scene. A virtual keyboard 3110 is included in the display and can be used to add alphanumeric information 3112 to the displayed image. Such a virtual keyboard allows users to input custom annotations, such as various check notes, tags, date / time stamps, or other data that can be stored with the image. In various examples, the virtual keyboard can be used to add text included in and / or appended to image data, such as through metadata associated with the displayed image.

[0275] Various devices can be used to present display images, which include various combinations of acoustic image data and other data, such as alphanumeric data, image data from one or more electromagnetic spectra, symbols, and so on. In some examples, a handheld acoustic imaging tool may include a built-in display for presenting the display image. In other examples, information to be displayed or data processed to generate the display (e.g., raw sensor data) may be transmitted to an external device for display. Such external devices may include, for example, smartphones, tablets, computers, wearable devices, and so on. In some embodiments, the display image is presented in conjunction with real-time electromagnetic image data (e.g., visible light image data) in an augmented reality type display.

[0276] Figure 32 A display is shown embedded in glasses 3210 that can be worn by a user. In some examples, the glasses may include one or more embedded imaging tools, such as those described in U.S. Patent Publication No. 20160076937 entitled “DISPLAY OF IMAGES FROM AN IMAGINGTOOL EMBEDDED OR ATTACHED TO A TEST AND MEASUREMENT TOOL”, which is assigned to the assignee of this application, the relevant portions of which are incorporated herein by reference. In some such examples, the integrated display may show real-time displayed images 3220. For example, the display may show electromagnetic image data (e.g., visible light image data) representing the scene faced by the user, and may simultaneously display (e.g., via blending, overlay, etc.) one or more additional data streams, such as acoustic image data (e.g., including indicator 3202), etc., to provide the user with added information. In some embodiments, such as Figure 32 The glasses shown include a transparent display screen, allowing the user to view the scene directly through their eyes when no image is presented to the monitor, rather than being presented with real-time visible light image data. In some such examples, additional data, such as acoustic image data, alphanumeric data, etc., can be displayed on a separately transparent display screen within the user's field of view, allowing the user to view this data in addition to viewing their scene view through the monitor.

[0277] As described elsewhere in this document, various data presented in a displayed image can be combined in a wide variety of ways, including mixing with other data streams (e.g., mixing acoustic image data with visible light image data). In some examples, the intensity of the mixing can vary between different locations within a single displayed image. In some embodiments, a user can manually adjust the mixing ratio at each of a plurality of locations (e.g., each of a plurality of indicators of a detected acoustic signal). Additionally or alternatively, the mixing can be a function of one or more parameters, such as frequency, amplitude, proximity, etc.

[0278] In some embodiments, the acoustic imaging tool can be configured to identify the extent to which the sensor array is directed at each of a plurality of locations where the detected acoustic signal is emitted, and accordingly mix the corresponding acoustic image data with, for example, visible light image data. Figure 33A An exemplary display is shown, which includes a first indicator 3302 and a second indicator 3304 representing sound signals in an acoustic scene. Figure 33AIn the comparison with pipe 2, the acoustic sensor points more directly to pipe 1, which corresponds to the position of the first indicator 3302, and pipe 2, which corresponds to the position of the second indicator 3304. Accordingly, in Figure 33A In the display scheme, the first indicator 3302 is displayed more prominently than the second indicator 3304 (e.g., with a higher blending factor or lower transparency). Conversely, in Figure 33B In the comparison with pipe 1, the acoustic sensor points more directly to pipe 2, which corresponds to the position of the second indicator 3304, while pipe 1 corresponds to the position of the first indicator 3302. Accordingly, in Figure 33B In the display scheme, the second indicator 3304 is displayed more prominently than the first indicator 3302 (e.g., with a higher blending coefficient or lower transparency). Generally, in some embodiments, the acoustic imaging system can determine a measure of the degree to which the indicator sensor is pointing to a given location (e.g., corresponding to an indicator in the acoustic image data) and adjust the blending ratio corresponding to such location accordingly (e.g., a greater degree of pointing corresponds to a higher blending ratio).

[0279] This document describes various systems and methods for performing acoustic imaging and generating and displaying acoustic image data. Exemplary systems may include an acoustic sensor array comprising a plurality of acoustic sensor elements configured to receive acoustic signals from an acoustic scene and output acoustic data based on the received acoustic signals.

[0280] The system may include an electromagnetic imaging tool configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representing the received electromagnetic radiation. Such an imaging tool may include an infrared imaging tool, a visible light imaging tool, an ultraviolet imaging tool, and so on, or a combination thereof.

[0281] The system may include a processor that communicates with an acoustic sensor array and an electromagnetic imaging tool. The processor may be configured to receive electromagnetic image data from the electromagnetic imaging tool and acoustic data from the acoustic sensor array. The processor may be configured to generate acoustic image data of a scene, for example, via backpropagation calculations, based on the received acoustic data and received distance information representing the distance to the target. The acoustic image data may include a visual representation of the acoustic data, such as through a color palette or color scheme, as described elsewhere herein.

[0282] The processor can be configured to combine the generated acoustic image data and the received electromagnetic image data to generate a display image that includes both the acoustic and electromagnetic image data, and to transmit the display image to a display. Combining the acoustic and electromagnetic image data may include, for example, correcting for parallax errors between the acoustic and electromagnetic image data based on received distance information.

[0283] In some examples, distance measurement tools that communicate with the processor (e.g., Figure 1A (110) receives distance information. The distance measurement tool may include, for example, an optical distance measurement device (such as a laser distance measurement device), and / or an acoustic distance measurement device. Additionally or alternatively, the user may manually enter the distance information, for example, via a user interface (e.g., via control 126 and / or a touchscreen display).

[0284] In some embodiments, the acoustic imaging system may include a laser pointer to help identify the location of a point of interest, such as a sound or sound profile, based on selected parameters or combinations thereof, such as frequency, decibel level, periodicity, distance, etc. Figure 34A An acoustic imaging device is shown that emits a laser toward a scene. In various embodiments, the laser pointer can be manually operated by a user and / or can be automatically controlled by the system, such as via a processor in the acoustic imaging tool.

[0285] like Figure 34A As shown, the acoustic imaging device 3412, including a laser pointer 3404, can emit a laser beam 3406 toward at least one point of interest 3408 in the scene 3410 (e.g., toward sound 3402). The laser pointer 3404 can be used to accurately locate / identify the point of interest and / or align the field of view of the scene with an appropriate acoustic visualization displayed on a monitor. This can be useful in environments where the object being examined is at a distance relative to the acoustic imaging device, or when the position of the acoustic visualization on the monitor relative to the actual scene is unclear. In some cases, the laser pointer can communicate with a processor of the acoustic imaging system, and the processor can be additionally configured to cause the laser pointer to emit a laser toward the target scene.

[0286] In some examples, the laser pointer can be visualized on the display, such as through a graphical indicator on the display. Such visualization may include generating a laser pointer point representing the laser pointer in the actual scene (e.g., via a processor). The orientation of the laser pointer can also be enhanced on the display, for example, by using an icon representing the laser pointer in the scene or another aligned display marker to better determine its position on the display relative to the actual scene.

[0287] Figure 34B Example display images including visible light image data and acoustic image data are shown, along with a representative visualization of laser points in the scene. As shown, the display images include electromagnetic image data and acoustic image data, the electromagnetic image data including visible light image data 3420 and the acoustic image data including an indicator 3422 representing sound 3402 within the scene. As shown, the images further include a marker 3424 indicating where laser 3406 is present in the scene. As shown, marker 3424 is aligned with indicator 3422 to indicate the presence of laser 3406 near the source of sound 3402.

[0288] As shown and discussed herein, certain scenes can be difficult to interpret when little or no light illuminates them. To improve the quality of image and video capture, illuminators, such as visible light illuminators or near-infrared emitters, can be used to illuminate the scene. Therefore, a system may additionally or alternatively include one or more illuminators, and a system processor may be configured to illuminate the target scene via the illuminators. Such illuminators may be configured to emit electromagnetic radiation toward the target scene at one or more wavelengths or wavelength ranges (e.g., wavelength bands). In some examples, the system may include a visible light illuminator, such as a flashlight or handlight, to illuminate the scene using light in the visible spectrum. The system may additionally or alternatively include one or more illuminators, such as near-infrared emitters, configured to emit electromagnetic radiation outside the visible spectrum. Such illumination can provide the system with additional capabilities and improved performance for capturing high-quality images and videos of scenes under poor imaging conditions.

[0289] In some examples, electromagnetic imaging tools can be configured to detect electromagnetic radiation (e.g., visible light radiation, near-infrared radiation, etc.) corresponding to electromagnetic radiation emitted from one or more associated illuminators, in order to generate electromagnetic image data representing the illuminated scene. Figures 35A-35C The representation of display images of scenes with various types and / or amounts of illumination is shown, including electromagnetic image data and acoustic image data.

[0290] Figure 35A A display image representing scene 3510 is shown, which includes acoustic image data and electromagnetic image data in the absence of lighting. Figure 35B An image depicting scene 3510 captured using a visible light illuminator. Figure 35C An image depicting scene 3510 captured using a near-infrared illuminator is shown.

[0291] As you can see, Figure 35A The displayed image includes an indicator 3504 representing acoustic image data; however, Figure 35AElectromagnetic image data of scene 3510, which is in an unlit state, is difficult to interpret, thus reducing the added scenes that can be provided by electromagnetic image data.

[0292] If it is possible Figures 35A-35C As can be seen, lighting scene 3510 can help provide better visualization of any point of interest, such as the location associated with an indicator 3504 representing sound signals in and around the scene. Figure 35A In this scenario, there is no illumination. Even if indicator 3504 is clearly displayed in the image representing the detected acoustic signal, it may be difficult to determine certain information about the associated acoustic signal, such as the location or environment around the source of the signal. To better visualize the scene, illuminators (e.g., visible light illuminators, near-infrared illuminators, etc.) can be used to provide more information about the scene. Figure 35B and 35C As shown, the illuminator provides more context to the scene, thus providing more information about the acoustic signals represented in the acoustic image data.

[0293] In some embodiments, the system may include a single illuminator configured to emit electromagnetic radiation across multiple wavelengths, such as electromagnetic radiation across both the visible and near-infrared wavelengths. Alternatively, the system may include multiple illuminators configured to emit electromagnetic radiation of different wavelengths. Similarly, in some examples, the system may include a single electromagnetic imaging tool capable of detecting electromagnetic radiation across multiple wavelengths, such as visible and near-infrared light. Alternatively, multiple electromagnetic imaging tools sensitive to different electromagnetic radiation wavelengths may be used.

[0294] Even if not in Figures 35A-35C As shown, using illuminators of different wavelengths (e.g., visible light, near-infrared) can be advantageous in different scenarios. In some examples, the user can manually switch between visible light illuminators, near-infrared illuminators, both visible and near-infrared illuminators, and no illumination. In some examples, the system can combine one or more illuminators to function as a momentary photographic flash.

[0295] In some embodiments, the system may include an ambient light sensor for collecting information about the ambient light level in a scene. Such light level information may be displayed to a user and / or used by a processor to automatically activate one or more illuminators, such as visible light illuminators and / or near-infrared illuminators, to improve image visualization, display, and capture. In some embodiments, the processor may be configured to illuminate a target scene in response to an ambient light level falling below a predetermined threshold. Additionally or alternatively, the processor may be configured to recommend that the user illuminate the target scene in response to an ambient light level falling below a predetermined threshold. In various examples, illumination from one or more electromagnetic illuminators may be implemented manually and / or automatically.

[0296] In some examples, the distance measuring tool and / or electromagnetic imaging tool may be located within the periphery of the acoustic sensor array. For example, in a handheld tool, various components such as the distance measuring tool, electromagnetic imaging tool, acoustic sensor array, processor, display, laser pointer, one or more illuminators (e.g., visible light illuminators and / or near-infrared illuminators), and / or ambient light sensor may be supported by a single housing. In other examples, one or more components may be located outside other components, such as an external display placed separately from the housing.

[0297] In some examples, the acoustic imaging system described herein may include one or more laser pointers, and in some embodiments, one or more corresponding scanning mechanisms are configured to adjust the direction of the corresponding laser pointer, for example, to manipulate the system laser pointer around the scene.

[0298] By incorporating a laser designator, an acoustic imaging system can accurately locate and align the field of view of a scene using the appropriate displayed sound visualization. This is particularly useful when the object being inspected is some distance from the acoustic imaging device, or when the position of the sound visualization on the display relative to the actual scene is unclear. In some examples, the laser designator can also function as a laser rangefinder.

[0299] In some examples, one or more laser pointers are attached to an electric actuator to assist the scanning process. One or more lasers can be used to help identify one or more locations of interest in a scene. For example, in some embodiments, a user can select an interest point (e.g., selecting a location of interest in acoustic image data via a virtual or physical user interface), and the scanning mechanism (e.g., an electric actuator) can be configured to direct the laser pointer to the selected interest point. In various examples, the user can transmit the selected interest point via a display or a set of controls integrated with the system. Additionally or alternatively, the user can select multiple interest points. In some such examples, the laser pointer can scan among multiple points, and / or multiple laser pointers can be used to identify multiple interest points (e.g., simultaneously). Various implementations can include a single fast-scanning laser design, as well as multiple laser designs.

[0300] In some examples, the acoustic imaging system can automatically identify points of interest, such as sounds that satisfy one or more selected and / or predetermined parameters (such as frequency, decibel level, periodicity, distance, etc.), and direct laser pointers to fire lasers toward the location of interest in the scene. In some examples, one or more laser pointers are directed toward a sound or sound profile based on a pre-programmed set of user-selected parameters or one or more parameters (e.g., parameters that satisfy one or more alarm or threshold conditions).

[0301] In an example where multiple locations of interest exist within the scene, a single laser pointer can be moved between multiple orientations to emit lasers toward each of the multiple locations of interest, thus enabling continuous laser emission toward multiple identified locations. Alternatively, multiple laser pointers can be used to simultaneously emit lasers toward multiple locations of interest. For example, each of the multiple locations of interest can be identified simultaneously, or a subset of the locations of interest can be identified simultaneously.

[0302] In some embodiments, for example, one or more laser pointers can be visualized or enhanced on the display using representative icons or other aligned display markers to better determine their position on the display relative to the actual scene, as discussed elsewhere in this document.

[0303] In some embodiments, one or more lasers may allow a user to select displayed acoustic image data (e.g., one or more locations of interest). This selection provides input that can be processed (e.g., via a system processor) to provide information to the laser system (e.g., one or more lasers and one or more corresponding actuators) about where the laser should be projected to locate an actual sound source in the actual scene. In some examples, this can be achieved using triangulation methods, such as those used when locating and visualizing sound displayed on a device's screen (e.g., when locating acoustic image data and registering the acoustic image data with one or more additional data streams). In some embodiments, information obtained from a laser rangefinder can further refine this location and orientation.

[0304] In some embodiments, in addition to or as an alternative to one or more laser pointers, the acoustic imaging system may include a projection system (e.g., including projector 114) configured to project a visual representation of one or more sets of image data onto a scene, such as projecting acoustic image data onto a surface or area in the scene that emits sound and / or a surface in the scene that reflects sound. The projection system may be attached to the acoustic imaging device and / or may be a separate projector device communicating with the acoustic imaging device. In an exemplary embodiment, the projection system may include a picoprojector embedded in the acoustic imaging device.

[0305] In some examples, the projection system can project acoustic image data onto one or more portions of a scene. In some embodiments, a user can select one or more points of interest in the scene (e.g., based on visualized or otherwise analyzed acoustic image data), wherein the projection system projects appropriate data (e.g., acoustic image data) into the scene. Additionally or alternatively, the system can be configured to automatically project acoustic image data onto the scene at one or more points of interest, such as detected sounds or sound profiles that satisfy one or more predetermined conditions (e.g., selected parameters such as frequency, decibel level, periodicity, distance, etc., satisfy one or more thresholds). In various examples, the projection system can additionally or alternatively be configured to project acoustic image data based on user-selected parameters or pre-programmed algorithms.

[0306] In various embodiments, the projection system is configured to project image data in real time and / or a representation of acoustic image data present on a display. In some examples, the projection system may project a static representation or video loop onto a scene. In some examples, the projection system may be configured to project image data onto one or more locations in the scene at a later time, for example, by using image recognition and alignment techniques. In some embodiments, the projection system may project light toward a determined location of an acoustic signal, which includes color tones representing acoustic parameters of the acoustic signal (e.g., frequency, decibel level, periodicity, distance, etc.).

[0307] As discussed elsewhere in this document, laser pointers can be used to identify sounds in a scene. Depending on the implementation, acoustic imaging tools initially receive acoustic signals and electromagnetic data from the surrounding environment. The acoustic and electromagnetic data can then be transmitted to a processor to generate a display image, which includes, for example, acoustic image data and electromagnetic image data.

[0308] Figure 36 An example implementation is shown where a user can select a location within a scene where the system can point a laser pointer. As illustrated, in an exemplary embodiment, the user can select one of a plurality of indicators presented on display 3616 (corresponding to sounds 3672, 3674, and 3676 respectively) (e.g., via a touch interface on display 3616 and / or one or more physical controls 3614). Figure 36 In the example shown, the user selects indicator 3676 on the display, which prompts the user to manipulate the laser pointer 3604 to point at sound 3686 in the scene (e.g., via a corresponding electric actuator), and fires laser 3606 toward sound 3686. Even if not shown, the user could select different indicators (e.g., 3672 or 3674) to prompt the user to manipulate the laser pointer to point at different sounds in the scene (e.g., sound 3682 or sound 3684).

[0309] In some embodiments, identifying multiple points in a scene may be advantageous. Figure 37 A non-limiting embodiment is shown, in which one or more laser pointers are used to identify three sounds (sounds 3782, 3784, and 3786). About Figure 37The acoustic imaging system 3712 can direct one or more laser pointers 304 to point one or more corresponding lasers 3706 at multiple identified sound sources (e.g., via one or more corresponding electric actuators). In a specific example, the acoustic imaging device identifies sound sources 3772, 3774, and 3776 and directs the laser pointers 3704 to point the lasers 3706 at sounds 3782, 3784, and 3786 in the scene. In some examples, multiple lasers 304 emit lasers 306 toward corresponding locations within the scene. Additionally or alternatively, one or more laser pointers 304 can be configured to move between different orientations and point to different locations at different times.

[0310] In some examples, the acoustic imaging system can be configured to point a laser at the location of each detected sound in the scene. In other examples, the laser can be pointed at the location of a sound having acoustic parameters that satisfy one or more predetermined conditions (e.g., frequency, decibel level, periodicity, distance, etc.). Additionally or alternatively, a user can select multiple sounds (e.g., via a touch interface on display 3716 and / or one or more physical controls 3714). The user's selection of multiple sounds can prompt manipulation of the laser pointer to point at the selected sound in the actual scene (e.g., sounds 3782, 3784, and 3786). In additional embodiments, the user can then select different sounds or different subsets of sounds to prompt manipulation of the laser pointer toward different sounds in the actual scene.

[0311] In some embodiments, in addition to using a laser pointer or as an alternative, it may be advantageous to use a projector system to project acoustic image data from the display image onto the actual scene. Figure 38 A projector system 3818 is shown that projects acoustic image data 3820 toward a location in a display image 3816 based on user selection. As described elsewhere herein, the display image 3816 is generated and presented to the user. As shown, the user selects a location on the display image 3816 corresponding to the location in the scene from which sound 3882 is emitted (e.g., indicator 3872), which prompts the acoustic imaging device 3812 to project acoustic image data 3820 representing sound 3882 onto the location in the scene from which sound 3882 originates. The acoustic imaging device 3812 can project such acoustic image data via a reprojector 3818. Even if not shown, the user can select a different location (e.g., indicator 3874) corresponding to a different sound (e.g., sound 3884) to prompt the system to project data toward the different sound (e.g., sound 3884) in the scene.

[0312] In some embodiments, it may be advantageous to identify multiple points in a scene via a projector system or multiple projector systems. Figure 39 An acoustic imaging device 3912 with a projector system 3918 is shown, which projects acoustic image data 3920A and 3920B representing sounds 3982 and 3984 toward the source of such sounds. In some examples, acoustic image data from locations 3972 and 3974 can be identified in the acoustic image data, and the acoustic image data can be projected onto a corresponding location in the scene associated with sounds 3982 and 3984. In some examples, the projector system can be configured to project acoustic image data toward each location in the scene associated with a detected acoustic signal. In other examples, the acoustic image data can be projected toward a location having an acoustic signal having one or more parameters satisfying one or more predetermined conditions (e.g., frequency, decibel level, periodicity, distance, etc.), and / or the acoustic signal satisfying one or more user-selected parameters or sound profiles based on one or more predetermined conditions.

[0313] In some examples, acoustic image data from locations 3972 and 3974 can be identified based on the generated acoustic image data associated with such locations, and the projector system 3918 projects acoustic image data 3920A and 3920B onto the corresponding locations in the actual scene associated with sounds 3982 and 3984. Additionally or alternatively, a user can select multiple sounds (e.g., via a touch interface and / or one or more controls) to project acoustic image data onto them. The user's selection of multiple sounds can prompt the projector system to direct the acoustic image data to the corresponding locations in the scene, such as acoustic image data 3920A and 3920B projected toward sounds 3982 and 3984. In additional embodiments, the user can select different sounds or different subsets of sounds to prompt the projector system to project acoustic image data toward different sounds in the actual scene.

[0314] As described elsewhere in this document, in various examples, an acoustic imaging system (e.g., a handheld acoustic imaging tool) can be configured to identify the location of one or more acoustic signals in an acoustic scene. In some examples, such a system may include one or more laser pointers and may be configured to project the laser pointer(s) toward locations in the scene corresponding to one or more identified acoustic signals. Additionally or alternatively, in some embodiments, the acoustic imaging system may be configured to determine the distance to a target from one or more locations in the scene, such as by means of a laser rangefinder, passive acoustic triangulation, active acoustic triangulation, or some combination of these.

[0315] For example, regarding Figure 36 , Figure 37And as described elsewhere in this document, the acoustic imaging system can be configured to detect one or more acoustic signals, visualize such sound via live or still video images (e.g., via generating / displaying acoustic image data in a display image), and direct one or more laser pointers toward the sound source, for example, based on one or more user-selected parameters. In some examples, such laser pointers can provide distance information to a target at each of multiple locations in a scene. In some embodiments, the acoustic imaging system can be configured to use one or more of laser distance measurement information, acoustic beamforming triangulation, etc., to calculate the orientation of such a location in volumetric (e.g., XYZ) space.

[0316] The system can be configured to determine a representative area map or representative volume map of the acoustic signal based on the determined location of one or more sound sources, or to determine the reflection based on the determined location and distance of the sound sources.

[0317] In some embodiments, such locations (including information about the distance to the target) can be laid out on a representative map, such as an area map (e.g., an XY-plane projected view) and / or a volume map (e.g., an XYZ three-dimensional representation). In some examples, the user can select from multiple views that show the location of one or more acoustic signal sources.

[0318] In some examples, representative area or volume maps can be stored for later use and / or displayed on the display of an acoustic imaging device. In some embodiments, various display modes are available for visualization of representative area or volume maps. In some examples, a user can manually select or deselect visualizations based on a selected representative area or volume map. For example, in some embodiments, a user can manually define an area or volume from which acoustic signals are visualized while excluding other acoustic signals. Additionally or alternatively, a user can select one or more acoustic signals (e.g., acoustic image data generated as described elsewhere herein) to include in or exclude from the area or volume representation. In some examples, a user can select one or more acoustic signals to define at least a minimum area or volume boundary such that the area or volume map at least includes the location of the selected acoustic signal. In some examples, the acoustic imaging device can be configured to visualize sound within an area or volume based on such sound having one or more parameters that satisfy one or more conditions, such as those relating to frequency, decibel level, periodicity, distance, etc.

[0319] In some embodiments, such as acoustic imaging systems, the system may be configured to determine the three-dimensional contours of an entire scene, but will utilize a “point projection system” using near-infrared, visible light, etc. An exemplary point projection system is described in U.S. Patent Publication No. 20160025993, filed July 28, 2014, entitled “OVERLAPPING PATTERN PROJECTOR”.

[0320] Additionally or alternatively, the system may use any of a variety of methods to determine the three-dimensional contours of the scene, such as those described in U.S. Patent No. 9,729,803, filed March 13, 2014, entitled “APPARATUS AND METHOD FOR MULTISPECTRALIMAGING WITH PARALLAX CORRECTION,” which is assigned to Infrared Integrated Systems, Inc., the relevant portions of which are incorporated herein by reference. In some embodiments, a three-dimensional representation of the scene structure found in the scene may be synchronized with acoustic image data and / or electromagnetic image data and displayed on a monitor or stored for later observation.

[0321] In some examples, users can choose how to view acoustic image data, such as in a volumetric representation with or without associated volumetric image data (e.g., a three-dimensional outline of the determined scene), or in a two-dimensional view from multiple different planar perspectives (e.g., from the perspective of the acoustic sensor array or from an XY “planar view”).

[0322] As described herein, in some embodiments, exemplary acoustic imaging tools may include sensor arrays, laser pointers / laser rangefinders, dot projectors (e.g., for emitting visible light or infrared dot patterns), and strobe lights / flashlights supported by a housing. The acoustic imaging tool may include a display and multiple interface controls (e.g., for receiving input from a user). In some embodiments, the acoustic imaging system includes a projection system configured to reproject image data (e.g., acoustic image data) onto a scene, such as... Figure 38 and 39 As shown, the point pattern is projected onto the scene, for example, to determine the depth profile of the scene.

[0323] Figure 40An exemplary embodiment is illustrated, wherein acoustic image data is mapped to a representative area map, for example, in the XY plane. In the illustrated example, the acoustic imaging system 4012 may (e.g., via a processor) map sounds 4082, 4084, and 4086 from an acoustic scene onto the XY plane to generate a two-dimensional orientation map based on the determined locations, for example, as described elsewhere herein. As shown, sounds 4082, 4084, and 4086 are displayed in the acoustic image data on the display 4016 of the acoustic imaging device 4012 via indicators 4072, 4074, and 4078, respectively. Such data is mapped to two-dimensional locations 4092, 4094, and 4098, respectively. The data mapped to two-dimensional areas may include acoustic image data, for example, color-coded to indicate one or more acoustic parameters, as described elsewhere herein. It will be appreciated that various other coordinate systems, such as polar coordinates, etc., may be used in addition to or as an alternative to the XY plane.

[0324] Understanding this means that a two-dimensional orientation map can be overlaid on a corresponding two-dimensional representation or a three-dimensional illustration and representation of a scene, such as a floor plan, architectural drawing, CAD drawing, etc. In some embodiments, users can rotate and / or zoom in / out on the two-dimensional map of the acoustic image data, and the two-dimensional map of the acoustic image data can be rotated to provide different viewpoints.

[0325] Figure 41 Sounds 4172, 4174, and 4076 are shown, which are mapped onto the XY region, and then the mapped sounds are overlaid onto a planar layout diagram. Such a planar layout diagram can show a top view or location of other components in the region.

[0326] Figure 41 A display 4112 of the acoustic system 4116 is shown, which maps sounds 4172, 4174, and 4176 onto an XY region and then overlays the mapped sounds onto a planar layout diagram 4124. Such a planar layout diagram can show a top view or location of other components in the region.

[0327] In some embodiments, in addition to or instead of two-dimensional maps, mapping sound onto a three-dimensional map such as a volumetric map may be advantageous. Figure 42An exemplary embodiment is illustrated, in which acoustic image data is mapped to a volumetric space. In the illustrated example, the acoustic imaging system 4216 maps sounds 4282, 4284, and 4286 to a volumetric XYZ space. In some examples, such sounds can be viewed in the acoustic image data indicated by indicators 4272, 4274, and 4276, as shown on the display 4216 of the acoustic imaging device 4212. Although the illustrated volumetric mapping is shown in Cartesian coordinates, it will be appreciated that various other coordinate systems, such as cylindrical coordinates, spherical coordinates, etc., can be used.

[0328] In some embodiments, the volumetric coordinates of the acoustic image data can be stored in, for example, memory associated with the acoustic image data and / or the displayed image. In some embodiments, the representation of the acoustic data can be viewed in volumetric representation, such as... Figure 43 The volume shown is a representation of the volume. Figure 43 An acoustic imaging device 4312 is illustrated, including a display 4316 that shows a volumetric representation of acoustic image data, including the three-dimensional orientation of detected sounds, as indicated by indicators 4372, 4374, and 4376. In some examples, such acoustic data is color-coded, as described elsewhere in this document regarding acoustic image data. In some embodiments, a user can rotate or zoom in / out on the volumetric representation to examine the positional relationships of the sound signals. Even without... Figure 42 As shown, a two-dimensional representation of the scene, such as a planar layout diagram, can also be included in the XY plane of the XYZ view (e.g., in...). Figure 41 (in the middle), to provide the added context to the volume display.

[0329] In some embodiments, it may be desirable or necessary to generate a two-dimensional or three-dimensional model of the scene instead of using a predetermined model. Figure 44 An acoustic imaging system 4412 is described, which can use a point projection system 4418 to project a point pattern 4428 onto a scene, for example, using wavelengths in the near-infrared spectrum, visible spectrum, etc. The acoustic imaging device can be configured to analyze the point pattern 4428 to create a three-dimensional model of the scene.

[0330] Furthermore, the acoustic imaging system can be configured to calculate the orientation of the identified sounds 4482, 4484, and 4486 in XYZ space using methods described elsewhere in this document. The acoustic imaging system can combine the volumetric location of the identified sounds with a 3D model of the scene to locate the sounds within the scene. In some examples, the system can be configured to display acoustic image data located within a 3D representation of the scene.

[0331] Figure 45An exemplary display image is shown, which includes a visualization of acoustic image data mapped onto a three-dimensional representation of the scene. In some examples, such as reference... Figure 44 As described elsewhere in this document or via other techniques, point projection can be used to generate a 3D model visualization of a scene. As shown, the acoustic imaging device 4512 includes a display 4516 that shows a volumetric representation of acoustic image data. This volumetric image data includes a 3D representation 4550, which includes a 3D view of electromagnetic image data, such as visible light image data mapped into 3D based on point projection analysis. Indicators 4582, 4584, and 4586 are included in the displayed image at locations corresponding to detected acoustic signals. As described elsewhere in this document, such locations can be determined, for example, via backpropagation techniques, and the acoustic image data can include a color-tuned representation of one or more acoustic parameters.

[0332] Such a display image can help users visualize the location(s) of sound signals, including depth(s), represented by indicators within the scene. In some examples, the 3D representation of the view can be modified, for example, by rotation and / or scaling, to allow users to examine different parts of the scene.

[0333] In various embodiments, the three-dimensional representation as described herein can be generated in real time via an acoustic imaging system such as a handheld acoustic imaging tool. Additionally or alternatively, data collected from the acoustic imaging system (such as acoustic data, distance data to the target, volumetric data representing the volumetric contours of the scene, etc.) can be stored in memory and analyzed / visualized later. In some embodiments, such data can be transmitted to an external device for processing and / or display, such as a smartphone, tablet, computer, or other processing / display device.

[0334] The various processes described herein can be embodied in a non-transitory computer-readable medium comprising executable instructions for causing one or more processors to perform such processes. A system may include one or more processors configured to perform such processes, for example, based on instructions stored integrally with or outside the processor's memory. In some cases, the various components may be distributed throughout the system. For example, a system may include multiple distributed processors, each configured to perform at least a portion of the overall process executed by the system. Additionally, it will be appreciated that the various features and functions described herein can be combined into a single acoustic imaging system, for example, embodied as a handheld acoustic imaging tool or a distributed system having various individual and / or separable components.

[0335] The various functions of the components described herein can be combined. In some embodiments, features described in this application may be combined with features described in a PCT application entitled “SYSTEMS AND METHODS FORTAGGING AND LINKING ACOUSTIC IMAGES”, filed on July 24, 2019, with attorney file number 56581.179.2, which has been assigned to the assignee of this application and is incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in a PCT application entitled “SYSTEMS AND METHODS FOR DETACHABLE AND ATTACHABLE ACOUSTIC IMAGING SENSORS”, filed on July 24, 2019, with attorney file number 56581.180.2, which has been assigned to the assignee of this application and is incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in a PCT application entitled "SYSTEMS AND METHODS FOR ANALYZING AND DISPLAYING ACOUSTICDATA", filed on July 24, 2019, with attorney file number 56581.181.2, which has been assigned to the assignee of this application and is incorporated herein by reference. In some embodiments, features described in this application may be combined with features described in a PCT application entitled "SYSTEMS AND METHODS FOR REPRESENTING ACOUSTICSIGNATURES FROM A TARGET SCENE", filed on July 24, 2019, with attorney file number 56581.182.2, which has been assigned to the assignee of this application and is incorporated herein by reference.

[0336] Various embodiments have been described. Such examples are non-limiting and do not in any way limit or restrict the scope of the invention.

Claims

1. An acoustic inspection system (200), comprising: An acoustic sensor array (202) includes a plurality of acoustic sensor elements, each of which is configured to receive an acoustic signal from a scene and the acoustic sensor array is configured to output acoustic data based on the acoustic signal; A light-emitting device (204) is configured to emit directional electromagnetic radiation; as well as A processor (212) that communicates with an acoustic sensor array and a light-emitting device, the processor being configured to: The origin of the acoustic signal in the scene and the acoustic parameters of the acoustic signal are determined based on acoustic data from an acoustic sensor array. The user receives a selection of a specific acoustic parameter, which is the distance from the acoustic sensor array to the origin of the acoustic signal in the scene; Determine the origin of an acoustic signal in a scene that has a pre-programmed set of acoustic parameters that satisfy a specific set of selected acoustic parameters; The light-emitting device emits electromagnetic radiation toward the origin of an acoustic signal in the scene, which has acoustic parameters that satisfy a pre-programmed set of selected specific acoustic parameters, in order to create an electromagnetic radiation point in the scene corresponding to the origin of the acoustic signal. as well as The electromagnetic radiation point is visualized on the display by using aligned display markers to determine its position on the display relative to the actual scene.

2. The system according to claim 1, wherein, The light-emitting device includes a distance measuring tool (204) configured to provide distance information representing the distance from the acoustic sensor array to the origin of the acoustic signal in the scene.

3. The system according to claim 1 or 2, wherein, The light-emitting device includes: a laser pointer having an adjustable direction controllable by a processor, wherein the processor is configured to cause laser light to be emitted from the laser pointer toward the origin of the acoustic signal in the scene.

4. The system according to claim 3, wherein, The laser pointer comprises only a single laser pointer, and the processor is configured to cause the single laser pointer to continuously point to multiple locations.

5. The system according to claim 1 or 2, wherein, The light-emitting device includes a projector that communicates with a processor and has an adjustable orientation, which can be controlled by the processor, wherein the processor is configured to emit light from the projector toward the source.

6. The system according to claim 5, wherein, The processor is further configured to visualize acoustic parameters associated with acoustic signals in light projected toward the origin.

7. The system according to claim 6, wherein, The light projected toward the origin includes: a color tone representing one or more acoustic parameters associated with the acoustic signal.

8. The system according to claim 5, wherein, The projector is configured to move between multiple orientations, and the processor is configured to cause the projector to emit light toward multiple locations in the scene by moving the projector between the multiple orientations.

9. The system according to claim 1 or 2, wherein, The light-emitting device includes a plurality of projectors, and the processor is configured to cause light to be emitted simultaneously with a corresponding one of the plurality of projectors toward each of a plurality of locations.

10. The system according to claim 1 or 2, further comprising: An electromagnetic imaging tool that communicates with a processor and is configured to receive electromagnetic radiation from a scene and output electromagnetic image data representing the electromagnetic radiation. as well as A display that communicates with a processor; and wherein said processor is further configured to: Receive electromagnetic image data representing the scene from an electromagnetic imaging tool; Generate acoustic image data based on acoustic data; Acoustic image data is combined with electromagnetic image data to generate a display image; as well as The image to be displayed is sent to the monitor.

11. A method for acoustic inspection, comprising: Receive scene-related acoustic signals from the acoustic sensor array (202); Based on the acoustic signals from the acoustic sensor array, determine the origin of the acoustic signals in the scene and the acoustic parameters of the acoustic signals; The user receives a selection of one or more specific acoustic parameters, the specific acoustic parameters being the distance from the acoustic sensor array to the origin of the acoustic signal in the scene; Determine the origin of an acoustic signal in a scene that has acoustic parameters that satisfy the selected specific acoustic parameters; The light-emitting device emits electromagnetic radiation toward the origin of an acoustic signal in the scene, which has acoustic parameters that satisfy a pre-programmed set of selected specific acoustic parameters, in order to create an electromagnetic radiation point in the scene corresponding to the origin of the acoustic signal. as well as The electromagnetic radiation point is visualized on the display by using aligned display markers to determine its position on the display relative to the actual scene.

12. The method of claim 11, further comprising: The origin of an acoustic signal can be determined by triangulation of active or passive acoustic signals.

Images