System and method for generating panoramic acoustic images and virtualizing an acoustic imaging device by segmentation.

Abstract

The acoustic analysis system includes an acoustic sensor array that receives acoustic signals from a target scene and outputs acoustic data to a processor. The processor generates acoustic image data of the target scene and divides the acoustic image data into a plurality of acoustic image portions, including a first acoustic image portion and a second acoustic image portion, the first acoustic image portion being different from the second acoustic image portion. The processor generates a display image by visually contrasting acoustic image data in the first acoustic image portion that meets a first predetermined set of acoustic criteria with acoustic image data in the first acoustic image portion that does not meet the first predetermined set of acoustic criteria, and visually contrasting acoustic image data in the second acoustic image portion that meets a second predetermined set of acoustic criteria with acoustic image data in the second acoustic image portion that does not meet the second predetermined set of acoustic criteria.

Description

[[BACKGROUND ART]]

[0001] When monitoring a certain area using various acoustic imaging methods, it may be necessary or desirable to monitor a wide area. In some cases, this area can be larger than the field of view (FOV) of existing systems and solutions. Additionally or alternatively, it may be necessary or desirable to monitor different regions of the area at different points in time. However, having multiple acoustic imaging systems installed to monitor such different regions may not be technically possible or commercially feasible.

[0002] In some area monitoring applications, it may be necessary to monitor an area where problems only become apparent over a certain period of time. Sometimes these problems can be intermittent and may be missed during periodic inspections.

[0003] Additionally or alternatively, in some cases, sounds emitted from different positions within a scene and / or at different times may have different acoustic parameters, which can make imaging such sounds difficult. Optimal imaging of one such sound (e.g., within a first frequency range) can result in suboptimal detection and / or display of a different sound having different acoustic parameters (e.g., within a second frequency range different from the first frequency range). [[BRIEF DESCRIPTION OF THE DRAWINGS]]

[0004] [Figure 1A] Shows a front view and a rear view of an exemplary acoustic imaging device. [Figure 1B] Shows a front view and a rear view of an exemplary acoustic imaging device. [Figure 2] It is a functional block diagram showing components of an example of an acoustic analysis system. [Figure 3A] Shows a schematic diagram of an exemplary acoustic sensor array configuration in an acoustic analysis system. [Figure 3B] Shows a schematic diagram of an exemplary acoustic sensor array configuration in an acoustic analysis system. [Figure 3C] A schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system is shown. [Figure 3D] A schematic diagram of an exemplary acoustic sensor array configuration within an acoustic analysis system is shown. [Figure 4A] This is a schematic diagram of parallax when generating frames for visible light image data and acoustic image data. [Figure 4B] This is a schematic diagram of parallax when generating frames for visible light image data and acoustic image data. [Figure 5A] This demonstrates parallax correction between visible light images and acoustic images. [Figure 5B] This demonstrates parallax correction between visible light images and acoustic images. [Figure 6] This process flow diagram illustrates an exemplary method for generating a final image by combining acoustic image data and electromagnetic image data. [Figure 7] This is a process flow diagram illustrating an exemplary process for generating acoustic image data from a received acoustic signal. [Figure 8] An exemplary lookup table is provided for determining the appropriate algorithm and sensor array to use during the acoustic imaging process. [Figure 9A] This is an illustrative plot of the frequency components of received image data over time in an acoustic scene. [Figure 9B] This shows an exemplary scene that includes multiple locations where acoustic signals are emitted. [Figure 9C] This shows multiple combinations of acoustic image data and visible light image data in multiple predetermined frequency ranges. [Figure 10A] This is an exemplary display image, including combined visible light image data and acoustic image data. [Figure 10B] This is an exemplary display image, including combined visible light image data and acoustic image data. [Figure 11A] This shows an example plot of frequency versus time for acoustic data in an acoustic scene. [Figure 11B]This shows an example plot of frequency versus time for acoustic data in an acoustic scene. [Figure 12] This shows a visualization of a horizontal panoramic scan using a linear horizontal scanning system. [Figure 13] This provides a simplified example of an acoustic image containing multiple acoustic image frames. [Figure 14] This shows visualizations of the field of view in acoustic and electromagnetic images. [Figure 15] This shows a visualization of a vertical panoramic scan using a pan-tilt system. [Figure 16] This demonstrates a large-area panoramic scanning method that uses a combination of horizontal and vertical panoramic scanning. [Figure 17] This document illustrates an exemplary embodiment of aligning acoustic images using beacon acoustic signals. [Figure 18A] This provides an exemplary embodiment for capturing multiple acoustic images within a target scene using one or more beacon acoustic signals. [Figure 18B] This provides an exemplary embodiment for capturing multiple acoustic images within a target scene using one or more beacon acoustic signals. [Figure 18C] This provides an exemplary embodiment for capturing multiple acoustic images within a target scene using one or more beacon acoustic signals. [Figure 19] This document illustrates an exemplary embodiment of capturing multiple acoustic images within a target scene using one or more beacon acoustic signals. [Figure 20A] This document illustrates an exemplary embodiment showing the display of acoustic signals currently outside the field of view. [Figure 20B] This document illustrates an exemplary embodiment showing the display of acoustic signals currently outside the field of view. [Figure 20C] This document illustrates an exemplary embodiment showing the display of acoustic signals currently outside the field of view. [Figure 21] This shows segmented fields of view (FOV) related to multiple acoustic image portions that provide a virtualized acoustic imaging device. [Modes for carrying out the invention]

[0005] 1A and 1B show a front view and a rear view of an exemplary acoustic imaging device. FIG. 1A shows the front of an acoustic imaging device 100 having a housing 102 that supports an acoustic sensor array 104 and an electromagnetic imaging tool 106. In some embodiments, the acoustic sensor array 104 includes a plurality of acoustic sensor elements, each of the plurality of acoustic sensor elements is configured to (e.g., passively) receive an acoustic signal emitted from an acoustic scene and output acoustic data based on the received acoustic signal. The electromagnetic imaging tool 106 may be configured to receive electromagnetic radiation from a target scene and output electromagnetic image data representing the received electromagnetic radiation. The electromagnetic imaging tool 106 may be configured to detect electromagnetic radiation in one or more of a plurality of wavelength ranges, such as visible light, infrared light, and ultraviolet light.

[0006] In the illustrated example, the acoustic imaging device 100 includes an ambient light sensor 108 and a position sensor 116 such as a GPS. In some embodiments, the device 100 includes a laser pointer 110 including a laser distance meter. The device 100 includes a torch 112 that can be configured to emit visible light toward a scene, and an infrared illuminator 118 that can be configured to emit infrared light toward the scene. In some examples, the device 100 may include an illuminator for illuminating the scene over any wavelength range. The device 100 further includes a projector 114 such as an image reprojector that can be configured to project a generated image such as a colored image onto the scene, and / or a dot projector configured to project a series of dots onto the scene to, for example, determine a depth profile of the scene.

[0007] FIG. 1B shows the back surface of an acoustic imaging device 100. As shown, the device includes a display 120 that can present images or other data. In some examples, the display 120 comprises a touch screen display. The acoustic imaging device 100 includes a speaker that can provide audio feedback signals to a user, and a wireless interface 124 that can enable wireless communication between the acoustic imaging device 100 and an external device. The device further includes a control unit 126 that may include one or more buttons, knobs, dials, switches, or other interface components that enable interfacing between a user and the acoustic imaging device 100. In some examples, the control unit 126 and the touch screen display are combined to provide a user interface for the acoustic imaging device 100.

[0008] In various embodiments, an acoustic imaging device need not include all elements shown in the embodiments of FIGS. 1A and 1B. One or more of the illustrated components may be omitted from the acoustic imaging device. In some examples, one or more components shown in the embodiments of FIGS. 1A and 1B may be included as part of an acoustic imaging system, but may be included separately from the housing 102. Such components can communicate with other components of the acoustic imaging system using wired or wireless communication techniques, for example, using the wireless interface 124.

[0009] Figure 2 is a functional block diagram showing the components of an example acoustic analysis system 200. The exemplary acoustic analysis system 200 in Figure 2 may include multiple acoustic sensors, such as microphones, MEMS, and transducers, arranged within an acoustic sensor array 202 to capture acoustic signals traveling through the air. Such arrays may be one-dimensional, two-dimensional, or three-dimensional. In various examples, the acoustic sensor array can define any appropriate size and shape. In some examples, the acoustic sensor array 202 includes multiple acoustic sensors arranged in a grid pattern, such as an array of sensor elements arranged in vertical columns and horizontal rows. In various examples, the acoustic sensor array 202 may include vertical column × horizontal row arrays, such as 8×8, 16×16, 32×32, 64×64, 128×128, 256×256, etc. Other examples are possible, and various sensor arrays do not necessarily have to include the same number of rows as columns. In some embodiments, such sensors may be located on a substrate, such as a printed circuit board (PCB).

[0010] In the configuration shown in Figure 2, a processor 212 communicating with the acoustic sensor array 202 can receive acoustic data from each of the multiple acoustic sensors. During exemplary operation of the acoustic analysis system 200, the processor 212 can communicate with the acoustic sensor array 202 to generate acoustic image data. For example, the processor 212 may be configured to analyze data received from each of the multiple acoustic sensors located within the acoustic sensor array and determine an acoustic scene by "backpropagating" the acoustic signals to their sources. In some embodiments, the processor 212 can generate digital "frames" of acoustic image data by identifying various source locations and intensities of the acoustic signals across a two-dimensional scene. By generating frames of acoustic image data, the processor 212 captures an acoustic image of the target scene at substantially a given point in time. In some examples, a frame contains multiple pixels that make up the acoustic image, each pixel representing a portion of the source scene from which the acoustic signals have been backpropagated.

[0011] The components described as processors within the acoustic analysis system 200, including processor 212, may be implemented as one or more processors, either individually or in any preferred combination, such as microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and programmable logic circuits. Processor 212 may also include memory for storing program instructions and associated data that, when executed by processor 212, cause the acoustic analysis system 200 and processor 212 to perform functions attributed to them in this disclosure. Examples of memory include any fixed or removable magnetic, optical, or electrical medium, such as RAM, ROM, CD-ROM, hard or floppy disks, and EEPROM. Memory may also include removable memory portions that can be used to update or increase memory capacity. Removable memory may also allow for easy transfer of acoustic image data to another computing device or removal before using the acoustic analysis system 200 for another purpose. Processor 212 may also be implemented as a system-on-a-chip, integrating some or all components of a computer or other electronic system onto a single chip. The processor 212 (processing circuit) may be configured to communicate the processed data to the display 214 or other output / control device 218.

[0012] In some embodiments, the acoustic sensors in the acoustic sensor array 202 generate a series of signals corresponding to the acoustic signals received by each acoustic sensor via the air to represent an acoustic image. A “frame” of the acoustic image data is generated when signals from each acoustic sensor are obtained by scanning all the rows that make up the acoustic sensor array 202. In some examples, the processor 212 can acquire acoustic image frames at a speed (e.g., 30 Hz or 60 Hz) sufficient to generate a video representation of the acoustic image data. Independent of specific circuitry, the acoustic analysis system 200 may be configured to manipulate acoustic data representing the acoustic profile of a target scene to provide an output that can be displayed, stored, transmitted, or otherwise utilized by the user.

[0013] In some embodiments, "backpropagation" of a received acoustic signal to generate acoustic image data includes, for example, analyzing the received signal in multiple acoustic sensors within an acoustic sensor array 202 using a processor. In various examples, performing backpropagation is a function of one or more parameters, such as the distance to the target, frequency, acoustic intensity (e.g., dB level), and the dimensions / configuration of the sensor array, such as the spacing and arrangement of individual sensors in one or more arrays. In some embodiments, such parameters can be pre-programmed into the system, for example, memory. For example, the characteristics of the acoustic sensor array 202 can be stored in memory, such as internal memory or memory specifically associated with the acoustic sensor array 202. Other parameters, such as the distance to the target, can be received in various ways. For example, in some examples, the acoustic analysis system 200 includes a distance measuring tool 204 that communicates with a processor 212. The distance measuring tool may be configured to provide distance information representing the distance from the distance measuring tool 204 to a specific location in the target scene. Various distance measuring tools may include laser rangefinders or other known distance measuring devices, such as other optical or acoustic distance measuring devices. In addition, or alternatively, the distance measuring tool may be configured to generate three-dimensional depth data such that each part of the target scene has a distance value to the relevant target. Thus, in some examples, the distance measurements to the target used herein may correspond to the distance to each position within the target scene. Such three-dimensional depth data may be generated, for example, by multiple imaging tools having different views of the target scene, or by other known distance scanning tools. In general, in various embodiments, the distance measuring tool may be used to perform one or more distance measuring functions, including but not limited to laser distance measuring, active sonic distance measuring, passive ultrasonic distance measuring, LIDAR distance measuring, RADAR distance measuring, millimeter-wave distance measuring, and the like.

[0014] Distance information from the distance measuring tool 204 can be used in backpropagation calculations. In addition, or alternatively, the system 200 may include a user interface 216 that allows the user to manually input distance parameters to a target. For example, the user may input a distance value to the target into the system 200 if the distance to the component suspected of generating the acoustic signal is known or if measurement using the distance measuring tool 204 is difficult.

[0015] In the illustrated embodiment, the acoustic analysis system 200 includes an electromagnetic imaging tool 203 for generating image data representing a target scene. An exemplary electromagnetic imaging tool may be configured to receive electromagnetic radiation from the target scene and generate electromagnetic image data representing the received electromagnetic radiation. In some examples, the electromagnetic imaging tool 203 may be configured to generate electromagnetic image data representing a specific wavelength range in the electromagnetic spectrum, such as infrared radiation, visible light radiation, and ultraviolet radiation. For example, in some embodiments, the electromagnetic timing tool 203 may include one or more camera modules configured to generate image data representing a specific wavelength range in the electromagnetic spectrum, such as a visible light camera module 206.

[0016] Visible light camera modules are generally well known. For example, various visible light camera modules are included in smartphones and many other devices. In some embodiments, the visible light camera module 206 may be configured to receive visible light energy from a target scene and concentrate the visible light energy into a visible light sensor to generate visible light energy data that can be displayed, for example, in the form of a visible light image on a display 214 and / or stored in memory. The visible light camera module 206 may include any suitable components to perform the functions resulting from the module described herein. In the example of Figure 2, the visible light camera module 206 is shown as including a visible light lens assembly 208 and a visible light sensor 210. In some such embodiments, the visible light lens assembly 208 includes at least one lens that captures visible light energy emitted by the target scene and concentrates the visible light energy into the visible light sensor 210. The visible light sensor 210 may include multiple visible light sensor elements, such as a CMOS detector, a CCD detector, a PIN diode, or an avalanche photodiode. The visible light sensor 210 responds to concentrated energy by generating an electrical signal that can be converted and displayed as a visible light image on the display 214. In some examples, the visible light module 206 is user-configurable and can provide output to the display 214 in various formats, for example. The visible light camera module 206 may include compensation functions for various lighting or other operating conditions or user settings. The visible light camera module may provide digital output such as image data that may contain data in various formats (e.g., RGB, CYMK, YCbCr, etc.).

[0017] In some exemplary visible light camera module 206 operation, light energy received from the target scene may pass through the visible light lens assembly 208 and be concentrated in the visible light sensor 210. When light energy enters the visible light sensor element of the visible light sensor 210, photons in the photodetector are emitted and can be converted into a detection current. The processor 212 can process this detection current to form a visible light image of the target scene.

[0018] During use of the acoustic analysis system 200, the processor 212 can control the visible light camera module 206 to generate visible light data from the captured target scene in order to create a visible light image. The visible light data may include luminosity data indicating the color associated with different parts of the captured target scene and / or the magnitude of light associated with different parts of the captured target scene. The processor 212 can generate a "frame" of visible light image data by measuring the response of each visible light sensor element of the acoustic analysis system 200 once. By generating a frame of visible light data, the processor 212 captures a visible light image of the target scene at a given point in time. The processor 212 may repeatedly measure the response of each visible light sensor element of the acoustic analysis system 200 to generate a dynamic visible light image (e.g., a video representation) of the target scene. In some examples, the visible light camera module 206 may include its own dedicated processor or other circuitry (e.g., ASIC) that can operate the visible light camera module 206. In some such embodiments, the dedicated processor communicates with the processor 212 to provide visible light image data (e.g., RGB image data) to the processor 212. In alternative embodiments, the dedicated processor for the visible light camera module 206 may be integrated into the processor 212.

[0019] Since each sensor element of the visible light camera module 206 functions as a sensor pixel, the processor 212 can generate a two-dimensional visible light image or picture representation from a target scene by converting the electrical response of each sensor element into a time-division multiplexed electrical signal that can be processed, for example, for visualization on the display 214 and / or storage in memory.

[0020] The processor 212 can control the display 214 to display at least a portion of the visible light image of the captured target scene. In some examples, the processor 212 controls the display 214 so that the electrical response of each sensor element of the visible light camera module 206 is associated with a single pixel on the display 214. In other examples, the processor 212 can increase or decrease the resolution of the visible light image so that the pixels displayed on the display 214 are greater than or less than the number of sensor elements in the visible light camera module 206. The processor 212 can control the display 214 to display the entire visible light image (e.g., all parts of the target scene captured by the acoustic analysis system 200) or less than the entire visible light image (e.g., less of the entire target scene captured by the acoustic analysis system 200).

[0021] In some embodiments, the processor 212 may control the display 214 to simultaneously display at least a portion of the visible light image captured by the acoustic analysis system 200 and at least a portion of the acoustic image generated by the acoustic sensor array 202. Such simultaneous display may be useful in that it may help an operator to refer to features displayed in the visible light image to see the source of the acoustic signal simultaneously displayed in the acoustic image. In some cases, the processor 212 is configured to recognize one or more features in the electromagnetic (e.g., visible light) image data and, based on the one or more recognized features, to specify (identify or depict) at least a portion of the acoustic image data. In various examples, the processor 212 may control the display 214 to display the visible light image and the acoustic image in a parallel arrangement, in a picture-in-picture arrangement where one image surrounds the other, or in any other preferred arrangement in which the visible light image and the acoustic image are displayed simultaneously.

[0022] For example, the processor 212 may control the display 214 to display a visible light image and an acoustic image in a combined arrangement. In such an arrangement, for a pixel or set of pixels in the visible light image that represents a portion of a target scene, there exists a corresponding pixel or set of pixels in the acoustic image that represents a substantially identical portion of the target scene. In various embodiments, the acoustic image and the visible light image do not need to be the same size and / or resolution. Therefore, there may be a set of pixels in one of the acoustic or visible light images, or a set of pixels of different sizes, that corresponds to a single pixel in the other acoustic or visible light image. Similarly, there may be pixels in one of the visible light or acoustic images that correspond to a set of pixels in the other image. Therefore, as used herein, correspondence does not require a one-to-one pixel relationship, but may include pixels or groups of pixels of mismatched sizes. Various combination techniques for mismatched image regions, such as upsampling or downsampling one of the images, or combining a pixel with the average value of the corresponding set of pixels, can be performed. Other examples are known and are within the scope of this disclosure.

[0023] Therefore, corresponding pixels do not need to have a direct one-to-one relationship. Rather, in some embodiments, a single acoustic pixel has multiple corresponding visible light pixels, or a visible light pixel has multiple corresponding acoustic pixels. In addition, or alternatively, in some embodiments, not all visible light pixels have corresponding acoustic pixels, and vice versa. Such embodiments may, for example, represent picture-in-picture type displays as described above. Therefore, visible light pixels do not necessarily have the same pixel coordinates in a visible light image, as do corresponding acoustic pixels. Thus, as used herein, corresponding pixels generally refer to pixels from any image (e.g., a visible light image, an acoustic image, a composite image, a display image, etc.) that contains information from substantially the same portion of the target scene. Such pixels do not need to have a one-to-one relationship between images, nor do they need to have similar coordinate positions within each image.

[0024] Similarly, images having corresponding pixels (i.e., pixels representing identical parts of the target scene) are sometimes called corresponding images. Thus, in some such arrangements, corresponding visible light and acoustic images may be superimposed on each other at their corresponding pixels. The operator can interact with the user interface 216 to control the transparency or opacity of one or both of the images displayed on the display 214. For example, the operator can interact with the user interface 216 to adjust the acoustic image between completely transparent and completely opaque, and also adjust the visible light image between completely transparent and completely opaque. Such exemplary combined arrangements may be called alpha-blended arrangements, and may allow the operator to adjust the display 214 to display an acoustic-only image, a visible light-only image, or any overlapping combination of two images between the two extremes of an acoustic-only image and a visible light-only image. The processor 212 can also combine scene information with other data, such as alarm data. Generally, alpha-blended combinations of visible light and acoustic images may include either 100 percent acoustic and 0 percent visible light to 0 percent acoustic and 100 percent visible light. In some embodiments, the amount of blending can be adjusted by the camera user. Thus, in some embodiments, the blended image can be adjusted between 100 percent visible light and 100 percent sound.

[0025] In addition, in some embodiments, the processor 212 can interpret and execute commands from the user interface 216 and / or the output / control device 218. Furthermore, input signals can be used to modify the processing of visible light and / or acoustic image data performed by the processor 212.

[0026] The operator may interact with the acoustic analysis system 200 via a user interface 216, which may include buttons, keys, or other mechanisms for receiving user input. The operator may receive output from the acoustic analysis system 200 via a display 214. The display 214 may be configured to display acoustic images and / or visible light images in any acceptable palette or color scheme, the palette may change, for example, in response to user control. In some embodiments, acoustic image data may be presented in a palette to represent varying magnitudes of acoustic data from different locations in a scene. For example, in some examples, the display 214 is configured to display acoustic images in a monochrome palette, such as grayscale. In other examples, the display 214 is configured to display acoustic images in a color palette, such as amber, ironbow, blue-red, or other high-contrast color schemes. A combination of grayscale and color palette display is also possible. In some examples, a display configured to display such information may include processing capabilities for generating and presenting such image data. In other examples, being configured to display such information may include the ability to receive image data from other components, such as a processor 212. For example, the processor 212 may generate a value (e.g., an RGB value, a grayscale value, or other display option) for each pixel to be displayed. The display 214 may receive such information and map each pixel to a visual display.

[0027] The processor 212 can control the display 214 to simultaneously display at least a portion of the acoustic image and at least a portion of the visible light image in any preferred arrangement, but picture-in-picture arrangement can help the operator easily focus on and / or interpret the acoustic image by displaying the corresponding visible images of the same scene adjacently.

[0028] A power supply (not shown) provides operating power to various components of the acoustic analysis system 200. In various examples, the power supply may include a rechargeable or non-rechargeable battery and a power generation circuit, AC power, an inductive power pickup, a photovoltaic power supply, or any other suitable power supply component. Combinations of power supply components are also possible, such as a rechargeable battery, another component configured to supply power to operate the device and / or to charge the rechargeable battery.

[0029] During the operation of the acoustic analysis system 200, the processor 212 controls the acoustic sensor array 202 and the visible light camera module 206 with the help of instructions related to program information stored in memory to generate visible light and acoustic images of the target scene. The processor 212 further controls the display 214 to display the visible light and / or acoustic images generated by the acoustic analysis system 200.

[0030] As described above, in some situations, it may be difficult to identify and distinguish the actual (visible) features of the target scene within the acoustic image. In addition to supplementing the acoustic image with visible light information, in some embodiments, it may be useful to highlight visible edges within the target scene. In some embodiments, known edge detection methods can be performed on the visible light image of the target scene. Due to the correspondence between the acoustic image and the visible light image, visible light pixels determined to represent visible edges in the target scene correspond to acoustic pixels that also represent visible edges in the acoustic image. As used herein, “edge” does not necessarily refer to the physical boundary of an object, but can refer to any sufficiently clear gradient within the visible light image. Examples include the physical boundary of an object, color changes within an object, and shading of the entire scene.

[0031] With respect to Figure 2, the system has been described in general terms as including a visible light camera module 206, but in some examples, the electromagnetic imaging tool 203 of the acoustic analysis system 200 may, in addition or alternatively, include imaging tools capable of generating image data representing various spectra. For example, in various examples, the electromagnetic imaging tool 203 may include one or more tools capable of generating infrared image data, visible light image data, ultraviolet image data, or any other useful wavelength, or a combination thereof. In some embodiments, the acoustic imaging system may include an infrared camera module having an infrared lens assembly and an infrared sensor array. For example, it may include additional components for interfacing with the infrared camera module, such as those described in U.S. Patent Application No. 14 / 837,757, filed on 27 August 2015, titled "EDGE ENHANCEMENT FOR THERMAL-VISIBLE COMBINED IMAGES AND CAMERAS," which has been assigned to the assignee of this application and is incorporated herein by reference in its entirety.

[0032] In some examples, two or more data streams may be blended and displayed. For example, an exemplary system including a visible light camera module 206, an acoustic sensor array 202, and an infrared camera module (not shown in Figure 2) may be configured to produce an output image that includes a blend of visible light (VL) image data, infrared (IR) image data, and acoustic image data. In the exemplary blend scheme, the displayed image may be represented as α × IR + β × VL + γ × Acoustic, where α + β + γ = 1. Generally, any number of data streams can be combined and displayed. In various embodiments, the blend ratios, such as α, β, and γ, are user-configurable. In addition, or alternatively, the configured display program may be configured to include different image data streams based on alarm conditions (e.g., one or more values ​​in one or more data streams satisfy a predetermined threshold) or other conditions, as described in U.S. Patent No. 7,538,326, titled "VISIBLE LIGHT AND IR COMBINED IMAGE CAMERA WITH A LASER POINTER," which is, for example, assigned to the assignee of this application and incorporated herein in its entirety by reference.

[0033] One or more components of the acoustic analysis system 200 described with respect to Figure 2 may be included in a portable (e.g., handheld) acoustic analysis tool. For example, in some embodiments, the portable acoustic analysis tool may include a housing 230 configured to house the components within the acoustic analysis tool. In some examples, one or more components of the system 200 may be located outside the housing 230 of the acoustic analysis tool. For example, in some embodiments, a processor 212, a display 214, a user interface 216, and / or an output control device 218 may be located outside the housing of the acoustic analysis tool and can communicate with various other system components, for example, via wireless communication (e.g., Bluetooth communication, Wi-Fi). Such components outside the acoustic analysis tool may be provided via external devices such as a computer, smartphone, tablet, or wearable device. In addition, or alternatively, other test and measurement or data acquisition tools configured to act as master or slave devices to the acoustic analysis tool can similarly provide various components of the acoustic analysis system outside the acoustic analysis tool. External devices can communicate with the portable acoustic analysis tool via wired and / or wireless connections and can be used to perform various processing, display, and / or interface steps.

[0034] In some embodiments, such external devices can provide redundant functionality as components housed within the portable acoustic analysis tool. For example, in some embodiments, the acoustic analysis tool may include a display for displaying acoustic image data and may be further configured to communicate the image data to the external device for storage and / or display. Similarly, in some embodiments, a user may interface with the acoustic analysis tool via an application ("app") running on a smartphone, tablet, computer, etc., to perform one or more functions that could also be performed by the acoustic analysis tool itself.

[0035] Figure 3A is a schematic diagram of an exemplary configuration of an acoustic sensor array in an acoustic analysis system. In the illustrated example, the acoustic sensor array 302 includes a plurality of first acoustic sensors (shown in white) and a plurality of second acoustic sensors (shaded). The first acoustic sensors are located in the first array 320, and the second acoustic sensors are located in the second array 322. In some examples, the first array 320 and the second array 322 may be used selectively to receive acoustic signals transmitted through the air, for example, passively, and generate corresponding acoustic image data. For example, in some configurations, the sensitivity of a particular acoustic sensor array to a particular acoustic frequency is a function of the distance between the acoustic sensor elements.

[0036] In some configurations, sensor elements that are more closely spaced apart (e.g., a second array 322) can resolve higher frequency acoustic signals (e.g., sounds with frequencies above 20 kHz, such as ultrasonic signals between 20 kHz and 100 kHz) better than sensor elements that are more far apart (e.g., a first array 320). Similarly, sensor elements that are more far apart (e.g., a first array 320) may be better suited to detecting lower frequency acoustic signals (e.g., <20 kHz) than sensor elements that are more closely spaced apart (e.g., a second array 322). Various acoustic sensor arrays can be provided with sensor elements spaced apart from each other to detect acoustic signals in various frequency ranges, such as ultra-low frequencies (<20 Hz), audible frequencies (approximately 20 Hz to 20 kHz), and ultrasonic frequencies (20 kHz to 100 kHz). In some embodiments, a partial array can be used to optimize detection of a specific frequency band (e.g., every other acoustic sensor from array 320).

[0037] Furthermore, in some examples, certain acoustic sensor elements may be better suited to detecting acoustic signals with different frequency characteristics, such as low or high frequencies. Therefore, in some embodiments, an array configured to detect low-frequency acoustic signals, such as a first array 320 having more widely spaced sensor elements, may include first acoustic sensor elements better suited to detecting low-frequency acoustic signals. Similarly, an array configured to detect higher-frequency acoustic signals, such as a second array 322, may include second acoustic sensor elements better suited to detecting high-frequency acoustic signals. Therefore, in some examples, the first array 320 and the second array 322 of acoustic sensor elements may include different types of acoustic sensor elements. Alternatively, in some embodiments, the first array 320 and the second array 322 may include the same type of acoustic sensor elements.

[0038] Therefore, in exemplary embodiments, the acoustic sensor array 302 may include multiple acoustic sensor element arrays, such as a first array 320 and a second array 322. In some embodiments, the arrays may be used individually or in combination. For example, in some examples, the user may choose to use the first array 320, the second array 322, or both the first and second arrays 322 simultaneously to perform an acoustic imaging procedure. In some examples, the user may select which array to use via a user interface. In addition, or alternatively, in some embodiments, the acoustic analysis system may automatically select the array to use based on an analysis of the received acoustic signal or other input data, such as an expected frequency range. The configuration shown in Figure 3A includes two arrays (the first array 320 and the second array 322) arranged in a rectangular grid, but it should be understood that multiple acoustic sensor elements can be grouped into any number of separate arrays of any shape. Furthermore, in some embodiments, one or more acoustic sensor elements may be included in multiple separate arrays that can be selected for operation. As described elsewhere in this specification, in various embodiments, the process for backpropagating acoustic signals to establish acoustic image data from a scene is based on the arrangement of acoustic sensor elements. Therefore, the arrangement of acoustic sensors may be known or otherwise accessible by the processor for performing the acoustic image generation method.

[0039] The acoustic analysis system in Figure 3A further includes a distance measuring tool 304 positioned within the acoustic sensor array 302 and a camera module 306. The camera module 306 may represent a camera module of an electromagnetic imaging tool (e.g., 203) and may include a visible light camera module, an infrared camera module, an ultraviolet camera module, and the like. Furthermore, although not shown in Figure 3A, the acoustic analysis system may include one or more additional camera modules of the same or different type as the camera module 306. In the illustrated example, the distance measuring tool 304 and the camera module 306 are positioned within the grid of acoustic sensor elements in the first array 320 and the second array 322. Although shown as being positioned between grid sites in the first array 320 and the second array 322, in some embodiments, one or more components (e.g., the camera module 306 and / or the distance measuring tool 304) may be positioned at one or more corresponding grid sites in the first array 320 and / or the second array 322. In some such embodiments, the components may be positioned at the grid sites instead of the acoustic sensor elements which would typically be located in such positions according to the grid arrangement.

[0040] As described elsewhere in this specification, an acoustic sensor array may include acoustic sensor elements arranged in any of a variety of configurations to receive acoustic signals transmitted through the air from an acoustic source located in or near a target scene. Figures 3B and 3C are schematic diagrams showing exemplary acoustic sensor array configurations. Figure 3B shows an acoustic sensor array 390 including a plurality of acoustic sensor elements evenly spaced in a substantially square grid. A distance measuring tool 314 and a camera array 316 are located within the acoustic sensor array 390. In the illustrated example, the acoustic sensor elements in the acoustic sensor array 390 are of the same type, but in some embodiments, different types of acoustic sensor elements may be used within the array 390.

[0041] Figure 3C shows several acoustic sensor arrays. Acoustic sensor arrays 392, 394, and 396 each contain multiple acoustic sensor elements arranged in arrays of different shapes. In the example in Figure 3C, acoustic sensor arrays 392, 394, and 396 can be used separately or together in any combination to create sensor arrays of various sizes. In the illustrated embodiment, the sensor elements of array 396 are spaced closer together than the sensor elements of array 392. In some examples, array 396 is designed to sense higher frequency acoustic signals, and array 392 is designed to sense lower frequency acoustic signals.

[0042] In various embodiments, arrays 392, 394, and 396 may include the same or different types of acoustic sensor elements. For example, acoustic sensor array 392 may include sensor elements having a lower frequency operating range than the sensor elements of acoustic sensor array 396.

[0043] As described elsewhere in this specification, in some examples, different acoustic sensor arrays (e.g., 392, 394, 396) can be selectively turned on and off during various operating modes (e.g., different desired frequencies being imaged). In addition, or alternatively, various acoustic sensor elements (e.g., some or all acoustic sensor elements in one or more sensor arrays) can be enabled or disabled according to the desired system operation. For example, in some acoustic imaging processes, data from a large number of sensor elements (e.g., densely arranged sensor elements, such as in sensor array 396) slightly improves the acoustic image data resolution, but at the expense of the processing required to extract acoustic image data from the data received by each sensor element. That is, in some examples, the increased processing requirements (e.g., in terms of cost, processing time, power consumption, etc.) required to process a large number of input signals (e.g., from a large number of acoustic sensor elements) are inferior to any additional signal resolution provided by the additional data stream. Therefore, in some embodiments, it may be worthwhile to disable or ignore data from one or more acoustic sensor elements depending on the desired acoustic imaging operation.

[0044] Similar to the systems in Figures 3A and 3B, the system in Figure 3C includes a distance measuring tool 314 and a camera array 316 located within acoustic sensor arrays 392, 394, and 396. In some examples, additional components, such as an additional camera array (e.g., used for imaging different parts of the electromagnetic spectrum from camera array 316), may be similarly located within the acoustic sensor arrays 392, 394, and 396. While Figures 3A to 3C show them located within one or more acoustic sensor arrays, it will be understood that the distance measuring tool and / or one or more imaging tools (e.g., a visible light camera module, an infrared camera module, an ultraviolet sensor, etc.) may be located outside the acoustic sensor arrays. In some such examples, the distance measuring tool and / or one or more imaging tools located outside the acoustic sensor arrays may be supported by an acoustic imaging tool, for example, by a housing that houses the acoustic sensor arrays, or they may be located outside the housing of the acoustic imaging tool.

[0045] Figures 3A to 3C provide exemplary acoustic sensor array configurations, but other acoustic sensor array configurations may also be used. Acoustic sensor array configurations can include various shapes and patterns, such as acoustic sensors arranged in a grid pattern, concentric circles, sunflower-shaped arrays, and regular circular arrays. Figure 3D provides an exemplary embodiment in which acoustic sensors 395 are arranged in a regular circular array configuration. A regular circular array as shown in Figure 3D can provide varying distances between acoustic sensors rather than uniform distances between them. Such assurance can help the acoustic sensor array identify a wider range of frequencies by utilizing the varying portions of the acoustic sensor array. For example, acoustic sensors that are placed closer together and further apart may be more effective in detecting higher frequencies. Similarly, acoustic sensors that are placed further apart may be more effective in detecting lower frequencies. Therefore, having varying distances between acoustic sensors (some closer together, some further apart) can help provide better detection for a wider range of frequencies. Furthermore, the regular circular array in Figure 3D can provide differences in arrival times relative to the sound location for different parts of the array, and thus can be useful for focusing the acoustic sensor array and / or determining the emission location of the acoustic signal. In addition, the regular circular array can be useful for resolving spatial aliasing and side lobes compared to, for example, a rectangular array.

[0046] In some cases, general misalignment between an acoustic sensor array and an imaging tool such as a camera module can lead to misalignment in the corresponding image data generated by the acoustic sensor array and the imaging tool. Figure 4A shows a schematic diagram of parallax when generating visible light image data and acoustic image data frames. Generally, the parallax can be vertical, horizontal, or both. In the illustrated embodiment, an imaging tool comprises an acoustic sensor array 420 and a visible light camera module 406. A visible light image frame 440 is shown as being captured according to the field of view 441 of the visible light camera module 406, and an acoustic image frame 450 is shown as being captured according to the field of view 451 of the acoustic sensor array 420.

[0047] As shown in the figure, the visible light image frame 440 and the acoustic image frame 450 are not aligned with each other. In some embodiments, a processor (e.g., processor 212 in Figure 2) is configured to manipulate one or both of the visible light image frame 440 and the acoustic image frame 450 in order to align the visible light image data and the acoustic image data. Such manipulation may include shifting one image frame relative to the other. The amount by which the image frames are shifted relative to each other may be determined based on various factors, such as the distance from the visible light camera module 406 and / or the acoustic sensor array 420 to the target. Such distance data can be determined, for example, using a distance measuring tool 404 or by receiving distance values ​​via a user interface (e.g., 216).

[0048] Figure 4B is a schematic diagram similar to Figure 4A, but includes a visible light image of the scene. In the example of Figure 4B, the visible light image 442 shows a scene of multiple power lines and support towers. The acoustic image 452 includes multiple locations 454, 456, and 458, and shows a large amount of acoustic data from these locations. As shown, both the visible light image 442 and the acoustic image 452 are displayed simultaneously. However, upon observation of both images, at least one acoustic image maxima is shown at location 458, which does not appear to coincide with any particular structure in the visible light image 442. Therefore, an observer of both images can conclude that there is a misalignment (e.g., parallax) between the acoustic image 452 and the visible light image 442.

[0049] Figures 5A and 5B illustrate parallax correction between a visible light image and an acoustic image. Figure 5A, similar to Figure 4B, shows the visible light image 542 and the acoustic image 552. The acoustic image 552 contains maxima at positions 554, 556, and 558. As can be seen from the figure, the maxima at positions 554 and 558 do not appear to coincide with any structure in the visible light image. In the example of Figure 5B, the visible light image 542 and the acoustic image 552 are aligned with respect to each other. Here, the maxima at positions 554, 556, and 558 in the acoustic image appear to coincide with various positions in the visible light image 542.

[0050] During use, the operator can determine the approximate location within the visible scene 542 that is likely to be the source of the received acoustic signal by looking at the representation in Figure 5B (for example, via the display 214). Such a signal may be further processed to determine information about the acoustic signatures of various components within the scene. In various embodiments, acoustic parameters such as frequency components, periodicity, and amplitude can be analyzed with respect to various locations within the acoustic image. If such parameters are overlaid on visible light data so that they can be associated with various system components, the acoustic image data may be used to analyze various characteristics (e.g., performance characteristics) of objects within the visible light image.

[0051] As shown in Figures 5A and 5B, positions 554, 556, and 558 exhibit a circular gradient. As described elsewhere in this specification, the acoustic image data may be presented visually according to a palettetization scheme in which each pixel of the acoustic image data is colored based on the acoustic intensity at the corresponding position. Thus, in the exemplary representations of Figures 5A and 5B, the circular gradient at positions 554, 556, and 558 generally represents a gradient of acoustic intensity on the imaging plane based on the backpropagated received acoustic signal.

[0052] While the illustrative figures in Figures 4A, 4B, 5A, and 5B illustrate acoustic image data and visible light image data, it will be understood that such processes can be similarly carried out using various electromagnetic image data. For example, as described elsewhere in this specification, in various embodiments, various such processes can be carried out using a combination of acoustic image data and one or more of the following: visible light image data, infrared image data, ultraviolet image data, etc.

[0053] As described elsewhere in this specification, in some embodiments, the backpropagation of an acoustic signal to form an acoustic image may be based on a distance value to a target. That is, in some examples, the backpropagation calculation may be distance-based and include determining a two-dimensional acoustic scene located at that distance from an acoustic sensor array. Given a two-dimensional imaging plane, spherical sound waves emitted from a wave source within that plane generally appear to have a circular cross-section and their intensity attenuates radially, as shown in Figures 5A and 5B.

[0054] In some such examples, portions of the acoustic scene representing data not located at the target distance used in the backpropagation calculation introduce errors into the acoustic image data, such as inaccuracies in the position of one or more sounds within the scene. Such errors can lead to parallax between the acoustic image data and other image data when the acoustic image data is displayed simultaneously with other image data (e.g., electromagnetic image data such as visible light, infrared, or ultraviolet image data) (e.g., blended, combined, etc.). Therefore, in some embodiments, several techniques for correcting parallax (e.g., as shown in Figures 5A and 5B) include adjusting the target distance values ​​used in the backpropagation calculation to generate the acoustic image data.

[0055] In some cases, the system may be configured to perform a backpropagation process using the distance value to the first target, and to display an image such as that shown in Figure 5A, where the acoustic image data and another data stream may not be aligned. The acoustic analysis system can then adjust the distance value to the target used for backpropagation, re-run the backpropagation, and update the display image with the new acoustic image data. This process can be repeated, and while the user observes the resulting display image on the display, the acoustic analysis system sequentially displays multiple distance values ​​to the targets. As the distance value to the targets changes, the user may observe a gradual transition from the display image shown in Figure 5A to the display image shown in Figure 5B. In some such cases, the user may visually observe a point in time when the acoustic image data appears to be properly aligned with another data stream, such as electromagnetic image data. The user may inform the acoustic analysis system that the acoustic image data appears to be properly aligned, indicating that the distance value to the target used to perform the most recent backpropagation is approximately correct, and that distance value can be stored in memory as the correct distance to the target. Similarly, the user may manually adjust the distance to the target as the displayed image is updated with the new distance values ​​in the updated backpropagation process, until the user observes that the acoustic image data is properly aligned. The user may choose to save the current distance to the target as the correct distance to the target in the acoustic analysis system.

[0056] In some examples, parallax correction may involve adjusting the position of acoustic image data relative to other image data (e.g., electromagnetic image data) by a predetermined amount and direction based on distance data to the target. In some embodiments, such adjustment is independent of the generation of acoustic image data by backpropagating the acoustic signal to the specified target distance.

[0057] In some embodiments, the distance value to a target may be used for other decisions in addition to being used to generate acoustic image data and to reduce parallax between the acoustic image data and other image data. For example, in some examples, as described in U.S. Patent No. 7,538,326 incorporated by reference, a processor (e.g., processor 212) may use the distance value to a target to focus on an image such as an infrared image, or to assist the user in focusing. As described in the said patent, this can similarly be used to correct parallax between visible light image data and infrared image data. Thus, in some examples, the distance value can be used to align acoustic image data with electromagnetic imaging data such as infrared image data and visible light image data.

[0058] As described elsewhere in this specification, in some examples, the distance measuring tool (e.g., distance measuring tool 204) is configured to generate acoustic image data and provide distance information that can be used by a processor (e.g., processor 212) for alignment. In some embodiments, the distance measuring tool comprises a laser rangefinder configured to emit light onto a target scene at a location where the distance is to be measured. In some such examples, the laser rangefinder may emit light in the visible spectrum so that a user can view a laser spot in the physical scene to confirm that the rangefinder is measuring the distance to a desired portion of the scene. In addition, or alternatively, the laser rangefinder is configured to emit light in the spectrum to which one or more imaging components (e.g., camera modules) are sensitive. Thus, a user viewing the target scene via an analysis tool (e.g., via a display 214) can observe a laser spot in the scene to confirm that the laser is measuring the distance to the correct location in the target scene. In some examples, the processor (e.g., 212) may be configured to generate a reference mark in the display image representing the position where the laser spot will be located within the acoustic scene, based on the current distance value (e.g., based on a known distance-based parallax relationship between the laser rangefinder and the acoustic sensor array). The position of the reference mark may be compared (e.g., graphically on the display and / or physically within the target scene), and the scene may be adjusted until the reference mark and the laser coincide. Such a process may be performed in a similar manner to the infrared alignment and focusing techniques described in U.S. Patent No. 7,538,326, which is incorporated by reference.

[0059] Figure 6 is a process flow diagram illustrating an exemplary method for generating a final image by combining acoustic image data and electromagnetic image data. The method includes the steps of receiving an acoustic signal via an acoustic sensor array (680) and receiving distance information (682). The distance information may be received, for example, via a distance measuring device and / or a user interface (such as via manual input or as a result of a distance adjustment process in which the distance is determined based on observed alignment).

[0060] The method further includes backpropagating the received acoustic signal to determine acoustic image data representing an acoustic scene (684). As described elsewhere in this specification, backpropagation may include analyzing multiple acoustic signals received by multiple sensor elements in an acoustic sensor array in combination with received distance information to determine the source pattern of the received acoustic signal.

[0061] The method in Figure 6 further includes the steps of capturing electromagnetic image data (686) and aligning acoustic image data with electromagnetic image data (688). In some embodiments, aligning acoustic image data with electromagnetic image data is performed as part of a backpropagation step (684) for generating acoustic image data. In other examples, aligning acoustic image data with electromagnetic image data is performed separately from the generation of acoustic image data.

[0062] The method in Figure 6 includes the step (690) of combining acoustic image data with electromagnetic image data to generate a display image. Combining electromagnetic image data and acoustic image data as described elsewhere in this specification may include alpha blending of the electromagnetic image data and acoustic image data. Combining image data may include overlaying one image dataset on the other in picture-in-picture mode or at locations where specific conditions (e.g., alarm conditions) are met. The display image may be presented to the user, for example, via a display supported by a housing supporting the acoustic sensor array and / or via a display separate from the sensor array, such as the display of an external device (e.g., a smartphone, tablet, computer, etc.).

[0063] In addition, or alternatively, the display image may be stored in local (e.g., onboard) memory and / or remote memory for future viewing. In some embodiments, the stored display image may include metadata that allows for future adjustment of the display image characteristics, such as blend ratio, backpropagation distance, or other parameters used to generate the image. In some examples, raw acoustic signal data and / or electromagnetic image data may be stored along with the display image for subsequent processing or analysis.

[0064] While Figure 6 demonstrates combining acoustic and electromagnetic image data as a method for generating a final image, it will be understood that the method can be used to combine acoustic image data with one or more image datasets spanning any portion of the electromagnetic spectrum, such as visible light image data, infrared image data, and ultraviolet image data. In some such examples, multiple image datasets, such as visible light image data and infrared image data, can all be combined with acoustic image data to generate a display image using a method similar to that described with respect to Figure 6.

[0065] In some examples, receiving an acoustic signal via a sensor array (680) may include the step of selecting an acoustic sensor array to receive the acoustic signal. For example, as described with respect to Figures 3A to 3C, the acoustic analysis system may include multiple acoustic sensor arrays which may be suitable for analyzing acoustic signals of fluctuating frequencies. In addition, or alternatively, in some examples, different acoustic sensor arrays may be useful for analyzing acoustic signals propagating from different distances. In some embodiments, different arrays can be nested one another. In addition, or alternatively, partial arrays can be selectively used to receive acoustic image signals.

[0066] For example, Figure 3A shows a first array 320 and a second array 322 nested within the first array. In an exemplary embodiment, the first array 320 may include (e.g., spaced apart) sensor arrays configured to receive acoustic signals and generate acoustic image data for frequencies within a first frequency range. The second array 322 may include a second sensor array configured to be used alone or in combination with all or part of the first array 320 to generate acoustic image data for frequencies within a second frequency range, for example.

[0067] Similarly, Figure 3C shows a first array 392, a second array 394 at least partially nested within the first array 392, and a third array 396 at least partially nested within the first array 392 and the second array 394. In some embodiments, the first array 392 may be configured to receive acoustic signals and generate acoustic image data for frequencies within a first frequency range. The second array 394 may be used in whole or in part with the first array 392 to receive acoustic signals and generate acoustic image data for frequencies within a second frequency range. The third array 396 may be used alone, in whole or in part with the second array 394, and / or in whole or in part with the first array 392 to receive acoustic signals and generate acoustic image data for frequencies within a third frequency range.

[0068] In some embodiments, in a nested array configuration, acoustic sensor elements from one array may be positioned between acoustic sensor elements, such as elements of a third array 396 located between elements of a first array 392. In some such examples, acoustic sensor elements in a nested array (e.g., a third array 396) may be positioned coplanar with, in front of, or behind, acoustic sensor elements in a nested array (e.g., a first array 392).

[0069] In various implementations, arrays used to sense higher frequency acoustic signals generally require shorter distances between individual sensors. Therefore, with respect to Figure 3C, for example, the third array 396 may be better suited for performing acoustic imaging processes involving higher frequency acoustic signals. Other sensor arrays (e.g., the first array 392) may be sufficient for performing acoustic imaging processes involving lower frequency signals and, compared to array 396, may be used to reduce the computational requirements for processing signals from fewer acoustic sensor elements. Thus, in some examples, high-frequency sensor arrays may be nested within low-frequency sensor arrays. As described elsewhere in this specification, such arrays can generally operate individually (e.g., by switching between active arrays) or together.

[0070] In addition to selecting an appropriate sensor array based on the expected / desired frequency spectrum for analysis, or alternatively, in some examples, different sensor arrays may be better suited to performing the acoustic imaging process at different distances to the target scene. For example, in some embodiments, when the distance between the acoustic sensor array and the target scene is small, the outer sensor elements within the acoustic sensor array may receive significantly less useful acoustic information from the target scene than the more centrally located sensor elements.

[0071] On the other hand, when the distance between the acoustic sensor array and the target scene is large, acoustic sensor elements that are placed close together but also spaced far apart may not provide separately useful information. That is, if the first and second acoustic sensor elements are close together and the target scene is generally far away, the second acoustic sensor element may not provide any information that is significantly different from that of the first acoustic sensor element. Therefore, the data streams from such first and second sensor elements are redundant and may unnecessarily consume processing time and resources for analysis.

[0072] As described elsewhere in this specification, the distance to the target can be used not only when determining which sensor array may be optimal for performing acoustic imaging, but also when performing backpropagation to determine acoustic image data from the received acoustic signal. However, in addition to being an input value to the backpropagation algorithm, the distance to the target can be used to select an appropriate backpropagation algorithm to use. For example, in some cases, at long distances, a spherically propagating sound wave can be approximated as substantially planar compared to the size of the acoustic sensor array. Therefore, in some embodiments, when the distance to the target is large, the backpropagation of the received acoustic signal may involve acoustic beamforming calculations. However, when closer to the source of the sound wave, the planar approximation of the sound wave may not be appropriate. Therefore, different backpropagation algorithms, such as near-field acoustic holography, may be used.

[0073] As mentioned above, the distance metric to the target can be used in various ways in the acoustic imaging process, such as determining the active sensor array, determining the backpropagation algorithm, executing the backpropagation algorithm, and / or aligning the resulting acoustic image with electromagnetic image data (e.g., visible light, infrared, etc.). Figure 7 is a process flow diagram illustrating an exemplary process for generating acoustic image data from received acoustic signals.

[0074] The process in Figure 7 includes, for example, receiving distance information from a distance measuring device or distance information input via a user interface (780). The method further includes the step (782) of selecting one or more acoustic sensor arrays to perform acoustic imaging based on the received distance information. As described, in various examples, the selected arrays may include a single array, a combination of multiple arrays, or a part of one or more arrays.

[0075] The method in Figure 7 further includes the step (784) of selecting a processing scheme for performing acoustic imaging based on the received distance information. In some examples, selecting a processing scheme may include selecting a backpropagation algorithm to generate acoustic image data from the acoustic signal.

[0076] After selecting an acoustic sensor array to perform acoustic imaging (782) and selecting a processing scheme (784), the method includes the step of receiving acoustic signals through the selected acoustic sensor array (786). The received acoustic signals are then backpropagated using distance and the selected processing scheme to determine acoustic image data (788).

[0077] In various embodiments, the steps in Figure 7 may be performed by a user, an acoustic analysis system (e.g., via processor 212), or a combination thereof. For example, in some embodiments, the processor may be configured to receive distance information via a distance measuring tool and / or user input (780). In some embodiments, for example, if the distance to an object is known and / or difficult to analyze using the distance measuring tool (e.g., the object is small and / or the distance to the target is large), the user may input a value to override the measured distance and use it as distance information. The processor may further be configured to automatically select an appropriate acoustic sensor array for performing acoustic imaging based on the received distance information, for example, using a lookup table or other database. In some embodiments, selecting an acoustic sensor array includes enabling and / or disabling one or more acoustic sensor elements to obtain the desired acoustic sensor array.

[0078] Similarly, in some examples, the processor may be configured to automatically select a processing scheme (e.g., a backpropagation algorithm) to perform acoustic imaging based on received distance information. In some such examples, this may involve selecting one from a number of known processing schemes stored in memory. In addition, or alternatively, selecting a processing scheme may ultimately involve adjusting parts of a single algorithm to arrive at the desired processing scheme. For example, in some embodiments, a single backpropagation algorithm may include multiple terms and variables (e.g., based on distance information). In some such examples, selecting a processing scheme (784) may involve defining one or more values ​​in a single algorithm, such as adjusting the coefficients of one or more terms (e.g., setting various coefficients to 0 or 1).

[0079] Therefore, in some embodiments, the acoustic imaging system can automate several steps of the acoustic imaging process by suggesting and / or automatically implementing a selected acoustic sensor array and / or processing scheme (e.g., a backpropagation algorithm) based on received distance data. This enables speeding up, improving, and simplifying the acoustic imaging process, eliminating the requirement for an acoustic imaging specialist to perform the process. Thus, in various examples, the acoustic imaging system can automatically implement such parameters, notify the user that such parameters are about to be implemented, request permission from the user to implement such parameters, and suggest such parameters for manual input by the user.

[0080] Automatic selection and / or suggestion of such parameters (e.g., processing scheme, sensor array) may be useful for optimizing the localization of acoustic image data with respect to other forms of image data, processing speed, and analysis of acoustic image data. For example, as described elsewhere in this specification, accurate backpropagation determination (e.g., using appropriate algorithms and / or accurate distance metrics) can reduce parallax between acoustic image data and other (e.g., electromagnetic) image data such as visible light and infrared light. In addition, the accuracy of thermal image data can be optimized by using appropriate algorithms and / or sensor arrays, such as those that can be automatically selected or suggested by the acoustic analysis system, enabling analysis of the received acoustic data.

[0081] As mentioned above, in some examples, the acoustic analysis system may be configured to automatically select an algorithm and / or sensor array to perform an acoustic imaging process based on the received distance information. In some such embodiments, the system includes a lookup table, for example, stored in memory, to determine which of a plurality of backpropagation algorithms and acoustic sensor arrays should be used to determine the acoustic image data. Figure 8 shows an exemplary lookup table for determining the appropriate algorithm and sensor array to use during the acoustic imaging process.

[0082] In the illustrated example, the lookup table in Figure 8 contains N columns, each column representing a different array, i.e., Array 1, Array 2, ..., Array N. In various examples, each array comprises a unique set of arranged acoustic sensor elements. Different arrays may include sensor elements arranged in a grid (e.g., Arrays 392 and Array 396 in Figure 3C). Arrays in the lookup table may also include combinations of sensor elements from one or more such grids. Generally, in some embodiments, each array, i.e., Array 1, Array 2, ..., Array N, corresponds to a unique combination of acoustic sensor elements. Some of these combinations may include an entire set of sensor elements arranged in a particular grid, or a subset of sensor elements arranged in a particular grid. Any of the various combinations of acoustic sensor elements is a possible choice for use as a sensor array in the lookup table.

[0083] The lookup table in Figure 8 further contains M rows, each row representing a different algorithm, i.e., algorithm 1, algorithm 2, ..., algorithm M. In some examples, different algorithms may involve different processes for performing backpropagation analysis of the received acoustic signal. As described elsewhere in this specification, in some examples, several different algorithms may be similar to one another, but have different coefficients and / or terms for correcting the backpropagation results.

[0084] The exemplary lookup table in Figure 8 contains M × N entries. In some embodiments, an acoustic analysis system using such a lookup table is configured to analyze received distance information and classify it into one of M × N bins, each bin corresponding to an entry in the lookup table in Figure 8. In such an example, when the acoustic analysis system receives distance information, the system can find the entry (i,j) in the lookup table corresponding to the bin in which the distance information resides and determine an appropriate algorithm and sensor array to use during the acoustic imaging process. For example, if the received distance information corresponds to a bin associated with entry (i,j), the acoustic analysis system may automatically use algorithm i and array j for the acoustic imaging process, or suggest using them.

[0085] In various such examples, distance information bins can correspond to distance ranges of uniform size, for example, the first bin corresponding to a distance of less than 1 foot, and the second bin corresponding to a distance between 1 and 2 feet. In other examples, the bins do not need to correspond to distance spans of uniform size. Furthermore, in some embodiments, fewer than M × N bins may be used. For example, in some embodiments, there may be an algorithm (e.g., algorithm x) that is never used in a particular array (e.g., array y). Therefore, in such examples, there will be no corresponding distance information bin for an entry (x,y) in the M × N lookup table.

[0086] In some embodiments, statistical analysis of additional distance bins may be used to identify the most common distance or distance range within the target scene. In some such embodiments, a distance bin having the maximum number of corresponding locations (e.g., the maximum number of locations with acoustic signals) may be used as distance information in the process shown in Figure 7. That is, in some embodiments, the acoustic sensor array and / or processing scheme used may be implemented and / or recommended based on statistical analysis of the distance distribution of various objects within the target scene. This can increase the likelihood that the sensor array and / or processing scheme used for acoustic imaging of the scene is suitable for the maximum number of locations within the acoustic scene.

[0087] In addition, or alternatively, parameters other than distance information can be used to select a suitable sensor array and / or processing scheme for use in generating acoustic image data. As described elsewhere in this specification, various sensor arrays may be configured to be sensitive to specific frequencies and / or frequency bands. In some examples, similar different backpropagation calculations can be used according to different acoustic signal frequency components. Thus, in some examples, one or more parameters can be used to determine the processing scheme and / or acoustic sensor array.

[0088] In some embodiments, an acoustic analysis system can be used to first analyze various parameters for processing / analyzing the received acoustic signal. Referring again to Figure 7, a method for generating acoustic image data may include the step (790) of analyzing the frequency components of the received signal after receiving the acoustic signal (786). In some such examples, if an acoustic sensor array and / or processing scheme has been selected (e.g., via steps 782 and / or 784, respectively), the method may include, for example, the step (792) of updating the selected array and / or the selected processing scheme based on the analyzed frequency components.

[0089] After updating the sensor array and / or processing scheme, the method can perform various actions using the updated parameters. For example, if the selected sensor array is updated based on the analyzed frequency components (790) (792), a new acoustic signal may be received from the (newly) selected acoustic sensor array (786), and then backpropagated to determine acoustic image data (788). Alternatively, if the processing scheme is updated in 792, the captured acoustic signal may be backpropagated according to the updated processing scheme to determine updated acoustic image data. If both the processing scheme and the sensor array are updated, a new acoustic signal may be received using the updated sensor array and backpropagated according to the updated processing scheme.

[0090] In some embodiments, the acoustic analysis system can receive frequency information (778) without analyzing the frequency components of the received acoustic signal (790). For example, in some examples, the acoustic analysis system can receive information about a desired or expected frequency range for future acoustic analysis. In some such examples, the desired or expected frequency information can be used to select one or more sensor arrays and / or processing schemes that best fit the frequency information. In some such examples, the steps of selecting an acoustic sensor array (782) and / or selecting a processing scheme (784) may be based on the received frequency information in addition to, or instead of, the received distance information.

[0091] In some examples, the received acoustic signal (e.g., received via an acoustic sensor element) may be analyzed by, for example, the processor of an acoustic analysis system (e.g., 210). Such analysis may be used to determine one or more characteristics of the acoustic signal, such as frequency, intensity, periodicity, apparent proximity (e.g., distance estimated based on the received acoustic signal), measured proximity, or any combination thereof. In some examples, the acoustic image data may be filtered to show only the acoustic image data representing an acoustic signal having, for example, a specific frequency component, periodicity, etc. In some examples, any number of such filters may be applied simultaneously.

[0092] As described elsewhere in this specification, in some embodiments, a series of frames of acoustic image data may be captured over time, similar to acoustic video data. In addition, or alternatively, in some examples, acoustic signals are repeatedly sampled and analyzed even when acoustic image data is not repeatedly generated. Thus, parameters of the acoustic data, such as frequency, can be monitored over time, with or without repeated generation of acoustic image data (e.g., video).

[0093] Figure 9A is an illustrative plot of the frequency components of received image data over time in an acoustic scene. As shown, the acoustic scene represented by the plot in Figure 9A generally contains four persistent frequencies over time, labeled as frequency 1, frequency 2, frequency 3, and frequency 4. Frequency data, such as the frequency components of the target scene, can be determined by processing the received acoustic signal, for example, using the Fast Fourier Transform (FFT) or other known frequency analysis methods.

[0094] Figure 9B shows an exemplary scene including multiple locations that emit acoustic signals. In the illustrated image, acoustic image data is combined with visible light image data to show acoustic signals present at locations 910, 920, 930, and 940. In some embodiments, the acoustic analysis system is configured to display acoustic image data for any detected frequency range. For example, in the exemplary embodiment, location 910 includes acoustic image data including frequency 1, location 920 includes acoustic image data including frequency 2, location 930 includes acoustic image data including frequency 3, and location 940 includes acoustic image data including frequency 4.

[0095] In some such examples, displaying acoustic image data representing a frequency range is a selectable mode of operation. Similarly, in some embodiments, the acoustic analysis system is configured to display acoustic image data representing only frequencies within a given frequency band. In some such examples, displaying acoustic image data representing a given frequency range involves selecting one or more acoustic sensor arrays to receive the acoustic signals from which the acoustic image data is generated. Such arrays may be configured to receive a selective frequency range. Similarly, in some examples, one or more filters can be used to limit the frequency components used to generate the acoustic image data. In addition, or alternatively, in some embodiments, acoustic image data containing information representing a wide range of frequencies may be analyzed and displayed only if the acoustic image data satisfies a certain condition (e.g., falls within a given frequency range).

[0096] Figure 9C shows multiple combined acoustic image data and visible light image data in multiple predetermined frequency ranges. The first image includes acoustic image data at a first position 910 containing a frequency component of frequency 1. The second image includes acoustic image data at a second position 920 containing a frequency component of frequency 2. The third image includes acoustic image data at a third position 930 containing a frequency component of frequency 3. The fourth image includes acoustic image data at a fourth position 940 containing a frequency component of frequency 4.

[0097] In an exemplary embodiment, the user may select various frequency ranges, such as a range including frequency 1, frequency 2, frequency 3, or frequency 4, in order to filter out acoustic image data that represents frequency components other than the selected frequency range. Thus, in such an example, as a result of the user selecting the desired frequency range, one of the first, second, third, or fourth images may be displayed.

[0098] In addition, or alternatively, in some examples, the acoustic analysis system may sequentially display multiple display images, each having different frequency components. For example, with respect to Figure 9C, in an exemplary embodiment, the acoustic analysis system may sequentially display the first, second, third, and fourth images, as indicated by the arrows in Figure 9C.

[0099] In some examples, the displayed image may include text or other indications representing the frequency components displayed within the image, allowing the user to observe which locations within the image contain acoustic image data representing specific frequency components. For example, with respect to Figure 9C, each image may show a text representation of the frequency represented in the acoustic image data. With respect to Figure 9B, an image showing multiple frequency ranges may include indications of frequency components at each location containing acoustic image data. In some such examples, the user may select a location within the image, for example, via a user interface, to view the frequency components present at that location in the acoustic scene. For example, the user may select a first location 910, and the acoustic analysis system may present the frequency component (e.g., frequency 1) at the first location. Thus, in various examples, the user can use the acoustic analysis system to analyze the frequency components of an acoustic scene by viewing the locations in the scene corresponding to specific frequency components and / or by viewing the frequency components present at various locations.

[0100] During exemplary acoustic imaging operations, filtering acoustic image data by frequency can help reduce image clutter from background or other unimportant sounds, for example. In exemplary acoustic imaging procedures, the user may want to remove background noise, such as floor noise in an industrial environment. In some such cases, background noise may consist mainly of low-frequency noise. Therefore, the user may choose to display acoustic image data that represents acoustic signals greater than a given frequency (e.g., 10 kHz). In another embodiment, the user may want to analyze a specific object that emits acoustic signals generally within a certain range, such as corona discharge from a transmission line (e.g., as shown in Figures 5A and 5B). In such an example, the user may select a specific frequency range for acoustic imaging (e.g., 11 kHz to 14 kHz in the case of corona discharge).

[0101] In some examples, an acoustic analysis system can be used to analyze and / or present information related to the intensity of a received acoustic signal. For example, in some embodiments, backpropagating a received acoustic signal may include determining acoustic intensity values ​​at multiple locations within an acoustic scene. In some examples, as with frequencies, acoustic image data is included in the display image only if the intensity of the acoustic signal meets one or more predetermined requirements.

[0102] In various such embodiments, the displayed image may include acoustic image data representing an acoustic signal above a predetermined threshold (e.g., 15 dB), an acoustic signal below a predetermined threshold (e.g., 100 dB), or an acoustic signal within a predetermined intensity range (e.g., 15 dB to 40 dB). In some embodiments, the threshold may be based on a statistical analysis of the acoustic scene, such as being above or below the standard deviation from the average acoustic intensity.

[0103] Similar to what was described above regarding frequency information, in some embodiments, restricting acoustic image data to represent acoustic signals that satisfy one or more intensity requirements may include filtering the received acoustic signals so as to generate acoustic image data using only received signals that satisfy predetermined conditions. In other examples, the acoustic image data is filtered to adjust the acoustic image data to be displayed.

[0104] In addition, or alternatively, in some embodiments, the acoustic intensity at multiple locations within an acoustic scene can be monitored over time (e.g., in conjunction with a video acoustic image representation, or by background analysis without necessarily updating the displayed image). In some such examples, a predetermined requirement for displaying acoustic image data may include the amount or rate of change in acoustic intensity at a certain location within the image.

[0105] Figures 10A and 10B are exemplary display images including combined visible light image data and acoustic image data. Figure 10A shows a display image including acoustic image data indicated at several positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090. In some examples, intensity values ​​can be palette-displayed, for example, acoustic intensity values ​​are assigned colors based on a predetermined palette display scheme. In exemplary embodiments, intensity values ​​may be classified according to intensity ranges (e.g., 10dB to 20dB, 20dB to 30dB, etc.). Each intensity range may be associated with a specific color according to the palette display scheme. Acoustic image data may include multiple pixels, each pixel being colored with a color associated with the intensity range in which the intensity represented by the pixel of the acoustic image data falls. In addition to, or instead of, distinction by color, different intensities may be distinguished according to other properties, such as transparency (e.g., in image overlays where acoustic image data is overlaid on other image data).

[0106] Additional parameters, such as the rate of change of sound intensity, may also be displayed in a palette. Similar to intensity, the fluctuating rate of change of sound intensity may be displayed in a palette so that parts of the scene showing various rates and / or amounts of change in sound intensity are displayed in different colors.

[0107] In the illustrated example, the acoustic image data is paletted according to an intensity palette, such that acoustic image data representing different acoustic signal intensities is shown in different colors and / or shades. For example, the acoustic image data at positions 1010 and 1030 shows a paletted representation of a first intensity, positions 1040, 1060, and 1080 show a paletted representation of a second intensity, and positions 1020, 1050, 1070, and 1090 show a paletted representation of a third intensity. As shown in the illustrative representation in Figure 10A, each position showing a paletted representation of the acoustic image data exhibits a circular pattern with a color gradient extending outward from the center. This may be due to the attenuation of acoustic intensity as the signal propagates from the acoustic signal source.

[0108] In the example shown in Figure 10A, acoustic image data is combined with visible light image data to generate a display image, which can be presented to the user, for example, via a display. The user may view the display image in Figure 10A to see the location in the visible scene where the acoustic signal is being emitted, and the intensity of such signal. Thus, the user can quickly and easily observe the location of the sound source and compare the intensity of sounds originating from various locations in the scene.

[0109] As described with respect to frequency elsewhere in this specification, in some embodiments, acoustic image data may be presented only when the corresponding acoustic signal satisfies a predetermined intensity condition. Figure 10B is similar to the display image in Figure 10A and shows an exemplary display image including visible light image data and acoustic images representing an acoustic signal exceeding a predetermined threshold. As shown, of the positions 1010, 1020, 1030, 1040, 1050, 1060, 1070, 1080, and 1090 in Figure 10A that include acoustic image data, only positions 1020, 1050, 1070, and 1090 include acoustic image data representing an acoustic signal that satisfies a predetermined condition.

[0110] In an exemplary scenario, Figure 10A may include all acoustic image data exceeding the noise floor threshold at each of the positions 1010–990, while Figure 10B shows the same scene as Figure 10A, but only includes acoustic image data with an intensity greater than 40 dB. This can help a user identify sound sources within the environment (e.g., within the target scenes in Figures 10A and 10B) that contribute to a particular sound (e.g., the loudest sound in the scene).

[0111] As described elsewhere in this specification, in addition to, or instead of, a direct comparison with an intensity threshold (e.g., 40 dB), in some such examples, a predetermined requirement for displaying acoustic image data may include the amount or rate of change in acoustic intensity at a location within the image. In some such examples, acoustic image data may be presented only if the rate or amount of change in acoustic intensity at a given location satisfies a predetermined condition (e.g., greater than a threshold, less than a threshold, within a predetermined range, etc.). In some embodiments, the amount or rate of change in acoustic intensity may be and may be displayed as intensity acoustic image data or in conjunction with it in a palette display. For example, in an exemplary embodiment, if the rate of change is used as a threshold for determining the location to include acoustic image data, the acoustic image data may include an intensity rate of change metric in a palette display for display.

[0112] In some examples, the user may manually set intensity requirements (e.g., minimum, maximum, range, rate of change, amount of change, etc.) for the displayed acoustic image data. As described elsewhere in this specification, including only acoustic image data that meets the intensity requirements can be achieved during the generation of the acoustic image data (e.g., by filtering the received acoustic signals) and / or by not displaying the generated acoustic image data that represents acoustic signals that do not meet the set requirements. In some such examples, filtering the display image according to intensity values ​​may be performed after the acoustic and visible light image data have been captured and stored in memory. That is, the data stored in memory can be used to generate a display image that includes any number of filtering parameters, such as displaying only acoustic image data that meets predetermined intensity conditions.

[0113] In some cases, setting a lower limit on the intensity within the acoustic image (e.g., displaying only acoustic image data representing acoustic signals above a certain intensity) can exclude unwanted background or ambient noise and / or sound reflections from the acoustic image data. In other cases, setting an upper limit on the intensity within the acoustic image (e.g., displaying only acoustic image data representing acoustic signals below a certain intensity) can exclude expected loud sounds from the acoustic image data in order to observe acoustic signals that would normally be masked by such loud sounds.

[0114] Several display functions are possible. For example, similar to the frequency analysis / display described with respect to Figure 9C, in some examples, the acoustic analysis system can sequentially display multiple display images, each showing acoustic image data that meets different intensity requirements. Similarly, in some examples, the user can scroll through a series of acoustic intensity ranges to view the location within the acoustic image data that has an acoustic intensity within a given range.

[0115] Another parameter that can be used in the analysis of acoustic data is the periodicity of the acoustic signal. Figures 11A and 11B show exemplary frequency-versus-time plots of acoustic data in an acoustic scene. As shown in the plot in Figure 11A, the acoustic data includes a signal at frequency X with a first periodicity, a signal at frequency Y with a second periodicity, and a signal at frequency Z with a third periodicity. In the illustrated example, acoustic signals with different frequencies may also contain different periodicities in the acoustic signals.

[0116] In some such cases, the acoustic signal may be filtered based on periodicity in addition to, or instead of, its frequency components. For example, in some cases, multiple sources of acoustic signals within an acoustic scene may generate acoustic signals at specific frequencies. If the user wishes to isolate one such sound source for acoustic imaging, the user may choose to include or exclude the acoustic image data from the final display image based on the periodicity associated with the acoustic data.

[0117] Figure 11B shows a frequency-versus-time plot of an acoustic signal. As shown, the frequency increases almost linearly over time. However, as also shown, the signal exhibits a nearly constant periodicity over time. Therefore, such a signal may or may not appear in the acoustic image, depending on the selected display parameters. For example, a signal may meet the frequency criteria for display at one point in time, but be outside the frequency range for display at another point in time. However, the user can choose to include or exclude such a signal from the acoustic image data based on the periodicity of the signal, regardless of its frequency components.

[0118] In some cases, extracting acoustic signals with a specific periodicity can be useful when analyzing a particular part of a target scene (e.g., a specific piece of equipment or a type of equipment that typically operates with a certain periodicity). For example, if an object of interest operates with a specific periodicity (e.g., once per second), the acoustic analysis of the object of interest can be improved by excluding signals with different periodicities. For example, referring to Figure 11B, if an object of interest operates with a periodicity of 4, the analysis of the object of interest can be improved by isolating signals with a periodicity of 4 for analysis. For example, as shown in Figure 11B, the object of interest emits a sound with a periodicity of 4, but this frequency may increase. This may suggest that the properties of the object may be changing (e.g., an increase in torque or load) and should be investigated.

[0119] In an exemplary acoustic imaging process, background noise (e.g., floor noise in an industrial environment, wind in an outdoor environment) is generally not periodic, but a specific object of interest in the scene emits periodic acoustic signals (e.g., a machine operating at regular intervals). Therefore, the user may choose to exclude non-periodic acoustic signals from the acoustic image to remove the background noise and more clearly present the acoustic data of interest. In another example, the user may be trying to identify a source of a constant tone and therefore choose to exclude periodic signals from the acoustic image data that could obscure the constant tone. Generally, the user may choose to include acoustic signals above, below, or within a desired periodic range in the acoustic image data. In various examples, periodicity can be defined by either the interval between periodic signals or the frequency of periodic signals. Similar to the frequency shown in Figure 11B, an analysis of intensity at a given periodicity (e.g., due to an object of interest operating at that periodicity) can be used to track changes in acoustic signals from an object over time. In general, in some embodiments, periodicity can be used to perform rate-of-change analysis of various parameters such as frequency and intensity.

[0120] In some cases, users may desire to capture multiple overlapping acoustic images. For example, in some cases, users may want to highlight different types of acoustic signals in various parts of a target scene, such as acoustic signals containing different acoustic parameters (e.g., frequency differences, sound intensity (e.g., dB level), distance to the target, periodicity, etc.). In addition, or alternatively, in some cases, users may desire to capture acoustic images representing an acoustic scene larger than the acoustic field of view (FOV) of the acoustic sensor array in use (e.g., acoustic sensor array 102). Capturing multiple acoustic images can provide information about parts of the target scene outside the FOV of a single such image. When viewed together, the acoustic image data from multiple such acoustic images can provide acoustic information for a larger target scene as a whole. Therefore, capturing multiple acoustic images can provide a virtual FOV larger than the individual FOV of the individual acoustic images when viewed together. To capture multiple acoustic images, panoramic scanning of the scene by the acoustic sensor array may be used, such as vertical panoramic scanning, horizontal panoramic scanning, or a combination thereof.

[0121] Panoramic scans can be captured by adjusting the position and / or orientation of the acoustic analysis system to adjust the field of view (FOV) of the acoustic sensor array. In some embodiments, the acoustic analysis system can be operated by a user, such as by being handheld or by being positioned on an adjustable stand (e.g., tripod, monopod, etc.). In addition, or alternatively, other systems such as linear movement systems, pan-tilt systems, and 360° rotation systems may be used. In some embodiments, such a movement system may be configured to support a housing containing the acoustic sensor array and / or an electromagnetic imaging tool, and may be configured to operate the housing so that the FOV of the acoustic sensor array and / or the electromagnetic imaging tool changes. In addition, or alternatively, in some examples, the acoustic analysis system may include an acoustic sensor array positioned on the movement system so that the array can be operated independently of the housing (e.g., pan, tilt, rotate, etc.). In various examples, the electromagnetic imaging tool and the acoustic sensor array may be fixedly coupled to each other so that operation of one (e.g., by pan, tilt, rotate) results in a similar operation of the other. In other examples, electromagnetic imaging tools and acoustic sensor arrays are not coupled to each other so that one such component can be operated independently of the other (e.g., pan, tilt, or rotate).

[0122] In some embodiments, the system may include a moving mechanism configured to operate an electromagnetic imaging tool and / or an acoustic sensor array to adjust its field of view (FOV). The moving mechanism may include, for example, an electric moving mechanism including a motor configured to move the electromagnetic imaging tool and / or acoustic sensor array (e.g., pan, tilt, rotate, etc.). In such embodiments, the electric moving mechanism may be automated or programmed to periodically adjust the FOV of the acoustic sensor array, for example, between capturing acoustic image data from a scene. Furthermore, the electric moving mechanism may be used to periodically scan a target scene (e.g., capture multiple acoustic images) based on a schedule or input from a controller or user. In embodiments including a schedule, the schedule may be automated by a processor or the like (e.g., processor 212, external processor, etc.).

[0123] Figure 12 provides an exemplary embodiment of an acoustic analysis system that uses panoramic scanning to capture multiple acoustic images of a target scene 1210. In some cases, the field of view (FOV) of the acoustic sensor array may not capture the entire target scene 1210 that is to be analyzed. In some embodiments, as shown in Figure 12, multiple acoustic images with different fields of view, such as acoustic images 1220-1228, may be captured to fully capture the data contained in the entire desired acoustic target scene 1210.

[0124] In some embodiments, one or more electromagnetic images of a target scene (e.g., target scene 1210) may be captured. For example, the electromagnetic images may be captured to correspond to one or more of the acoustic images 1220-1228. In such embodiments, one or more electromagnetic images may be blended with the acoustic images and displayed on a display, as described elsewhere in this specification. Furthermore, in some embodiments, the system may not capture any electromagnetic images, or may not capture any electromagnetic images that represent a different field of view than the corresponding acoustic image, such as a larger or smaller field of view.

[0125] Referring to Figure 12, acoustic images 1220-1228 may be captured by the same acoustic analysis system (e.g., acoustic imaging device 100, acoustic analysis system 200, etc.), or one or more such acoustic images may be captured by using multiple acoustic analysis systems. In some embodiments, the acoustic analysis system is adjusted before each acoustic image is captured to capture different parts of the target scene. Such adjustments may be made by the user or by an electrically operated moving mechanism as described herein. For example, acoustic image 1220 may be captured, and then the acoustic analysis system may be panned, tilted, and / or rotated to capture acoustic image 1222. Similarly, the acoustic analysis system may be adjusted to capture each of the other acoustic images (e.g., acoustic images 1224, 1226, and 1228).

[0126] Two or more acoustic images (e.g., acoustic images 1220-1228) may include overlapping portions of a desired target scene, such as target scene 1210. In Figure 12, each acoustic image is shown overlapping with an adjacent acoustic image. More specifically, acoustic images 1220 and 1222 include an overlapping portion 1231. Similarly, acoustic images 1222 and 1224 may include an overlapping portion 1233. Acoustic images 1224 and 1226 may include an overlapping portion 1235, and acoustic images 1226 and 1228 may include an overlapping portion 1237. The overlapping portion may include acoustic information from the acoustic data forming the adjacent acoustic image (e.g., the overlapping portion 1231 may include acoustic information from either acoustic image 1220 or acoustic image 1222, or from a combination of acoustic images 1220 and 1222).

[0127] As shown in Figure 12, two or more acoustic images (e.g., acoustic images 1220-1228) can be combined into a panorama, such as panorama 1215. Panorama 1215 may include the field of view (FOV) of the combined acoustic images, and as a result, the panorama includes a virtual FOV that is larger than the individual FOV of each acoustic image.

[0128] In some embodiments, combining acoustic images to form a panorama includes aligning the acoustic images so that they are aligned. In some embodiments, aligning acoustic images includes spatially aligning the acoustic data based on the FOV and / or orientation / position of the acoustic sensor array of each acoustic image at the time of acoustic data capture. For example, in some embodiments, the pan, tilt, and rotation of each captured acoustic image can be measured using various optical or mechanical systems. Such pan, tilt, and rotation information associated with each image can be used to position each acoustic image relative to each other in the panorama. For example, an inertial measurement unit (IMU) can be used to obtain the position and / or orientation on the acoustic sensor array, and / or the relative changes of such information between captured datasets. Such information can be used to determine and position the orientation of each acoustic image relative to other acoustic images in the panorama. In embodiments where the FOV of the acoustic analysis system is adjusted using an electrically operated moving mechanism, each adjustment (e.g., change in position, pan, tilt, and rotation) can be recorded and used to align the acoustic images when combining them into a panorama. In some examples, the motorized moving mechanism is configured to adjust the FOV of the acoustic analysis system by a predetermined amount, and the acoustic analysis system is configured to use the predetermined amount of FOV adjustment to create a panorama when aligning the subsequently captured data.

[0129] In some embodiments involving electromagnetic images, electromagnetic image data may be used when combining acoustic images. For example, an electromagnetic image may be captured along with each acoustic image. The associated electromagnetic images may be aligned (for example, based on overlapping portions of the electromagnetic images), and such alignment information may be used to similarly align the acoustic images.

[0130] In some embodiments, after alignment of the acoustic images, the acoustic images can be combined to form a panorama. Furthermore, the panorama can be presented to the user, such as by displaying it on a display, and / or stored in memory for later use. In various examples, the display may be integrated with the acoustic imaging device and / or may include an external display such as a smart device, computer, or tablet.

[0131] For simplicity, Figure 12 shows matching acoustic images (e.g., acoustic images 1220-1228) and matching overlapping portions (e.g., overlapping portions 1231-1237). However, different sizes and shapes may be used for both the acoustic images and the overlapping portions. For example, each acoustic image may be captured after different amounts of shift / pan so that one or more of the overlapping portions 1231-1237 differ from one or more of the other overlapping portions 1231-1237. Furthermore, in some examples, the field of view of each of the different images may have different sizes or shapes due to changes in distance to different parts of the scene, for example, when the system is tilted. In addition, or alternatively, in some cases, one or more captured acoustic images may not simply overlap with any of the other captured acoustic images (e.g., as shown in Figure 16).

[0132] In some embodiments, each of the acoustic images 1220-1228 may be captured using acoustic sensor arrays at different positions and / or orientations, for example, by adjusting the acoustic analysis system as described above. In addition, or alternatively, the analysis of acoustic data received by the acoustic sensor arrays may be adjusted for each acoustic image 1220-1228, and in some such examples, such acoustic images may be captured without physically adjusting the acoustic sensor arrays. For example, in some embodiments, the field of view of each acoustic image (e.g., acoustic images 1220-1228) may be determined using different sensor arrays (e.g., different subsets of available acoustic sensors) and / or different backpropagation algorithms.

[0133] In some embodiments, the acoustic analysis system may be configured to display or otherwise capture (for example, to create a video) multiple acoustic image frames associated with one or more fields of view from acoustic images 1220-1228, such as including frames of acoustic data captured sequentially. Figure 13 provides a simplified example of an acoustic image 1320 containing multiple acoustic image frames. As shown, the acoustic image 1320 contains nine acoustic image frames that show substantially identical fields of view. Although the illustrated acoustic image 1320 contains nine acoustic image frames, more or fewer acoustic image frames may be included in the acoustic image. The acoustic image 1320 may contain multiple acoustic image frames (e.g., acoustic image frames 1-9) that are captured in succession, such as 24 acoustic image frames per second, 30 acoustic image frames per second, or 60 acoustic image frames per second. Multiple acoustic frames (e.g., acoustic image frames 1-9) may also not be captured in succession, but rather may be captured over time, such as in sequence.

[0134] In some embodiments, other acoustic image frames / acoustic images may be captured between acoustic image frames used in acoustic image 1320, such as acoustic images that include other fields of view. For example, in some examples, the system may be configured to use multiple fields of view sequentially (e.g., to create a panorama). The system may be configured to associate data (e.g., acoustic image data) captured from the same field of view, even if the intervening dataset is captured from other fields of view. In some such examples, this is similar to time-series data collection for multiple fields of view.

[0135] In some embodiments, each acoustic image (e.g., acoustic image 1320) may be displayed by sequentially displaying multiple acoustic imaging frames, such as in a video. In some such examples, the frames may be repeated such that when the final acoustic image frame in the sequence (e.g., acoustic image frame 9) is displayed, the first acoustic image frame (e.g., acoustic image frame 1) is displayed again. In addition, or alternatively, acoustic images may be repeated based on received input, such as user input via a user interface or processor. Acoustic images may be displayed in real time, accelerated, or decelerated to provide the user with ease of analysis.

[0136] With respect to Figure 12, each acoustic image (e.g., acoustic images 1220-1228) may include a single still acoustic image frame or multiple acoustic image frames (e.g., acoustic image 1320 includes acoustic image frames 1-9). In some embodiments, each of acoustic images 1220-1228 may include multiple acoustic image frames. For example, in some examples, the acoustic analysis system may be configured to store multiple frames of acoustic image data for a given acoustic field of view. In some such examples, even if the sensor array moves to a new field of view, the multiple acoustic image frames from the previous field of view can be viewed in a loop (as described elsewhere in this specification) or otherwise displayed upon selection (e.g., via a user interface). For example, in one example, the user can select acoustic image 1220 via a user interface. The system may be configured to provide the user with the option to view the multiple acoustic image frames associated with image 1220 by scrolling through such image or viewing such image as a video.

[0137] In addition, or alternatively, in some examples, the current field of view of an acoustic sensor array may be displayed to show multiple acoustic images (e.g., as a live acoustic video), while other acoustic images may be displayed as a single still acoustic image (e.g., the most recent acoustic image data captured from the FOV associated with the acoustic image). For example, if an acoustic sensor array is positioned at a given time to generate acoustic image data of the field of view of acoustic image 1224, acoustic image 1224 may include multiple acoustic image frames displayed as a live video, while acoustic images 1220, 1222, 1226, and 1228 are shown as still acoustic images in a panorama.

[0138] In some embodiments, an acoustic image comprising multiple acoustic image frames may have a corresponding electromagnetic image comprising multiple electromagnetic image frames. In addition, or alternatively, an electromagnetic image representing the current field of view of an electromagnetic imaging tool (e.g., 203) may comprise multiple electromagnetic image frames. For example, similar to the acoustic image data described above, in some embodiments, in a panoramic context, the current field of view of the electromagnetic imaging tool may be displayed in real time (e.g., as live video), and electromagnetic image data outside the current FOV may be displayed as still images. Similar to the acoustic image data described elsewhere in this specification, the acoustic analysis system may be configured to store multiple electromagnetic images of such past FOVs that are accessible to the user.

[0139] In some embodiments, the acoustic sensor array may capture a larger field of view (FOV) than the associated electromagnetic imaging tool. For example, in some examples, the FOV of the acoustic sensor array may be larger than the FOV of the electromagnetic imaging tool. In such embodiments, the panorama may include multiple electromagnetic images associated with each acoustic image. In such embodiments, the panorama may be captured by adjusting the position and / or orientation of the electromagnetic imaging tool (e.g., 203) to cover a larger portion of the acoustic FOV. For example, the electromagnetic imaging tool may be manually operated by a user, such as being handheld or positioned on an adjustable stand (e.g., tripod, monopod, etc.). In addition, or alternatively, other systems such as linear movement systems, pan-tilt systems, and 360° rotation systems may be used. In some embodiments, the system may include a motorized movement mechanism system. In such embodiments, the motorized movement mechanism may be automated or programmed to periodically adjust the field of view of the electromagnetic imaging tool, for example, between capturing acoustic image data. In general, the processes described herein for generating a panorama may be carried out in some such systems by changing the FOV of the electromagnetic imaging tool.

[0140] Figure 14 shows the field of view for acoustic imaging and electromagnetic imaging. In the illustrated example, the FOV of acoustic imaging 1410 is larger than the FOV of electromagnetic imaging 1420. In some examples, the acoustic analysis system may be configured to manipulate the FOV of electromagnetic imaging to acquire electromagnetic image data for portions of the acoustic imaging FOV 1410 that are larger than a single electromagnetic image by sequentially collecting electromagnetic image data in several fields of view. In some examples, the system may be configured to adjust the FOV of the electromagnetic imaging tool once for a given set of fields of view to create a panorama of electromagnetic image data corresponding to portions of the acoustic imaging FOV 1410.

[0141] In some embodiments, the acoustic analysis system may be configured to recognize an acoustic signal (e.g., at position 1425) within the FOV 1410 of the acoustic imaging that is outside the field of view 1420 of the electromagnetic imaging. The system may be configured to operate the electromagnetic imaging tool (e.g., via a motorized movement mechanism) to move it to a new FOV (e.g., 1430) that contains an acoustic signal that was not previously within the FOV 1420 of the electromagnetic imaging. In some such examples, the system may be configured to determine the position of the acoustic signal relative to the FOV of the electromagnetic imaging based on information received from an acoustic sensor array and to operate the electromagnetic imaging tool accordingly.

[0142] In addition, or alternatively, the system may be configured to capture multiple electromagnetic images with different fields of view to create an electromagnetic panorama that covers all or part of the FOV of an acoustic image. Thus, electromagnetic image data corresponding to an acoustic signal outside the FOV of one electromagnetic image (e.g., at position 1425) may still be included in the larger electromagnetic image panorama. For example, in some examples, a system including a motorized moving mechanism can be used to periodically scan a target scene based on a schedule or input from a controller or user. In embodiments including a schedule, the schedule may be automated by a processor or the like (e.g., processor 212, external processor, etc.).

[0143] In some embodiments, the electromagnetic imaging tool may be located independently of the acoustic sensor array, for example, in a separate housing. For example, the acoustic analysis system may be configured such that the adjustment mechanism for manipulating the field of view (FOV) of the electromagnetic imaging tool is not attached to the acoustic sensor array. In addition, or alternatively, the acoustic sensor array may have an adjustment mechanism configured to adjust the acoustic sensor array in reverse such that the FOV of the acoustic sensor array remains relatively constant when the electromagnetic imaging tool is repositioned and the FOV of the electromagnetic imaging tool is changed.

[0144] As described herein, panoramas can be generated using various panoramic scans. Figure 15 provides an exemplary embodiment of capturing multiple acoustic images of a target scene 1510 using a vertical panoramic scan. As shown, the desired acoustic target scene 1510 is captured using multiple acoustic images (e.g., acoustic images 1520-1526). Multiple acoustic images (e.g., acoustic images 1520-1528) may be captured using systems and methods described herein, such as a pan-tilt mechanism.

[0145] As shown in Figure 15, parts of the scene associated with each acoustic image (1520-1526) are shown at a different size than the viewpoint in the figure. In some cases, when a panorama includes images spanning various different distances to the target, the area within the field of view of each image changes. Furthermore, in acoustic imaging, acoustic signals from closer sources may appear stronger than acoustic signals emitted from further away. In some examples, distance information associated with each image and / or various objects within the image can be determined and used, for example, to appropriately combine the images (to generate a panorama) and / or to accurately determine acoustic intensity information. In various examples, such distance information may include three-dimensional depth information relating to various parts of the target scene. In addition, or alternatively, the user may measure (e.g., using a laser distance tool) or manually input distance values ​​associated with multiple locations and / or multiple acoustic image portions within the target scene. In some examples, such different distance values ​​for various parts of the target scene can be used to adjust backpropagation calculations at those locations to correspond to specific distance values ​​at those locations or within those acoustic image portions.

[0146] In some embodiments, a portion of the target scene may be captured using various techniques. With respect to Figure 16, multiple acoustic images (e.g., acoustic images 1620-1628) may be captured so as to cover part or all of the target scene (e.g., target scene 1610). As shown, a portion of an acoustic image may overlap with one or more other acoustic images, such as acoustic images 1620-1626. In addition, one or more acoustic images may be separated from other acoustic images, such as acoustic image 1628.

[0147] In some embodiments, capturing a portion of a target scene, as described herein, may involve adjusting the position and / or orientation of an acoustic analysis system, such as adjusting the position of an acoustic sensor array. In some embodiments, as shown in Figures 17-19, different portions of a target scene may be selected and / or images may be stitched together based on a predetermined acoustic signal (referred to herein as a beacon acoustic signal) emitted from the target scene. In some examples, the beacon acoustic signal may include sounds emitted from one or more devices present in the target scene, which may provide the acoustic analysis system with context regarding the location of the target of interest, how to stitch together the acoustic images, the size and / or location of the target scene, etc. In some embodiments, the beacon device may be placed in the target scene by a user or the like and emit a beacon acoustic signal. In such embodiments, the beacon device may emit acoustic signals continuously or intermittently (e.g., on a schedule) based on inputs received from the acoustic analysis system, a user, or the like. The beacon acoustic signal may be unique to other acoustic signals present in the target scene. For example, a beacon acoustic signal may include a unique frequency, periodicity, etc., which may enable an acoustic analysis system to easily distinguish the beacon acoustic signal from other acoustic signals emitted from within the target scene using filtering techniques, etc., as described herein. Furthermore, the acoustic parameters of the beacon acoustic signal may be provided to the acoustic analysis system, for example, via the user and / or via a communication connection (e.g., wireless, wired, etc.) between the beacon device and the acoustic analysis system.

[0148] In some embodiments, the acoustic analysis system may be configured to recognize any number of different beacon acoustic signals. For example, in various examples, one or more identifiable beacon acoustic signals may have the same or different acoustic parameters. A first beacon device in the scene may emit a first beacon acoustic signal, and a second beacon device in the scene may emit a second beacon acoustic signal different from the first beacon acoustic signal. The acoustic analysis system may be configured to identify the first and second beacon acoustic signals in the scene. In addition, or alternatively, in some examples, the beacon acoustic signal may be emitted from other objects present in the scene, including a predetermined sound. For example, in some embodiments, a motor, vent, belt, other mechanism, etc., may emit a predetermined sound and / or an identifiable sound. Such acoustic signals may be used as beacon acoustic signals for various processes described herein.

[0149] Figure 17 shows an exemplary scene in which multiple acoustic images, such as acoustic images 1720A-D, are captured using a beacon acoustic signal 1725. As shown, the beacon acoustic signal 1725 is located at various positions within acoustic images 1720A-D, such as the lower left corner of acoustic image 1720A, the lower right corner of acoustic image 1720B, the upper left corner of acoustic image 1720C, and the upper right corner of acoustic image 1720D. The beacon acoustic signal 1725 can be used to align acoustic images by aligning the position of the beacon in one image with the position of the beacon in another image. In some such examples, as shown in Figure 17, the aligned acoustic images 1720A-D can be stitched together to form a composite acoustic image 1730 based on aligning the position of the beacon acoustic signal 1725 within each acoustic image 1720A-D.

[0150] One or more beacon signals may also be used to determine a portion of a scene that may be of interest. For example, Figures 18A and 18B show a target scene containing multiple beacon acoustic signals 1825A-E. In some such cases, the acoustic analysis system may be configured to capture images 1820A-E encompassing each beacon acoustic signal 1825A-E by manipulating the field of view (e.g., by manipulating the orientation or position of the acoustic sensor array, or by changing the active portion of the acoustic sensor array).

[0151] For example, with respect to Figure 18A, the acoustic analysis system may be positioned and / or oriented so that beacon acoustic signal 1825A captures acoustic image 1820A where it is centered. The acoustic analysis system may then be positioned and / or oriented so that beacon acoustic signal 1825B captures acoustic image 1820B where it is centered. This process may be repeated until all desired beacon acoustic signals (e.g., beacon acoustic signals 1825A-E) are captured in the acoustic image. In some embodiments, all beacon acoustic signals (e.g., beacon acoustic signals 1825A-E) may be emitted simultaneously. Alternatively, each beacon acoustic signal may be emitted when such beacon acoustic signal should be captured. For example, when capturing acoustic image 1820A, beacon acoustic signal 1825A may be emitted, but beacon acoustic signals 1825B-E may not be emitted. Next, when capturing acoustic image 1820B, beacon acoustic signal 1825B may be emitted, but beacon acoustic signals 1825A or 1825C-E may not be emitted, and so on. In some such embodiments, a single beacon device may be used and repositioned between locations corresponding to beacon acoustic signals 1825A-E. In addition, or alternatively, one or more beacon devices, e.g., one or more beacon devices positioned on a line, belt, or other moving mechanism, may be movable through the target scene. For example, in an exemplary example, a beacon device may start at the location of beacon acoustic signal 1825A, traverse the target scene, and finally reach the location of beacon acoustic signal 1825E, and multiple acoustic images (e.g., acoustic images 1820A-E) may be captured during the movement of the beacon device. In some embodiments, the beacon device may pause during the capture of an acoustic image encompassing the beacon acoustic signals.

[0152] Figure 18B shows an exemplary synthesized acoustic image and / or target scene including the beacon acoustic signals shown in Figure 18A. As shown, the beacon acoustic signals 1825A-E may be used to determine the portion of the scene to be captured for inclusion in a synthesized image (e.g., a panorama). The acoustic images (e.g., 1820A-E) may be stitched together in any of various ways, including the method described herein. Figure 18C shows the resulting panorama 1850 including the beacon acoustic signals 1825A-E.

[0153] In some embodiments, the acoustic analysis system may be configured to generate a panorama including representations of beacon acoustic signals 1825A-E, as shown in Figure 18C. In addition, or alternatively, in some examples, the system may be configured to exclude beacon acoustic signals from the resulting image data. For example, in some examples, an acoustic beacon device may be used to guide the system to capture image data from multiple fields of view (e.g., corresponding to acoustic images 1820A-E) using, for example, a recognizable acoustic signature in one or more beacon signals within the area to be analyzed. However, in some such examples, acoustic data from the beacon itself is irrelevant to the scene. Therefore, in some embodiments, acoustic signals matching one or more acoustic signatures of the beacon device may be excluded from the generated acoustic image data.

[0154] Examples in Figures 18A to 18C show embodiments in which the beacon acoustic signals are positioned horizontally to each other. However, in some embodiments, the beacon acoustic signals may be positioned at other locations relative to each other and, in general, at any location within the area to be analyzed.

[0155] Figure 19 provides an exemplary embodiment in which beacon acoustic signals are located at various positions and depths within the target scene 1910. As indicated by the size of the beacon acoustic signals, beacon acoustic signal 1925A is located at a greater distance, beacon acoustic signal 1925D is located at a closer distance, and beacon acoustic signals 1925B, 1925C are located in between (e.g., at an intermediate distance). In some embodiments, the acoustic analysis system may determine the distance to the beacon device by determining the distance to the target, as described herein. In addition, or alternatively, the distance to the beacon may be provided to the acoustic analysis system by other means, for example, when the beacon device is at a predetermined distance from the acoustic analysis system. The beacon devices may be located at various distances within the scene to help focus the acoustic analysis system on the subject of interest across the entire target scene. For example, the beacon acoustic signal 1925A may be located near the acoustic signal of an object of interest located further away and / or the object of interest, while the acoustic signal 1925D may be located near the acoustic signal of an object of interest located closer to the acoustic analysis system. Such acoustic signals can be useful in focusing the acoustic analysis system on the acoustic signal of an object of interest and / or the object of interest, for example, when capturing an acoustic image of a scene (e.g., acoustic images 1920A-D).

[0156] Although not explicitly illustrated, the acoustic analysis system may filter out one or more beacon acoustic signals when presenting an acoustic image to the user. For example, with respect to Figure 19, one or more of the beacon acoustic signals 1925A-D may be filtered out. In such an example, it may be possible for the user to better distinguish a certain beacon acoustic signal from the acoustic signal of interest. In embodiments where one or more beacon acoustic signals are of interest (for example, when a beacon device is of interest in the scene), one or more beacon acoustic signals may not be filtered out. In some embodiments, the user may be able to toggle the presence of one or more beacon acoustic signals in the scene via a user interface or the like.

[0157] In some embodiments, the acoustic analysis system may be tuned to include another or subsequent beacon acoustic signal, as is known. For example, as shown in Figure 18A, each beacon acoustic signal is located to the right of the previous beacon acoustic signal. In addition, or alternatively, the subsequent beacon acoustic signal may be either present or partially present within the previous acoustic image, such as being located along the periphery of the acoustic image or immediately beyond the periphery of the acoustic image, so that the subsequent beacon acoustic signal is partially present or blends into the acoustic image.

[0158] Figure 20A shows an exemplary scene with acoustic signals present at positions 2025, 2035, and 2045. Figure 20B shows an acoustic image 2000 including the acoustic signal at 2025. As shown, the acoustic signal at position 2035 is not within the field of view of the acoustic image 2000. However, in some examples, the acoustic imaging system is configured to detect the presence of acoustic signals outside the field of view of the acoustic image. For example, in some examples, an active acoustic sensor in an active acoustic sensor array can detect a sound that is not within the field of view. In some such examples, precise localization or analysis of the out-of-field sound is not possible, but the system may still be configured to determine that such a sound is present in the environment. In some examples, the acoustic analysis system may be configured to display an indication on a display showing the presence of such a sound currently outside the field of view.

[0159] For example, in Figure 20B, the acoustic analysis system is configured to generate an acoustic image 2000, which includes a display of an acoustic signal in 2025 and further includes a display 2030 on the right side of the screen, indicating the presence of an acoustic signal outside the field of view to the right of the image. This can show the user that an acoustic signal of interest may be present outside the current field of view. The user can adjust the field of view of the acoustic analysis system (for example, by changing the orientation or direction of the system) to attempt to capture such additional acoustic signals.

[0160] Figure 20C shows an acoustic image 2010, which includes a display of an acoustic signal at position 2035 after the system has adjusted its field of view in response to, for example, a notification display 2030 in image 2000. As shown, the acoustic image 2010 includes displays 2020 and 2040 (for example, at positions 2025 and 2045, respectively) indicating the presence of sounds outside the current field of view. Such displays can help the user observe all sounds of interest within the area by, for example, showing the user how to adjust the field of view to acquire additional acoustic signals. In some examples, the user can adjust the system's field of view to capture multiple acoustic images in response to a notification display suggesting the presence of sounds outside the current field of view. Using such multiple acoustic images, a panorama including such sounds can be generated, for example, as described elsewhere in this specification. In various examples, displays such as 2020, 2030, and 2040 indicating the presence of additional acoustic signals outside the field of view may be included in the captured acoustic image or excluded from the captured acoustic image despite being present as a display on the display, to assist the user in locating the acoustic signals.

[0161] In some embodiments, the acoustic signals indicated by the indicators (e.g., 2020, 2030, 2040) may not necessarily be outside the field of view of the acoustic sensor array, but may be outside the field of view of the electromagnetic imaging tool that generates electromagnetic image data for combination with the acoustic image data. For example, as described elsewhere in this specification, in some examples, the acoustic sensor array may have a larger field of view than the associated electromagnetic imaging tool. The acoustic analysis system may be configured to determine that the acoustic signals are located outside the FOV of the electromagnetic imaging tool and to provide an indicator showing the current location of the acoustic signals outside the FOV of the electromagnetic imaging tool.

[0162] In addition, or alternatively, in some embodiments, indicators such as those described with respect to Figures 20A-C may indicate acoustic signals outside the FOV (e.g., the FOV of an acoustic sensor array and / or the FOV of an electromagnetic imaging tool) that have one or more acoustic parameters satisfying one or more predetermined conditions (e.g., acoustic signals having a predetermined frequency component, minimum amplitude, etc.). In some embodiments, the acoustic analysis system may determine whether a beacon acoustic signal, such as a beacon acoustic signal as described elsewhere in this specification, is located outside the FOV, and if so, may provide an indicator indicating the direction of the beacon acoustic signal.

[0163] In some embodiments, the location of such an acoustic signal outside the field of view of the current image may provide the acoustic analysis system with information on how to adjust the field of view to capture the next acoustic image (e.g., by rearranging or otherwise adjusting the acoustic sensor array). As shown, when capturing acoustic image 2000, the acoustic signal at position 2035 is located near or beyond the right periphery of the field of view. The acoustic analysis system may be configured to detect the presence of such an acoustic signal and use this information to determine, for example, that it is necessary to adjust the field of view to the right to determine that the location of the detected acoustic signal is outside the field of view of the current acoustic image. Using such a process, multiple acoustic images may be automatically captured by continuing to adjust the system's field of view to capture acoustic image data of various acoustic signals. Using such multiple acoustic images, a panorama can be generated, such as a panorama as described elsewhere in this specification.

[0164] In some embodiments, the acoustic analysis system is configured to automatically adjust its field of view to capture multiple acoustic images based on acoustic signals detected outside the current field of view. In some such examples, the acoustic analysis system is configured to limit the amount by which the field of view changes before a new acoustic image dataset is captured while navigating to detected acoustic signals, so that, for example, the system can create a panorama that includes the entire area between the initial and final fields of view.

[0165] In some embodiments, each of the multiple acoustic images constituting a panorama of a target scene can be captured sequentially in a short period of time, for example, using a single imaging array, or using multiple imaging arrays simultaneously or nearly simultaneously. For example, in some examples, each of the acoustic image sequences shown in Figures 12, 15, and 16-19 (e.g., acoustic images 1220-1228, 1520-1526, 1620-1628, 1720A-D, 1820A-E, and 1920A-D) can be captured nearly simultaneously, and thus they provide a representation of the acoustic signals in the target scene at nearly a single point in time or over a short period of time.

[0166] However, in some embodiments, one or more acoustic images may be captured at different times. For example, the acoustic image of a particular portion of a panorama may be updated over time or in response to external events (e.g., acoustic changes in a target scene, at a predetermined time, after a predetermined time, user input, instructions from a processor, etc.). In addition, or alternatively, in some examples, different images used to create a panoramic representation are captured at different times. In some embodiments, a display may be presented to indicate the time each acoustic image was captured and / or the time each portion of the panorama was last updated. For example, when an acoustic panorama is displayed on a display, the panorama may include the time the last image was captured, in real time or in comparison to other acoustic images. Such a display may represent the “freshness” of a given portion of the acoustic image and / or panorama. In some embodiments, “freshness” may be a display of the time elapsed since the acoustic image and / or panorama was captured. For example, the time may be the difference between the time the acoustic image was captured and real time, or the most recently captured acoustic image compared to a predetermined time.

[0167] For example, in Figure 16, acoustic image 1626 is shown to have a thick border relative to other acoustic images (e.g., acoustic images 1620, 1622, 1624, and 1628). The thick border of acoustic image 1626 may provide the user with an indication that acoustic images 1620, 1622, 1624, and 1628 have been updated recently or within a given time, while acoustic image 1626 has not been updated recently. In addition, or alternatively, other indicators, such as the use of various colors, shading, opacity, timestamps, and countdown timers, may be used to indicate whether a portion of the panorama has been updated recently (e.g., within a given time). In some cases, the acoustic analysis system may further be configured to switch the display between the updated panorama and the previous panorama based on user input received via the user interface. In addition, or alternatively, as described elsewhere in this specification, in some examples, each FOV may be associated with multiple frames of image data stored in memory. In some examples, each portion of the panorama may involve sequentially using multiple frames associated with the FOV corresponding to such portion of the panorama. In some examples, the rate at which such frames are used sequentially depends on the freshness of the acoustic image data. For example, as described elsewhere in this specification, in some examples, a “live” FOV may be displayed as “real-time” video. In some such examples, a portion of a panorama from a previously captured FOV may display frames that are used sequentially at a rate different from real-time, such as looping slower or faster than the rate at which such frames were captured.

[0168] In some embodiments, a visual representation of data freshness may be incorporated into the visualization of acoustic data, as shown at positions 554, 556, and 558 in Figure 5B. For example, in an exemplary embodiment, the freshness of acoustic image data may be represented based on the transparency of the visualization of acoustic data at a given position in the image. A portion of an acoustic panorama captured more recently may contain acoustic image data that is relatively less opaque than acoustic image data in a portion of the panorama captured earlier. In some examples, temporal information (e.g., the time when a given dataset was captured, its freshness, etc.) may be included in the metadata of the acoustic image, for example.

[0169] In some embodiments, the user may be requested to capture an acoustic image of a portion of a target scene that has not been recently updated (e.g., via a display). In embodiments comprising a motorized mobile mechanism, the motorized mobile mechanism may be requested to automatically adjust the acoustic analysis system via a controller or processor, as described herein, to capture the portion that has not been recently updated.

[0170] In some embodiments, different acoustic images used to create an acoustic panorama include different parameters, such as different frequency information or distance values ​​to targets. In some embodiments, the acoustic imaging system may be configured to adjust data acquisition and / or representation based on one or more such parameters. For example, in some embodiments, the system may be configured to determine distance values ​​to targets associated with various locations within individual acoustic images, and similarly for panoramas containing such acoustic image data. The system may be configured to take different distance values ​​to targets into account when presenting acoustic image data, for example, when determining the acoustic intensity associated with each of several locations. For example, with respect to Figure 15, the acoustic signal in image 1526 is closer than the acoustic signal in image 1520. Acoustic signals with the same intensity in each image will appear quieter in image 1520 due to their distance, without correction. The system may be configured to use distance information to adjust intensity across different locations and different images used to create the panorama.

[0171] In addition, or alternatively, in some embodiments, the acoustic analysis system may be configured to adjust its operation during the acquisition of multiple images for creating a panorama using one or more parameters of the scene, such as detected frequency components or distance information. For example, capturing each acoustic image may include determining and presenting the frequency bands associated with a particular field of view of that image. In some examples, the system may be configured to select an acoustic sensor array (e.g., 320, 322 in Figure 3) for capturing each acoustic image to be used when creating the acoustic panorama. Thus, in various examples, the system may be configured to automatically change the frequency bands (e.g., in data acquisition via the selected acoustic sensor array and / or via representations in the acoustic image data) to compensate for changes in distance to the target as the system collects acoustic information from multiple fields of view. In addition, or alternatively, the system may be configured to display the acoustic image data for each image based on acoustic parameters associated with that image, such as based on distance to the target and / or frequency component information. Such image-specific display settings may be included in the corresponding positions within the acoustic panorama.

[0172] In some embodiments, the target scene may include one or more locations of interest (e.g., locations within the target scene that emit an acoustic signal of interest). Furthermore, one or more locations of interest may include various acoustic signals having different acoustic parameters or criteria (e.g., frequency difference, sound intensity (e.g., dB level), distance to target, periodicity, etc.). Therefore, it may be beneficial to acquire different acoustic parameters between acoustic images used to construct an acoustic panorama, such as different frequency ranges, decibel ranges, distance to target, periodicity, etc. In such embodiments, it may be beneficial to compensate for differences in acoustic parameters when the acoustic analysis system is adjusted to capture an acoustic image of the target scene (e.g., adjusting the sound intensity level based on the distance to target, or adjusting the size of the acoustic sensor).

[0173] In some embodiments, different acoustic parameters may be displayed and / or captured for different parts of a single acoustic image (including an acoustic panorama). In some examples, the field of view of a single acoustic image may include one or more locations of interest (e.g., locations in a target scene that emit an acoustic signal of interest). Such one or more locations of interest may include a variety of different acoustic parameters, some of which may be more important or of higher interest at some locations compared to others. Therefore, it may be beneficial to display different acoustic parameters (e.g., frequency difference, sound intensity, distance to target, periodicity, etc.) at different parts of the acoustic image, and / or to display acoustic image data representing an acoustic signal that satisfies one or more predetermined acoustic parameter criteria (e.g., above a threshold, below a threshold, within a certain range, etc.).

[0174] In addition, or alternatively, in some embodiments, the acoustic analysis system may be configured to display or highlight one or more acoustic parameters for a given portion of an acoustic image (e.g., a panorama) based on one or more parameters detected within the portion of the scene. For example, in some examples, a given portion may be configured to display acoustic parameters that reflect the acoustic parameters of the sound source with the greatest acoustic signal within that portion (e.g., a frequency band containing the frequency information of the loudest sound) or the acoustic parameters of the highest frequency within that portion.

[0175] In some embodiments, different acoustic parameters may be displayed in such a way that acoustic signals satisfying one or more predetermined criteria within a particular area are visually contrasted with acoustic signals in areas or other areas that do not satisfy the criteria. In some embodiments, this may include displaying acoustic signals that satisfy the criteria and not displaying acoustic signals that do not satisfy the criteria. Alternatively, acoustic signals that satisfy the criteria may be displayed with a different palette, size, and / or opacity than acoustic signals that do not satisfy the criteria. In some embodiments, such criteria may be used to define an acoustic profile that may include one or more criteria related to one or more acoustic parameters. Acoustic signals that satisfy each of the one or more criteria defining the acoustic profile will match the acoustic profile. In various embodiments, the acoustic profile may include one or more frequency ranges, one or more intensity ranges, and / or one or more distance ranges. Such ranges may include closed ranges (e.g., between a minimum intensity level and a maximum intensity level) or may be defined by a single threshold (e.g., above a threshold intensity level).

[0176] Figure 21 provides an exemplary embodiment in which the target scene 2110 is segmented or divided into multiple acoustic image portions (e.g., acoustic image portions 2120-2126). As shown, each acoustic image portion (e.g., acoustic image portions 2120-2126) may have a border with one another. However, in various embodiments, other configurations may be used, such as having overlapping acoustic image portions and / or one or more separate acoustic image portions.

[0177] In various examples, the acoustic image portion may correspond to a single acoustic image (e.g., in the case of an acoustic panorama). In addition, or alternatively, the acoustic image portion may be located within the same acoustic image. In some embodiments, the acoustic image portion may extend to multiple acoustic image portions (e.g., a portion of a panorama containing acoustic image data from separately captured images).

[0178] In some embodiments, acoustic image segments (e.g., acoustic image segments 2120-2126) may be defined by the user, for example, via a user interface (e.g., via a touchscreen, one or more buttons, a mouse, etc.). In some such embodiments, the user may segment the acoustic image segments via the user interface using freeform and / or predetermined geometric shapes. In addition, or alternatively, the user may specify segments of the acoustic image based on one or more acoustic parameters related to the target scene, such as distance values ​​to the target or acoustic intensity values. As can be understood from the description herein, by segmenting or dividing the acoustic image into multiple acoustic image segments, the acoustic analysis system can be made to operate substantially as if the system were composed of multiple acoustic imaging devices, even if the system contains fewer acoustic imaging devices or only one acoustic imaging device. Each of the multiple acoustic image segments corresponds substantially to a “virtual” acoustic imaging device. Different criteria may be established for each acoustic image segment that determine when and how to display (or not display) the representation of the acoustic data received from the target scene within the “field of view” of the corresponding virtual acoustic imaging device (i.e., within each acoustic image segment).

[0179] In addition, or alternatively, in some examples, acoustic image segments (e.g., acoustic image segments 2120-2126) may be automatically created. For example, in some embodiments, the acoustic analysis system may be configured to divide or segment the acoustic image of a target scene into multiple parts based on the location or acoustic parameters of acoustic signals within the target scene, such as based on current and / or historical information about the target scene. In some embodiments, the acoustic image segments may be defined by the location of objects within the target scene, the orientation of objects within the target scene relative to the acoustic analysis system, etc. For example, if the target scene includes several different machines emitting acoustic signals, the acoustic analysis system may be configured to divide the acoustic image into multiple parts (e.g., by acoustic signature recognition) such that each machine is in a separate acoustic image segment. In addition, or alternatively, in embodiments including an electromagnetic image, the acoustic analysis system may be configured to identify features or objects (e.g., several machines) within the electromagnetic image and automatically specify and divide the corresponding acoustic image into multiple parts based on the location of one or more identified objects within the electromagnetic image.

[0180] In some embodiments, the acoustic analysis system may be configured to automatically identify objects in a scene, for example, by machine learning methods. In various examples, the system may be configured to identify one or more locations in a scene using machine learning methods.

[0181] In addition, or alternatively, in some examples, the user can provide the acoustic analysis system with information about the position of objects in a target scene via a user interface, and the system can generate a portion of the image based on the information received from the user.

[0182] In some embodiments, an acoustic image portion may be defined, for example, by coordinates relative to a part of an acoustic analysis system. For example, the system may be configured to define a portion based on area and / or volume within a target scene (e.g., within a range of horizontal and / or vertical angles from a direction perpendicular to the acoustic sensing array or electromagnetic imaging surface). In some such examples, the system may be configured to adjust the orientation of an acoustic sensor array and / or electromagnetic imaging tool by a predetermined amount based on such predetermined portions in order to collect data from multiple portions. Such data may be combined, for example, based on a predetermined amount by which the orientation of the system components is adjusted, to form a panorama including acoustic and / or electromagnetic image data, such as data described elsewhere in this specification. Such information may be used to construct such a panorama by aligning data captured from multiple fields of view.

[0183] As described herein, an acoustic image portion (e.g., acoustic image portions 2120-2126) may be configured to display specific acoustic parameters and / or acoustic signals having acoustic parameters that satisfy predetermined criteria within a given image portion. For example, a predetermined set of acoustic criteria may be determined (e.g., identified, accessed, or received), and acoustic signals may be filtered so that acoustic signals having acoustic parameters within the predetermined set of acoustic criteria are displayed. Acoustic signals that do not satisfy such predetermined criteria are not displayed. In various embodiments, such criteria may include one or more parameters having a specific value (e.g., 60 Hz, 13 dB), one or more parameters satisfying one or more corresponding threshold conditions, or one or more parameters each within a corresponding range of values ​​(e.g., 50-100 Hz, 10-15 dB). Generally, such criteria may be based on any number of different acoustic parameters. For example, in one example, a region may represent acoustic signals within a specific distance and display only acoustic image data having specific frequency components. Criteria may also be defined by a logical combination of criteria, such as acoustic signals satisfying any one or more of a predetermined set of criteria. In some examples, the criteria may include matching the acoustic signal to one or more acoustic signatures, such as acoustic signatures of acoustic signals captured in previous recordings (e.g., recordings of machines operating normally and / or recordings of machines with identified problems).

[0184] As described elsewhere in this specification, in some embodiments, one or more criteria may define an acoustic profile. In some embodiments, an acoustic image portion (e.g., acoustic image portions 2120-2126) may have an associated acoustic profile specifying one or more criteria. In some such examples, a given portion of an acoustic image contains only acoustic image data representing acoustic signals within the associated acoustic profile. In some embodiments, the system may be configured to generate and / or access existing acoustic profiles by analyzing an acoustic scene (e.g., by determining common ranges such as frequencies and intensities present in the scene). Existing acoustic profiles may be pre-generated by the system or loaded from another source.

[0185] In some cases, machine learning methods can be used to generate one or more profiles. For example, a system may be configured to analyze acoustic information representing a portion of a scene, a specific piece of equipment, etc., and determine one or more profiles associated with it. Such profiles may include, for example, information about acoustic parameters representing normal and / or abnormal operating conditions.

[0186] Similarly, in some embodiments, a user may generate such a profile and / or input information into a machine learning process that generates such a profile. For example, in some cases, a user may, based on their own experience, identify a portion of an acoustic scene operating in a particular state or severity level (e.g., good, moderate problem, serious problem, critical problem, failure). The user may capture acoustic image data and / or other acoustic parameters (e.g., frequency and intensity information) representing such portion of the scene and input information about the scene's status (e.g., critical operational problems). The acoustic analysis system may be configured to create a profile based on the received status and acoustic information. In addition, or alternatively, such input may serve as data points analyzed using a machine learning system to generate such a profile. Such an acoustic profile may relate to one or more portions of a scene (e.g., if the system is programmed to capture a given acoustic dataset from multiple given fields of view) and / or to a particular piece of equipment or object within the scene. The profile may be used by the system and / or the user to analyze similar portions of the scene and / or objects in the future.

[0187] In some embodiments, the system may be configured to construct profiles of one or more parts of an acoustic scene, such as user-defined and / or automatically defined parts. For example, in some embodiments, a user can define parts of an acoustic scene, and the acoustic analysis system may be configured to sample acoustic data from such parts and generate acoustic profiles therefrom. In addition, or alternatively, in some examples, the system may be configured to sample information from a scene and identify different parts within the scene using machine learning or other analytical processes based on information received from different parts of the scene, for example. In some such cases, the system may be further configured to associate a profile with each such part of the scene based on the sampled data. Thus, in some examples, the acoustic analysis system may be configured to automatically divide a scene into multiple parts and associate acoustic profiles with one or more such parts by machine learning or other analytical processes that sample data from the scene.

[0188] In addition, or alternatively, in some embodiments, the user can generate an acoustic profile and assign it to a specific part of an acoustic scene, and / or associate an existing acoustic profile with a part of the scene. In some examples, the system may be configured to automatically associate an acoustic profile with such part of a scene, for example, by recognizing a specific piece of equipment within that part (e.g., by recognizing an acoustic signature or a specific piece of equipment shape or configuration), or by receiving input from the user indicating that such equipment is present within that part.

[0189] In some examples, a predetermined criterion may be based on an alarm profile of a given portion of an acoustic image and / or target scene. For example, in one example, an acoustic signal is presented within the acoustic image data when the acoustic signal matches an alarm profile. The alarm profile may include a set of alarm acoustic parameter criteria similar to the predetermined acoustic parameter criteria described herein. In some examples, if an acoustic signal contains acoustic parameters in the alarm profile, the acoustic analysis system may be configured to notify the user of a satisfied alarm profile.

[0190] For example, in some embodiments, an acoustic analysis system may be configured to notify the user when alarm conditions for an alarm profile associated with a portion of a scene are met. The notification may include visual, auditory, and / or tactical notifications, such as on-display notifications, tones, and / or vibrations. In some embodiments, acoustic image data representing acoustic signals that satisfy the alarm conditions may be presented on a display image within the portion of the scene having the associated alarm profile in a manner that distinguishes them from acoustic image data representing acoustic signals that do not satisfy the alarm conditions. In various examples, distinguishing acoustic signals that satisfy the alarm conditions may include providing different palette displays, opacity, color intensity, periodic flashing, and the like.

[0191] In addition, or alternatively, in some examples, acoustic signals that satisfy alarm conditions may be represented by acoustic image data on a display within a portion of the scene having the associated alarm profile, while acoustic signals that do not satisfy alarm conditions are not presented within such portion. In some embodiments, the user may receive notifications via a user interface, display, etc., integrated with the acoustic analysis system. In addition, or alternatively, other notification mechanisms may be used, such as notifications transmitted to a central station via an operation management system, a computerized maintenance management system, or a smart device (e.g., a tablet, telephone, wearable device, computer, etc.).

[0192] In some embodiments, the alarm condition may include a hysteresis component. For example, in some embodiments, the alarm condition is met when an acoustic parameter (e.g., intensity in a given frequency range) meets a threshold (e.g., in dB units) a predetermined number of times within a given time frame. In an exemplary embodiment, the acoustic analysis system may detect an alarm condition if the acoustic intensity within a given frequency range meets a predetermined intensity threshold more than 10 times a day. Other counts and periods are also possible. In some embodiments, such counts and periods may be selected by the user. In some examples, multiple such alarm conditions may be used simultaneously. For example, an alarm condition may be met not only when the acoustic intensity meets a first threshold a first predetermined number of times, but also when the acoustic intensity meets a second threshold a second predetermined number of times. For example, in addition to the alarm condition for meeting a predetermined intensity threshold more than 10 times a day, an alarm condition may be detected when the intensity meets a second, higher predetermined intensity threshold five times a day.

[0193] In some examples, a relevance threshold can be used to generate alarm conditions based on intermittent signals, such as intermittent signals described elsewhere in this specification. For example, as described herein, in some examples, an acoustic analysis system can generate and display a display image having acoustic image data that shows intermittent acoustic signals that have been detected to date but are not present in real time. In some embodiments, such intermittent signals can satisfy alarm conditions that occurred, for example, within a predetermined time range. The system may be configured to include acoustic image data representing such acoustic signals as signals that satisfy alarm conditions, for example, even if such acoustic signals are not present when the acoustic image data is generated or viewed. Similarly, such timing (e.g., relevance) thresholds can be used as parameters in multimodal alarms. For example, in some embodiments, the system may be configured to detect alarm conditions when an acoustic signal meets a threshold intensity within a relevance threshold (e.g., within the last day, within the last hour, etc.).

[0194] In some embodiments, the location of an alarm condition may be labeled on a display in the portion of the acoustic image having the alarm condition, or near it. In addition, or alternatively, the acoustic signal satisfying the alarm condition may be labeled on a display, user interface, etc. In various examples, the label may include information such as a title / brief description (e.g., pipe gasket, motor 1), one or more values ​​of acoustic parameters (e.g., dB level, frequency, etc.), past minimum / maximum values ​​of acoustic parameters, one or more alarm conditions (e.g., minimum, maximum, etc.), and alarm history, such as one or more times the alarm condition was met. In embodiments where an area / object has multiple alarm conditions, the label may include such multiple alarm conditions. In addition, or alternatively, the user may be able to switch between each alarm condition via a user interface, etc. Similarly, in some embodiments, the acoustic analysis system may be configured to sequentially display multiple such labels and provide the labels to the user in sequence.

[0195] In some cases, one or more acoustic profiles may be manually defined by the user and / or automatically selected or recommended by the acoustic analysis system. For example, in some cases, the acoustic analysis system may determine the acoustic profiles of one or more acoustic image portions based on objects located within corresponding portions of a scene (e.g., machines, vents, wires, other electronic equipment, etc.). The acoustic analysis system may be configured to create such profiles by, for example, sampling a target scene over time to build such profiles (e.g., establishing a baseline "normal" sound profile). In addition, or alternatively, the system may access such profiles stored in memory or from a database (e.g., information on how a particular piece of equipment behaves acoustically during normal operation).

[0196] In some embodiments, a predetermined acoustic criterion may be used based on a location of interest within each acoustic image portion. For example, referring to Figure 21, acoustic image portion 2124 contains acoustic signals from machines closer to the acoustic analysis system compared to acoustic image portions 2120 and 2121. Acoustic image portion 2125 contains acoustic signals from a specific machine. One or more such acoustic image portions may have different acoustic parameters of interest associated with them. For example, a specific machine (e.g., within acoustic image portions 2120 and 2121) may be of interest when its acoustic signal is lower in intensity than that of other machines (e.g., within acoustic image portion 2124). In such an example, different acoustic profiles may be associated with such portions so that acoustic signals with lower intensities than those in acoustic image portion 2124 can be displayed in acoustic image portions 2120 and 2121. Similarly, different machines within each such portion may be associated with one or more alarms that can be implemented within a particular portion of the image (e.g., incorporated into the associated acoustic profile). For example, the acoustic analysis system may be configured to provide a visual display in section 2120 when the acoustic signal in section 2120 exceeds a first intensity threshold, and / or in section 2124 when the acoustic signal in section 2124 exceeds a second threshold different from the first threshold. Furthermore, when observing the instrument in section 2125, different acoustic parameters may be of interest (e.g., periodicity rather than intensity). Therefore, acoustic signals containing different acoustic parameters may be displayed in image section 2125.

[0197] As described herein, in some embodiments, acoustic image data of a given acoustic signal may be filtered according to one or more acoustic parameters based on the portion of the acoustic image in which it is located. Such parameters may include, for example, intensity, frequency, periodicity, apparent proximity, measured proximity, sound pressure, particle velocity, particle displacement, acoustic power, acoustic energy, acoustic energy density, sound exposure, pitch, amplitude, luminance, harmonics, and the rate of change of any such parameter. For example, a particular portion of the acoustic image may show acoustic image data representing an acoustic signal having a specific frequency component, while another portion of the acoustic image may be configured to show only acoustic image data representing an acoustic signal having an intensity within a specific range. In addition, or alternatively, textual representations of one or more acoustic parameters may be included, the parameters presented in text being based on the acoustic profile associated with the portion of the image.

[0198] In some examples, the user may combine various acoustic parameters to define an acoustic profile using any suitable logical join, such as AND, OR, XOR, etc. For example, the user may want to display an acoustic signal in acoustic image portions 2123, 2124, and 2126 having an intensity within a first intensity range and (AND) a frequency within a first frequency range, and then display the acoustic signal (sequentially or simultaneously) in acoustic image portion 2125 having a second frequency range.

[0199] As with some embodiments described elsewhere in this specification, in some examples, acoustic image portions (e.g., acoustic image portions 2120-2126) may be associated with different distances from the acoustic analysis system. For example, the distance information may include three-dimensional depth information relating to various parts of the target scene. In addition, or alternatively, the user may measure (e.g., using a laser distance tool) or manually input distance values ​​relating to multiple locations and / or multiple acoustic image portions within the target scene. In some examples, such different distance values ​​for various parts of the scene can be used to adjust backpropagation calculations at such locations to match specific distance values ​​at those locations or within those acoustic image portions.

[0200] The various functions of the components described herein can be combined. In some embodiments, the features described in this application can be combined with features described in the following applications, each filed on July 24, 2019, assigned to the assignee of this application, and incorporated herein by reference: A PCT application titled "SYSTEMS AND METHODS FOR PROJECTING AND DISPLAYING ACOUSTIC DATA" with WIPO publication number WO2020 / 023622; A PCT application titled "SYSTEMS AND METHODS FOR TAGGING AND LINKING ACOUSTIC IMAGES" with the same application number WO2020 / 023633; A PCT application titled "SYSTEMS AND METHODS FOR DETACHABLE AND ATTACHABLE ACOUSTIC IMAGING SENSORS" with the same application number WO2020 / 023631; A PCT application titled "SYSTEMS AND METHODS FOR ANALYZING AND DISPLAYING ACOUSTIC DATA" with the same application number WO2020 / 023627; A PCT application titled "SYSTEMS AND METHODS FOR REPRESENTING ACOUSTIC SIGNATURES FROM A TARGET SCENE" has the same application number WO2020 / 023629.

[0201] In addition, or alternatively, the features of this disclosure may be used in combination with features described in one or more of the following patent applications, which were filed concurrently with this specification, assigned to the assignee of this application, and each is incorporated herein by reference: U.S. Patent Provisional Application No. 63 / 077,441, filed on September 11, 2020, entitled "SYSTEMS AND METHODS FOR GENERATING PANORAMIC AND / OR SEGMENTED ACOUSTIC IMAGES"; Filed on September 11, 2020, patent no. 63 / 077,445, titled "ACOUSTIC IMAGING WITH ACCUMULATED-TIME VIEW"; Filing dated September 11, 2020, patent no. 63 / 077,449, titled "SYSTEMS AND METHODS FOR GENERATING PANORAMIC AND / OR SEGMENTED ACOUSTIC IMAGES".

[0202] Various examples of embodiments have been described. Such examples are not limiting and do not in any way define or limit the scope of the present invention.

[0203] For example, various embodiments of the acoustic analysis system described herein may include any of the following features individually or in any combination: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of which is configured to receive an acoustic signal from a target scene and to output acoustic data based on the received acoustic signal; a display; a processor communicating with the acoustic sensor array; a processor communicating with the acoustic sensor array and the display; the processor configured to receive a first acoustic dataset, the first acoustic dataset representing a first portion of a target scene and having a first field of view (FOV); The processor is configured to: generate a first acoustic image based on a first acoustic dataset received; receive a second acoustic dataset, the second acoustic dataset representing a second portion of a target scene and having a second FOV, the second FOV being different from the first FOV; generate a second acoustic image based on the received second acoustic dataset; align the first and second acoustic images to form an aligned first acoustic image and an aligned second acoustic image; generate a panorama including the aligned first and second acoustic images; and present the panorama on a display.

[0204] The acoustic analysis system may further include an electromagnetic imaging tool configured to generate electromagnetic image data of a target scene, and a processor is configured to communicate with the electromagnetic imaging tool, receive first electromagnetic image data from the electromagnetic imaging tool, and combine the electromagnetic image data and acoustic image data to create a display image.

[0205] Various embodiments of the acoustic analysis system may further include any of the following features individually or in any combination: a panorama includes electromagnetic image data and a housing configured to support elements such as an acoustic sensor array, an electromagnetic imaging tool, a display, and a processor; a first acoustic image and a second acoustic image include overlapping portions within a target scene, the overlapping portions having acoustic information from the first acoustic dataset and the second acoustic dataset; the processor is further configured to determine a first predetermined acoustic reference set and a second predetermined acoustic reference set, and the displayed panorama includes in the first acoustic image data representing acoustic signals that satisfy the first predetermined acoustic reference set. The acoustic sensor array includes acoustic image data and acoustic image data in a second acoustic image that represents only acoustic signals that satisfy a second predetermined acoustic reference set; the acoustic sensor array is configured to capture acoustic signals that include a first predetermined acoustic reference set in a first acoustic dataset and to capture acoustic signals that satisfy a second predetermined acoustic reference set in a second acoustic dataset; the first predetermined acoustic reference set includes one or more frequency ranges, one or more decibel ranges, one or more target distance ranges, and / or one or more periodicity ranges, and the second predetermined acoustic reference set includes one or more frequency ranges, one or more decibel ranges, one or more target distance ranges, and / or one or more periodicity ranges.

[0206] In addition, the acoustic analysis system may be handheld by the user; the user positions the acoustic analysis system to capture a first FOV of a first acoustic dataset and a second FOV of a second acoustic dataset; the acoustic analysis system further comprises an adjustment mechanism, which is configured to position the acoustic analysis system to capture a first FOV of a first acoustic dataset and to position the acoustic analysis system to capture a second FOV of a second acoustic dataset; the processor is configured to communicate with the adjustment mechanism and receive a first position information set relating to the position of the acoustic analysis system for the first FOV and a second position information set relating to the position of the acoustic analysis system for the second FOV, and the alignment of the first acoustic image and the second acoustic image is based on the first and second position information sets, and the adjustment mechanism includes a linear movement system, a pan-tilt system, and / or a 360-degree rotation system.

[0207] The processor of the acoustic analysis system may be configured to receive an updated first acoustic dataset, the updated first acoustic dataset representing a first portion of a target scene captured at a different time than the first acoustic dataset, generate an updated first acoustic image based on the received first acoustic dataset, align the updated first acoustic image and the second acoustic image, and generate an updated panorama including the updated first acoustic image data and the second acoustic image.

[0208] The acoustic analysis system may further include a display on a screen and / or a user interface that represents the time when the first acoustic image was last updated, and the processor may further be configured to switch between an updated panorama and a previous panorama based on the input received via the user interface, and the processor may further be configured to receive an updated second acoustic dataset that represents a second portion of a target scene and was captured at a different time than the second acoustic dataset, generate an updated second acoustic image based on the received second acoustic dataset, align the updated first acoustic image and the updated second acoustic image, generate a second updated panorama including the aligned, updated first acoustic image data and updated second acoustic image data portions, and present the updated panorama on the display.

[0209] In some cases, the acoustic analysis system is configured to receive one or more beacon acoustic signals emanating from a target scene, where at least one of a first FOV and a second FOV is specified based on the location of one or more beacon acoustic signals, where one or more beacon acoustic signals include predetermined acoustic signals, where one or more beacon acoustic signals are distinguishable from other acoustic signals in the target scene, and where one or more beacon acoustic signals include different acoustic parameters from other acoustic signals in the target scene.

[0210] The acoustic analysis system may further comprise one or more beacon devices positioned within a target scene, each configured to emit one or more beacon acoustic signals. These one or more beacon acoustic signals are located in both a first acoustic image and a second acoustic image. The first and second acoustic images are aligned by aligning the one or more beacon acoustic signals.

[0211] The processor of the acoustic analysis system may be configured to determine a freshness value of acoustic image data within a panorama, which represents the relative relevance of the acoustic image data, and to display the freshness data on a display so that more recently captured acoustic image data is visually distinguishable from earlier captured acoustic image data, wherein at least one of the first and second acoustic image data portions includes multiple acoustic image frames, and the acoustic analysis system is positioned such that the current FOV of the acoustic sensor array is the first FOV and the first acoustic image data portion includes multiple acoustic image frames.

[0212] Further embodiments of the acoustic analysis system described herein may include any of the following features individually or in any combination: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of which is configured to receive an acoustic signal from a target scene and to output acoustic data based on the received acoustic signal; a display; a processor communicating with the acoustic sensor array and the display; the processor receiving acoustic data from the acoustic sensor array, generating acoustic image data of a target scene based on the received acoustic data, and dividing the acoustic image data into a plurality of acoustic image portions, the plurality of acoustic image portions including a first acoustic image portion and a second acoustic image portion. The system is configured to: the first acoustic image portion is different from the second acoustic image portion; determine a first predetermined acoustic reference set; determine a second predetermined acoustic reference set; generate a display image, wherein generating the display image includes visually comparing acoustic image data in the first acoustic image portion that satisfies the first predetermined acoustic parameter set with acoustic image data in the first acoustic image portion that does not satisfy the first predetermined acoustic reference set; visually comparing acoustic image data in the second acoustic image portion that satisfies the second predetermined acoustic reference set with acoustic image data in the second acoustic image portion that does not satisfy the second predetermined acoustic reference set; and present the display image on a display.

[0213] The acoustic analysis system may further include any of the following features individually or in any combination: dividing acoustic image data into multiple acoustic image parts, which includes receiving a selection of a first acoustic image part and receiving a selection of a second acoustic image part; receiving a selection of a first acoustic image part, which includes receiving a first selection from a user via a user interface, and receiving a selection of a second acoustic image part, which includes receiving a second selection from a user via a user interface; determining a first predetermined acoustic reference set, which includes receiving a first predetermined acoustic reference set from a user via a user interface, and determining a second predetermined acoustic reference set, which includes receiving a first predetermined acoustic reference set from a user via a user interface This includes receiving a second predetermined acoustic reference set from the user via a first acoustic image portion; visually comparing acoustic image data in a first acoustic image portion that satisfies a first predetermined acoustic parameter set with acoustic image data in a first acoustic image portion that does not satisfy the first predetermined acoustic reference set, which includes displaying acoustic image data in the first acoustic image portion for acoustic signals that satisfy the first predetermined acoustic reference set, and not displaying acoustic image data in the first acoustic image portion for acoustic signals that do not satisfy the first predetermined acoustic reference set; the first acoustic image portion includes acoustic image data from a first field of view, and the second acoustic image portion includes acoustic image data from a second field of view; and the displayed image includes a panorama including acoustic image data from the first and second fields of view.

[0214] The acoustic analysis system may further include the following features individually or in any combination: at least one of the first acoustic image portion and the second acoustic image portion includes a plurality of acoustic image frames; a first predetermined acoustic reference set includes one or more frequency ranges such that an acoustic signal not included in any of one or more frequency ranges does not satisfy the first predetermined reference set; a first predetermined acoustic reference set includes a first decibel range and a first frequency range such that an acoustic signal not within a first decibel range or not within a first frequency range does not satisfy the first predetermined acoustic reference set; the first predetermined acoustic reference set corresponds to a predetermined acoustic signature; the first predetermined acoustic reference set includes one or more alarm profiles associated with the first acoustic image portion; one or more alarm profiles correspond to objects in the first acoustic image portion, and one or more alarm profiles are associated with a predetermined location in the first acoustic image portion.

[0215] Further embodiments of the acoustic analysis system described herein may include any of the following features individually or in any combination: an acoustic sensor array comprising a plurality of acoustic sensor elements, each of which is configured to receive an acoustic signal from a target scene and to output acoustic data based on the received acoustic signal; a processor configured to communicate with the acoustic sensor array and receive acoustic data from the acoustic sensor array, generate acoustic image data of a target scene based on the received acoustic data, divide the acoustic image data into a plurality of acoustic image portions, associate an acoustic profile with each of the plurality of acoustic image portions, each acoustic profile comprising one or more predetermined acoustic criteria; and for each of the plurality of acoustic image portions, generate a display image comprising acoustic image data representing an acoustic signal that satisfies one or more predetermined acoustic criteria corresponding to the acoustic profile associated with the acoustic image portion, wherein such acoustic image data is presented within the corresponding acoustic image portion in a manner that distinguishes it from an acoustic signal that does not satisfy one or more predetermined acoustic criteria corresponding to the acoustic profile.

[0216] In addition, the acoustic analysis system may further comprise an electromagnetic imaging tool configured to communicate with a processor and generate electromagnetic image data of a target scene, the processor being configured to receive electromagnetic image data from the electromagnetic imaging tool and to align the electromagnetic image data and acoustic image data; the display image may further comprise a display that includes electromagnetic image data and acoustic image data and communicates with the processor, the processor being configured to present the display image on the display, and a user interface being further comprised, the processor being configured to receive input from the user interface that defines one or more acoustic image portions; the processor being configured to recognize one or more features in the received electromagnetic image data and to specify at least one portion of the acoustic image data based on the recognized one or more features; a single acoustic profile may comprise multiple acoustic image portions Related to minutes; at least one acoustic profile includes one or more alarm conditions; presenting acoustic image data that satisfies one or more predetermined acoustic criteria corresponding to an acoustic profile within an acoustic image portion in a manner that distinguishes it from acoustic signals that do not satisfy one or more predetermined acoustic criteria corresponding to an acoustic profile includes not displaying acoustic image data within an acoustic image portion that does not satisfy one or more predetermined acoustic criteria of the acoustic profile associated with that portion; the processor is further configured to sample acoustic data from a target scene over time and to automatically divide the acoustic image data into multiple acoustic image portions based on the sampling of acoustic data over time; and the processor is further configured to define at least one acoustic profile associated with one of the multiple acoustic image portions based on the sampling of acoustic data over time.

[0217] Further embodiments can be provided by combining the various embodiments described above. All U.S. and foreign patents, patent application publications, and non-patent publications mentioned herein and / or listed in the application data sheet are incorporated herein by reference in their entirety. Further embodiments can be provided by modifying the aspects of the embodiments and, where necessary, by using concepts from various patents, specifications, and publications.

[0218] In consideration of the details described above, these and other modifications can be made to the embodiments. In general, the terms used in the following claims should not be interpreted as limiting the claims to the specific embodiments disclosed in the specification and claims, but rather as including all possible embodiments along the entire scope of equivalents to which such claims are granted. Accordingly, the claims are not limited by the disclosure herein.

Claims

1. It is an acoustic analysis system, An acoustic sensor array configured to receive acoustic signals from a target scene and output acoustic data based on the acoustic signals, The acoustic sensor array communicates with the aforementioned acoustic sensor array, Receiving acoustic data from the aforementioned acoustic sensor array, Based on the aforementioned acoustic data, generate acoustic image data of the target scene. Identifying multiple parts of the target scene based on the aforementioned acoustic image data or the aforementioned acoustic data, Associating an acoustic profile with each part of the target scene, wherein each acoustic profile includes one or more acoustic criteria, An acoustic analysis system comprising: a processor configured to generate a display image including the plurality of parts of the target scene, wherein the acoustic image data is visually represented in each part of the target scene according to the acoustic profile associated with each part of the target scene.

2. The acoustic analysis system according to claim 1, wherein at least two acoustic profiles, each of which is the aforementioned acoustic profile, have one or more different acoustic criteria.

3. The acoustic analysis system according to claim 1 or 2, wherein the displayed image shows the boundaries of each of the plurality of parts of the target scene.

4. The acoustic analysis system according to any one of claims 1 to 3, wherein the plurality of parts of the target scene are identified based on one or more of the following: the position of the acoustic signal in the target scene, the acoustic parameters of the acoustic signal in the target scene, the orientation of an object in the target scene with respect to the acoustic analysis system, and one or more features or objects identified in the target scene.

5. The acoustic analysis system according to any one of claims 1 to 4, wherein the processor is configured to define at least one acoustic profile associated with at least one portion of the target scene based on one or more parameters of the acoustic data received from the acoustic sensor array.

6. The acoustic analysis system according to any one of claims 1 to 5, wherein an alarm profile is associated with a portion of the target scene, and the alarm profile includes an acoustic criterion such that the acoustic analysis system notifies of an alarm state when the acoustic signal in the portion of the target scene is filled.

7. The acoustic analysis system according to any one of claims 1 to 6, wherein acoustic image data of different parts of the target scene are represented in different manner in the display image according to different acoustic criteria of each acoustic profile.

8. The acoustic analysis system according to any one of claims 1 to 7, wherein each acoustic profile has a periodicity criterion, and acoustic image data representing an acoustic signal that does not satisfy the periodicity criterion is not represented in the display image, or is visually represented in the display image in a manner different from the acoustic image data representing an acoustic signal that satisfies the periodicity criterion.

9. The acoustic analysis system according to any one of claims 1 to 8, wherein each acoustic profile has a frequency reference, and acoustic image data representing an acoustic signal that does not satisfy the frequency reference is not represented in the display image, or is visually represented in the display image in a manner different from the acoustic image data representing an acoustic signal that satisfies the frequency reference.

10. The acoustic analysis system according to any one of claims 1 to 9, wherein each acoustic profile has an intensity standard, and acoustic image data representing an acoustic signal that does not meet the intensity standard is not represented in the display image, or is visually represented in the display image in a manner different from the acoustic image data representing an acoustic signal that meets the intensity standard.

11. The acoustic analysis system according to any one of claims 1 to 10, wherein each acoustic profile has an acoustic signature, and acoustic image data representing an acoustic signal that does not match the acoustic signature is not represented in the display image, or is visually represented in the display image in a manner different from the acoustic image data representing an acoustic signal that matches the acoustic signature.

12. A method of acoustic analysis, Receiving acoustic data from an acoustic sensor array, Based on the aforementioned acoustic data, generate acoustic image data of the target scene. Identifying multiple parts of the target scene based on the aforementioned acoustic image data or the aforementioned acoustic data, Associating acoustic profiles with each part of the target scene, wherein at least two acoustic profiles differ in one or more acoustic criteria, and The method involves generating a display image that includes the plurality of parts of the target scene, wherein the acoustic image data in each part of the target scene is visually represented in the display image according to the respective acoustic profile associated with the part of the target scene. A method for acoustic analysis that includes the ability to perform the following actions.

13. The acoustic analysis method according to claim 12, wherein identifying the plurality of parts of the target scene includes identifying the plurality of parts based on one or more of the following: the position of an acoustic signal in the target scene, the acoustic parameters of an acoustic signal in the target scene, the orientation of an object in the target scene, and one or more features or objects identified in the target scene.

14. The acoustic analysis method according to claim 12 or 13, further comprising: each acoustic profile having an alarm criterion; and notifying an alarm state when the alarm criterion is met.

15. The acoustic analysis method according to any one of claims 12 to 14, wherein each acoustic profile has an acoustic criterion that includes one or more of the following: a periodicity criterion, a frequency criterion, an intensity criterion, or an acoustic signature, and acoustic image data representing an acoustic signal that does not satisfy the acoustic criterion is excluded from the display image or is visually represented in the display image in a manner different from the acoustic image data representing an acoustic signal that satisfies the acoustic criterion.

Images