A kind of acoustic and optical multi-source imaging method and system

Abstract

The embodiments of this specification mainly relate to the field of acoustic-optical multi-source imaging technology, specifically, an acoustic-optical multi-source imaging method and system. An acoustic-optical multi-source imaging method, the method comprising: L1. A first control processing module acquires a visible light image through a visible light image acquisition module, and acquires an infrared image through an infrared image acquisition module at the same time; a second control processing module acquires sound data through a sound data acquisition module; L2. After acquiring the sound data, the second control processing module determines whether the sound data triggers a threshold; when the sound data triggers the threshold, the second control processing module generates a threshold triggered signal, and sends the threshold triggered signal to the first control processing module. The acoustic-optical multi-source imaging method and system of the embodiments of this specification can save a large number of steps of converting sound data into sound source images and combining the sound source images with visible light images, thereby effectively reducing the amount of calculation of imaging fusion in the acoustic-optical multi-source imaging process.

Description

Technical Field

[0001] The embodiments of this specification mainly relate to the technical field of acoustic-optical multi-source imaging, and specifically to an acoustic-optical multi-source imaging method and system. Background Art

[0002] In the power industry, gas detection industry, machinery industry and other industries, the method of acoustic and optical multi-source imaging is usually used to locate defects such as partial discharge, gas leakage, vibration and abnormal noise on the equipment that needs to be inspected. During acoustic and optical multi-source imaging, it is necessary to first convert the sound source data into a sound source image, then combine the sound source image with the visible light image to obtain an acoustic image, and then combine the acoustic image with the infrared image to obtain the final image, and finally determine whether the inspected equipment has defects through the final imaging. The imaging fusion calculation amount of this acoustic and optical multi-source imaging method is large (especially when the light source image and sound source data are acquired at high frequency, the imaging fusion calculation amount is large), resulting in low imaging efficiency. In addition, the acquisition frequency of the light source image and the sound source data is usually different, which makes acoustic and optical multi-source imaging more difficult. Summary of the invention

[0003] In view of the problems existing in the prior art, the embodiments of this specification propose an acoustic-optical multi-source imaging method and system to solve the problem of large imaging fusion calculation amount in the acoustic-optical multi-source imaging process.

[0004] In a first aspect, an embodiment of the present specification provides an acoustic-optical multi-source imaging method, the method comprising:

[0005] L1. The first control processing module acquires a visible light image through a visible light image acquisition module and acquires an infrared image through an infrared image acquisition module; the second control processing module acquires sound data through a sound data acquisition module;

[0006] L2. After acquiring the sound data, the second control processing module determines whether the sound data triggers the threshold; when the sound data triggers the threshold, the second control processing module generates a threshold triggered signal, and sends the threshold triggered signal to the first control processing module, and the second control processing module sends the sound data to the first control processing module; when the sound data does not trigger the threshold, the second control processing module generates a threshold not triggered signal, and sends the threshold not triggered signal to the first control processing module;

[0007] L3. After receiving the threshold triggered signal and sound data, the first control processing module generates an acoustic image based on the sound data and the visible light image, and sends the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, the first control processing module sends the visible light image and the infrared image to the image processing module.

[0008] Preferably, L1 further comprises:

[0009] The first control processing module generates a sampling signal and synchronously sends the sampling signal to the second control processing module; after generating the sampling signal, the first control processing module obtains a visible light image through the visible light image acquisition module, and simultaneously obtains an infrared image through the infrared image acquisition module; after receiving the sampling signal, the second control processing module obtains a piece of sound data through the sound data acquisition module.

[0010] Preferably, in L2, determining whether the sound data triggers the threshold specifically includes:

[0011] L21. When the sound data includes S-channel sound data, the S-channel sound data are discretely calculated by FFT, and the amplitude of each frequency point after the S-channel sound data are discretely calculated by the following amplitude matrix;

[0012]

[0013] Wherein, M represents the amplitude matrix of the recorded amplitude; S represents the number of sub-sound data in the sound data; N represents the number of frequency points after the discrete FFT calculation of the sub-sound data; Indicates the amplitude of the Nth frequency point of the Sth channel sound data;

[0014] L22. Determine the lower limit frequency value f1 and the upper limit frequency value f2, and calculate the energy value of each channel of sound data based on the lower limit frequency value f1 and the upper limit frequency value f2 and the amplitude matrix of the sound data;

[0015] L23. Determine a first threshold value, and calculate the deviation value based on the first threshold value and the energy value of each channel sound data;

[0016] L24. Determine the second threshold based on the first threshold, and compare the deviation degree value with the second threshold; when the deviation degree value is greater than the second threshold, determine that the sound data triggers the threshold; otherwise, determine that the sound data does not trigger the threshold.

[0017] Preferably, in L22, the energy value of each channel of sound data is calculated by the following formula:

[0018]

[0019] in, represents the energy value of the ith sub-sound data; N1 represents the frequency point value corresponding to the lower limit frequency value f1 in the ith sub-sound data; N2 represents the frequency point value corresponding to the upper limit frequency value f2 in the ith sub-sound data; Represents the amplitude of the jth frequency point of the i-th sub-sound data, where i ranges from 1 to S; j ranges from N1 to N2; and N represents the total number of frequency points after the discrete FFT calculation of the sub-sound data.

[0020] Preferably, in L23, the deviation degree value is calculated by the following formula:

[0021]

[0022] in, Indicates the degree of deviation; Represents the energy value of the i-th channel sound data; represents the first threshold; S represents the number of sub-sound data in the sound data.

[0023] Preferably, in L24, the second threshold is 0.5 to 0.7 times the first threshold.

[0024] Preferably, in L3, generating an acoustic image based on sound data and a visible light image specifically includes:

[0025] L31. Processing the sound data by a beamforming algorithm to obtain a sound intensity data matrix, and converting the sound intensity data matrix into a sound intensity image;

[0026] L32. Acquire an image area having an acoustic intensity value greater than a third threshold value on the acoustic intensity image, and combine the image area with the visible light image to obtain an acoustic image.

[0027] Preferably, the method further comprises:

[0028] L4. The image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain a final image, and sends the final image to the display module for display.

[0029] Preferably, in L4, the image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain the final image, specifically including:

[0030] L41. Adjusting the aspect ratio of the infrared image to obtain an infrared adjusted image;

[0031] L42. Adjusting the center position and aspect ratio of the acoustic image to obtain an acoustically adjusted image or adjusting the center position and aspect ratio of the visible light image to obtain a visible light adjusted image;

[0032] L43. Stitch the infrared adjusted image with the acoustic adjusted image or stitch the infrared adjusted image with the visible light adjusted image to obtain the final image.

[0033] In a second aspect, an embodiment of the present specification provides an acoustic-optical multi-source imaging system, the system comprising:

[0034] A visible light image acquisition module, used for acquiring visible light images;

[0035] Infrared image acquisition module, used for acquiring infrared images;

[0036] A sound data collection module, used for collecting sound data;

[0037] A first control processing module is electrically connected to the visible light image acquisition module and the infrared image acquisition module;

[0038] A second control processing module is electrically connected to the sound data acquisition module and the first control processing module;

[0039] An image processing module, electrically connected to the second control processing module;

[0040] A display module electrically connected to the image processing module;

[0041] The first control processing module is used to generate a sampling signal and synchronously send the sampling signal to the second control processing module; and after generating the sampling signal, a visible light image is acquired through the visible light image acquisition module, and an infrared image is acquired through the infrared image acquisition module;

[0042] The second control processing module is used to obtain sound data through the sound data acquisition module after receiving the sampling signal; and after obtaining the sound data, determine whether the sound data triggers the threshold; when the sound data triggers the threshold, generate a threshold triggered signal and send the threshold triggered signal to the first control processing module, and send the sound data to the first control processing module; when the sound data does not trigger the threshold, generate a threshold not triggered signal and send the threshold not triggered signal to the first control processing module;

[0043] The first control processing module is further used to generate an acoustic image based on the sound data and the visible light image after receiving the threshold triggered signal and the sound data, and send the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, send the visible light image and the infrared image to the image processing module;

[0044] The image processing module is used to process the infrared image and the acoustic image or the infrared image and the visible light image to obtain a final image, and send the final image to the display module for display.

[0045] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least:

[0046] A second control processing module is provided, and the second control processing module first determines whether the sound data triggers a threshold. If the sound data triggers the threshold, the second control processing module sends the sound data and the threshold triggered signal to the first control processing module. After receiving the threshold triggered signal, the first control processing module converts the sound data into a sound source image and combines the sound source image with the visible light image to generate an acoustic image, and finally sends the acoustic image and the infrared image to the image processing module for combination to form a final imaging; if the sound data does not trigger the threshold, the second control processing module will not send the sound data to the first control processing module, but only sends the threshold not triggered signal to the first control processing module. After receiving the threshold not triggered signal, the first control processing module directly sends the visible light image and the infrared image to the image processing module for combination to form a final imaging; that is, through the acoustic-optical multi-source imaging method and system of the embodiment of the present specification, a large number of steps of converting the sound data into the sound source image and combining the sound source image with the visible light image can be omitted, thereby effectively reducing the amount of calculation of imaging fusion in the acoustic-optical multi-source imaging process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an acoustic-optical multi-source imaging method provided in an embodiment of this specification;

[0048] Figure 2 A schematic diagram of the composition of the acousto-optic multi-source imaging system provided in an embodiment of this specification. DETAILED DESCRIPTION

[0049] Embodiments of the present specification will be described in more detail below with reference to the accompanying drawings.

[0050] Embodiment 1:

[0051] An acoustic-optical multi-source imaging method, combining Figure 1 and Figure 2 As shown, the method includes:

[0052] L1. The first control processing module acquires a visible light image through the visible light image acquisition module, and acquires an infrared image through the infrared image acquisition module; the second control processing module acquires sound data through the sound data acquisition module.

[0053] L2. After acquiring the sound data, the second control processing module determines whether the sound data triggers the threshold; when the sound data triggers the threshold, the second control processing module generates a threshold triggered signal, and sends the threshold triggered signal to the first control processing module, and at the same time the second control processing module sends the sound data to the first control processing module; when the sound data does not trigger the threshold, the second control processing module generates a threshold not triggered signal, and sends the threshold not triggered signal to the first control processing module.

[0054] L3. After receiving the threshold triggered signal and sound data, the first control processing module generates an acoustic image based on the sound data and the visible light image, and sends the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, the first control processing module sends the visible light image and the infrared image to the image processing module.

[0055] The acoustic-optical multi-source imaging method of this embodiment is implemented by an acoustic-optical multi-source imaging device, which includes a movable component and a display component. The movable component includes a frame, a visible light image acquisition module, an infrared image acquisition module, a sound data acquisition module, a first control processing module, a second control processing module and an image processing module arranged on the frame. The display component includes a display screen. The display component and the movable component can be connected by wire or wirelessly.

[0056] The visible light image acquisition module may be a visible light camera, the infrared image acquisition module may be a thermal imaging camera, and the sound data acquisition module may be a large-scale sound array. The first control processing module may be a central processing unit, and the second control processing module may be a field programmable logic gate array.

[0057] Assuming that a certain area needs to be inspected, the staff only needs to use the movable component to "scan" the inspected equipment. The movable component can form a "final image" based on the acquired visible light image, infrared image, and sound data and send the "final image" to the display component for display. The staff can use the "final image" to determine whether there is a problem with the inspected equipment.

[0058] In the prior art, each acquired sound data needs to be converted into a sound source image first, and then the sound source image is combined with the visible light image to generate an acoustic image, and finally the acoustic image is combined with the infrared image to obtain the final image. This method makes the imaging fusion process of acoustic and optical multi-source imaging computationally intensive.

[0059] To this end, the present embodiment sets a second control processing module, through which the second control processing module first determines whether the sound data triggers the threshold (i.e., determines whether the sound data has a problem). If the sound data triggers the threshold (i.e., there is a problem), the second control processing module sends the sound data and the threshold triggered signal to the first control processing module. After receiving the threshold triggered signal, the first control processing module converts the sound data into a sound source image and combines the sound source image with the visible light image to generate an acoustic image, and finally sends the acoustic image and the infrared image to the image processing module for combination to form the final imaging. If the sound data does not trigger the threshold (i.e., there is no problem), the second control processing module will not send the sound data to the first control processing module, but only send the threshold not triggered signal to the first control processing module. After receiving the threshold not triggered signal, the first control processing module directly sends the visible light image and the infrared image to the image processing module for combination to form the final imaging.

[0060] That is, the acousto-optic multi-source imaging method of this embodiment can save a lot of steps of converting sound data into sound source images and combining the sound source images with visible light images, thereby effectively reducing the amount of calculation of imaging fusion in the acousto-optic multi-source imaging process.

[0061] Furthermore, L1 in this embodiment also includes:

[0062] The first control processing module generates a sampling signal and synchronously sends the sampling signal to the second control processing module; after generating the sampling signal, the first control processing module obtains a visible light image through the visible light image acquisition module, and simultaneously obtains an infrared image through the infrared image acquisition module; after receiving the sampling signal, the second control processing module obtains a piece of sound data through the sound data acquisition module.

[0063] In the prior art, the acquisition frequencies of visible light images, infrared images and sound data are usually different. If the acquisition frequencies differ greatly, there is a problem that the visible light image, infrared image and sound data acquired for the second time do not correspond to the same detected position. Then, when performing image fusion processing, it is necessary to calculate based on the frequency difference to find the visible light image, infrared image and sound data belonging to the same detected position, which makes the acoustic and optical multi-source imaging more difficult.

[0064] In this embodiment, a timer is set inside the first control processing module, and a sampling signal is periodically generated by the timer. After the first control processing module and the second control processing module obtain a sampling signal, they will correspondingly obtain a visible light image, an infrared image and a piece of sound data. This setting makes the acquisition frequency of the visible light image, infrared image and sound data almost the same. That is, the visible light image, infrared image and sound data obtained at a certain time point correspond to the same detected position, and the visible light image, infrared image and sound data obtained each time can be directly processed as a group of data for image fusion, which greatly reduces the difficulty of sound and light multi-source imaging.

[0065] Furthermore, in L2 of this embodiment, determining whether the sound data triggers the threshold specifically includes:

[0066] L21. When the sound data includes S-channel sound data, the S-channel sound data are discretely calculated by FFT, and the amplitude of each frequency point after the S-channel sound data are discretely calculated by the following amplitude matrix;

[0067]

[0068] Wherein, M represents the amplitude matrix of the recorded amplitude; S represents the number of sub-sound data in the sound data; N represents the number of frequency points after the discrete FFT calculation of the sub-sound data; Indicates the amplitude of the Nth frequency point of the Sth channel sound data.

[0069] The sound data acquisition module (i.e., large-scale sound array) in this embodiment can be a 128-channel microphone array, which is equivalent to installing 128 microphones for sound acquisition. When the large-scale sound array acquires a section of sound data (approximately 10ms), the large-scale sound array includes several microphones, and then this section of sound data includes several sub-sound data. Assuming it is a 128-channel microphone array, the sound data includes 128 sub-sound data. Assuming it is a 168-channel microphone array, the sound data includes 168 sub-sound data.

[0070] This step is used to perform discrete FFT calculation on each channel of sound data. Since the length of each channel of sound data is the same, the frequency points (including the number of frequency points and the frequency value of each frequency point) obtained after the discrete FFT calculation of each channel of sound data are the same. Then, the amplitude of each frequency point of each channel of sound data is obtained to form an amplitude matrix.

[0071] L22. Determine the lower limit frequency value f1 and the upper limit frequency value f2, and calculate the energy value of each channel of sound data based on the lower limit frequency value f1 and the upper limit frequency value f2 and the amplitude matrix of the sound data.

[0072] The lower limit frequency value f1 and the upper limit frequency value f2 can be determined according to the actual detected device. The frequency of the sound emitted by the detected device is between the lower limit frequency value f1 and the upper limit frequency value f2. In different application scenarios (i.e., problem detection for different devices), the determined lower limit frequency value f1 and upper limit frequency value f2 are different. This setting can filter out other sounds in each channel of sound data as much as possible.

[0073] Taking the first channel sound data as an example, after performing discrete FFT calculation on the first channel sound data, we can get: the first frequency point (its frequency point value is 1, the frequency value is a1), the second frequency point (its frequency point value is 2, the frequency value is a2), the third frequency point (its frequency point value is 3, the frequency value is a3) ... the Nth frequency point (its frequency point value is N, the frequency value is an). Then compare the lower limit frequency value f1 and the upper limit frequency value f2 with the frequency values ​​of different frequency points. When the lower limit frequency value f1 is the same as the frequency value a11, and the upper limit frequency value f2 is the same as the frequency value a28, then the frequency point value corresponding to the lower limit frequency value f1 in the first channel sound data is 11 (that is, N1 is 11), and the frequency point value corresponding to the upper limit frequency value f2 in the first channel sound data is 28 (that is, N2 is 28).

[0074] In L22 of this embodiment, the energy value of each channel of sound data is calculated by the following formula:

[0075]

[0076] in, represents the energy value of the ith sub-sound data; N1 represents the frequency point value corresponding to the lower limit frequency value f1 in the ith sub-sound data; N2 represents the frequency point value corresponding to the upper limit frequency value f2 in the ith sub-sound data; Represents the amplitude of the jth frequency point of the i-th sub-sound data, where i ranges from 1, 2, 3 to S; j ranges from N1 to N2; N represents the total number of frequency points after discrete FFT calculation of the sub-sound data.

[0077] Taking the first channel of sound data as an example, assuming that N1 is 11 and N2 is 28, then we can find the amplitude corresponding to the 11th row of the 1st column, the 12th row of the 1st column, the 13th row of the 1st column, and so on, the 28th row of the 1st column from the amplitude matrix. Assuming that N is 100, we can calculate the energy value of the first channel of sound data using the above formula.

[0078] In a similar manner, the energy value of each channel of sound data can be calculated through step L22.

[0079] L23. Determine a first threshold, and calculate a deviation degree value based on the first threshold and the energy value of each channel of sound data.

[0080] In L23 of this embodiment, the deviation degree value is calculated by the following formula:

[0081]

[0082] in, Indicates the degree of deviation; Represents the energy value of the i-th channel sound data; represents the first threshold; S represents the number of sub-sound data in the sound data.

[0083] The first threshold value needs to be determined in advance based on a large number of tests, and the first threshold values ​​of different detected devices may be different. In this step, it is only necessary to first calculate the difference between the energy value of each channel of sound data and the first threshold value to obtain the energy difference value of each channel of sound data, and then calculate the square of the energy difference value of each channel of sound data to obtain the square of the energy difference value, and finally sum up all the square of the energy difference values ​​to obtain the deviation degree value of the sound data.

[0084] L24. Determine a second threshold based on the first threshold, and compare the deviation value with the second threshold. When the deviation value is greater than the second threshold, determine that the sound data triggers the threshold; otherwise, determine that the sound data does not trigger the threshold.

[0085] The second threshold is 0.5 to 0.7 times the first threshold, and the specific second threshold is 0.6 times the first threshold. When the first threshold is determined, the second threshold can also be determined.

[0086] This step only requires comparing the deviation value with the second threshold value. When the deviation value is greater than the second threshold value, it is determined that the sound data triggers the threshold value; otherwise, it is determined that the sound data does not trigger the threshold value.

[0087] When the sound data triggers the threshold, the second control processing module generates a threshold triggered signal and sends the threshold triggered signal to the first control processing module, and the second control processing module sends the sound data to the first control processing module; when the sound data does not trigger the threshold, the second control processing module generates a threshold not triggered signal and sends the threshold not triggered signal to the first control processing module.

[0088] If the first control processing module receives the threshold triggered signal and the sound data, it generates an acoustic image based on the sound data and the visible light image, and sends the acoustic image and the infrared image to the image processing module. If the first control processing module only receives the threshold untriggered signal, it directly sends the visible light image and the infrared image to the image processing module.

[0089] Furthermore, in L3 of this embodiment, generating an acoustic image based on the sound data and the visible light image specifically includes:

[0090] L31. Process the sound data through a beamforming algorithm to obtain a sound intensity data matrix, and convert the sound intensity data matrix into a sound intensity image.

[0091] This embodiment can directly adopt the existing beamforming algorithm. In this step, it is only necessary to process the current segment of sound data through the beamforming algorithm to obtain a sound intensity data matrix. Then, the sound intensity values ​​in the sound intensity data matrix are mapped to color values ​​to convert the sound intensity matrix into a sound intensity image (one sound intensity value is converted into one color value).

[0092] L32. Acquire an image area having an acoustic intensity value greater than a third threshold value on the acoustic intensity image, and combine the image area with the visible light image to obtain an acoustic image.

[0093] The third threshold needs to be determined in advance based on a large number of tests, and the third threshold of different detected devices will be different. This step only requires marking the image contour of the image area (assuming it is abnormal area a) with a sound intensity value greater than the third threshold on the sound intensity image, and then obtaining the contour position information of the image contour (the contour position information can be represented by the SHP graphic file), and finally combining the SHP graphic file with the visible light image, that is, automatically marking the abnormal area b (corresponding to the abnormal area a) on the visible light image according to the contour position information provided by the SHP graphic file, and automatically filling the mask in the image contour of the abnormal area b.

[0094] The acoustic image of this step is equivalent to marking the abnormal area on the visible light image. The abnormal area includes the outline of the abnormal area and the prompt color filled in the outline of the abnormal area. The outline of the abnormal area is determined according to the SHP graphic file, and this SHP graphic file is obtained based on the sound intensity image (first find the abnormal area on the sound intensity image, and then use the SHP graphic file to represent the outline position information of the abnormal area on the sound intensity image).

[0095] In short, when there is an abnormality in the sound data (that is, when the sound data triggers the threshold), the sound data is first combined with the visible light image to obtain the acoustic image, and then the acoustic image and the infrared image are sent to the image processing module. When there is no abnormality in the sound data (that is, when the sound data does not trigger the threshold), the visible light image and the infrared image are directly sent to the image processing module.

[0096] Furthermore, the acousto-optic multi-source imaging method of this embodiment further includes:

[0097] L4. The image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain the final image, and sends the final image to the display module for display. In L4, the image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain the final image specifically including:

[0098] L41. Adjust the aspect ratio of the infrared image to obtain an infrared adjusted image.

[0099] L42. Adjust the center position and the aspect ratio of the acoustic image to obtain an acoustically adjusted image, or adjust the center position and the aspect ratio of the visible light image to obtain a visible light adjusted image.

[0100] L43. Stitch the infrared adjusted image with the acoustic adjusted image or stitch the infrared adjusted image with the visible light adjusted image to obtain the final image.

[0101] In order to avoid the final image being distorted on the display screen, the aspect ratio of the final image needs to be set to the same as the aspect ratio of the display screen. Assuming the aspect ratio of the display screen is 3:2, the aspect ratio of the final image also needs to be 3:2. The final image is obtained by splicing the infrared adjusted image and the acoustic adjusted image / visible light adjusted image in the width direction, so the aspect ratio of the infrared adjusted image needs to be 1:2, and the aspect ratio of the acoustic adjusted image / visible light adjusted image needs to be 2:2.

[0102] That is, in this embodiment, the aspect ratio of the infrared image needs to be adjusted first through step L41 to obtain an infrared adjusted image with an aspect ratio of 1:2. Then, the aspect ratio of the acoustic image / visible light image is adjusted through step L42 to obtain an acoustic adjusted image / visible light adjusted image with an aspect ratio of 2:2. Finally, through step L43, the infrared adjusted image with an aspect ratio of 1:2 and the acoustic adjusted image / visible light adjusted image with an aspect ratio of 2:2 are spliced ​​to obtain a final image with an aspect ratio of 3:2. The final image will not be distorted when displayed on the display screen.

[0103] In addition, since the installation positions of the visible light image acquisition module and the infrared image acquisition module on the frame are different, the center position of the collected visible light image and the center position of the collected infrared image are different. For example, the center position of the infrared image is the middle position of object A, while the center position of the visible light image is slightly above the middle position of object A. Then, the center position of the visible light image needs to be adjusted so that the center position of the visible light image is also the middle position of object A, and finally the center position of the visible light image is roughly aligned with the center position of the infrared image.

[0104] In short, this step enables the detected device to be better displayed in the final image.

[0105] Embodiment 2:

[0106] An acoustic-optical multi-source imaging system, used to implement the acoustic-optical multi-source imaging method in embodiment 1, combined with Figure 2 As shown, the system includes:

[0107] A visible light image acquisition module, used for acquiring visible light images;

[0108] Infrared image acquisition module, used for acquiring infrared images;

[0109] A sound data collection module, used for collecting sound data;

[0110] A first control processing module is electrically connected to the visible light image acquisition module and the infrared image acquisition module;

[0111] A second control processing module is electrically connected to the sound data acquisition module and the first control processing module;

[0112] An image processing module, electrically connected to the second control processing module;

[0113] A display module electrically connected to the image processing module;

[0114] The first control processing module is used to generate a sampling signal and synchronously send the sampling signal to the second control processing module; and after generating the sampling signal, a visible light image is acquired through the visible light image acquisition module, and an infrared image is acquired through the infrared image acquisition module;

[0115] The second control processing module is used to obtain sound data through the sound data acquisition module after receiving the sampling signal; and after obtaining the sound data, determine whether the sound data triggers the threshold; when the sound data triggers the threshold, generate a threshold triggered signal and send the threshold triggered signal to the first control processing module, and send the sound data to the first control processing module; when the sound data does not trigger the threshold, generate a threshold not triggered signal and send the threshold not triggered signal to the first control processing module;

[0116] The first control processing module is further used to generate an acoustic image based on the sound data and the visible light image after receiving the threshold triggered signal and the sound data, and send the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, send the visible light image and the infrared image to the image processing module;

[0117] The image processing module is used to process the infrared image and the acoustic image or the infrared image and the visible light image to obtain a final image, and send the final image to the display module for display.

[0118] Although some embodiments of the present specification are shown in the drawings, it should be understood that the present specification can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. Instead, these embodiments are provided for a more thorough and complete understanding of the present specification. It should be understood that the drawings and embodiments of the present specification are only for exemplary purposes and are not intended to limit the scope of protection of the present specification.

Claims

1. An acousto-optic multi-source imaging method, characterized in that: The method comprises: L1. The first control processing module acquires a visible light image through a visible light image acquisition module and acquires an infrared image through an infrared image acquisition module; the second control processing module acquires sound data through a sound data acquisition module; L2. After acquiring the sound data, the second control processing module determines whether the sound data triggers a threshold; when the sound data triggers the threshold, the second control processing module generates a threshold triggered signal, and sends the threshold triggered signal to the first control processing module, and the second control processing module sends the sound data to the first control processing module; when the sound data does not trigger the threshold, the second control processing module generates a threshold not triggered signal, and sends the threshold not triggered signal to the first control processing module; L3. After receiving the threshold triggered signal and the sound data, the first control processing module generates an acoustic image based on the sound data and the visible light image, and sends the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, the first control processing module sends the visible light image and the infrared image to the image processing module; In the L2, determining whether the sound data triggers a threshold specifically includes: L21. When the sound data includes S-way sound data, the S-way sound data are discrete FFT calculations, and the amplitude of each frequency point after the S-way sound data are discretely calculated by the following amplitude matrix; Wherein, M represents the amplitude matrix of the recorded amplitude; S represents the number of sub-sound data in the sound data; N represents the number of frequency points after the discrete FFT calculation of the sub-sound data; Indicates the amplitude of the Nth frequency point of the Sth channel sound data; L22. Determine a lower frequency value f1 and an upper frequency value f2, and calculate the energy value of each channel of sound data based on the lower frequency value f1 and the upper frequency value f2 and the amplitude matrix of the sound data; L23. Determine a first threshold value, and calculate the deviation degree value based on the first threshold value and the energy value of each channel sound data; L24. Determine a second threshold based on the first threshold, and compare the deviation value with the second threshold; when the deviation value is greater than the second threshold, determine that the sound data triggers the threshold; otherwise, determine that the sound data does not trigger the threshold.

2. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: The L1 also includes: The first control processing module generates a sampling signal and synchronously sends the sampling signal to the second control processing module; after generating the sampling signal, the first control processing module obtains a visible light image through the visible light image acquisition module and obtains an infrared image through the infrared image acquisition module; after receiving the sampling signal, the second control processing module obtains a piece of sound data through the sound data acquisition module.

3. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: In L22, the energy value of each channel of sound data is calculated specifically by the following formula: in, represents the energy value of the ith sub-sound data; N1 represents the frequency point value corresponding to the lower limit frequency value f1 in the ith sub-sound data; N2 represents the frequency point value corresponding to the upper limit frequency value f2 in the ith sub-sound data; Represents the amplitude of the jth frequency point of the i-th sub-sound data, where i ranges from 1 to S; j ranges from N1 to N2; and N represents the total number of frequency points after the discrete FFT calculation of the sub-sound data.

4. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: In the L23, the deviation degree value is calculated specifically by the following formula: in, Indicates the degree of deviation; Represents the energy value of the i-th channel sound data; represents the first threshold; S represents the number of sub-sound data in the sound data.

5. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: In L24, the second threshold is 0.5 to 0.7 times the first threshold.

6. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: In the L3, generating an acoustic image based on the sound data and the visible light image specifically includes: L31. The sound data is processed by a beamforming algorithm to obtain an intensity data matrix, and the intensity data matrix is ​​converted into an intensity image; L32. Acquire an image area having an acoustic intensity value greater than a third threshold value on the acoustic intensity image, and combine the image area with the visible light image to obtain an acoustic image.

7. The method for acousto-optic multi-source imaging according to claim 1, characterized in that: The method further comprises: L4. The image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain a final image, and sends the final image to the display module for display.

8. The method for acousto-optic multi-source imaging according to claim 7, characterized in that: In the L4, the image processing module processes the infrared image and the acoustic image or processes the infrared image and the visible light image to obtain the final image, specifically including: L41. The aspect ratio of the infrared image is adjusted to obtain an infrared adjusted image; L42. Adjusting the center position and the aspect ratio of the acoustic image to obtain an acoustically adjusted image or adjusting the center position and the aspect ratio of the visible light image to obtain a visible light adjusted image; L43. Stitch the infrared adjusted image with the acoustic adjusted image or stitch the infrared adjusted image with the visible light adjusted image to obtain a final image.

9. An acoustic-optical multi-source imaging system, characterized in that: The system comprises: A visible light image acquisition module, used for acquiring visible light images; Infrared image acquisition module, used for acquiring infrared images; A sound data collection module, used for collecting sound data; A first control processing module, electrically connected to the visible light image acquisition module and the infrared image acquisition module; A second control processing module, electrically connected to the sound data acquisition module and the first control processing module; An image processing module, electrically connected to the second control processing module; A display module, electrically connected to the image processing module; The first control processing module is used to generate a sampling signal and synchronously send the sampling signal to the second control processing module; and after generating the sampling signal, a visible light image is acquired through the visible light image acquisition module, and an infrared image is acquired through the infrared image acquisition module; The second control processing module is used to obtain sound data through the sound data acquisition module after receiving the sampling signal; and after obtaining the sound data, determine whether the sound data triggers a threshold; when the sound data triggers the threshold, generate a threshold triggered signal and send the threshold triggered signal to the first control processing module, and send the sound data to the first control processing module; when the sound data does not trigger the threshold, generate a threshold not triggered signal and send the threshold not triggered signal to the first control processing module; The first control processing module is further configured to generate an acoustic image based on the sound data and the visible light image after receiving the threshold triggered signal and the sound data, and send the acoustic image and the infrared image to the image processing module; or, after receiving the threshold untriggered signal, send the visible light image and the infrared image to the image processing module; The image processing module is used to process the infrared image and the acoustic image or the infrared image and the visible light image to obtain a final image, and send the final image to the display module for display; Determining whether the sound data triggers a threshold specifically includes: When the sound data includes S-channel sound data, discrete FFT calculations are performed on the S-channel sound data respectively, and the amplitude of each frequency point after the discrete FFT calculations are performed on the S-channel sound data respectively is recorded through the following amplitude matrix; Wherein, M represents the amplitude matrix of the recorded amplitude; S represents the number of sub-sound data in the sound data; N represents the number of frequency points after the discrete FFT calculation of the sub-sound data; Indicates the amplitude of the Nth frequency point of the Sth channel sound data; Determine a lower limit frequency value f1 and an upper limit frequency value f2, and calculate the energy value of each channel of sound data based on the lower limit frequency value f1 and the upper limit frequency value f2 and the amplitude matrix of the sound data; Determine a first threshold, and calculate a deviation degree value based on the first threshold and the energy value of each channel of sound data; A second threshold is determined based on the first threshold, and the deviation degree value is compared with the second threshold; when the deviation degree value is greater than the second threshold, the sound data is determined to have triggered a threshold; otherwise, it is determined that the sound data has not triggered a threshold.

Images