Multi-modal space-time-frequency visual fusion reconstruction method based on acoustic image-infrared-visible light
Through the multi-modal space-time and time-frequency visual fusion reconstruction technology of sound-image-infrared-visible light, the problems of inaccurate fault detection and difficult operation and maintenance in power equipment are solved, more accurate fault identification and detection are achieved, and power equipment failures are improved Accuracy and reliability of the detection.
Patent Information
- Application Number
- CN202410861627.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-28
AI Technical Summary
In power equipment, due to the complex working environment and high background noise interference, traditional fault identification methods are difficult to meet monitoring requirements, defect signals are easily overwhelmed, resulting in increased operation and maintenance difficulties, inaccurate fault detection and reduced sensitivity.
Using multimodal space-time video fusion reconstruction technology based on audio-image-infrared-visible light, the camera parameters and pose parameters are calibrated, and the forward and reverse projection of the image is carried out to generate matching The accurate infrared and acoustic images are fused with the original visible light image to form a multimodal registration reconstruction image.
It realizes more comprehensive and accurate identification of power equipment status, improves the accuracy and reliability of fault detection, solves the problems of low data dimensions and weak cross-domain analysis capabilities, and provides unmanned intelligent power fault location data sets Collect support.
Smart Images

Figure CN118918422A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal fusion of electric power equipment, and in particular to a multimodal space-time frequency visual fusion reconstruction technology based on audio-visual-infrared-visible light. Background Art
[0002] In the field of electric power, modern power systems have many types of power equipment, complex application environments, and a huge amount of equipment status information. During the actual operation of the equipment, due to the complex working environment and large background noise interference, defect signals are easily submerged and difficult to detect, which greatly increases the difficulty of operation and maintenance. Traditional fault identification methods are difficult to meet the requirements of power equipment monitoring, so it is necessary to combine them with methods based on intelligent information fusion processing for research.
[0003] With the advancement of various sensing imaging technologies, digital image processing technologies and other related technologies, intelligent analysis technologies based on multimodal fusion data have gradually been widely used in power equipment fault diagnosis in recent years. Common ones include the combined visible light and infrared imaging to detect temperature anomalies, the combined visible light and ultraviolet spectrum to detect the external corona discharge state of high-voltage electrical equipment, and the acoustic imaging technology used for discharge, abnormal sound and vibration detection is becoming more mature and is used in various scenarios of the power grid.
[0004] In the field of intelligent analysis of power data, the fusion of multimodal data has become a key technology, among which the multimodal fusion method of visible light, infrared and acoustic data has received widespread attention. The use of rail-mounted visible light and infrared motion real-time monitoring solutions can dynamically perceive the working status of power equipment components. In addition, combined with acoustic imaging monitoring technology, the background interference can be effectively filtered out from the time-space-frequency three-modal fusion analysis to form a comprehensive perception of the overall situation of the scene. By comparing the synchronous detection of different signals, the problem of inaccurate detection or reduced sensitivity caused by interference with a single spectral signal can be reduced. In addition, for the dynamic change process of electrical equipment failure, the time-space-frequency change trend of the power equipment status data can accurately reflect the development of the failure, which is of great significance for the comprehensive diagnosis and early warning of the failure. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multimodal spatiotemporal-frequency visual fusion reconstruction method based on audio-visual-infrared-visible light.
[0006] The purpose of the present invention can be achieved by the following technical solutions:
[0007] The present invention provides a multimodal temporal-spatial-frequency visual fusion reconstruction method based on audio-visual-infrared-visible light, comprising the following steps:
[0008] S1. Select a visible light camera as a reference imaging camera, calibrate the parameters of the visible light camera, infrared camera and audio-visual instrument as well as the posture parameters between the cameras, and map the infrared camera and audio-visual instrument to the visible light camera for the first time;
[0009] S2, forward projecting each integer pixel coordinate in the infrared image, the visible light image, and the acoustic image to the corresponding 3D coordinates, and performing a second mapping of the 3D coordinates of the infrared camera and the acoustic imager to the 3D coordinates of the visible light camera;
[0010] S3, taking the 3D coordinates of the visible light camera as world coordinates, and reversely projecting the world coordinates to the infrared camera and the sound imager to obtain pixel coordinates of the world coordinates reversely projected to the infrared camera and the sound imager;
[0011] S4, using the forward projection of the infrared camera and the acoustic imager in step S2 and the reverse projection in step S3, obtain a third mapping of the grayscale value of the original infrared image pixel to the grayscale value of the visible light image pixel and a third mapping of the RGB value of the original acoustic image pixel to the RGB value of the visible light image pixel, and generate a registered infrared image and a registered acoustic image respectively;
[0012] S5, fusing the original visible light image with the registered infrared image and the registered acoustic image in step S4 to obtain a multi-modal registered reconstructed image.
[0013] The parameters for calibrating the visible light camera, the infrared camera and the acoustic imager include internal parameters and external parameters for calibrating the visible light camera, the infrared camera and the acoustic imager.
[0014] The visible light camera intrinsic parameters are determined by the horizontal focal length, vertical focal length, horizontal offset, and vertical offset of the visible light camera.
[0015] The optical camera internal parameters are expressed by the following formula:
[0016]
[0017] Among them, f x is the horizontal focal length of the visible light camera, f y is the vertical focal length of the visible light camera, c x is the horizontal offset of the optical axis of the visible light camera in the image coordinate system, in pixels, c y It is the vertical offset of the visible light camera optical axis in the image coordinate system, in pixels.
[0018] The optical camera external parameters include the position and posture parameters between cameras.
[0019] The pose parameters between the cameras are determined by a rotation matrix vector and a horizontal matrix vector.
[0020] The pose parameters between the cameras are expressed by the following formula:
[0021]
[0022] in, c M0 is the pose parameter between cameras, r ij (i, j = 1, 2, 3) are the vectors of the rotation matrix, t x ,t x ,t x are the vectors of the translation matrix respectively.
[0023] Convert the pixel points in the infrared camera and the imager to the coordinates corresponding to the pixel points in the visible light camera image.
[0024] Preferably, the pixels in the infrared camera and the imager are converted to coordinates corresponding to the pixels in the visible light camera image by matrix multiplication.
[0025] In a second aspect, the present invention provides a multimodal spatiotemporal-frequency visual fusion reconstruction system based on audio-visual-infrared-visible light, comprising: a visible light camera, an infrared camera, an audio-visual device, a mapping module, a projection module, a registration module, and an image fusion module. The system is used to implement any of the methods described above.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1. Through the fusion of multimodal data, the present invention can identify the equipment status more comprehensively and accurately, promote the improvement of fault target detection and processing capabilities from the source, and achieve more comprehensive understanding and prevention of fault formation and development.
[0028] 2. The present invention has significant application potential in the field of power equipment fault detection. It solves the problems of low dimensionality of data collection for equipment status perception, weak cross-domain analysis and dynamic evolution capabilities, thereby effectively improving the accuracy and reliability of power equipment fault detection.
[0029] 3. The present invention adopts the key technology of visual cross-modal fusion analysis, provides a prototype deployment strategy for the future visible light-infrared-audio-visual combined intelligent perception data fusion analysis, and provides data set collection support for further unmanned intelligent power fault location.
[0030] 4. The present invention designs a multimodal space-time frequency visual fusion reconstruction system based on audio-visual-infrared-visible light. It senses the relative position of the terminal space, projects it into the world coordinate system of the three-dimensional modeling of the equipment, performs coordinate transformation, target scaling and offset processing, and then fuses the multimodal data, thereby solving the problems of low dimensionality of collected data, weak cross-domain analysis capability, and weak dynamic evolution capability in the state perception of power grid equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a back-projection transformation algorithm registration process in the prior art;
[0032] Figure 2 This is a registration process based on the multimodal spatiotemporal-frequency visual fusion reconstruction technology of audio-visual-infrared-visible light in the present invention;
[0033] Figure 3 The camera module internal parameters, distortion coefficients and the relationship between the pose parameters of the three modes of audio-visual-infrared-visible light are as follows;
[0034] Figure 4 This is the registration result of the fusion reconstruction of three images: acoustic image, infrared image and visible light image. DETAILED DESCRIPTION
[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0036] This embodiment provides a multi-modal spatiotemporal-frequency visual fusion reconstruction technology based on audio-visual-infrared-visible light. For the fusion reconstruction of the three modal images of audio-visual-infrared-visible light, it is necessary to use the visible light camera as the reference imaging camera, and its corresponding visible light RGB image as the reference image, while the infrared camera and the audio-visual camera are used as the cameras to be registered. The specific implementation is as follows: Figure 2 As shown, the following steps are included:
[0037] Step 1: Pre-calibration.
[0038] The visible light camera is selected as the reference imaging camera, and the internal parameters, distortion coefficients and inter-camera pose parameters of the visible light camera, infrared camera and audio-visual camera are obtained (mapping 1).
[0039] The output results are a rotation vector and a translation vector.
[0040] Among them, the relationship between the camera module internal parameters, distortion coefficients, and posture parameters between cameras corresponding to the three modes of audio-visual-infrared-visible light is expressed by the following formula:
[0041]
[0042] Among them, s is the distortion factor, u is the u-axis coordinate of the visible light camera pixel coordinate, v is the v-axis coordinate of the visible light camera pixel coordinate, and f x is the horizontal focal length of the visible light camera, f y is the vertical focal length of the visible light camera, c xis the horizontal offset of the optical axis of the visible light camera in the image coordinate system, in pixels, c y is the vertical offset of the optical axis of the visible light camera in the image coordinate system, in pixels, r ij (i, j = 1, 2, 3) are the vectors of the rotation matrix, t x ,t x ,t x are the vectors of the translation matrix, X0, Y0, Z0 are the coordinates of the camera coordinate system of the visible light camera, K is the internal parameter of the visible light camera module, c M0 is the pose parameter between cameras.
[0043] Step 2: Project the camera pixel coordinates into the corresponding camera 3D coordinates.
[0044] Using the parameters of the infrared camera, visible light camera and acoustic imager in step 1, project each integer pixel coordinate in the infrared image, visible light image and acoustic image to the corresponding 3D coordinate. Because the visible light camera is selected as the reference camera, the 3D coordinate of the visible light camera is used as the world coordinate.
[0045] Step 3: World coordinates to individual camera pixel coordinates.
[0046] Using the transformation parameters between the visible light camera, infrared camera and imager obtained by calibration in step 1, the 3D coordinates of the infrared camera and imager coordinate system are transformed to the 3D coordinates (world coordinates) of the visible light camera coordinate system (mapping 2). Combined with the projection of the infrared camera and imager pixel coordinates to the camera 3D coordinates in step 2, the pixel coordinates of the infrared camera and imager are projected to the world coordinates, and then the back projection is solved to obtain the pixel coordinates of the infrared camera and imager that are back-projected to the infrared camera and imager.
[0047] c M0 can be expressed in a homogeneous form, and the points in the corresponding 3D coordinates can be converted to the corresponding camera images. The coordinate relationship between the points in the 3D coordinates of the three modal images of audio-visual-infrared-visible light and the corresponding camera images is shown in the following formula:
[0048]
[0049] Step 4: Generate the target image.
[0050] Through matrix multiplication, the pixels in the infrared camera and the imager are converted to the coordinates corresponding to the pixels in the visible light camera image, such as Figure 3 shown.
[0051] As shown in the following formula, the projected 3D points in the camera image to be registered are converted into points corresponding to the registered image according to the coordinate relationship between the reference imaging camera image and the camera image to be registered:
[0052]
[0053] C1 M0 is the camera to be registered; C2 M0 is the reference imaging camera.
[0054] Then, the visible light image is kept unchanged, while the original image coordinates of the infrared image and the acoustic image need to be mapped to the new pixel coordinates in step 3, so that the source pixel points are converted to the coordinates corresponding to the pixel points of the visible light camera image.
[0055] By using the forward projection in step 2 and the back projection in step 3 of the infrared camera and the acoustic imager, a mapping 3 (generally requiring interpolation) of the grayscale value of the pixel point of the infrared image source image to the grayscale value of the pixel point of the target image (which can be the light image) and a mapping 3 of the RGB value of the pixel point of the acoustic image source image to the RGB value of the pixel point of the target image are obtained, and a registered infrared image and a registered acoustic image are generated respectively.
[0056] Step 5: Image fusion and reconstruction
[0057] The original visible light image is fused with the registered infrared image and the registered acoustic image obtained by mapping in step 4, as shown in Figure 4 As shown, the audio-visual-infrared-visible light registration fusion reconstructed image material is obtained for further unmanned intelligent power fault location.
[0058] Step 6: Deploy the application
[0059] The above algorithms and logical steps contained in the fusion reconstruction technology are deployed on the corresponding hardware equipment. Based on the images that can be transmitted back by the optical camera, infrared camera and audio-visual device, the technology will automatically perform the registration, fusion and reconstruction work on the hardware, and generate the required audio-visual-infrared-visible light registration, fusion and reconstruction image materials.
[0060] Example 2
[0061] The present embodiment provides a multimodal spatiotemporal-frequency visual fusion reconstruction system based on audio-visual-infrared-visible light, including: a visible light camera, an infrared camera, an audio-visual device, a mapping module, a projection module, a registration module, and an image fusion module. The system is used to implement the method described in the above embodiment, and the similarities are not repeated here.
[0062] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A multimodal spatiotemporal-frequency visual fusion reconstruction method based on audio-visual-infrared-visible light, characterized in that: The following steps are involved: S1. Select a visible light camera as a reference imaging camera, calibrate the parameters of the visible light camera, infrared camera and audio-visual instrument as well as the posture parameters between the cameras, and map the infrared camera and audio-visual instrument to the visible light camera for the first time; S2, forward projecting each integer pixel coordinate in the infrared image, the visible light image, and the acoustic image to the corresponding 3D coordinates, and performing a second mapping of the 3D coordinates of the infrared camera and the acoustic imager to the 3D coordinates of the visible light camera; S3, taking the 3D coordinates of the visible light camera as world coordinates, and reversely projecting the world coordinates to the infrared camera and the sound imager to obtain pixel coordinates of the world coordinates reversely projected to the infrared camera and the sound imager; S4, using the forward projection of the infrared camera and the acoustic imager in step S2 and the reverse projection in step S3, obtain a third mapping of the grayscale value of the original infrared image pixel to the grayscale value of the visible light image pixel and a third mapping of the RGB value of the original acoustic image pixel to the RGB value of the visible light image pixel, and generate a registered infrared image and a registered acoustic image respectively; S5, fusing the original visible light image with the registered infrared image and the registered acoustic image in step S4 to obtain a multi-modal registered reconstructed image.
2. According to claim 1, a multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light is characterized in that: The parameters for calibrating the visible light camera, the infrared camera and the acoustic imager include internal parameters and external parameters for calibrating the visible light camera, the infrared camera and the acoustic imager.
3. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 2 is characterized in that: The visible light camera intrinsic parameters are determined by the horizontal focal length, vertical focal length, horizontal offset, and vertical offset of the visible light camera.
4. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 3 is characterized in that: The optical camera internal parameters are expressed by the following formula: Among them, f x is the horizontal focal length of the visible light camera, f y is the vertical focal length of the visible light camera, c x is the horizontal offset of the optical axis of the visible light camera in the image coordinate system, in pixels, c y It is the vertical offset of the visible light camera optical axis in the image coordinate system, in pixels.
5. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 2 is characterized in that: The optical camera external parameters include the position and posture parameters between cameras.
6. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 5 is characterized in that: The pose parameters between the cameras are determined by a rotation matrix vector and a horizontal matrix vector.
7. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 6 is characterized in that: The pose parameters between the cameras are expressed by the following formula: in, c M0 is the pose parameter between cameras, r ij (i, j = 1, 2, 3) are the vectors of the rotation matrix, t x ,t x ,t x are the vectors of the translation matrix respectively.
8. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 1 is characterized in that: Convert the pixel points in the infrared camera and the imager to the coordinates corresponding to the pixel points in the visible light camera image.
9. The multimodal space-time frequency visual fusion reconstruction method based on audio-visual-infrared-visible light according to claim 8 is characterized in that: The pixel points in the infrared camera and the imager are converted to the coordinates corresponding to the pixel points in the visible light camera image through matrix multiplication.
10. A multimodal spatiotemporal-frequency visual fusion reconstruction system based on audio-visual-infrared-visible light, comprising: Visible light camera, infrared camera, audio-visual device, mapping module, projection module, registration module, image fusion module, characterized in that the system is used to implement the method as claimed in any one of claims 1-9.
Citation Information
Patent Citations
Automatic identification method for damage area of aerospace composite material
CN112818822A
Method and device for detecting partial discharge of transformer substation
CN115825665A
Power distribution and utilization equipment fault detection method based on acousto-optic-thermal multi-physics field fusion
CN116912644A
Systems and methods for illumination correction of an image
US20110293179A1
Infrared and visible light fusion method
US20220044374A1
Cited By
Dynamic reconstruction method and system for state perception of lithium iron phosphate battery pack
CN121955788A
