An infrared and visible light image fusion processing method for dike leakage hazards

Abstract

The present application provides an infrared and visible light image fusion processing method for levee leakage danger situations. This method obtains an infrared monitoring image of the levee structure to be monitored, converts the infrared monitoring image into a target color space to enhance the luminance component in the target color space and generate an infrared processed image. Then, feature point detection is performed on the infrared processed image to determine an infrared feature point set. Next, a visible light monitoring image of the levee structure to be monitored is obtained, and feature point detection is performed on the visible light monitoring image to determine a visible light feature point set. Furthermore, a preset data fusion model is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, so that the fused monitoring image has high practicability and accuracy in subsequent levee leakage danger situation monitoring.

Description

Technical Field

[0001] This application relates to data processing technologies, and in particular to an infrared and visible light image fusion processing method for levee seepage danger situations. Background Art

[0002] Due to various reasons such as natural environmental factors, construction quality, operation and maintenance, etc., levees may have danger situations such as seepage during operation. Timely detection and handling of these danger situations are of great significance for ensuring the safe operation of levees. Traditional levee seepage danger situation monitoring methods mainly include manual inspections and sensor monitoring. Although manual inspections can directly observe the seepage conditions on the surface of the levee, it is difficult to achieve comprehensive and efficient monitoring due to limitations in human, material, and time costs. Although sensor monitoring can obtain physical parameters inside the levee (such as humidity, temperature, etc.) in real time, the deployment and maintenance costs of sensors are relatively high, and it is difficult to cover all potential seepage areas.

[0003] With the continuous development of image processing technologies, infrared thermal imaging technology and visible light imaging technology have been applied in levee seepage danger situation monitoring. Infrared thermal imaging technology can capture the thermal radiation information on the surface of the levee and identify seepage areas through temperature differences. However, infrared images often have problems such as low resolution and insufficient detailed information, making it difficult to accurately judge the severity and scope of seepage. Visible light imaging technology can clearly reflect the texture and detailed information on the surface of the levee, but it is greatly affected by lighting conditions and has poor monitoring effects at night or in bad weather.

[0004] In order to overcome the limitations of single imaging technologies, researchers have begun to explore infrared and visible light image fusion processing methods. By fusing infrared images and visible light images, the advantages of both types of images can be fully utilized to generate a fusion image that not only contains thermal radiation information but also has high clarity. Such a fusion image can not only more accurately identify seepage areas but also provide more detailed seepage information, providing strong support for subsequent seepage danger situation assessment and handling.

[0005] However, existing infrared and visible light image fusion processing uses fusion in the RGB space. In the RGB color space, there is a strong correlation between the three components of the RGB color space. This correlation may lead to information redundancy or information loss during the image fusion process, affecting the quality of the fusion image. Summary of the Invention

[0006] This application provides an infrared and visible light image fusion processing method for levee seepage danger situations, which combines the advantages of infrared images and visible light images, retains the visible light morphological information of the levee structure, and integrates the temperature feature information in the infrared images, so that the fusion monitoring image has high practicability and accuracy in subsequent levee seepage danger situation monitoring.

[0007] In a first aspect, the present application provides an infrared and visible light image fusion processing method for dike leakage hazards, including:

[0008] Obtain an infrared monitoring image of the dike structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space to generate an infrared processed image;

[0009] Detect feature points in the infrared processed image to determine an infrared feature point set;

[0010] Obtain a visible light monitoring image of the dike structure to be monitored, and detect feature points in the visible light monitoring image to determine a visible light feature point set;

[0011] Use a preset data fusion model, and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fusion monitoring image, where the preset data fusion model is a data fusion model established based on an adversarial network.

[0012] In the above solution, first, an infrared monitoring image of the levee structure to be monitored is obtained and converted into a target color space. The key to this step is that the target color space (such as the HSV color space) can more intuitively separate the brightness information of the image compared to the original red, green, and blue (RGB) color space, facilitating subsequent enhancement processing. By enhancing the brightness component, the generated infrared processed image is more prominent in brightness features, which is beneficial for subsequent feature point detection and image fusion processes. This technical effect significantly improves the visibility and information extraction ability of infrared images in complex environments. Feature point detection is performed on the infrared processed image and the visible light monitoring image respectively to determine the infrared feature point set and the visible light feature point set. As the key information points in the image, feature points ensure the effective alignment of infrared and visible light images in the feature space through feature point detection technology, laying a solid foundation for subsequent data fusion. This technical effect makes the image fusion process more stable and reliable, improving the quality of the fused image. Then, a data fusion model established based on the adversarial network is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image. As an advanced deep learning model, the adversarial network can automatically learn and optimize the fusion strategy during the data fusion process, enabling the fused image to effectively integrate and enhance information while retaining the information of the original images. This not only improves the clarity and information content of the fused image but also enhances the image's ability to identify levee seepage hazards. The generated fused monitoring image combines the advantages of infrared and visible light images, retaining both the visible light morphological information of the levee structure and incorporating the temperature feature information in the infrared image. This makes the fused monitoring image highly practical and accurate in levee seepage hazard monitoring. Through the fused image, the temperature distribution and morphological changes of the levee structure can be visually observed, effectively identifying potential seepage hazard locations.

[0013] Optionally, the converting the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image includes:

[0014] Converting the infrared monitoring image from the current color space into the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel;

[0015] Determining a corresponding feature illumination component according to the pixel values of each pixel point in the monitoring image to be processed in the target color space;

[0016] Enhance the brightness component of each pixel point in the monitoring image to be processed in the target color space according to the described feature illumination component to generate the infrared processed image.

[0017] In the above solution, first, the infrared monitoring image is converted from the current color space (RGB color space) to the target color space (such as HSV color space). Although the RGB color space is widely used in the field of image processing, the information of its red, green, and blue channels is intertwined, which is not conducive to the separate processing of brightness information. The HSV color space represents the image through three independent channels: hue, saturation, and value, enabling the brightness information to be separately extracted and enhanced. This conversion process not only simplifies the brightness processing steps but also improves the accuracy and efficiency of processing. Second, according to the pixel values of each pixel point in the monitoring image to be processed in the target color space, the corresponding feature illumination component is determined. The feature illumination component is an important parameter reflecting the brightness characteristics of pixel points. It considers the gradient information of pixel values in multiple directions, thus being able to more accurately describe the brightness changes of pixel points. By calculating the feature illumination component, a reliable basis is provided for subsequent brightness enhancement processing. Finally, according to the feature illumination component, the brightness component of each pixel point in the monitoring image to be processed in the target color space is enhanced to generate the infrared processed image. This step makes the brightness information in the infrared image more prominent by adjusting the value of the brightness component, which is beneficial to subsequent feature point detection and image fusion processes. At the same time, since the brightness enhancement is based on the feature illumination component, it can avoid the distortion problem caused by over-enhancement while maintaining the image details.

[0018] Optionally, the detecting feature points of the infrared processed image to determine the infrared feature point set includes:

[0019] Detect calibration points of the infrared processed image to determine the infrared feature point set, where at least one calibration point is preset on the embankment structure to be monitored;

[0020] Correspondingly, the detecting feature points of the visible light monitoring image to determine the visible light feature point set includes:

[0021] Detect calibration points of the visible light monitoring image to determine the visible light feature point set.

[0022] In the above solution, first, for the infrared processed image, a method of detecting calibration points is used to determine the set of infrared feature points. The calibration points are known points preset on the dike structure to be monitored, and their positions are fixed in both infrared and visible light images. By detecting these calibration points, the key feature points in the infrared image can be quickly and accurately determined. This method has higher accuracy and reliability compared with traditional feature point detection methods, especially when facing complex and variable dike structures. Second, for the visible light monitoring image, the method of detecting calibration points is also used to determine the set of visible light feature points. Since there is a certain correlation in content between the visible light image and the infrared image, by detecting the calibration points in the visible light image, the visible light feature points corresponding to the infrared feature points can be obtained. This process not only realizes the effective matching of feature points between the two images but also provides the necessary data support for subsequent image fusion. In addition, the method of determining the set of feature points by detecting calibration points also has the advantages of fast processing speed and strong adaptability. It can complete the feature point detection task of a large number of images in a short time and can adapt to image inputs with different resolutions and different lighting conditions. This is of great significance for real-time monitoring of dike leakage risks and can ensure the stable operation of the system in a complex environment.

[0023] Optionally, before fusing the infrared processed image and the visible light monitoring image by using the preset data fusion model and according to the set of infrared feature points and the set of visible light feature points, it further includes:

[0024] Determine a spatial transformation matrix according to the set of infrared feature points and the set of visible light feature points;

[0025] Perform a spatial transformation on each pixel point in the infrared processed image by using the spatial transformation matrix to align the transformed infrared processed image with the visible light monitoring image to generate data of the image to be fused, and the data of the image to be fused includes the infrared image to be fused and the visible light image to be fused.

[0026] In the above solution, first, a spatial transformation matrix is determined based on the infrared feature point set and the visible light feature point set. The solution of this matrix is based on the correspondence between the two sets of feature points and is continuously optimized through an iterative algorithm. The spatial transformation matrix can accurately reflect the spatial position differences between the infrared image and the visible light image, providing a basis for subsequent image alignment. Secondly, using the obtained spatial transformation matrix, spatial transformation is performed on each pixel point in the infrared processed image. This process is essentially to adjust the spatial coordinate system of the infrared image to be consistent with that of the visible light image, thus achieving the alignment of the two images in the physical space. After the spatial transformation, the infrared image and the visible light image are more visually matched, creating favorable conditions for subsequent image fusion. Furthermore, the introduction of the image alignment step makes the generated image data to be fused (including the infrared image to be fused and the visible light image to be fused) more consistent in content and the correspondence between feature points more accurate. This helps to improve the fusion effect of the data fusion model, enabling the generated fused monitoring image to better integrate the advantages of the two images while retaining the original image information, improving the image quality and information content. In addition, the application of the spatial transformation and image alignment technology also enhances the adaptability of the above solution to images obtained in different environments. Due to the differences in the imaging principles of infrared images and visible light images, there is often a certain spatial misalignment between the two images in practical applications. By introducing the spatial transformation and image alignment steps, the image position can be automatically adjusted to eliminate the misalignment phenomenon, thereby improving the accuracy and stability of image fusion.

[0027] Optionally, the preset data fusion model includes a generator and a discriminator;

[0028] The output end of the generator is connected to the discriminator, and the output end of the discriminator is respectively connected to the generator to form the adversarial network.

[0029] In the above solution, first, as the front-end part of the adversarial network, the generator is responsible for integrating the information in the infrared processed image and the visible light monitoring image and generating a fused monitoring image. The design of the generator makes full use of the neural network structure in deep learning technology and can automatically learn and extract the key features in the two images, thus achieving effective information fusion. Secondly, the discriminator plays the role of a "supervisor". It discriminates the authenticity of the fused monitoring image generated by the generator, that is, determines whether the image truly reflects the leakage danger of the dike structure. The design of the discriminator is also based on a deep neural network and has strong image recognition capabilities. Through continuous adversarial training with the generator, the discriminator can gradually improve its discrimination accuracy of the authenticity of the fused image. Furthermore, the adversarial network formed by the feedback connection between the generator and the discriminator enables the generator, when generating the fused image, not only to consider how to integrate the information in the infrared and visible light images but also to deceive the discriminator as much as possible so that it is difficult to judge the authenticity of the fused image. And the discriminator continuously improves its discrimination ability to cope with the deception of the generator. This process of mutual competition and co-evolution prompts the generator to continuously generate higher-quality fused monitoring images. In addition, the preset data fusion model based on the adversarial network also has good generalization ability. Since both the generator and the discriminator in the adversarial network learn based on a large amount of training data, they can automatically adapt to the image features in different environments and generate corresponding fused monitoring images. This makes it have a wide application prospect in the monitoring of dike leakage danger.

[0030] Optionally, before using the preset data fusion model and fusing the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, it further includes:

[0031] Initializing the generator and the discriminator;

[0032] Inputting the training data in the image fusion dataset into the generator to generate fused training images. The training data includes infrared training images and visible light training images, and both the infrared training images and the visible light training images include the dike structure to be monitored;

[0033] Inputting the fused training images and the fused verification images in the fusion dataset into the discriminator to determine the loss function value;

[0034] Iteratively updating the model parameters in the generator and the discriminator according to the loss function value and the gradient descent algorithm until the preset convergence condition is met to generate the preset data fusion model.

[0035] In the above solution, first, initializing the generator and discriminator is the starting point of the training process. The initialization process sets the initial model parameters for these two core components, providing a basis for subsequent training. Reasonable initialization helps to accelerate the training speed and improve the convergence of the model. Next, the training data in the image fusion dataset (including infrared training images and visible light training images) is input into the generator to generate fused training images. This step is crucial for the generator to learn how to fuse the information of the two types of images. Through the input of a large amount of training data, the generator can gradually master the features of infrared and visible light images and learn how to effectively fuse these features together to generate fused training images with higher quality and more information. Then, the fused training images and the fused validation images in the fusion dataset are input into the discriminator to determine the loss function value. The discriminator evaluates the quality of the images generated by the generator by comparing the differences between the fused training images and the fused validation images and calculates the loss function value. This loss function value reflects the gap between the current fusion ability of the generator and the ideal fusion effect. Finally, based on the loss function value and the gradient descent algorithm, the model parameters in the generator and discriminator are iteratively updated. By continuously adjusting the model parameters, the generator can gradually improve the quality of its fused images, while the discriminator can more accurately judge the authenticity of the fused images. The iterative update process continues until the preset convergence condition is met. At this time, the generated preset data fusion model already has high fusion ability and discrimination accuracy.

[0036] Optionally, after using the preset data fusion model to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the method further includes:

[0037] Using a preset danger recognition model to recognize the fused monitoring image to determine the leakage danger location.

[0038] In the above solution, first, the fused monitoring image is used as the input for danger identification, providing more comprehensive and accurate information about the dike structure. Through the fusion of infrared and visible light images, not only the thermal radiation information of the infrared image is retained, but also the detailed texture information of the visible light image is fused, making the fused image clearer visually and richer in information. This provides a more reliable data basis for subsequent danger identification. Second, the preset danger identification model is constructed based on deep learning or machine learning techniques and has powerful image recognition and analysis capabilities. Through learning a large amount of training data, this model has mastered the characteristic laws of seepage danger and can automatically extract the feature information related to danger from the fused monitoring image. During the identification process, the preset danger identification model scans and analyzes the fused monitoring image pixel by pixel or region by region. By comparing the similarity between the features in the image and the known danger features, it determines whether there is a seepage danger and locates the specific position of the danger. This identification method not only improves the accuracy of identification but also achieves precise positioning of the danger location. In addition, the introduction of the preset danger identification model also improves the automation and intelligence level of the entire dike seepage danger monitoring system. In traditional monitoring methods, it is often necessary to manually observe and analyze images, which is not only time-consuming and laborious but also easily affected by human factors. However, in the present invention, through the automatic identification of the preset danger identification model, rapid and accurate monitoring of seepage danger is achieved, greatly improving the monitoring efficiency and accuracy.

[0039] In a second aspect, the present application provides an infrared and visible light image fusion processing device for dike seepage danger, comprising:

[0040] An acquisition module, configured to acquire an infrared monitoring image of the dike structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space and generate an infrared processed image;

[0041] A processing module, configured to perform feature point detection on the infrared processed image to determine an infrared feature point set;

[0042] The acquisition module is further configured to acquire a visible light monitoring image of the dike structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set;

[0043] The processing module is further configured to use a preset data fusion model and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0044] Optionally, the processing module is specifically configured to:

[0045] Convert the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel;

[0046] Determine a corresponding characteristic illumination component according to the pixel values of each pixel point in the monitoring image to be processed in the target color space;

[0047] Enhance the brightness component of each pixel point in the monitoring image to be processed in the target color space according to the characteristic illumination component to generate the infrared processed image.

[0048] Optionally, the processing module is specifically configured to:

[0049] Detect calibration points on the infrared processed image to determine the set of infrared feature points, where at least one calibration point is preset on the dike structure to be monitored;

[0050] Correspondingly, the detecting feature points of the visible light monitoring image to determine a set of visible light feature points includes:

[0051] Detect calibration points on the visible light monitoring image to determine the set of visible light feature points.

[0052] Optionally, the processing module is specifically configured to:

[0053] Determine a spatial transformation matrix according to the set of infrared feature points and the set of visible light feature points;

[0054] Perform a spatial transformation on each pixel point in the infrared processed image by using the spatial transformation matrix to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, where the image data to be fused includes an infrared image to be fused and a visible light image to be fused.

[0055] Optionally, the preset data fusion model includes a generator and a discriminator;

[0056] The output end of the generator is connected to the discriminator, and the output end of the discriminator is respectively connected to the generator to form the adversarial network.

[0057] Optionally, the processing module is specifically configured to:

[0058] Initialize the generator and the discriminator;

[0059] Input the training data in the image fusion dataset into the generator to generate fused training images. The training data includes infrared training images and visible light training images, and both the infrared training images and the visible light training images include the dike structure to be monitored.

[0060] Input the fused training images and the fused verification images in the fusion dataset into the discriminator to determine the loss function value.

[0061] Iteratively update the model parameters in the generator and the discriminator according to the loss function value and the gradient descent algorithm until the preset convergence condition is met to generate the preset data fusion model.

[0062] Optionally, the processing module is specifically configured to:

[0063] Use a preset danger identification model to identify the fused monitoring images to determine the leakage danger location.

[0064] In a third aspect, the present application provides an electronic device, including:

[0065] A processor; and,

[0066] A memory for storing the executable instructions of the processor;

[0067] Wherein, the processor is configured to execute any possible method described in the first aspect by executing the executable instructions.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0069] The infrared and visible light image fusion processing method for levee seepage danger provided by this application obtains the infrared monitoring image of the levee structure to be monitored, converts the infrared monitoring image into a target color space, enhances the brightness component in the target color space to generate an infrared processed image. Then, feature point detection is performed on the infrared processed image to determine the infrared feature point set. Next, the visible light monitoring image of the levee structure to be monitored is obtained, and feature point detection is performed on the visible light monitoring image to determine the visible light feature point set. Then, using a preset data fusion model, the infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, thereby combining the advantages of infrared images and visible light images, retaining both the visible light morphological information of the levee structure and incorporating the temperature feature information in the infrared image, so that the fused monitoring image has high practicability and accuracy in subsequent levee seepage danger monitoring. Brief Description of the Drawings

[0070] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0071] Figure 1 is a schematic flowchart of the infrared and visible light image fusion processing method for levee seepage danger shown according to an exemplary embodiment of this application;

[0072] Figure 2 is a schematic flowchart of the infrared and visible light image fusion processing method for levee seepage danger shown according to another exemplary embodiment of this application;

[0073] Figure 3 is a schematic structural diagram of the infrared and visible light image fusion processing device for levee seepage danger shown according to an exemplary embodiment of this application;

[0074] Figure 4 is a schematic structural diagram of an electronic device shown according to an exemplary embodiment of this application.

[0075] Through the above accompanying drawings, the clear embodiments of this application have been shown, and there will be more detailed descriptions later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0076] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0077] To solve the above problems, the embodiments provided in the present application generate a fused image that contains both thermal radiation information and high definition by fusing infrared images and visible light images, thereby more accurately identifying leakage areas. To achieve this goal, the embodiments provided in the present application adopt a series of innovative technical means, specifically including:

[0078] Image color space conversion and brightness enhancement: Convert the infrared monitoring image from the original color space (such as the RGB color space) to the target color space (such as the HSV color space) to more intuitively separate the brightness information of the image. In the target color space, enhance the brightness component to improve the brightness contrast of the leakage area in the infrared image, making it easier to be identified and detected.

[0079] Feature point detection and image registration: Perform feature point detection on the infrared processed image and the visible light monitoring image respectively to determine the infrared feature point set and the visible light feature point set. Determine the spatial transformation matrix according to the feature point set, and use this matrix to perform spatial transformation on the infrared processed image to align it with the visible light monitoring image, laying a foundation for subsequent image fusion.

[0080] Image fusion based on adversarial network: Construct a data fusion model based on an adversarial network, which includes a generator and a discriminator. The generator is responsible for integrating the information in the infrared processed image and the visible light monitoring image to generate a fused monitoring image; the discriminator discriminates the authenticity of the generated fused monitoring image, and continuously improves the quality of the fused image through adversarial training with the generator.

[0081] Leakage danger identification: Use a preset danger identification model to identify the fused monitoring image and determine the location of the leakage danger. By setting a preset brightness component threshold and a preset pixel point threshold, accurately screen out the target pixel points in the water leakage area, and determine the specific location of the leakage danger according to the location of the target pixel points.

[0082] Figure 1 is a schematic flowchart of a method for fusing infrared and visible light images of dike leakage danger according to an exemplary embodiment of the present application. As Figure 1 shown, the method for fusing infrared and visible light images of dike leakage danger provided in this embodiment includes:

[0083] S101. Obtain an infrared monitoring image of the dike structure to be monitored, and convert the infrared monitoring image into a target color space.

[0084] In this step, obtain an infrared monitoring image of the dike structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space and generate an infrared processed image.

[0085] Specifically, an infrared thermal imager can be used to photograph the dike structure to be monitored to obtain its infrared monitoring image. The infrared thermal imager can capture the thermal radiation information on the surface of the dike structure and form an infrared image with temperature distribution.

[0086] Convert the obtained infrared monitoring image from the original color space (such as grayscale space or the color space output by a specific infrared device) into a target color space, such as YUV, HSV, etc. These color spaces usually contain a luminance component and a chrominance component, which are convenient for separately processing the luminance.

[0087] In the target color space, enhance the luminance component. Methods such as histogram equalization, linear stretching, or adaptive enhancement can be used to improve the contrast and clarity of the luminance component, making the thermal radiation information in the infrared image more prominent.

[0088] Optionally, combine the enhanced luminance component with the original chrominance component. For example, perform luminance addition to generate an infrared processed image. Based on retaining the original infrared information, the luminance of this image is enhanced, which is convenient for subsequent feature point detection and image fusion.

[0089] S102. Detect feature points in the infrared processed image to determine an infrared feature point set.

[0090] Specifically, according to the characteristics of the infrared processed image, select a suitable feature point detection algorithm, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), or ORB (Oriented FAST and Rotated BRIEF), etc. Apply the selected feature point detection algorithm to process the infrared processed image and extract the feature points in the image. These feature points usually have scale invariance, rotation invariance, and robustness to noise. Organize and classify the detected feature points to form an infrared feature point set. This set contains information such as the positions, scales, and directions of all the detected feature points in the infrared processed image.

[0091] Optionally, the Harris corner detection algorithm can also be used to detect feature points in the infrared processed image. The Harris corner detection algorithm is a commonly used corner detection algorithm, which has advantages such as rotation invariance and illumination invariance, and is suitable for infrared image processing. By calculating the gradient covariance matrix of each pixel point in the image and judging whether the pixel point is a corner according to the corner response function. The detected corners are used as infrared feature points to form an infrared feature point set.

[0092] S103. Obtain the visible light monitoring image of the dike structure to be monitored, and perform feature point detection on the visible light monitoring image to determine the visible light feature point set.

[0093] Specifically, a visible light camera can be used to photograph the dike structure to be monitored to obtain its visible light monitoring image. The visible light camera can capture the appearance form and detailed texture information of the dike structure.

[0094] Similar to the infrared processed image, apply a feature point detection algorithm to the visible light monitoring image to extract the feature points in the image. Organize and classify the detected feature points to form a visible light feature point set. This set contains information such as the positions, scales, and directions of all detected feature points in the visible light monitoring image.

[0095] Optionally, the Harris corner detection algorithm can also be used to detect feature points in the visible light monitoring image. The detected corners are used as visible light feature points to form a visible light feature point set.

[0096] S104. Use a preset data fusion model, and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image.

[0097] In this step, use a preset data fusion model, and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, where the preset data fusion model is a data fusion model established based on an adversarial network.

[0098] Specifically, construct a preset data fusion model based on an adversarial network. The adversarial network consists of a generator and a discriminator. The generator is responsible for fusing infrared and visible light images to generate a fused image, and the discriminator is responsible for judging whether the input image is a real fused image or a fake image generated by the generator.

[0099] In the model training stage, use a large number of labeled infrared and visible light image pairs for training. By continuously adjusting the parameters of the generator and the discriminator, the generator can generate more real and clear fused images, and at the same time, the discriminator can more accurately judge the authenticity of the input image.

[0100] In the image fusion stage, the infrared feature point set and the visible light feature point set are used as inputs, and image fusion is performed through the generator of the preset data fusion model. According to the information in the feature point set, the generator fuses the useful information in the infrared and visible light images together to generate a fused monitoring image. The fused monitoring image not only retains the thermal radiation information of the infrared image but also fuses the texture and detail information of the visible light image, providing a more comprehensive and accurate data basis for subsequent leakage danger identification.

[0101] After the preset data fusion model is trained, when new infrared monitoring images and visible light monitoring images are input, the generator can generate corresponding fused monitoring images. These images can be used for the monitoring and analysis of dike leakage dangers.

[0102] In this embodiment, by obtaining the infrared monitoring image of the dike structure to be monitored and converting the infrared monitoring image into a target color space to enhance the luminance component in the target color space to generate an infrared processed image, then performing feature point detection on the infrared processed image to determine the infrared feature point set, then obtaining the visible light monitoring image of the dike structure to be monitored and performing feature point detection on the visible light monitoring image to determine the visible light feature point set, and then using the preset data fusion model and fusing the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, thereby combining the advantages of infrared images and visible light images, retaining both the visible light morphological information of the dike structure and incorporating the temperature feature information in the infrared image, so that the fused monitoring image has high practicability and accuracy in subsequent dike leakage danger monitoring.

[0103] Figure 2 is a schematic flowchart of a method for fusing infrared and visible light images of dike leakage dangers according to another exemplary embodiment of the present application. As Figure 2 shown, the method for fusing infrared and visible light images of dike leakage dangers provided in this embodiment includes:

[0104] S201. Obtain the infrared monitoring image of the dike structure to be monitored and convert the infrared monitoring image into a target color space.

[0105] In this step, obtain the infrared monitoring image of the dike structure to be monitored and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space to generate an infrared processed image.

[0106] Specifically, the infrared monitoring image is converted from the current color space to the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel;

[0107] Using formula 1 and based on the pixel values of each pixel point in the monitoring image to be processed in the target color space Determine the corresponding characteristic illumination component , where formula 1 is:

[0108] ;

[0109] Among them, Is the brightness component in the pixel value , Is the total number of channels in the target color space, Is the pixel value Of the th component corresponds to the preset weighting coefficient, Is the pixel value Of the th component in the Direction of the pixel gradient, Is the pixel value Of the th component in the Direction of the pixel gradient, Is the pixel value Of the th component corresponds to the preset first weight coefficient, Is the pixel value Of the th component corresponds to the preset second weight coefficient, Is the pixel value Of the th component corresponds to the preset third weight coefficient;

[0110] Using formula 2 and based on the characteristic illumination component Enhance the brightness component of each pixel point in the monitoring image to be processed in the target color space to generate an infrared processed image, where formula 2 is:

[0111] ;

[0112] Among them, Is the enhanced brightness component of each pixel point in the monitoring image to be processed, Is the maximum value of the characteristic illumination component in the monitoring image to be processed, Is the preset brightness enhancement coefficient.

[0113] In the above step, the infrared monitoring image is converted from the current color space (including the red channel, green channel, and blue channel) to the target color space (including the hue channel, saturation channel, and brightness channel) to generate a monitoring image to be processed. This step aims to convert the infrared monitoring image from the commonly used RGB color space to the HSV color space which is more suitable for processing brightness information. In the RGB color space, the brightness information of the infrared image is scattered in the three color channels of red, green, and blue, making it difficult to directly perform targeted enhancement processing; while in the HSV color space, the brightness information of the image is centrally represented through an independent brightness channel (Value), facilitating subsequent processing.

[0114] It is worth noting that infrared thermal imaging technology can capture the thermal radiation information on the surface of the levee and identify leakage areas through temperature differences. However, infrared images often have problems such as low resolution and insufficient detail information, and their original image format is mostly in the RGB color space. Directly using them for image fusion processing may not fully utilize the advantages of the thermal radiation information of the infrared image. In contrast, the HSV color space decomposes the image information into three components: hue, saturation, and brightness. Among them, the brightness component is more in line with the thermal radiation information of the infrared image and can more intuitively reflect the temperature difference. Therefore, converting the infrared monitoring image from the RGB color space to the HSV color space and enhancing the brightness component helps to improve the effect of the infrared image in image fusion.

[0115] Visible light imaging technology can clearly reflect the texture and detail information on the surface of the levee, but it is also greatly affected by lighting conditions. Fusing the infrared image and the visible light image can make full use of the advantages of both to generate a fused image that contains both thermal radiation information and high definition, thus more accurately identifying leakage areas.

[0116] However, in the process of fusing infrared and visible light images, different color space processing strategies will have a significant impact on the fusion result. When fusing the infrared monitoring image based on RGB with the visible light image, due to the strong correlation between the components in the RGB color space and the dispersion of the brightness information of the infrared image in the three color channels, the fused image may have deficiencies in terms of brightness contrast and retention of detail information. In contrast, when fusing the infrared monitoring image based on HSV (especially after enhancing the brightness component) with the visible light image, it can better retain the thermal radiation information of the infrared image and combine the texture and detail information of the visible light image to generate a higher-quality fused image.

[0117] Specifically, there may be the following defects in fusing the infrared monitoring image based on RGB with the visible light image:

[0118] Brightness information dispersion: In the RGB color space, the brightness information of the infrared image is dispersed in the red, green, and blue color channels, making it difficult to directly perform enhancement processing, thus affecting the brightness contrast of the fused image.

[0119] Color component correlation: There is a strong correlation between the three components in the RGB color space. This correlation may lead to information redundancy or information loss during the image fusion process, affecting the quality of the fused image.

[0120] Insufficient retention of detail information: Due to the low resolution and insufficient detail information of the infrared image, directly fusing based on the RGB color space may not fully retain the detail information in the visible light image.

[0121] However, fusing the infrared monitoring image (after enhancing the brightness component) based on HSV with the visible light image can overcome the above defects:

[0122] Brightness information concentration: In the HSV color space, the brightness information is concentrated in the brightness component (Value), which is convenient for enhancement processing, thereby improving the brightness contrast of the fused image.

[0123] Color component independence: The hue, saturation, and brightness components in the HSV color space are relatively independent. This independence helps to reduce information redundancy and information loss during the image fusion process and improve the quality of the fused image.

[0124] Sufficient retention of detail information: By enhancing the brightness component of the infrared image and combining the texture and detail information of the visible light image, a fused image that contains both thermal radiation information and high definition can be generated, providing a more accurate and detailed data basis for subsequent leakage danger identification.

[0125] In addition, in the above steps, Formula 1 is used, and the corresponding characteristic illumination component is determined according to the pixel values of each pixel point in the to-be-processed monitoring image in the target color space. Formula 1 comprehensively considers the brightness component of the pixel value and its gradient information in the x and y directions, and calculates the characteristic illumination component by weighted summation. Specifically, the term combines the gradient information of the i-th component (i.e., hue, saturation, or brightness component) of the pixel value in the x and y directions and the original value of this component, and performs weighted summation through a preset weighting coefficient to obtain the comprehensive feature representation of this component. Then, the characteristic illumination component is calculated using these comprehensive feature representations and the brightness component. This process effectively extracts the feature information in the image and provides a basis for subsequent brightness enhancement processing.

[0126] Then, using Formula 2 and based on the characteristic illumination component, the brightness component of each pixel point in the to-be-processed monitoring image in the target color space is enhanced to generate the infrared-processed image. Formula 2 performs non-linear enhancement on the brightness component by introducing the characteristic illumination component, where is the maximum value of the characteristic illumination component, which is used for normalization to ensure that the enhanced brightness component is within a reasonable range; is a preset brightness enhancement coefficient, which is used to adjust the enhancement intensity. In this step, the enhancement of the brightness component is guided by the characteristic illumination component, so that the leakage area (usually manifested as an area with abnormal temperature) in the infrared image is highlighted in brightness, while other important detail information in the image is retained.

[0127] By converting the infrared monitoring image from the RGB color space to the HSV color space and focusing on the processing of the brightness channel, the centralized and targeted enhancement of the brightness information of the infrared image is achieved. This helps to improve the brightness contrast of the leakage area in the image, making it easier to be identified and detected. And, Formula 1 effectively extracts the feature information in the image by comprehensively considering the brightness component and gradient information of the pixel value. These feature information provide an important basis for the subsequent brightness enhancement processing, making the enhancement process more accurate and effective. Formula 2 performs non-linear enhancement on the brightness component by introducing the characteristic illumination component, significantly improving the brightness of the leakage area in the infrared image. This enhancement effect not only makes the leakage area more prominent, but also helps to retain other important detail information in the image, improving the overall quality of the image.

[0128] S202. Detect feature points in the infrared-processed image to determine the set of infrared feature points.

[0129] Specifically, it can be to detect calibration points in the infrared-processed image to determine the set of infrared feature points, where at least one calibration point is preset on the to-be-monitored dike structure. During the dike monitoring process, in order to accurately perform image registration and fusion, at least one calibration point needs to be preset on the to-be-monitored dike structure. These calibration points can be obvious physical marks (such as stickers with strong reflectivity, marker objects with special shapes, etc.), or inherent and easily recognizable feature points on the dike structure (such as the inflection points of the dike, the vertices of special structures, etc.). The presetting of the calibration points should ensure that they are clearly visible in both infrared and visible light images and the positions are fixed and unchanged. It is worth noting that in order to better display the calibration points in the infrared-processed image, a temperature control device can also be set on the calibration points to control the temperature of the calibration points within a preset temperature range.

[0130] S203. Obtain the visible light monitoring image of the to-be-monitored dike structure, and detect feature points in the visible light monitoring image to determine the set of visible light feature points.

[0131] Specifically, in the visible light monitoring image, the same feature point detection algorithm as that for the infrared processed image is used to detect the calibration points. Since the calibration points are preset and fixed in position on the dike structure, they should also be clearly visible in the visible light image. Through the feature point detection algorithm, all feature points in the visible light monitoring image are extracted, including the preset calibration points. These feature points are formed into a visible light feature point set for subsequent image registration and fusion processes. To facilitate better recognition, the calibration points can be set to a specific shape.

[0132] S204. Determine the spatial transformation matrix based on the infrared feature point set and the visible light feature point set.

[0133] Specifically, iteration is performed using Equation 3, and the spatial transformation matrix is determined based on the infrared feature point set and the visible light feature point set , where Equation 3 is:

[0134] ;

[0135] where is the number of characteristic points in the infrared feature point set and the visible light feature point set, is the position of the th feature point in the infrared feature point set, is the position of the th feature point in the visible light feature point set, and the initial value of the iteration of the spatial transformation matrix is the identity matrix.

[0136] S205. Perform spatial transformation on each pixel point in the infrared processed image using the spatial transformation matrix to align the transformed infrared processed image with the visible light monitoring image to generate the image data to be fused.

[0137] Specifically, perform spatial transformation on each pixel point in the infrared processed image using the spatial transformation matrix to align the transformed infrared processed image with the visible light monitoring image to generate the image data to be fused, and the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

[0138] In the above S204 - S205, through the iterative calculation of Formula 3, the spatial transformation matrix can be determined. This formula finds the optimal spatial transformation matrix by minimizing the sum of the squared Euclidean distances after transformation between infrared feature points and visible - light feature points. This process effectively optimizes the matching accuracy between feature points and provides a solid foundation for subsequent image alignment. The spatial transformation matrix not only considers simple image transformations such as translation and rotation, but also can adapt to more complex deformations, such as scaling, affine transformation, etc. This makes the present invention have stronger adaptability and robustness when dealing with image differences caused by factors such as shooting angle, distance, or deformation of the dike structure itself.

[0139] Using the determined spatial transformation matrix to perform spatial transformation on each pixel point in the infrared - processed image can achieve precise alignment of the infrared image and the visible - light image at the pixel level. This alignment method ensures that the information sources of corresponding pixel points in the fused image are consistent, avoiding fusion artifacts or information loss caused by image misalignment. By achieving precise alignment of the infrared image and the visible - light image, it provides more accurate and consistent input data for the subsequent data fusion model, thus significantly improving the quality of the fused image. The fused image will simultaneously retain the thermal radiation information of the infrared image and the high - resolution detail information of the visible - light image, providing more reliable data support for the accurate identification of dike leakage hazards.

[0140] S206. Use a preset data fusion model and, based on the infrared feature point set and the visible - light feature point set, fuse the infrared - processed image and the visible - light monitoring image to generate a fused monitoring image.

[0141] In this step, use a preset data fusion model and, based on the infrared feature point set and the visible - light feature point set, fuse the infrared - processed image and the visible - light monitoring image to generate a fused monitoring image, where the preset data fusion model is a data fusion model established based on an adversarial network.

[0142] Among them, the preset data fusion model includes a generator and a discriminator. The output end of the generator is connected to the discriminator, and the output end of the discriminator is respectively connected to the generator to form an adversarial network.

[0143] First, as the front-end part of the adversarial network, the generator is responsible for integrating the information in the infrared processed image and the visible light monitoring image and generating a fused monitoring image. The design of the generator makes full use of the neural network structure in deep learning technology and can automatically learn and extract the key features in the two images, thus achieving effective information fusion. Secondly, the discriminator plays the role of a "supervisor". It discriminates the authenticity of the fused monitoring image generated by the generator, that is, judges whether the image truly reflects the leakage danger of the dike structure. The design of the discriminator is also based on a deep neural network and has powerful image recognition capabilities. Through continuous adversarial training with the generator, the discriminator can gradually improve its discrimination accuracy of the authenticity of the fused image. Furthermore, the adversarial network formed by the feedback connection between the generator and the discriminator enables the generator, when generating the fused image, to not only consider how to integrate the information in the infrared and visible light images but also try to deceive the discriminator as much as possible so that it is difficult to judge the authenticity of the fused image. And the discriminator continuously improves its discrimination ability to cope with the deception of the generator. This process of mutual competition and co-evolution prompts the generator to continuously generate higher-quality fused monitoring images. In addition, the preset data fusion model based on the adversarial network also has good generalization ability. Since both the generator and the discriminator in the adversarial network are trained based on a large amount of training data, they can automatically adapt to the image features in different environments and generate corresponding fused monitoring images. This makes it have a wide application prospect in the monitoring of dike leakage danger.

[0144] Regarding the specific training process of the preset data fusion model, it can be to initialize the generator and the discriminator. Input the training data in the image fusion dataset into the generator to generate fused training images. The training data includes infrared training images and visible light training images, and both the infrared training images and the visible light training images include the dike structure to be monitored. Input the fused training images and the fused verification images in the fusion dataset into the discriminator to determine the loss function value. Iteratively update the model parameters in the generator and the discriminator according to the loss function value and the gradient descent algorithm until the preset convergence condition is met to generate the preset data fusion model. Optionally, the loss of the above discriminator can also be one or a combination of cross-entropy loss, binary cross-entropy loss, and binary cross-entropy loss.

[0145] During the training process, first of all, initializing the generator and discriminator is the starting point of the training process. The initialization process sets the initial model parameters for these two core components, providing a basis for subsequent training. Reasonable initialization helps to accelerate the training speed and improve the convergence of the model. Next, the training data in the image fusion dataset (including infrared training images and visible light training images) is input into the generator to generate fused training images. This step is crucial for the generator to learn how to fuse the information of the two types of images. Through the input of a large amount of training data, the generator can gradually master the characteristics of infrared and visible light images and learn how to effectively fuse these characteristics to generate fused training images with higher quality and more information. Then, the fused training images and the fused validation images in the fusion dataset are input into the discriminator to determine the loss function value. The discriminator evaluates the quality of the images generated by the generator by comparing the differences between the fused training images and the fused validation images and calculates the loss function value. This loss function value reflects the gap between the current fusion ability of the generator and the ideal fusion effect. Finally, based on the loss function value and the gradient descent algorithm, the model parameters in the generator and discriminator are iteratively updated. By continuously adjusting the model parameters, the generator can gradually improve the quality of its fused images, while the discriminator can more accurately judge the authenticity of the fused images. The iterative update process continues until the preset convergence condition is met. At this time, the generated preset data fusion model already has high fusion ability and discrimination accuracy.

[0146] It should be noted that the above-generated fused monitoring images are images that add temperature features to each pixel point on the basis of visible light monitoring images, and the temperature features are associated with the luminance component in the target color space.

[0147] By integrating the temperature features in the infrared image into the visible light monitoring image, the information richness of the fused monitoring image is significantly improved. The infrared image can reflect the thermal radiation information of the dike structure. Especially in the seepage water area, due to reasons such as water evaporation, its temperature distribution often differs from that of the normal area. The visible light image provides high-resolution detail information of the dike structure, such as texture, shape, etc. After fusing the two, the fused monitoring image not only retains the detail information of the visible light image but also adds temperature features, providing more comprehensive and accurate data support for subsequent leakage danger detection.

[0148] The temperature features in the fusion monitoring image are associated with the luminance components in the visible light image. This design enables the water leakage areas to exhibit unique features in the fusion image. Since the water leakage areas usually have a lower temperature, in the fusion image, the luminance components of these areas will decrease accordingly, forming obvious luminance differences. This luminance difference provides an important feature basis for subsequent leakage danger detection algorithms, enabling the algorithms to more accurately identify the water leakage areas and improving the detection accuracy.

[0149] In practical applications, the dike monitoring environment is often complex and changeable, with various interference factors such as light changes and shadow occlusions. These factors may have a greater impact on the quality of visible light images, thereby affecting the detection effect of leakage dangers. However, since infrared images are not restricted by lighting conditions, the fusion monitoring image can still maintain stable quality under interference factors such as light changes and shadow occlusions. In addition, by associating the temperature features with the luminance components, the anti-interference ability of the fusion monitoring image is further enhanced, enabling the algorithm to accurately identify the water leakage areas in a complex environment.

[0150] In the field of image processing, the choice of color space has an important impact on the performance and effect of algorithms. By associating the temperature features with the luminance components in the visible light image, the luminance component information of the target color space is fully utilized. This not only makes the fusion monitoring image more visually intuitive and easy to understand, but also improves the utilization rate of color space information by the algorithm, providing a more convenient and efficient method for subsequent image processing and analysis.

[0151] S207. Use a preset danger recognition model to recognize the fusion monitoring image to determine the location of the leakage danger.

[0152] Specifically, it can be determined that the pixel points in the fusion monitoring image whose luminance components in the target color space are greater than the preset luminance component threshold are target pixel points to form a set of target pixel points;

[0153] Determine the location of the leakage danger according to the positions of the target pixel points in the set of target pixel points. Among them, the corresponding position of the leakage danger location in the fusion monitoring image includes a number of target pixel points greater than the preset pixel point threshold.

[0154] By setting a preset brightness component threshold, it is possible to accurately screen out the pixels with higher brightness components in the fused monitoring image as target pixels. Since the water leakage area usually appears as a low-temperature area in the infrared image, and in the fused monitoring image, these low-temperature areas are associated with the brightness components of the visible light image, so their brightness components will decrease accordingly. By setting a reasonable threshold, the interference of background noise and other non-water leakage areas can be effectively excluded, so as to accurately identify the target pixels in the water leakage area. Further, according to the positions of the pixels in the target pixel set, the specific position of the leakage danger can be determined, improving the accuracy of identification.

[0155] In practical applications, the dike monitoring environment is complex and changeable, and factors such as lighting conditions and shadow occlusion may affect the image quality. However, since the fused monitoring image is used for leakage danger identification, this image combines the advantages of infrared images and visible light images and can resist these interference factors to a certain extent. In addition, by setting a preset pixel threshold, it is required that the corresponding position of the leakage danger location in the fused monitoring image must contain a sufficient number of target pixels, and this requirement further enhances the robustness of the identification. Even if there are individual noise pixels or local lighting changes, as long as the number of pixels in the overall water leakage area meets the threshold requirement, the leakage danger location can still be accurately identified.

[0156] By directly analyzing the brightness components in the fused monitoring image, the target pixels are quickly screened out, and the leakage danger location is determined according to the positions of the target pixels. This method avoids the complex image processing and feature extraction processes and greatly improves the identification efficiency. Especially in real-time monitoring scenarios, fast and accurate leakage danger identification is of great significance for taking timely countermeasures and ensuring the safety of the dike.

[0157] In addition, by determining the target pixel set and marking the leakage danger location, it not only provides quantitative information about the leakage danger (such as position, scope, etc.), but also provides an intuitive visualization effect. In the fused monitoring image, the leakage danger location can be clearly displayed, which is convenient for the monitoring personnel to quickly understand the safety status of the dike and take corresponding maintenance measures. In addition, the visualization effect also helps the monitoring personnel to further analyze and study the leakage danger, providing strong support for the safety management of the dike.

[0158] Figure 3 It is a schematic structural diagram of an infrared and visible light image fusion processing device for dike leakage danger according to an exemplary embodiment of the present application. As Figure 3 shown, the infrared and visible light image fusion processing device 300 for dike leakage danger provided in this embodiment includes:

[0159] An acquisition module 310, configured to acquire an infrared monitoring image of a dike structure to be monitored, and convert the infrared monitoring image into a target color space to enhance a luminance component in the target color space, so as to generate an infrared processed image;

[0160] A processing module 320, configured to perform feature point detection on the infrared processed image to determine an infrared feature point set;

[0161] The acquisition module 310 is further configured to acquire a visible light monitoring image of the dike structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set;

[0162] The processing module 320 is further configured to use a preset data fusion model, and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, where the preset data fusion model is a data fusion model established based on a confrontation network.

[0163] Optionally, the processing module 320 is specifically configured to:

[0164] Convert the infrared monitoring image from a current color space to the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a luminance channel;

[0165] Determine a corresponding feature illumination component according to pixel values of each pixel point in the monitoring image to be processed in the target color space;

[0166] Enhance the luminance component of each pixel point in the monitoring image to be processed in the target color space according to the feature illumination component to generate the infrared processed image.

[0167] Optionally, the processing module 320 is specifically configured to:

[0168] Perform calibration point detection on the infrared processed image to determine the infrared feature point set, where at least one calibration point is preset on the dike structure to be monitored;

[0169] Correspondingly, performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes:

[0170] Perform calibration point detection on the visible light monitoring image to determine the visible light feature point set.

[0171] Optionally, the processing module 320 is specifically configured to:

[0172] Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set;

[0173] Perform a spatial transformation on each pixel point in the infrared processed image by using the spatial transformation matrix, so as to align the transformed infrared processed image with the visible light monitoring image, and generate image data to be fused, where the image data to be fused includes an infrared image to be fused and a visible light image to be fused.

[0174] Optionally, the preset data fusion model includes a generator and a discriminator;

[0175] The output end of the generator is connected to the discriminator, and the output end of the discriminator is respectively connected to the generator to form the adversarial network.

[0176] Optionally, the processing module 320 is specifically configured to:

[0177] Initialize the generator and the discriminator;

[0178] Input the training data in the image fusion dataset into the generator to generate a fused training image, where the training data includes an infrared training image and a visible light training image, and the infrared training image and the visible light training image both include the dike structure to be monitored;

[0179] Input the fused training image and the fused verification image in the fusion dataset into the discriminator to determine a loss function value;

[0180] Iteratively update the model parameters in the generator and the discriminator according to the loss function value and the gradient descent algorithm until a preset convergence condition is met, so as to generate the preset data fusion model.

[0181] Optionally, the processing module 320 is specifically configured to:

[0182] Use a preset danger situation recognition model to recognize the fused monitoring image to determine the leakage danger situation location.

[0183] Figure 4 It is a schematic structural diagram of an electronic device shown according to an exemplary embodiment of the present application. As Figure 4 shown, an electronic device 400 provided in this embodiment includes: a processor 401 and a memory 402; where:

[0184] The memory 402 is used to store a computer program, and this memory may also be a flash (flash memory).

[0185] A processor 401 is configured to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0186] Optionally, the memory 402 can be either independent or integrated with the processor 401.

[0187] When the memory 402 is a device independent of the processor 401, the electronic device 400 may further include:

[0188] A bus 403 for connecting the memory 402 and the processor 401.

[0189] This embodiment also provides a readable storage medium storing a computer program, and when at least one processor of the electronic device executes the computer program, the electronic device executes the methods provided by the above various embodiments.

[0190] This embodiment also provides a program product including a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and execution of the computer program by at least one processor causes the electronic device to implement the methods provided by the above various embodiments.

[0191] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and embodiments are to be considered as exemplary only, and the true scope and spirit of the present application are pointed out by the claims.

[0192] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An infrared and visible light image fusion processing method for dike leakage danger, characterized in that Including: Obtain an infrared monitoring image of the embankment structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space, generating an infrared processed image; Perform feature point detection on the infrared processed image to determine an infrared feature point set; Obtain a visible light monitoring image of the embankment structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; Use a preset data fusion model, and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, where the preset data fusion model is a data fusion model established based on an adversarial network; Determine the pixel points in the fused monitoring image whose luminance component in the target color space is greater than a preset luminance component threshold as target pixel points to form a target pixel point set; Determine the leakage danger location according to the positions of the respective target pixel points in the target pixel point set, where the corresponding position of the leakage danger location in the fused monitoring image includes that the number of the target pixel points is greater than a preset pixel point threshold; The converting the infrared monitoring image into a target color space to enhance the luminance component in the target color space and generating an infrared processed image includes: Convert the infrared monitoring image from the current color space into the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a luminance channel; Using formula 1, according to each pixel point in the to-be-processed monitoring image determine the corresponding characteristic illumination component based on the pixel value in the target color space , and formula 1 is as follows: ； Wherein, is the luminance component of the pixel value in, is the total number of channels of the target color space, is the th preset weighting coefficient corresponding to the component of the pixel value is the pixel gradient of the th component of the pixel value in the direction, is the pixel gradient of the th component of the pixel value in the direction, is the preset first weight coefficient corresponding to the th component of the pixel value is the preset second weight coefficient corresponding to the th component of the pixel value is the preset third weight coefficient corresponding to the component; Using formula 2, according to the described characteristic illumination component Enhance the brightness component of each pixel point in the target color space in the to-be-processed monitoring image to generate the infrared-processed image. The formula 2 is as follows: ； wherein, is the enhanced luminance component of each pixel point in the to-be-processed monitoring image, is the maximum value of the feature illumination component in the to-be-processed monitoring image, is a preset luminance enhancement coefficient.

2. The infrared and visible light image fusion processing method for dike leakage danger according to claim 1, characterized in that The performing feature point detection on the infrared processed image to determine an infrared feature point set includes: Perform calibration point detection on the infrared processed image to determine the infrared feature point set, where at least one calibration point is preset on the embankment structure to be monitored; Correspondingly, the performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes: Perform calibration point detection on the visible light monitoring image to determine the visible light feature point set.

3. The infrared and visible light image fusion processing method for dike leakage hazards according to claim 1, characterized in that, Before the using the preset data fusion model and fusing the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set, it further includes: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set; Use the spatial transformation matrix to perform spatial transformation on each pixel point in the infrared processed image to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, and the image data to be fused includes an infrared image to be fused and a visible light image to be fused.

4. The infrared and visible light image fusion processing method for dike leakage danger according to any one of claims 1-3, characterized in that The preset data fusion model includes a generator and a discriminator; The output end of the generator is connected to the discriminator, and the output end of the discriminator is respectively connected to the generator to form the adversarial network.

5. The infrared and visible light image fusion processing method for the levee seepage danger situation according to claim 4, wherein Before fusing the infrared processed image and the visible light monitoring image by using the preset data fusion model and according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the following steps are further included: Initialize the generator and the discriminator; Input the training data in the image fusion dataset into the generator to generate fused training images, where the training data includes infrared training images and visible light training images, and the infrared training images and the visible light training images both include the embankment structure to be monitored; Input the fused training images and the fused verification images in the fusion dataset into the discriminator to determine the loss function value; Iteratively update the model parameters in the generator and the discriminator according to the loss function value and the gradient descent algorithm until the preset convergence condition is met to generate the preset data fusion model.

6. The infrared and visible light image fusion processing method for the levee seepage danger situation according to any one of claims 1-3, characterized in that After fusing the infrared processed image and the visible light monitoring image by using the preset data fusion model and according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the following steps are further included: Use a preset danger identification model to identify the fused monitoring image to determine the leakage danger location.

7. An infrared and visible light image fusion processing device for dike leakage danger, characterized in that It includes: An acquisition module, configured to acquire an infrared monitoring image of the embankment structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the luminance component in the target color space to generate an infrared processed image; A processing module, configured to perform feature point detection on the infrared processed image to determine an infrared feature point set; The acquisition module is further configured to acquire a visible light monitoring image of the embankment structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The processing module is further configured to use a preset data fusion model and fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, where the preset data fusion model is a data fusion model established based on an adversarial network; The processing module is further specifically configured to: Determine the pixel points in the fused monitoring image whose luminance component in the target color space is greater than a preset luminance component threshold as target pixel points to form a target pixel point set; Determine the leakage danger location according to the positions of the target pixel points in the target pixel point set, where the corresponding position of the leakage danger location in the fused monitoring image includes the number of the target pixel points greater than a preset pixel point threshold; The processing module is further specifically configured to: Convert the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, where the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a luminance channel; Using formula 1, according to each pixel point in the monitoring image to be processed Determine the corresponding characteristic illumination component based on the pixel value in the target color space , and formula 1 is as follows: ； Among them, is the luminance component of the pixel value in, is the total number of channels in the target color space, is the th preset weighting coefficient corresponding to the th component of the pixel value in the th direction, is the pixel gradient of the th component of the pixel value in the direction, is the preset first weight coefficient corresponding to the th component of the pixel value is the preset second weight coefficient corresponding to the th component of the pixel value is the preset third weight coefficient corresponding to the th component of the pixel value; Using formula 2, based on the described characteristic illumination component Enhance the luminance component of each pixel point in the to-be-processed monitoring image in the target color space to generate the infrared-processed image, and formula 2 is as follows: ； wherein, is the enhanced luminance component of each pixel point in the monitoring image to be processed, is the maximum value of the characteristic illumination component in the monitoring image to be processed, is a preset luminance enhancement coefficient.

8. An electronic device, characterized in that, It includes: A processor; And, A memory, configured to store the executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images