Infrared and visible light image fusion processing method for dike leakage dangerous case

Abstract

The invention provides a dike leakage dangerous case infrared and visible light image fusion processing method. The method comprises the following steps: acquiring an infrared monitoring image of a dike structure to be monitored, converting the infrared monitoring image into a target color space, performing enhancement processing on a brightness component in the target color space to generate an infrared processing image, then performing feature point detection on the infrared processing image to determine an infrared feature point set, and determining the dike structure to be monitored according to the infrared feature point set. Then, acquiring a visible light monitoring image of the to-be-monitored dike structure, performing feature point detection on the visible light monitoring image to determine a visible light feature point set, and fusing the visible light feature point set by using a preset data fusion model; and according to the infrared feature point set and the visible light feature point set, the infrared processing image and the visible light monitoring image are fused to generate a fused monitoring image, so that the fused monitoring image has relatively high practicability and accuracy in subsequent dike leakage dangerous case monitoring.

Description

Technical Field

[0001] The present application relates to data processing technology, and more particularly to a method for fusing infrared and visible light images of dike leakage hazards. Background Art

[0002] Due to various reasons such as natural environmental factors, construction quality, operation and maintenance, leakage and other dangerous conditions may occur in the operation of the levee. Timely detection and handling of these dangerous conditions are of great significance to ensure the safe operation of the levee. Traditional methods for monitoring levee leakage hazards mainly include manual inspections and sensor monitoring. Although manual inspections can visually observe the leakage on the surface of the levee, it is difficult to achieve comprehensive and efficient monitoring due to the limitations of manpower, material resources and time costs. Although sensor monitoring can obtain the physical parameters inside the levee (such as humidity, temperature, etc.) in real time, the deployment and maintenance costs of sensors are high, and it is difficult to cover all potential leakage areas.

[0003] With the continuous development of image processing technology, infrared thermal imaging technology and visible light imaging technology have been applied in the monitoring of levee leakage. Infrared thermal imaging technology can capture the thermal radiation information on the surface of the levee and identify the leakage area through temperature differences. However, infrared images often have problems such as low resolution and insufficient detail information, making it difficult to accurately judge the severity and scope of the leakage. Visible light imaging technology can clearly reflect the texture and detail information of the levee surface, but it is greatly affected by lighting conditions and has poor monitoring effect at night or in bad weather.

[0004] In order to overcome the limitations of a single imaging technology, researchers began to explore the fusion processing method of infrared and visible light images. By fusing infrared images and visible light images, the advantages of both images can be fully utilized to generate a fused image that contains both thermal radiation information and high definition. This fused image can not only identify the leakage area more accurately, but also provide more detailed leakage information, providing strong support for subsequent leakage risk assessment and treatment.

[0005] However, the existing infrared and visible light image fusion processing adopts fusion in RGB space. In RGB color space, there is a strong correlation between the three components of RGB color space. This correlation may cause information redundancy or information loss during the image fusion process, affecting the quality of the fused image. Summary of the invention

[0006] The present application provides a method for fusing infrared and visible light images of levee leakage hazards, which combines the advantages of infrared images and visible light images, retains the visible light morphological information of the levee structure, and incorporates the temperature characteristic information in the infrared image, so that the fused monitoring image has high practicality and accuracy in subsequent levee leakage hazard monitoring.

[0007] In a first aspect, the present application provides a method for fusing infrared and visible light images of dike leakage hazards, comprising: Acquire an infrared monitoring image of the levee structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image; Performing feature point detection on the infrared processed image to determine a set of infrared feature points; Acquire a visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set by using a preset data fusion model to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0008] In the above scheme, the infrared monitoring image of the levee structure to be monitored is first obtained and converted into the target color space. The key to this step is that the target color space (such as HSV color space) can more intuitively separate the brightness information of the image compared to the original red, green and blue (RGB) color space, which is convenient for subsequent enhancement processing. By enhancing the brightness component, the infrared processed image generated is more prominent in brightness characteristics, which is conducive to the subsequent feature point detection and image fusion process. This technical effect significantly improves the visibility and information extraction ability of infrared images in complex environments. Feature point detection is performed on the infrared processed image and the visible light monitoring image respectively to determine the infrared feature point set and the visible light feature point set. As the key information point in the image, the feature point detection technology ensures the effective alignment of the infrared and visible light images in the feature space, laying a solid foundation for subsequent data fusion. This technical effect makes the image fusion process more stable and reliable, and improves the quality of the fused image. Then, the infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set using the preset data fusion model established based on the adversarial network to generate a fused monitoring image. As an advanced deep learning model, the adversarial network can automatically learn and optimize the fusion strategy during the data fusion process, so that the fused image can retain the original image information while achieving effective information integration and enhancement, which not only improves the clarity and information content of the fused image, but also enhances the image's ability to identify levee leakage hazards. The generated fused monitoring image combines the advantages of infrared images and visible light images, retaining the visible light morphological information of the levee structure and incorporating the temperature characteristic information in the infrared image. This makes the fused monitoring image highly practical and accurate in levee leakage hazard monitoring. By fusing the image, the temperature distribution and morphological changes of the levee structure can be intuitively observed, thereby effectively identifying the potential leakage hazard location.

[0009] Optionally, converting the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image includes: Converting the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, wherein the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel; Determine the corresponding characteristic illumination component according to the pixel value of each pixel point in the monitoring image to be processed in the target color space; The brightness component of each pixel in the monitored image to be processed in the target color space is enhanced according to the characteristic illumination component to generate the infrared processed image.

[0010] In the above scheme, first, the infrared monitoring image is converted from the current color space (RGB color space) to the target color space (such as HSV color space). Although the RGB color space is widely used in the field of image processing, the information of its three channels of red, green and blue are intertwined, which is not conducive to the separate processing of brightness information. The HSV color space represents the image through three independent channels of hue, saturation and brightness, so that the brightness information can be extracted and enhanced separately. This conversion process not only simplifies the brightness processing steps, but also improves the accuracy and efficiency of the processing. Secondly, according to the pixel value of each pixel in the monitoring image to be processed in the target color space, the corresponding characteristic illumination component is determined. The characteristic illumination component is an important parameter reflecting the brightness characteristics of the pixel point. It takes into account the gradient information of the pixel value in multiple directions, so that the brightness change of the pixel point can be more accurately described. By calculating the characteristic illumination component, a reliable basis is provided for the subsequent brightness enhancement processing. Finally, according to the characteristic illumination component, the brightness component of each pixel in the monitoring image to be processed in the target color space is enhanced to generate an infrared processing image. This step makes the brightness information in the infrared image more prominent by adjusting the value of the brightness component, which is beneficial to the subsequent feature point detection and image fusion process. At the same time, since the brightness enhancement is based on the characteristic illumination component, it can avoid the distortion caused by over-enhancement while maintaining the image details.

[0011] Optionally, the performing feature point detection on the infrared processed image to determine an infrared feature point set includes: Performing calibration point detection on the infrared processed image to determine the infrared feature point set, wherein at least one calibration point is preset on the embankment structure to be monitored; Correspondingly, the performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes: Calibration point detection is performed on the visible light monitoring image to determine the visible light feature point set.

[0012] In the above scheme, firstly, for the infrared processing image, the calibration point detection method is used to determine the infrared feature point set. The calibration points are known points preset on the levee structure to be monitored, and their positions are fixed in both infrared and visible light images. By detecting these calibration points, the key feature points in the infrared image can be quickly and accurately determined. Compared with the traditional feature point detection method, this method has higher accuracy and reliability, especially when facing complex and changeable levee structures. Secondly, for the visible light monitoring image, the calibration point detection method is also used to determine the visible light feature point set. Since the visible light image and the infrared image have a certain correlation in content, by detecting the calibration points in the visible light image, the visible light feature points corresponding to the infrared feature points can be obtained. This process not only realizes the effective matching of feature points between the two images, but also provides the necessary data support for subsequent image fusion. In addition, the method of determining the feature point set by calibration point detection also has the advantages of fast processing speed and strong adaptability. It can complete the feature point detection task of a large number of images in a short time, and can adapt to image inputs with different resolutions and different lighting conditions. This is of great significance for real-time monitoring of embankment leakage risks and can ensure the stable operation of the system in a complex environment.

[0013] Optionally, before using the preset data fusion model and fusing the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set, the method further includes: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set; The spatial transformation matrix is ​​used to perform spatial transformation on each pixel in the infrared processed image to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, wherein the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

[0014] In the above scheme, first, the spatial transformation matrix is ​​determined according to the infrared feature point set and the visible light feature point set. The solution of this matrix is ​​based on the correspondence between the two sets of feature points and is obtained through continuous optimization by an iterative algorithm. The spatial transformation matrix can accurately reflect the difference in spatial position between the infrared image and the visible light image, providing a basis for subsequent image alignment. Secondly, the obtained spatial transformation matrix is ​​used to perform spatial transformation on each pixel in the infrared processed image. This process essentially adjusts the spatial coordinate system of the infrared image to be consistent with the visible light image, thereby achieving the alignment of the two images in physical space. The infrared image and the visible light image after spatial transformation are more visually matched, creating favorable conditions for subsequent image fusion. Furthermore, the introduction of the image alignment step makes the generated image data to be fused (including the infrared image to be fused and the visible light image to be fused) more consistent in content, and the correspondence between the feature points is more accurate. This helps to improve the fusion effect of the data fusion model, so that the generated fused monitoring image can better integrate the advantages of the two images while retaining the original image information, thereby improving the image quality and information volume. In addition, the application of spatial transformation and image alignment technology also enhances the adaptability of the above scheme to images acquired in different environments. Due to the difference in imaging principles between infrared images and visible light images, there is often a certain spatial misalignment between the two images in practical applications. By introducing spatial transformation and image alignment steps, the image position can be automatically adjusted to eliminate the misalignment phenomenon, thereby improving the accuracy and stability of image fusion.

[0015] Optionally, the preset data fusion model includes a generator and a discriminator; The output end of the generator is connected to the discriminator, and the output ends of the output end of the discriminator are respectively connected to the generator to form the adversarial network.

[0016] In the above scheme, firstly, the generator, as the front-end part of the adversarial network, is responsible for integrating the information in the infrared processing image and the visible light monitoring image, and generating a fused monitoring image. The design of the generator makes full use of the neural network structure in deep learning technology, which can automatically learn and extract the key features in the two images, thereby realizing the effective fusion of information. Secondly, the discriminator plays the role of a "supervisor", which distinguishes the authenticity of the fused monitoring image generated by the generator, that is, whether the image truly reflects the leakage danger of the levee structure. The design of the discriminator is also based on a deep neural network and has a strong image recognition ability. Through continuous adversarial training with the generator, the discriminator can gradually improve its accuracy in distinguishing the authenticity of the fused image. Furthermore, the adversarial network formed by the feedback connection between the generator and the discriminator enables the generator to not only consider how to integrate the information in the infrared and visible light images when generating the fused image, but also to deceive the discriminator as much as possible, making it difficult to judge the authenticity of the fused image. The discriminator continuously improves its own discrimination ability to cope with the deception of the generator. This process of mutual competition and co-evolution prompts the generator to continuously generate higher-quality fused monitoring images. In addition, the preset data fusion model based on the adversarial network also has good generalization ability. Since the generator and discriminator in the adversarial network are learned based on a large amount of training data, they can automatically adapt to the image features in different environments and generate corresponding fused monitoring images. This makes it have a wide range of application prospects in embankment leakage monitoring.

[0017] Optionally, before using the preset data fusion model and fusing the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the method further includes: Initializing the generator and the discriminator; Inputting the training data in the image fusion data set into the generator to generate a fused training image, wherein the training data includes an infrared training image and a visible light training image, and both the infrared training image and the visible light training image include the levee structure to be monitored; Inputting the fused training image and the fused verification image in the fused data set into the discriminator to determine a loss function value; The model parameters in the generator and the discriminator are iteratively updated according to the loss function value and the gradient descent algorithm until the preset convergence conditions are met to generate the preset data fusion model.

[0018] In the above scheme, first of all, initializing the generator and the discriminator is the starting point of the training process. The initialization process sets the initial model parameters for these two core components and provides a basis for subsequent training. Reasonable initialization helps to speed up the training and improve the convergence of the model. Next, the training data in the image fusion dataset (including infrared training images and visible light training images) are input into the generator to generate fused training images. This step is the key for the generator to learn how to fuse the two image information. Through the input of a large amount of training data, the generator can gradually master the characteristics of infrared and visible light images, and learn how to effectively fuse these features together to generate fused training images with higher quality and more information. Then, the fused training image and the fused verification image in the fusion dataset are input into the discriminator to determine the loss function value. The discriminator evaluates the image quality generated by the generator by comparing the difference between the fused training image and the fused verification image, and calculates the loss function value. This loss function value reflects the gap between the current fusion capability of the generator and the ideal fusion effect. Finally, according to the loss function value and the gradient descent algorithm, the model parameters in the generator and the discriminator are iteratively updated. By continuously adjusting the model parameters, the generator can gradually improve the quality of its fused image, while the discriminator can more accurately judge the authenticity of the fused image. The iterative update process continues until the preset convergence conditions are met, at which point the generated preset data fusion model already has a high fusion capability and discrimination accuracy.

[0019] Optionally, after the preset data fusion model is used and the infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the method further includes: The fused monitoring image is identified using a preset danger recognition model to determine the location of the leakage danger.

[0020] In the above scheme, firstly, the fused monitoring image is used as the input for the hazard identification, providing more comprehensive and accurate information about the levee structure. By fusing the infrared and visible light images, not only the thermal radiation information of the infrared image is retained, but also the detailed texture information of the visible light image is fused, making the fused image visually clearer and richer in information. This provides a more reliable data basis for the subsequent hazard identification. Secondly, the preset hazard identification model is built based on deep learning or machine learning technology, and has powerful image recognition and analysis capabilities. Through the learning of a large amount of training data, the model has mastered the characteristic laws of leakage hazards and can automatically extract the characteristic information related to the hazard from the fused monitoring image. During the identification process, the preset hazard identification model scans and analyzes the fused monitoring image pixel by pixel or region by region, and judges whether there is a leakage hazard by comparing the similarity between the features in the image and the known hazard features, and determines the specific location of the hazard. This identification method not only improves the accuracy of identification, but also realizes the precise positioning of the hazard location. In addition, the introduction of the preset hazard identification model also improves the automation and intelligence level of the entire levee leakage hazard monitoring system. In traditional monitoring methods, manual observation and analysis of images are often required, which is not only time-consuming and labor-intensive, but also easily affected by human factors. However, the present invention realizes rapid and accurate monitoring of leakage hazards through automatic recognition of preset hazard recognition models, greatly improving monitoring efficiency and accuracy.

[0021] In a second aspect, the present application provides a device for fusion processing of infrared and visible light images of dike leakage hazards, comprising: An acquisition module is used to acquire an infrared monitoring image of the levee structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image; A processing module, used for performing feature point detection on the infrared processed image to determine a set of infrared feature points; The acquisition module is further used to acquire the visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The processing module is also used to use a preset data fusion model to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0022] Optionally, the processing module is specifically used to: Converting the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, wherein the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel; Determine the corresponding characteristic illumination component according to the pixel value of each pixel point in the monitoring image to be processed in the target color space; The brightness component of each pixel in the monitored image to be processed in the target color space is enhanced according to the characteristic illumination component to generate the infrared processed image.

[0023] Optionally, the processing module is specifically used to: Performing calibration point detection on the infrared processed image to determine the infrared feature point set, wherein at least one calibration point is preset on the embankment structure to be monitored; Correspondingly, the performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes: Calibration point detection is performed on the visible light monitoring image to determine the visible light feature point set.

[0024] Optionally, the processing module is specifically used to: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set; The spatial transformation matrix is ​​used to perform spatial transformation on each pixel in the infrared processed image to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, wherein the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

[0025] Optionally, the preset data fusion model includes a generator and a discriminator; The output end of the generator is connected to the discriminator, and the output ends of the output end of the discriminator are respectively connected to the generator to form the adversarial network.

[0026] Optionally, the processing module is specifically used to: Initializing the generator and the discriminator; Inputting the training data in the image fusion data set into the generator to generate a fused training image, wherein the training data includes an infrared training image and a visible light training image, and both the infrared training image and the visible light training image include the levee structure to be monitored; Inputting the fused training image and the fused verification image in the fused data set into the discriminator to determine a loss function value; The model parameters in the generator and the discriminator are iteratively updated according to the loss function value and the gradient descent algorithm until the preset convergence conditions are met to generate the preset data fusion model.

[0027] Optionally, the processing module is specifically used to: The fused monitoring image is identified using a preset danger recognition model to determine the location of the leakage danger.

[0028] In a third aspect, the present application provides an electronic device, including: processor; and, A memory, configured to store executable instructions of the processor; The processor is configured to perform any possible method described in the first aspect by executing the executable instructions.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0030] The present application provides a method for fusing infrared and visible light images of dike leakage hazards, which obtains an infrared monitoring image of the dike structure to be monitored, and converts the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image. Then, feature point detection is performed on the infrared processed image to determine an infrared feature point set. Next, a visible light monitoring image of the dike structure to be monitored is obtained, and feature point detection is performed on the visible light monitoring image to determine a visible light feature point set. A preset data fusion model is then used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image. The advantages of infrared images and visible light images are combined, and the visible light morphological information of the dike structure is retained, and the temperature feature information in the infrared image is integrated, so that the fused monitoring image has high practicality and accuracy in subsequent dike leakage hazard monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] Figure 1 It is a flowchart of a method for fusing infrared and visible light images of a dike leakage hazard according to an exemplary embodiment of the present application; Figure 2is a flow chart of a method for fusing infrared and visible light images of dike leakage hazards according to another exemplary embodiment of the present application; Figure 3 It is a structural schematic diagram of an infrared and visible light image fusion processing device for levee leakage danger according to an exemplary embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0033] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] In order to solve the above problems, the embodiments provided in this application fuse infrared images with visible light images to generate a fused image that contains both thermal radiation information and high definition, thereby more accurately identifying the leakage area. To achieve this goal, the embodiments provided in this application adopt a series of innovative technical means, including: Image color space conversion and brightness enhancement: Convert the infrared monitoring image from the original color space (such as RGB color space) to the target color space (such as HSV color space) to more intuitively separate the brightness information of the image. In the target color space, enhance the brightness component to improve the brightness contrast of the leakage area in the infrared image, making it easier to identify and detect.

[0036] Feature point detection and image registration: Feature point detection is performed on the infrared processed image and the visible light monitoring image respectively to determine the infrared feature point set and the visible light feature point set. The spatial transformation matrix is ​​determined based on the feature point set, and the matrix is ​​used to perform spatial transformation on the infrared processed image to achieve alignment with the visible light monitoring image, laying the foundation for subsequent image fusion.

[0037] Image fusion based on adversarial network: Construct a data fusion model based on adversarial network, which consists of two parts: generator and discriminator. The generator is responsible for integrating the information in the infrared processed image and the visible light monitoring image to generate a fused monitoring image; the discriminator distinguishes the authenticity of the generated fused monitoring image, and improves the quality of the fused image by continuously conducting adversarial training with the generator.

[0038] Leakage hazard identification: Use the preset hazard identification model to identify the fused monitoring image and determine the location of the leakage hazard. By setting the preset brightness component threshold and the preset pixel threshold, the target pixel points in the leakage area are accurately screened out, and the specific location of the leakage hazard is determined based on the location of the target pixel points.

[0039] Figure 1 FIG. 1 is a flow chart of a method for fusing infrared and visible light images of levee leakage hazards according to an exemplary embodiment of the present application. Figure 1 As shown, the method for fusing infrared and visible light images of dike leakage danger provided by this embodiment includes: S101, obtaining an infrared monitoring image of the levee structure to be monitored, and converting the infrared monitoring image into a target color space.

[0040] In this step, an infrared monitoring image of the levee structure to be monitored is acquired, and the infrared monitoring image is converted into a target color space to enhance the brightness component in the target color space to generate an infrared processed image.

[0041] Specifically, an infrared thermal imager can be used to photograph the levee structure to be monitored to obtain its infrared monitoring image. The infrared thermal imager can capture the thermal radiation information on the surface of the levee structure and form an infrared image with temperature distribution.

[0042] The acquired infrared monitoring image is converted from the original color space (such as grayscale space or the color space output by a specific infrared device) to the target color space, such as YUV, HSV, etc. These color spaces usually contain brightness components and chromaticity components, which facilitates the separate processing of brightness.

[0043] In the target color space, the brightness component is enhanced. Histogram equalization, linear stretching or adaptive enhancement can be used to improve the contrast and clarity of the brightness component, making the thermal radiation information in the infrared image more prominent.

[0044] Optionally, the enhanced brightness component is combined with the original chrominance component, for example, brightness addition is performed to generate an infrared processed image. The image retains the original infrared information and has enhanced brightness, which is convenient for subsequent feature point detection and image fusion.

[0045] S102: Perform feature point detection on the infrared processed image to determine a set of infrared feature points.

[0046] Specifically, according to the characteristics of the infrared processed image, select a suitable feature point detection algorithm, such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Rapid Rotation Invariant Features). Apply the selected feature point detection algorithm to process the infrared processed image and extract the feature points in the image. These feature points usually have scale invariance, rotation invariance and robustness to noise. The detected feature points are sorted and classified to form an infrared feature point set. The set contains information such as the position, scale and direction of all detected feature points in the infrared processed image.

[0047] Optionally, the Harris corner detection algorithm can be used to detect feature points of the infrared processed image. The Harris corner detection algorithm is a commonly used corner detection algorithm with advantages such as rotation invariance and illumination invariance, and is suitable for infrared image processing. The gradient covariance matrix of each pixel in the image is calculated, and whether the pixel is a corner point is determined according to the corner response function. The detected corner points are used as infrared feature points to form an infrared feature point set.

[0048] S103: Obtain a visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set.

[0049] Specifically, a visible light camera can be used to photograph the levee structure to be monitored to obtain its visible light monitoring image. The visible light camera can capture the appearance and detailed texture information of the levee structure.

[0050] Similar to infrared processing images, feature point detection algorithms are applied to visible light monitoring images to extract feature points in the image. The detected feature points are sorted and classified to form a visible light feature point set. This set contains information such as the position, scale, and direction of all detected feature points in the visible light monitoring image.

[0051] Optionally, the Harris corner point detection algorithm can also be used to detect feature points of the visible light monitoring image. The detected corner points are used as visible light feature points to form a visible light feature point set.

[0052] S104: Using a preset data fusion model, and according to the infrared feature point set and the visible light feature point set, fuse the infrared processed image and the visible light monitoring image to generate a fused monitoring image.

[0053] In this step, a preset data fusion model is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0054] Specifically, a preset data fusion model is constructed based on the adversarial network. The adversarial network consists of two parts: a generator and a discriminator. The generator is responsible for fusing infrared and visible light images to generate a fused image, and the discriminator is responsible for judging whether the input image is a real fused image or a fake image generated by the generator.

[0055] During the model training phase, a large number of labeled infrared and visible light image pairs are used for training. By continuously adjusting the parameters of the generator and discriminator, the generator can generate more realistic and clear fused images, while the discriminator can more accurately judge the authenticity of the input image.

[0056] In the image fusion stage, the infrared feature point set and the visible light feature point set are used as inputs, and the image fusion is performed through the generator of the preset data fusion model. The generator fuses the useful information in the infrared and visible light images based on the information in the feature point set to generate a fused monitoring image. The fused monitoring image not only retains the thermal radiation information of the infrared image, but also integrates the texture and detail information of the visible light image, providing a more comprehensive and accurate data basis for the subsequent leakage risk identification.

[0057] When the preset data fusion model is trained, the generator can generate the corresponding fused monitoring image by inputting new infrared monitoring images and visible light monitoring images. The image can be used to monitor and analyze the leakage risk of the embankment.

[0058] In this embodiment, an infrared monitoring image of the levee structure to be monitored is obtained, and the infrared monitoring image is converted into a target color space to enhance the brightness component in the target color space to generate an infrared processed image. Then, feature point detection is performed on the infrared processed image to determine an infrared feature point set. Next, a visible light monitoring image of the levee structure to be monitored is obtained, and feature point detection is performed on the visible light monitoring image to determine a visible light feature point set. Then, a preset data fusion model is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image. The advantages of infrared images and visible light images are combined, and the visible light morphological information of the levee structure is retained, and the temperature feature information in the infrared image is integrated, so that the fused monitoring image has high practicality and accuracy in subsequent levee leakage hazard monitoring.

[0059] Figure 2FIG. 1 is a flow chart of a method for fusing infrared and visible light images of levee leakage hazards according to another exemplary embodiment of the present application. Figure 2 As shown, the method for fusing infrared and visible light images of dike leakage danger provided by this embodiment includes: S201, obtaining an infrared monitoring image of the levee structure to be monitored, and converting the infrared monitoring image into a target color space.

[0060] In this step, an infrared monitoring image of the levee structure to be monitored is acquired, and the infrared monitoring image is converted into a target color space to enhance the brightness component in the target color space to generate an infrared processed image.

[0061] Specifically, the infrared monitoring image is converted from a current color space to a target color space to generate a monitoring image to be processed, wherein the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel; Using formula 1, and according to the pixel value of each pixel in the monitored image to be processed in the target color space Determine the corresponding characteristic irradiation component , where Formula 1 is: ; in, is the pixel value The brightness component in is the total number of channels in the target color space, is the pixel value No. The preset weighting coefficients corresponding to the components, is the pixel value No. The amount in The pixel gradient in the direction, is the pixel value No. The amount in The pixel gradient in the direction, is the pixel value No. The preset first weight coefficient corresponding to the component, is the pixel value No. The preset second weight coefficient corresponding to the component, is the pixel value No. A preset third weight coefficient corresponding to each component; Using formula 2, and according to the characteristic illumination component The brightness component of each pixel in the monitored image to be processed in the target color space is enhanced to generate an infrared processed image, where Formula 2 is: ; in, is the enhanced brightness component of each pixel in the monitored image to be processed, is the maximum value of the characteristic illumination component in the monitoring image to be processed, It is the preset brightness enhancement factor.

[0062] In the above steps, the infrared monitoring image is converted from the current color space (including red channel, green channel and blue channel) to the target color space (including hue channel, saturation channel and brightness channel) to generate a monitoring image to be processed. This step aims to convert the infrared monitoring image from the commonly used RGB color space to the HSV color space that is more suitable for processing brightness information. In the RGB color space, the brightness information of the infrared image is scattered in the three color channels of red, green and blue, making it difficult to directly perform targeted enhancement processing; while the HSV color space centrally represents the brightness information of the image through an independent brightness channel (Value), which is convenient for subsequent processing.

[0063] It is worth noting that infrared thermal imaging technology can capture the thermal radiation information on the surface of the levee and identify the leakage area through temperature differences. However, infrared images often have problems such as low resolution and insufficient detail information, and their original image format is mostly in RGB color space. Directly using them for image fusion processing may not give full play to the advantages of thermal radiation information of infrared images. In contrast, the HSV color space decomposes image information into three components: hue, saturation, and brightness. Among them, the brightness component is more consistent with the thermal radiation information of infrared images and can more intuitively reflect temperature differences. Therefore, converting infrared monitoring images from RGB color space to HSV color space and enhancing the brightness component can help improve the effect of infrared images in image fusion.

[0064] Visible light imaging technology can clearly reflect the texture and detail information of the levee surface, but it is also greatly affected by lighting conditions. Fusion of infrared images and visible light images can make full use of the advantages of both to generate a fused image that contains both thermal radiation information and high definition, thereby more accurately identifying the leakage area.

[0065] However, in the process of infrared and visible light image fusion, different color space processing strategies will have a significant impact on the fusion results. The fusion of RGB-based infrared monitoring images and visible light images may cause the fused images to have deficiencies in brightness contrast and detail information retention due to the strong correlation between the components of the RGB color space and the fact that the brightness information of the infrared image is scattered in three color channels. In contrast, the fusion of HSV-based infrared monitoring images (especially after brightness component enhancement) and visible light images can better retain the thermal radiation information of the infrared image, while combining the texture and detail information of the visible light image to generate a higher quality fused image.

[0066] Specifically, the fusion of RGB-based infrared monitoring images and visible light images may have the following defects: Brightness information dispersion: In the RGB color space, the brightness information of the infrared image is dispersed in the three color channels of red, green, and blue, making it difficult to directly enhance the information, thus affecting the brightness contrast of the fused image.

[0067] Color component correlation: There is a strong correlation between the three components of the RGB color space. This correlation may cause information redundancy or information loss during the image fusion process, affecting the quality of the fused image.

[0068] Insufficient detail information preservation: Due to the low resolution and insufficient detail information of infrared images, direct fusion based on the RGB color space may not be able to fully preserve the detail information in the visible light image.

[0069] The fusion of HSV-based infrared monitoring images (after brightness component enhancement) and visible light images can overcome the above defects: Brightness information is concentrated: In the HSV color space, brightness information is concentrated in the brightness component (Value), which facilitates enhancement processing, thereby improving the brightness contrast of the fused image.

[0070] Independence of color components: The hue, saturation, and brightness components of the HSV color space are relatively independent. This independence helps reduce information redundancy and information loss during image fusion and improves the quality of the fused image.

[0071] The detail information is fully preserved: by enhancing the brightness component of the infrared image and combining it with the texture and detail information of the visible light image, a fused image that contains both thermal radiation information and high definition can be generated, providing a more accurate and detailed data basis for subsequent leakage risk identification.

[0072] In addition, in the above steps, Formula 1 is used to determine the corresponding characteristic illumination component according to the pixel value of each pixel point in the monitored image to be processed in the target color space. Formula 1 comprehensively considers the brightness component of the pixel value and its gradient information in the x and y directions, and calculates the characteristic illumination component by weighted summation. Specifically, The term combines the gradient information of the i-th component of the pixel value (i.e., hue, saturation, or brightness component) in the x and y directions and the original value of the component, and performs weighted summation through a preset weighting coefficient to obtain a comprehensive feature representation of the component. Then, the characteristic illumination component is calculated using these comprehensive feature representations and the brightness component. This process effectively extracts the feature information in the image and provides a basis for subsequent brightness enhancement processing.

[0073] Then, using Formula 2, the brightness component of each pixel in the monitored image to be processed in the target color space is enhanced according to the characteristic illumination component to generate the infrared processed image. Formula 2 nonlinearly enhances the brightness component by introducing the characteristic illumination component, where is the maximum value of the characteristic illumination component, which is used for normalization processing to ensure that the enhanced brightness component is within a reasonable range; The preset brightness enhancement coefficient is used to adjust the enhancement intensity. This step guides the enhancement of the brightness component through the characteristic illumination component, so that the leakage area in the infrared image (usually manifested as an abnormal temperature area) is highlighted in brightness while retaining other important detail information in the image.

[0074] By converting the infrared monitoring image from RGB color space to HSV color space and focusing on the processing of the brightness channel, the brightness information of the infrared image is enhanced in a centralized and targeted manner. This helps to improve the brightness contrast of the leakage area in the image, making it easier to identify and detect. In addition, Formula 1 effectively extracts the feature information in the image by comprehensively considering the brightness component and gradient information of the pixel value. These feature information provides an important basis for the subsequent brightness enhancement processing, making the enhancement process more accurate and effective. Formula 2 significantly improves the brightness of the leakage area in the infrared image by introducing the characteristic illumination component to perform nonlinear enhancement on the brightness component. This enhancement effect not only makes the leakage area more prominent, but also helps to retain other important detail information in the image, improving the overall quality of the image.

[0075] S202: Perform feature point detection on the infrared processed image to determine a set of infrared feature points.

[0076] Specifically, the infrared processed image may be subjected to calibration point detection to determine a set of infrared feature points, wherein at least one calibration point is preset on the levee structure to be monitored. In the process of levee monitoring, in order to accurately perform image registration and fusion, it is necessary to preset at least one calibration point on the levee structure to be monitored. These calibration points may be obvious physical marks (such as highly reflective stickers, markers of special shapes, etc.), or they may be inherent and easily identifiable feature points on the levee structure (such as the turning point of the levee, the vertex of a special structure, etc.). The preset calibration points should ensure that they are clearly visible in both infrared and visible light images, and their positions are fixed and unchanged. It is worth noting that in order to better display the calibration points in the infrared processed image, a temperature control device may also be provided on the calibration points to control the temperature of the calibration points to a preset temperature range.

[0077] S203: Obtain a visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set.

[0078] Specifically, in the visible light monitoring image, the calibration points are detected using the same feature point detection algorithm as the infrared processing image. Since the calibration points are preset on the levee structure and fixed in position, they should also be clearly visible in the visible light image. Through the feature point detection algorithm, all feature points in the visible light monitoring image, including the preset calibration points, are extracted. These feature points are combined into a visible light feature point set for subsequent image registration and fusion processes. For better recognition, the calibration points can be set to a specific shape.

[0079] S204: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set.

[0080] Specifically, Formula 3 is used for iteration, and the spatial transformation matrix is ​​determined based on the infrared feature point set and the visible light feature point set: , where Formula 3 is: ; in, is the number of features in the infrared feature point set and the visible light feature point set, is the first The feature point locations, is the first in the visible light feature point set feature point positions, spatial transformation matrix The initial value of the iteration is the identity matrix.

[0081] S205. Using the space transformation matrix Each pixel in the infrared processed image is spatially transformed to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused.

[0082] Specifically, using the space transformation matrix Each pixel in the infrared processed image is spatially transformed to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, and the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

[0083] In the above S204-S205, the spatial transformation matrix can be determined by iterative calculation of formula 3. This formula finds the optimal spatial transformation matrix by minimizing the sum of squares of the transformed Euclidean distances between infrared feature points and visible light feature points. This process effectively optimizes the matching accuracy between feature points and provides a solid foundation for subsequent image alignment. The spatial transformation matrix not only takes into account simple image transformations such as translation and rotation, but can also adapt to more complex deformations such as scaling, affine transformation, etc. This makes the present invention more adaptable and robust when dealing with image differences caused by factors such as shooting angle, distance or deformation of the levee structure itself.

[0084] By using a certain spatial transformation matrix to perform spatial transformation on each pixel in the infrared processed image, accurate alignment of the infrared image and the visible light image at the pixel level can be achieved. This alignment method ensures that the information source of the corresponding pixels in the fused image is consistent, avoiding fusion artifacts or information loss caused by image misalignment. By achieving accurate alignment of the infrared image and the visible light image, more accurate and consistent input data is provided for the subsequent data fusion model, thereby significantly improving the quality of the fused image. The fused image will retain both the thermal radiation information of the infrared image and the high-resolution detail information of the visible light image, providing more reliable data support for the accurate identification of levee leakage hazards.

[0085] S206: Using a preset data fusion model, and according to the infrared feature point set and the visible light feature point set, fuse the infrared processed image and the visible light monitoring image to generate a fused monitoring image.

[0086] In this step, a preset data fusion model is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0087] The preset data fusion model includes a generator and a discriminator. The output end of the generator is connected to the discriminator, and the output end of the output end of the discriminator is respectively connected to the generator to form an adversarial network.

[0088] First, as the front-end part of the adversarial network, the generator is responsible for integrating the information in the infrared processed image and the visible light monitoring image, and generating a fused monitoring image. The design of the generator makes full use of the neural network structure in deep learning technology, which can automatically learn and extract the key features of the two images, thereby realizing the effective fusion of information. Secondly, the discriminator plays the role of a "supervisor". It distinguishes the authenticity of the fused monitoring image generated by the generator, that is, whether the image truly reflects the leakage danger of the levee structure. The design of the discriminator is also based on a deep neural network and has a strong image recognition ability. Through continuous adversarial training with the generator, the discriminator can gradually improve its accuracy in distinguishing the authenticity of the fused image. Furthermore, the adversarial network formed by the feedback connection between the generator and the discriminator enables the generator to not only consider how to integrate the information in the infrared and visible light images when generating the fused image, but also to deceive the discriminator as much as possible, making it difficult to judge the authenticity of the fused image. The discriminator continuously improves its own discrimination ability to cope with the deception of the generator. This process of mutual competition and co-evolution prompts the generator to continuously generate higher-quality fused monitoring images. In addition, the preset data fusion model based on the adversarial network also has good generalization ability. Since the generator and discriminator in the adversarial network are learned based on a large amount of training data, they can automatically adapt to the image features in different environments and generate corresponding fused monitoring images. This makes it have a wide range of application prospects in embankment leakage monitoring.

[0089] As for the specific training process of the preset data fusion model, the generator and the discriminator can be initialized. The training data in the image fusion data set is input into the generator to generate a fusion training image. The training data includes infrared training images and visible light training images. Both the infrared training images and the visible light training images include the embankment structure to be monitored. The fusion training image and the fusion verification image in the fusion data set are input into the discriminator to determine the loss function value. The model parameters in the generator and the discriminator are iteratively updated according to the loss function value and the gradient descent algorithm until the preset convergence conditions are met to generate the preset data fusion model. Optionally, the loss of the above-mentioned discriminator can still be one or more combinations of cross entropy loss, binary cross entropy loss, and binary cross entropy loss.

[0090] During the training process, first of all, initializing the generator and the discriminator is the starting point of the training process. The initialization process sets the initial model parameters for these two core components and provides a basis for subsequent training. Reasonable initialization helps to speed up the training and improve the convergence of the model. Next, the training data in the image fusion dataset (including infrared training images and visible light training images) are input into the generator to generate fused training images. This step is the key for the generator to learn how to fuse the two image information. Through the input of a large amount of training data, the generator can gradually master the characteristics of infrared and visible light images, and learn how to effectively fuse these features together to generate fused training images with higher quality and more information. Then, the fused training image and the fused verification image in the fusion dataset are input into the discriminator to determine the loss function value. The discriminator evaluates the image quality generated by the generator by comparing the difference between the fused training image and the fused verification image, and calculates the loss function value. This loss function value reflects the gap between the current fusion capability of the generator and the ideal fusion effect. Finally, according to the loss function value and the gradient descent algorithm, the model parameters in the generator and the discriminator are iteratively updated. By continuously adjusting the model parameters, the generator can gradually improve the quality of its fused image, while the discriminator can more accurately judge the authenticity of the fused image. The iterative update process continues until the preset convergence conditions are met, at which point the generated preset data fusion model already has a high fusion capability and discrimination accuracy.

[0091] It is worth noting that the fused monitoring image generated above is an image in which temperature features are added to each pixel point on the basis of the visible light monitoring image, and the temperature features are associated with the brightness component in the target color space.

[0092] By integrating the temperature features in the infrared image into the visible light monitoring image, the information richness of the fused monitoring image is significantly improved. Infrared images can reflect the thermal radiation information of the levee structure, especially the temperature distribution of the leakage area is often different from that of the normal area due to water evaporation and other reasons. The visible light image provides high-resolution detail information of the levee structure, such as texture and shape. After fusing the two, the fused monitoring image not only retains the detailed information of the visible light image, but also adds temperature features, providing more comprehensive and accurate data support for subsequent leakage risk detection.

[0093] The temperature features in the fused monitoring image are associated with the brightness components in the visible light image. This design makes the water leakage area present unique features in the fused image. Since the water leakage area is usually at a lower temperature, the brightness components of these areas will be reduced accordingly in the fused image, forming a significant brightness difference. This brightness difference provides an important feature basis for the subsequent leakage risk detection algorithm, enabling the algorithm to more accurately identify the water leakage area and improve the accuracy of detection.

[0094] In practical applications, the levee monitoring environment is often complex and changeable, with various interference factors, such as illumination changes and shadow occlusion. These factors may have a significant impact on the quality of visible light images, thereby affecting the detection effect of leakage hazards. However, since infrared images are not restricted by lighting conditions, the fused monitoring image can still maintain stable quality under interference factors such as illumination changes and shadow occlusion. In addition, by associating temperature features with brightness components, the anti-interference ability of the fused monitoring image is further enhanced, enabling the algorithm to accurately identify water leakage areas in complex environments.

[0095] In the field of image processing, the choice of color space has an important impact on the performance and effect of the algorithm. By associating the temperature feature with the brightness component in the visible light image, the brightness component information of the target color space is fully utilized, which not only makes the fused monitoring image more visually intuitive and easy to understand, but also improves the algorithm's utilization of color space information, providing a more convenient and efficient method for subsequent image processing and analysis.

[0096] S207: Use a preset hazard recognition model to identify the fused monitoring image to determine the location of the leakage hazard.

[0097] Specifically, it may be to determine that the pixel points of the fused monitoring image in the target color space whose brightness component is greater than a preset brightness component threshold are target pixel points, so as to form a target pixel point set; The leakage danger position is determined according to the position of each target pixel point in the target pixel point set, wherein the number of target pixel points included in the corresponding position of the leakage danger position in the fused monitoring image is greater than a preset pixel point threshold.

[0098] By setting a preset brightness component threshold, the pixels with higher brightness components in the fused monitoring image can be accurately screened out as target pixels. Since the water leakage area is usually manifested as a low temperature area in the infrared image, and in the fused monitoring image, these low temperature areas are associated with the brightness component of the visible light image, their brightness components will be reduced accordingly. By setting a reasonable threshold, the interference of background noise and other non-water leakage areas can be effectively eliminated, so as to accurately identify the target pixels in the water leakage area. Furthermore, according to the position of each pixel in the target pixel set, the specific location of the leakage risk can be determined, thereby improving the accuracy of recognition.

[0099] In practical applications, the levee monitoring environment is complex and changeable, and factors such as lighting conditions and shadows may affect the image quality. However, due to the use of fused monitoring images for leakage risk identification, the image combines the advantages of infrared images and visible light images and can resist these interference factors to a certain extent. In addition, by setting a preset pixel threshold, it is required that the corresponding position of the leakage risk location in the fused monitoring image must contain a sufficient number of target pixels. This requirement further enhances the robustness of recognition. Even if there are individual noise pixels or local lighting changes, as long as the number of pixels in the overall leakage area meets the threshold requirements, the leakage risk location can still be accurately identified.

[0100] By directly analyzing the brightness components in the fused monitoring image, the target pixels are quickly screened out, and the leakage danger location is determined according to the location of the target pixels. This method avoids the complex image processing and feature extraction process and greatly improves the efficiency of recognition. Especially in real-time monitoring scenarios, fast and accurate leakage danger recognition is of great significance for taking timely response measures and ensuring the safety of levees.

[0101] In addition, by determining the target pixel set and marking the leakage risk location, it not only provides quantitative information of the leakage risk (such as location, range, etc.), but also provides intuitive visualization effects. In the fused monitoring image, the leakage risk location can be clearly displayed, which is convenient for monitoring personnel to quickly understand the safety status of the levee and take corresponding maintenance measures. In addition, the visualization effect also helps monitoring personnel to conduct further analysis and research on the leakage risk, providing strong support for the safety management of the levee.

[0102] Figure 3 FIG. 1 is a schematic diagram of a structure of an infrared and visible light image fusion processing device for levee leakage danger according to an exemplary embodiment of the present application. Figure 3 As shown, the embankment leakage danger infrared and visible light image fusion processing device 300 provided in this embodiment includes: The acquisition module 310 is used to acquire an infrared monitoring image of the levee structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image; A processing module 320 is used to perform feature point detection on the infrared processed image to determine a set of infrared feature points; The acquisition module 310 is further used to acquire the visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The processing module 320 is also used to use a preset data fusion model to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

[0103] Optionally, the processing module 320 is specifically configured to: Converting the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, wherein the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel; Determine the corresponding characteristic illumination component according to the pixel value of each pixel point in the monitoring image to be processed in the target color space; The brightness component of each pixel in the monitored image to be processed in the target color space is enhanced according to the characteristic illumination component to generate the infrared processed image.

[0104] Optionally, the processing module 320 is specifically configured to: Performing calibration point detection on the infrared processed image to determine the infrared feature point set, wherein at least one calibration point is preset on the embankment structure to be monitored; Correspondingly, the performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes: Calibration point detection is performed on the visible light monitoring image to determine the visible light feature point set.

[0105] Optionally, the processing module 320 is specifically configured to: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set; The spatial transformation matrix is ​​used to perform spatial transformation on each pixel in the infrared processed image to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, wherein the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

[0106] Optionally, the preset data fusion model includes a generator and a discriminator; The output end of the generator is connected to the discriminator, and the output ends of the output end of the discriminator are respectively connected to the generator to form the adversarial network.

[0107] Optionally, the processing module 320 is specifically configured to: Initializing the generator and the discriminator; Inputting the training data in the image fusion data set into the generator to generate a fused training image, wherein the training data includes an infrared training image and a visible light training image, and both the infrared training image and the visible light training image include the levee structure to be monitored; Inputting the fused training image and the fused verification image in the fused data set into the discriminator to determine a loss function value; The model parameters in the generator and the discriminator are iteratively updated according to the loss function value and the gradient descent algorithm until the preset convergence conditions are met to generate the preset data fusion model.

[0108] Optionally, the processing module 320 is specifically configured to: The fused monitoring image is identified using a preset danger recognition model to determine the location of the leakage danger.

[0109] Figure 4 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 4 As shown, an electronic device 400 provided in this embodiment includes: a processor 401 and a memory 402; wherein: The memory 402 is used to store computer programs, and the memory may also be a flash memory.

[0110] The processor 401 is used to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant description in the above method embodiment.

[0111] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0112] When the memory 402 is a device independent of the processor 401, the electronic device 400 may further include: The bus 403 is used to connect the memory 402 and the processor 401 .

[0113] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the methods provided in the above-mentioned various implementation modes.

[0114] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the methods provided in the above various embodiments.

[0115] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0116] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for fusing infrared and visible light images of dike leakage hazards, characterized in that: include: Acquire an infrared monitoring image of the levee structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image; Performing feature point detection on the infrared processed image to determine a set of infrared feature points; Acquire a visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set by using a preset data fusion model to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

2. The method for fusion processing of infrared and visible light images of dike leakage danger according to claim 1 is characterized in that: The step of converting the infrared monitoring image into a target color space to enhance the brightness component in the target color space and generate an infrared processed image includes: Converting the infrared monitoring image from the current color space to the target color space to generate a monitoring image to be processed, wherein the current color space includes a red channel, a green channel, and a blue channel, and the target color space includes a hue channel, a saturation channel, and a brightness channel; Determine the corresponding characteristic illumination component according to the pixel value of each pixel point in the monitoring image to be processed in the target color space; The brightness component of each pixel in the monitored image to be processed in the target color space is enhanced according to the characteristic illumination component to generate the infrared processed image.

3. The method for fusion processing of infrared and visible light images of dike leakage danger according to claim 1 is characterized in that: The performing feature point detection on the infrared processed image to determine the infrared feature point set includes: Performing calibration point detection on the infrared processed image to determine the infrared feature point set, wherein at least one calibration point is preset on the embankment structure to be monitored; Correspondingly, the performing feature point detection on the visible light monitoring image to determine a visible light feature point set includes: Calibration point detection is performed on the visible light monitoring image to determine the visible light feature point set.

4. The method for fusion processing of infrared and visible light images of dike leakage danger according to claim 1 is characterized in that: Before the preset data fusion model is used to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set, the method further includes: Determine a spatial transformation matrix according to the infrared feature point set and the visible light feature point set; The spatial transformation matrix is ​​used to perform spatial transformation on each pixel in the infrared processed image to align the transformed infrared processed image with the visible light monitoring image to generate image data to be fused, wherein the image data to be fused includes the infrared image to be fused and the visible light image to be fused.

5. The method for fusion processing of infrared and visible light images of dike leakage danger according to any one of claims 1 to 4, characterized in that: The preset data fusion model includes a generator and a discriminator; The output end of the generator is connected to the discriminator, and the output ends of the output end of the discriminator are respectively connected to the generator to form the adversarial network.

6. The method for fusion processing of infrared and visible light images of dike leakage danger according to claim 5 is characterized in that: Before the method uses the preset data fusion model and fuses the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the method further includes: Initializing the generator and the discriminator; Inputting the training data in the image fusion data set into the generator to generate a fused training image, wherein the training data includes an infrared training image and a visible light training image, and both the infrared training image and the visible light training image include the levee structure to be monitored; Inputting the fused training image and the fused verification image in the fused data set into the discriminator to determine a loss function value; The model parameters in the generator and the discriminator are iteratively updated according to the loss function value and the gradient descent algorithm until the preset convergence conditions are met to generate the preset data fusion model.

7. The method for fusion processing of infrared and visible light images of dike leakage danger according to any one of claims 1 to 4, characterized in that: After the preset data fusion model is used and the infrared processed image and the visible light monitoring image are fused according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, the method further includes: The fused monitoring image is identified using a preset danger recognition model to determine the location of the leakage danger.

8. A device for fusion processing of infrared and visible light images of dike leakage danger, characterized in that: include: An acquisition module is used to acquire an infrared monitoring image of the levee structure to be monitored, and convert the infrared monitoring image into a target color space to enhance the brightness component in the target color space to generate an infrared processed image; A processing module, used for performing feature point detection on the infrared processed image to determine a set of infrared feature points; The acquisition module is further used to acquire the visible light monitoring image of the levee structure to be monitored, and perform feature point detection on the visible light monitoring image to determine a visible light feature point set; The processing module is also used to use a preset data fusion model to fuse the infrared processed image and the visible light monitoring image according to the infrared feature point set and the visible light feature point set to generate a fused monitoring image, wherein the preset data fusion model is a data fusion model established based on an adversarial network.

9. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images