Dike abnormal target object detection method based on infrared and visible light image fusion

Abstract

The embodiment of the present application provides a kind of abnormal target object detection method of dike based on infrared and visible light image fusion, for identifying the dike target object required to be detected in dike disaster scene, method includes: obtaining infrared image and visible light image;Infrared image and visible light image are registered using feature matching algorithm;Registered infrared image and visible light image are fused using image fusion algorithm to obtain fusion image;The following segmentation optimization operation is iteratively executed until the calculated value of the objective function of mean shift algorithm is minimum, to obtain the image segmentation result of fusion image: the fusion image is segmented using mean shift algorithm to obtain current segmentation result, the calculated value of objective function is determined according to current segmentation result, the hyperparameter of mean shift algorithm is optimized according to the calculated value of objective function.This scheme can accurately segment small size dike target object and can overcome the problem of over segmentation.

Description

Technical Field

[0001] This invention relates to the field of dike engineering technology, and more specifically to a method for detecting abnormal targets on dikes based on infrared and visible light image fusion, electronic equipment, storage medium, and computer program products. Background Technology

[0002] Earth-rock dams are prone to seepage, erosion, seepage, and piping. Failure to detect piping or seepage in time can lead to dam collapse. Therefore, real-time detection of piping and seepage is crucial. Traditionally, the physical properties of dams have been used to monitor seepage and piping, employing methods such as thermal, electromagnetic, geophysical, and radiological methods. However, these methods typically require specialized and expensive equipment or modifications to existing infrastructure, hindering their widespread application during flood season. In fact, in emergency response scenarios, dangerous and costly manual inspections remain the primary method. In flood monitoring, satellite remote sensing technology is widely used due to its wide coverage. However, its application is limited by insufficient spatial resolution, poor flexibility, susceptibility to cloud cover, and high data latency. In recent years, unmanned aerial vehicle (UAV) remote sensing, a flexible and rapid technology, has been widely applied in small-scale, high-resolution, and rapid-response fields such as forest fire prevention, power line inspection, and agricultural management. Therefore, UAV remote sensing holds promise for playing a key role in monitoring dam risks.

[0003] Thermal infrared (TIR) ​​remote sensing is a technique that generates thermal distribution images (i.e., infrared images) by detecting the infrared radiation emitted by objects. It has been used to identify piping and seepage. TIR utilizes thermal anomalies caused by the difference in specific heat capacity between leaking and intact sections of the dike to detect piping or seepage points. However, numerous studies have shown that relying solely on infrared images makes it difficult to distinguish genuine piping / seepage heat sources from "false heat sources" caused by other environmental factors (such as vegetation, waterlogged areas, shadows, and exposed areas of different materials), resulting in low reliability in identifying piping and seepage. Furthermore, relying solely on visible light images makes it difficult to visually distinguish between moist soil (a sign of seepage) and normal damp soil or shadows. Summary of the Invention

[0004] This invention addresses the aforementioned problems. It provides a method, electronic device, storage medium, and computer program product for detecting abnormal targets on dikes based on the fusion of infrared and visible light images. This approach combines the image information from both visible and infrared images to accurately segment small dike targets while overcoming the problem of over-segmentation.

[0005] According to one aspect of the present invention, a method for detecting abnormal target objects on dikes based on the fusion of infrared and visible light images is provided. This method is used to identify target objects on dikes to be detected in dike disaster scenarios. The method includes: acquiring infrared and visible light images of the dike area to be tested; registering the infrared and visible light images using a feature matching algorithm; fusing the registered infrared and visible light images using an image fusion algorithm to obtain a fused image; iteratively performing the following segmentation optimization operations until the calculated value of the objective function of the mean-shift algorithm is minimized to obtain the image segmentation result of the fused image: segmenting the fused image using the mean-shift algorithm to obtain the current segmentation result; determining the calculated value of the objective function based on the current segmentation result; and optimizing the hyperparameters of the mean-shift algorithm based on the calculated value of the objective function. The current segmentation result indicates the image location and object category of the target object on the dike in the fused image. The objective function includes an entropy exponent and a penalty term. The entropy exponent indicates the uncertainty of the object category in the current segmentation result, and the penalty term indicates the degree to which the number of object categories in the current segmentation result exceeds a preset number of categories. The image segmentation result is the current segmentation result obtained when the segmentation optimization operation iteration is completed.

[0006] Optionally, the objective function is the sum of the entropy exponent and the penalty term.

[0007] Alternatively, the objective function can be expressed by the following formula:

[0008] J(C,L)=EI(C,L)+βP(C)

[0009]

[0010] P(C) = max{0, M - α}

[0011] Where J(C,L) represents the objective function, C represents the set of clusters in the current segmentation result, C={c1,c2,……c M}, where M represents the number of clusters, and L represents the set of labels for each pixel in the current segmentation result, L = {l1, l2, ..., l...} N}, where N represents the number of labels, EI(C,L) represents the entropy exponent, and H(C,L) represents the conditional entropy. It is an indicator function, where k represents the k-th label, v represents any pixel value v in the fused image, and C L p(v|C) represents the cluster center of the cluster corresponding to label L. L p(v|C) represents the frequency of occurrence of a pixel with pixel value v in the cluster corresponding to label L. L)>0 indicates that there is a pixel with pixel value v in the cluster corresponding to label L, β represents the preset coefficient, P(C) represents the penalty term, and α represents the preset number of categories; where each cluster represents the set of pixels corresponding to an object category.

[0012] Optionally, the mean-shift algorithm is used to segment the fused image to obtain the current segmentation result, including: repeatedly performing the classification operation until all pixels in the fused image are classified to obtain the current segmentation result; randomly selecting any pixel from the fused image; iteratively calculating the kernel density estimate of the pixel using the mean-shift algorithm to determine the local density maxima of the pixel; if the local density maxima of the pixel belongs to the cluster center of any existing cluster, adding the pixel to the existing cluster; if the local density maxima of the pixel does not belong to the cluster center of any existing cluster, creating a new cluster and adding the pixel to the newly created cluster; wherein, each cluster represents a set of pixels corresponding to an object category.

[0013] Optionally, a feature matching algorithm is used to register the infrared image and the visible light image, including: inputting the infrared image and the visible light image into the coarse-grained feature extraction module of the feature matching algorithm respectively, to obtain a first coarse-grained feature and a second coarse-grained feature output by the coarse-grained feature extraction module that correspond one-to-one with the infrared image and the visible light image; inputting the infrared image and the visible light image into the fine-grained feature extraction module of the feature matching algorithm respectively, to obtain a first fine-grained feature and a second fine-grained feature that correspond one-to-one with the infrared image and the visible light image; inputting the first coarse-grained feature and the second coarse-grained feature into the encoding and decoding module of the feature matching algorithm, to obtain a dense deformation field and a matching score output by the encoding and decoding module, wherein the dense deformation field is used to represent the image of the visible light image. A one-to-one mapping relationship between pixels in the visible light image and pixels in the infrared image is established. The matching score is used to represent the confidence level of the matching ability between corresponding pixels in the visible light image and the infrared image. The feature refinement operation is performed iteratively until the preset refinement conditions are met: the first fine-grained feature, the second fine-grained feature, the dense deformation field, and the matching score are input into the feature refinement module of the feature matching algorithm to obtain the new first fine-grained feature, the new second fine-grained feature, the new dense deformation field, and the new matching score output by the feature refinement module. Based on the latest dense deformation field and the latest matching score at the end of the feature refinement operation iteration, the image coordinates of the matching pixels in the visible light image and the infrared image are aligned using a perspective transformation algorithm.

[0014] Optionally, based on the latest dense deformation field and the latest matchability score at the end of the feature refinement operation iteration, a perspective transformation algorithm is used to align the image coordinates of mutually matching pixels in the visible light image and the infrared image. This includes: randomly selecting multiple matching pixel pairs based on the latest dense deformation field, where each matching pixel pair includes a pixel in the visible light image and a pixel in the infrared image that has a mapping relationship with that pixel; filtering out matching pixel pairs whose matchability scores do not meet preset requirements; calculating the perspective transformation matrix using the filtered matching pixel pairs; and aligning the image coordinates of mutually matching pixels in the visible light image and the infrared image based on the perspective transformation matrix.

[0015] Optionally, an image fusion algorithm is used to fuse the registered infrared image and the visible light image to obtain a fused image, including: inputting the registered visible light image into the decomposition module of the image fusion algorithm to obtain the luminance component and chrominance component of the visible light image output by the decomposition module; inputting the luminance component and the infrared image into the channel stitching module of the image fusion algorithm to obtain a stitched image output by the channel stitching module; inputting the stitched image into the generator of the image fusion algorithm to obtain an intermediate image output by the generator; and inputting the intermediate image and the chrominance component into the fusion module of the image fusion algorithm to output the fused image by the fusion module.

[0016] Optionally, the generator training process includes: acquiring infrared sample images and visible light sample images collected for the sample embankment area; stitching the brightness components of the infrared sample images and visible light sample images along the channel dimension to obtain a stitched sample image; inputting the stitched sample image into the generator to output a predicted fused image; inputting the visible light sample image and the predicted fused image into a discriminator to output a discrimination result, the discrimination result including a first discrimination result corresponding to the visible light sample image and a second discrimination result corresponding to the predicted fused image, the discrimination result representing the confidence level that the corresponding image belongs to the real visible light image; calculating the first loss value of the generator and the second loss value of the discriminator based on the discrimination result; tuning the generator according to the first loss value and the discriminator according to the second loss value, and returning to execute the operation of inputting the stitched sample image into the generator until the first loss value of the generator and the second loss value of the discriminator meet the preset loss value requirements.

[0017] According to another aspect of the present invention, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described method for detecting abnormal target objects on dikes based on infrared and visible light image fusion.

[0018] According to another aspect of the present invention, a storage medium is provided that stores a computer program / instructions, which, when executed, are used to perform the above-described method for detecting abnormal target objects on dikes based on infrared and visible light image fusion.

[0019] According to another aspect of the present invention, a computer program product is provided, including computer program instructions, which, when executed, are used to perform the above-described method for detecting abnormal target objects on dikes based on infrared and visible light image fusion.

[0020] The aforementioned technical solution, by aligning and fusing visible light and infrared images, can fully combine the image information of the two images, enriching the image information in the fused image. In particular, this helps to enrich the detailed information about small-sized dike targets in the fused image, which is beneficial to the accuracy of subsequent segmentation of such small-sized dike targets. On the other hand, by employing the mean-shift algorithm, dike targets belonging to each object category in the fused image can be automatically, quickly, and accurately segmented. In particular, when the hyperparameters of the mean-shift algorithm are optimized using an objective function that includes an entropy exponent and a penalty term, the entropy exponent can guide the mean-shift algorithm to focus on small-sized dike targets in the fused image, and the penalty term helps to overcome the problem of over-segmentation of the fused image by the mean-shift algorithm, thereby obtaining highly accurate image segmentation results. Attached Figure Description

[0021] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0022] Figure 1 A schematic flowchart of a method for detecting abnormal target objects on a dike based on infrared and visible light image fusion according to an embodiment of the present invention is shown.

[0023] Figure 2 A schematic diagram of the module architecture of a feature matching algorithm according to an embodiment of the present invention is shown;

[0024] Figure 3 A schematic diagram of the processing flow of an image fusion algorithm according to an embodiment of the present invention is shown;

[0025] Figure 4 This diagram illustrates how the loss values ​​of the generator and discriminator change with training steps according to an embodiment of the present invention.

[0026] Figure 5 This diagram illustrates a comparison between the embankment target object recognition method using an embodiment of the present invention and image segmentation results obtained using related techniques.

[0027] Figure 6 The diagram illustrates the intersection-union ratio (IUGR) of the levee target object recognition method using an embodiment of the present invention and the image segmentation results obtained using related techniques.

[0028] Figure 7 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0030] To at least partially address the aforementioned problems, embodiments of the present invention provide a method, electronic device, storage medium, and computer program product for detecting abnormal targets on dikes based on the fusion of infrared and visible light images. This solution can combine the image information from both visible light and infrared images to accurately segment small dike targets and overcome the problem of over-segmentation.

[0031] Please see Figure 1 The diagram shown is a schematic flowchart of a method for detecting abnormal target objects on a dike based on infrared and visible light image fusion according to an embodiment of the present invention. According to one aspect of the present invention, a method for detecting abnormal target objects on a dike based on infrared and visible light image fusion is provided for identifying target objects on a dike to be detected in a dike disaster scenario. The method includes steps S110, S120, S130, and S140.

[0032] For example, in a levee disaster scenario, the levee targets to be detected may include water bodies, property, and piping. Water bodies refer to natural water bodies such as rivers, lakes, and oceans. Piping occurs when water flows through the pores between soil particles in a levee or its foundation, and due to the high flow velocity, small soil particles are carried out, gradually forming a conduit. Property refers to all material assets located within the levee's protection area, including but not limited to residential buildings, commercial buildings, agricultural land, and infrastructure (such as roads and bridges). Among these, piping is typically considered a key levee anomaly target to be detected in levee disaster scenarios.

[0033] In step S110, infrared and visible light images of the dike area to be tested are acquired.

[0034] For example, a visible light imaging device (e.g., a wide-angle camera) can be used to acquire a visible light image of the area of ​​the dike to be measured, while an infrared imaging device (e.g., a thermal imaging camera) can be used to acquire an infrared image of the same area of ​​the dike. It should be noted that the infrared image and the visible light image are images acquired for the same area of ​​the dike to be measured. In some embodiments, the infrared image can be an image obtained by grayscale processing of the original acquired image using an infrared imaging device.

[0035] In step S120, the infrared image and the visible light image are registered using a feature matching algorithm.

[0036] For example, because infrared and visible light images are imaged using different devices, their original forms (e.g., color spaces, image resolutions, etc.) may be incompatible, making direct registration difficult. Therefore, a feature matching algorithm is needed to register the infrared and visible light images. Feature matching algorithms can be any algorithm capable of registering infrared and visible light images, such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB). In some embodiments, the image coordinates of pixels in the infrared image can be registered to the image coordinates of matching pixels in the visible light image. In other embodiments, the image coordinates of pixels in the visible light image can be registered to the image coordinates of matching pixels in the infrared image.

[0037] In step S130, an image fusion algorithm is used to fuse the registered infrared image and the visible light image to obtain a fused image.

[0038] For example, by fusing an infrared image and a visible light image, the resulting fused image can retain at least some image information from both the infrared and visible light images. The image fusion algorithm used can be any algorithm capable of fusing registered infrared and visible light images, such as pixel-level fusion, feature-level fusion, Laplacian pyramid fusion, or deep learning-based fusion.

[0039] In step S140, the following segmentation optimization operation is performed iteratively until the calculated value of the objective function of the mean-shift algorithm is minimized to obtain the image segmentation result of the fused image: the mean-shift algorithm is used to segment the fused image to obtain the current segmentation result; the calculated value of the objective function is determined based on the current segmentation result; and the hyperparameters of the mean-shift algorithm are optimized based on the calculated value of the objective function. The current segmentation result indicates the image location and object category of the target object in the fused image. The objective function includes an entropy exponent and a penalty term. The entropy exponent indicates the uncertainty of the object category in the current segmentation result, and the penalty term indicates the degree to which the number of object categories in the current segmentation result exceeds the preset number of categories. The image segmentation result is the current segmentation result obtained when the segmentation optimization operation iteration is completed.

[0040] For example, after acquiring the fused image, the Mean Shift (MS) algorithm can be used to segment the fused image. The hyperparameters of the Mean Shift algorithm (e.g., bandwidth, kernel function) can have preset initial values. It can be understood that when the Mean Shift algorithm is used for the first segmentation of the fused image, the hyperparameters are set to their initial values. Each segmentation of the fused image yields a segmentation result. Based on the segmentation result, the calculated value of the objective function can be determined. Based on this calculated value, the hyperparameters can be adjusted. The hyperparameters involved in the adjustment typically include bandwidth. The adjusted hyperparameters can be used as the hyperparameters for the next segmentation of the fused image. It can be understood that changes in the hyperparameter values ​​can lead to changes in the segmentation result, and correspondingly, changes in the calculated value of the objective function. When the calculated value of the objective function is minimized, the hyperparameter values ​​can be considered to meet the requirements, and the corresponding segmentation result can be used as the final image segmentation result of the fused image. For example, after obtaining the segmentation result of the fused image each time, the current hyperparameters can be adjusted according to the calculated value of the objective function until the hyperparameters corresponding to the minimum calculated value of the objective function are found. In a specific embodiment, the hyperparameters can be systematically optimized using a Bayesian optimization algorithm, and the Bayesian optimization algorithm can use the PTESampler probabilistic surrogate model to model the objective function, thereby obtaining the hyperparameters that minimize the calculated value of the objective function.

[0041] To more clearly explain the guiding role of the objective function in the mean shift algorithm of this invention, the design principles of the objective function used in related technologies are first introduced below. In image segmentation algorithms, the objective function used to optimize the hyperparameters of the image segmentation algorithm usually follows the design principle of "maximizing inter-class differences and minimizing intra-class differences." Based on this design principle, the objective function constructs a metric for the feature space, prompting the algorithm to cluster pixels with similar features into one class, while maximizing the feature distance between pixels of different classes. In actual segmentation, this optimization strategy naturally tends to form a few large clusters because large-scale homogeneous regions are more likely to satisfy the condition of high intra-class feature similarity, and the feature boundaries between large clusters are often more obvious, facilitating the maximization of inter-class differences. For example, in the classification of land features in remote sensing images, this objective function can efficiently segment large areas of water, vegetation, and built-up land, forming regions with clear outlines and complete boundaries. However, this traditional objective function reveals significant shortcomings when dealing with small objects. Guided by the principle of "maximizing inter-class differences," image segmentation algorithms prioritize large objects that occupy the main part of the image, while treating small objects as noise or edge outliers. During iterative optimization, the extraction of image features from small objects is gradually weakened. At the same time, the requirement of "minimizing intra-class differences" makes image segmentation algorithms tend to merge neighboring background pixels, which further leads to the pixels occupied by small objects being "assimilated" by surrounding pixels, ultimately making it impossible to segment small objects.

[0042] The objective function of this invention can consist of at least two parts: an entropy exponent and a penalty term. The calculated values ​​of both the entropy exponent and the penalty term are positively correlated with the calculated value of the objective function, and both can be determined based on the current segmentation result. Specifically, the entropy exponent indicates the uncertainty of the object category in the current segmentation result. In this invention, the smaller the calculated value of the entropy exponent, the greater the uncertainty of the object category in the current segmentation result. It should be noted that the target objects in this invention typically include extremely small objects. For example, a piping failure may occupy only tens to hundreds of pixels in an image, while the background area may include tens to hundreds of thousands of pixels. Furthermore, the spectral or textural features of the area occupied by the piping failure in the image are usually small compared to the surrounding area, for example, only showing subtle grayscale changes or local texture anomalies. In this context, the lower the uncertainty of the object category in the current segmentation result (i.e., the larger the calculated entropy exponent), the more likely the mean-shift algorithm is to ignore small-sized embankment target objects. Conversely, the higher the uncertainty of the object category in the current segmentation result (i.e., the smaller the calculated entropy exponent), the more likely the mean-shift algorithm will focus on small-sized embankment target objects. To avoid ignoring small-sized embankment target objects (such as piping), the calculated entropy exponent should be as small as possible. However, an excessively small entropy exponent can easily lead to oversegmentation, with the most extreme case being each pixel in an image corresponding to one object category. To constrain this oversegmentation, the number of object categories in the segmentation result should be guided to avoid exceeding a preset number of categories. Therefore, a penalty term can be introduced into the objective function, indicating the extent to which the number of object categories in the current segmentation result exceeds the preset number of categories. To overcome the oversegmentation problem, the calculated value of the penalty term should be as small as possible. Since the objective function is determined by both the entropy exponent and the penalty term, the segmentation objective of "segmenting small dike target objects" and the segmentation objective of "avoiding over-segmentation" can be well balanced.

[0043] For example, the mean shift algorithm can be used to classify pixels in the fused image into clusters. Each cluster can be considered as a region belonging to a certain object category. The image position of each region corresponding to each cluster in the fused image can represent the image position of the dike target object belonging to the corresponding object category in the image. In other words, the segmentation result of the fused image can indicate the image position and object category of the dike target object in the fused image. In some embodiments, the registration in step S120 is to register the image coordinates of the infrared image to the image coordinates of the visible light image. In this case, the image coordinates of the visible light image are consistent with the image coordinates of the fused image, and the segmentation result of the fused image can also indicate the image position and object category of the dike target object in the visible light image. In other embodiments, the registration in step S120 is to register the image coordinates of the visible light image to the image coordinates of the infrared image. In this case, the image coordinates of the infrared image are consistent with the image coordinates of the fused image, and the segmentation result of the fused image can also indicate the image position and object category of the dike target object in the infrared image. In some embodiments, the image segmentation result can be a mask image; in other embodiments, the image segmentation result can be a result image generated based on the mask image and the fused image, in which the pixel values ​​of the image regions of the target objects of different object categories are different.

[0044] The aforementioned technical solution, by aligning and fusing visible light and infrared images, can fully combine the image information of the two images, enriching the image information in the fused image. In particular, this helps to enrich the detailed information about small-sized dike targets in the fused image, which is beneficial to the accuracy of subsequent segmentation of such small-sized dike targets. On the other hand, by employing the mean-shift algorithm, dike targets belonging to each object category in the fused image can be automatically, quickly, and accurately segmented. In particular, when the hyperparameters of the mean-shift algorithm are optimized using an objective function that includes an entropy exponent and a penalty term, the entropy exponent can guide the mean-shift algorithm to focus on small-sized dike targets in the fused image, and the penalty term helps to overcome the problem of over-segmentation of the fused image by the mean-shift algorithm, thereby obtaining highly accurate image segmentation results.

[0045] Optionally, the objective function is the sum of the entropy exponent and the penalty term.

[0046] For example, when the calculated value of the entropy exponent is at its minimum, the calculated value of the penalty term may not be at its minimum, and vice versa. In this embodiment of the invention, the sum of the calculated value of the entropy exponent and the calculated value of the penalty term is used as the calculated value of the objective function.

[0047] The above technical solution uses the sum of the entropy exponent and the penalty term as the objective function, which helps to "segment small-sized target objects while avoiding over-segmentation as much as possible".

[0048] Alternatively, the objective function can be expressed by the following formula:

[0049] J(C,L)=EI(C,L)+βP(C)

[0050]

[0051] P(C) = max{0, M - α}

[0052] Where J(C,L) represents the objective function, C represents the set of clusters in the current segmentation result, C={c1,c2,……c M}, where M represents the number of clusters, and L represents the set of labels for each pixel in the current segmentation result, L = {l1, l2, ..., l...} N}, where N represents the number of labels, EI(C,L) represents the entropy exponent, and H(C,L) represents the conditional entropy. It is an indicator function, k represents the k-th label, v represents any pixel value v in the fused image, C L p(v|C) represents the cluster center of the cluster corresponding to label L. L p(v|C) represents the frequency of occurrence of a pixel with pixel value v in the cluster corresponding to label L. L )>0 indicates that there is a pixel with pixel value v in the cluster corresponding to label L, β represents the preset coefficient, P(C) represents the penalty term, and α represents the preset number of categories; where each cluster represents the set of pixels corresponding to an object category.

[0053] To facilitate understanding of the above formulas, the mean shift algorithm is introduced first. In the mean shift algorithm, each pixel of the fused image can be represented as a vector in the feature space, for example, a 5-dimensional vector (R, G, B, x, y), where R, G, and B can represent the pixel value (i.e., color value), and x and y can represent the image position of the pixel. For any pixel in the fused image, the mean shift algorithm iteratively calculates the weighted average of the vectors of all pixels in the pixel's neighborhood (defined by a kernel function) until the weighted average vector converges (i.e., no longer changes significantly). The resulting weighted average vector can be used as the local density maxima of that pixel. By statistically analyzing the local density maxima of each pixel, the pixels can be classified into clusters. Pixels within the same cluster have consistent local density maxima (consistency can mean completely identical or minimally different, meeting the cluster merging criteria), and the local density maxima can be used as the cluster center of the corresponding cluster. It can be understood that each cluster can correspond to an object category, and each cluster can represent a set of pixels corresponding to an object category. Each pixel within the same pixel set can have a label to identify its cluster, and all pixels within the same pixel set have the same label. In the above formula, if the fused image contains 1000 pixels, then the set of labels (i.e., L) can have 1000 labels. That is, the number of labels is consistent with the number of pixels in the fused image, and there is a one-to-one correspondence between pixels and labels, with each label identifying the cluster to which the corresponding pixel belongs. Specifically, in the above formula, the calculated value of the entropy exponent is usually greater than 0 and less than or equal to 1. In the formula for calculating the conditional entropy H(C,L), "st" represents the constraint condition, p(v|C L )>0 indicates that there exists a pixel with pixel value v in the cluster corresponding to label L. Correspondingly, p(v|C) L A value of 0 indicates that there is no pixel with a value of v in the cluster corresponding to label L; that is, the pixel with a value of v does not belong to the cluster corresponding to label L. For example, each cluster can have a cluster center, which represents the central position of the cluster in the feature space (i.e., color space). Indicates the label is l k The cluster center of the cluster to which the pixel belongs. express The value is 1 when it matches the local density maxima corresponding to pixel value v, and 0 otherwise. For example, α can be defined according to the actual segmentation requirements; in a specific embodiment, α can be equal to 6.

[0054] Optionally, the mean-shift algorithm is used to segment the fused image to obtain the current segmentation result, including: repeatedly performing the classification operation until all pixels in the fused image are classified to obtain the current segmentation result; randomly selecting any pixel from the fused image; iteratively calculating the kernel density estimate of the pixel using the mean-shift algorithm to determine the local density maxima of the pixel; if the local density maxima of the pixel belongs to the cluster center of any existing cluster, adding the pixel to the existing cluster; if the local density maxima of the pixel does not belong to the cluster center of any existing cluster, creating a new cluster and adding the pixel to the newly created cluster; wherein, each cluster represents a set of pixels corresponding to an object category.

[0055] For example, the mean shift algorithm specifically finds the local density maxima of a pixel by calculating the kernel density estimate. The formula for calculating the kernel density estimate is as follows:

[0056]

[0057] In the formula, s is the vector of the currently selected pixel in the feature space, and K h (·) denotes a kernel function with bandwidth h. Typically, a Gaussian kernel is used, and its expression is as follows:

[0058]

[0059] For example, in the above formula, μ represents the mean and Σ represents the variance. By iteratively calculating the gradient of f(s) (i.e., the mean shift vector), pixels can be moved to positions with higher density until they converge to a local density maxima. For the current pixel, if the local density maxima of the pixel coincides with the cluster center of any existing cluster, the pixel can be directly added to that existing cluster. If the local density maxima of the pixel does not coincide with the cluster center of any existing cluster, a new cluster can be created, and the pixel can be added to the new cluster. Accordingly, the local density maxima of the pixel can be directly used as the cluster center of the new cluster. The classification operation can be repeated until every pixel in the fused image is classified according to a cluster. It can be understood that after each pixel in the fused image is classified according to a cluster, each cluster has a different image position in the fused image, and each cluster can correspond to an object category. That is to say, obtaining the pixel classification result is equivalent to obtaining the image segmentation result.

[0060] In the above technical solutions, the mean shift algorithm does not require pre-specifying the number of clusters to be formed, and it can identify clusters of any shape, exhibiting good robustness.

[0061] Optionally, a feature matching algorithm is used to register the infrared image and the visible light image, including: inputting the infrared image and the visible light image into the coarse-grained feature extraction module of the feature matching algorithm respectively, to obtain a first coarse-grained feature and a second coarse-grained feature output by the coarse-grained feature extraction module that correspond one-to-one with the infrared image and the visible light image; inputting the infrared image and the visible light image into the fine-grained feature extraction module of the feature matching algorithm respectively, to obtain a first fine-grained feature and a second fine-grained feature that correspond one-to-one with the infrared image and the visible light image; inputting the first coarse-grained feature and the second coarse-grained feature into the encoding and decoding module of the feature matching algorithm, to obtain a dense deformation field and a matching score output by the encoding and decoding module, wherein the dense deformation field is used to represent the image of the visible light image. A one-to-one mapping relationship between pixels in the visible light image and pixels in the infrared image is established. The matching score is used to represent the confidence level of the matching ability between corresponding pixels in the visible light image and the infrared image. The feature refinement operation is performed iteratively until the preset refinement conditions are met: the first fine-grained feature, the second fine-grained feature, the dense deformation field, and the matching score are input into the feature refinement module of the feature matching algorithm to obtain the new first fine-grained feature, the new second fine-grained feature, the new dense deformation field, and the new matching score output by the feature refinement module. Based on the latest dense deformation field and the latest matching score at the end of the feature refinement operation iteration, the image coordinates of the matching pixels in the visible light image and the infrared image are aligned using a perspective transformation algorithm.

[0062] Please see Figure 2 The diagram illustrates the module architecture of a feature matching algorithm according to an embodiment of the present invention. The feature matching algorithm may include a coarse-grained feature extraction module, a fine-grained feature extraction module, an encoding / decoding module, and a feature refinement module. The coarse-grained feature module may, for example, be a self-supervised learning-based visual model (DIstillation with NOlabels.v2, DINOv2). When processing infrared images and visible images using the feature matching algorithm, the coarse-grained feature extraction module can extract a first coarse-grained feature from the infrared image and a second coarse-grained feature from the visible image. The first and second coarse-grained features are... Figure 2 This is shown as CoarseFeatures. The fine-grained feature extraction module could, for example, be a Visual Geometry Group 19-layer model (VGG19). The fine-grained feature module can extract first fine-grained features from the infrared image and second fine-grained features from the visible light image. The first and second fine-grained features are... Figure 2This is shown as Fine Features. The encoding / decoding module can, for example, consist of a Gaussian process-based encoder and a Transformer-based decoder. The encoding / decoding module can output a dense deformation field based on the first and second coarse features. Figure 2 Shown as Warp Matrix W0) and the matching score ( Figure 2 Shown as Matchability P0, the dense deformation field can represent the one-to-one mapping relationship between pixels in the visible light image and pixels in the infrared image. The matchability score can represent the confidence level of the matchability between pixels with a mapping relationship indicated by the dense deformation field. The feature refinement module can be, for example, a refiner based on a convolutional neural network (Warp Refiner), which can iteratively process the first fine-grained feature, the second fine-grained feature, the dense deformation field, and the matchability score to obtain new first fine-grained features, new second fine-grained features, new dense deformation field, and new matchability score. The feature refinement operation iteration ends when a preset refinement condition is met, such as the number of iterations reaching a preset number. Based on the latest dense deformation field at the end of the feature refinement operation iteration ( Figure 2 The diagram is shown as Warp Matrix W. n ) and the latest matchability score ( Figure 2 Shown as

[0063] Matchability P n The perspective transformation algorithm can be used to align the image coordinates of matching pixels in visible light images and infrared images.

[0064] Specifically, the expressions for extracting the first coarse-grained feature and extracting the second coarse-grained feature are as follows:

[0065]

[0066] In the above formula, I inf Indicating an infrared image, I vis Represents a visible light image. and These are the first coarse-grained features extracted by DINOv2 for infrared images and the second coarse-grained features extracted for visible light images, respectively. and These are the first fine-grained features extracted by VGG19 for infrared images and the second fine-grained features extracted for visible light images, respectively.

[0067] Specifically, the expressions for the dense deformation field and the matchability score output by the encoding / decoding module are as follows:

[0068]

[0069] In the formula, D transformer Indicates decoder, E GaussianProcess Indicates encoder, Represents a dense deformation field. This represents the matchability score.

[0070] Specifically, the iterative process of the feature refinement module is represented by the following formula:

[0071]

[0072] In the formula, R k This represents a refiner based on a convolutional network, where k represents the number of iterations.

[0073] The registration method employed in the above technical solution can effectively capture the common structures (such as edges and textures) of visible light images and infrared images, reducing interference caused by modal differences between the two images. It can achieve high matching accuracy when processing low-contrast images (such as infrared images). Furthermore, this registration method can use an encoding / decoding module for global matching, providing an initial dense deformation field, which helps avoid local optima traps. It can also use a feature refinement module for local refinement, iteratively optimizing the dense deformation field and the matchability score to make the registration of each local region more accurate, thereby obtaining a high-precision registration result.

[0074] Optionally, based on the latest dense deformation field and the latest matchability score at the end of the feature refinement operation iteration, a perspective transformation algorithm is used to align the image coordinates of mutually matching pixels in the visible light image and the infrared image. This includes: randomly selecting multiple matching pixel pairs based on the latest dense deformation field, where each matching pixel pair includes a pixel in the visible light image and a pixel in the infrared image that has a mapping relationship with that pixel; filtering out matching pixel pairs whose matchability scores do not meet preset requirements; calculating the perspective transformation matrix using the filtered matching pixel pairs; and aligning the image coordinates of mutually matching pixels in the visible light image and the infrared image based on the perspective transformation matrix.

[0075] For example, after obtaining the latest dense deformation field, multiple matching pixel pairs can be randomly selected based on the latest dense deformation field, each matching pixel pair having a corresponding matching score. Matching pixel pairs whose matching scores do not meet preset requirements can be filtered out. This embodiment of the invention does not limit the specific preset requirements. For example, the preset requirements could be that the matching score is greater than or equal to a preset minimum score threshold, or that the selected matching pixel pairs are sorted from high to low according to their corresponding matching scores, with the corresponding arrangement position preceding a preset arrangement position. The filtered matching pixel pairs can be used to calculate a perspective transformation matrix, and based on the perspective transformation matrix, the image coordinates of mutually matching pixels in the visible light image and the infrared image can be aligned. In a specific embodiment, the perspective transformation matrix can be used to transform the image coordinates of pixels in the visible light image; this transformation process can be considered as the process of registering the visible light image to the infrared image. The above-mentioned sampling pixel pairs and perspective transformation process are described in... Figure 2 The middle is shown as

[0076] Sample & Transform.

[0077] The above technical solution can obtain a high-accuracy perspective transformation matrix by filtering out matching pixel pairs with low matching scores, thereby enabling more precise alignment of two images.

[0078] Optionally, an image fusion algorithm is used to fuse the registered infrared image and the visible light image to obtain a fused image, including: inputting the registered visible light image into the decomposition module of the image fusion algorithm to obtain the luminance component and chrominance component of the visible light image output by the decomposition module; inputting the luminance component and the infrared image into the channel stitching module of the image fusion algorithm to obtain a stitched image output by the channel stitching module; inputting the stitched image into the generator of the image fusion algorithm to obtain an intermediate image output by the generator; and inputting the intermediate image and the chrominance component into the fusion module of the image fusion algorithm to output the fused image by the fusion module.

[0079] For example, the image fusion algorithm may include a decomposition module, a channel stitching module, a generator, and a fusion module. In step S130, the decomposition module may employ a Luminance / Chrominance Separation module (LCSM), which first decomposes the visible light image into a luminance component and a chrominance component. The luminance component retains the texture information of the visible light image, and the chrominance component retains the color information of the visible light image. The channel stitching module can stitch the channels of the luminance component and the channels of the infrared image together. The resulting stitched image can be input into the generator, which can pre-fuse the texture information of the luminance component and the thermal radiation information of the infrared image based on the stitched image to output a corresponding intermediate image. The fusion module may employ a Luminance / Chrominance Merging module (LCMM), which can fuse the intermediate image and the chrominance component to output the final fused image. It can be understood that the resulting fused image may include the texture and color information of the visible light image and the thermal radiation information of the infrared image.

[0080] The above technical solution decomposes the visible light image into a luminance component and a chrominance component. By stitching the infrared image with the luminance component, the feature alignment process of multimodal images can be simplified. The generator can be trained to make its fusion rules more adaptable to the luminance components of the infrared and visible light images. In addition, by independently participating in the final fusion, the chrominance component can avoid the infrared image from contaminating the color information and ensure that the fused image conforms to the visual habits of the human eye.

[0081] Optionally, the generator training process includes: acquiring infrared sample images and visible light sample images collected for the sample embankment area; stitching the brightness components of the infrared sample images and visible light sample images along the channel dimension to obtain a stitched sample image; inputting the stitched sample image into the generator to output a predicted fused image; inputting the visible light sample image and the predicted fused image into a discriminator to output a discrimination result, the discrimination result including a first discrimination result corresponding to the visible light sample image and a second discrimination result corresponding to the predicted fused image, the discrimination result representing the confidence level that the corresponding image belongs to the real visible light image; calculating the first loss value of the generator and the second loss value of the discriminator based on the discrimination result; tuning the generator according to the first loss value and the discriminator according to the second loss value, and returning to execute the operation of inputting the stitched sample image into the generator until the first loss value of the generator and the second loss value of the discriminator meet the preset loss value requirements.

[0082] For example, when training the generator, the generator can be connected to the discriminator, and the generator and discriminator can be trained using adversarial training. In a specific embodiment, the generator and discriminator can be generators and discriminators in a Generative Adversarial Network (GAN). Specifically, the methods for acquiring infrared sample images, visible light sample images, and stitched sample images can refer to the methods for acquiring infrared images, visible light images, and stitched images in the foregoing embodiments, respectively, and will not be repeated here. The stitched sample image can be input into the generator, and the generator outputs a predicted fused image based on the input stitched sample image. The predicted fused image and the visible light sample image can be input into the discriminator, and the discriminator can output a first discrimination result corresponding to the predicted fused image and a second discrimination result corresponding to the visible light sample image. Based on the discrimination results, a first loss value of the generator can be calculated, and a second loss value of the discriminator can be calculated.

[0083] For example, the expression for the second loss value of the discriminator is as follows:

[0084]

[0085] In the formula, V D Let x represent the second loss value of the discriminator, p. data (x) represents a portion or all of the images randomly selected from the visible light sample images, D(x) is the output probability of the discriminator for image x (i.e., the second discrimination result), E[*] represents the mean of *, and z ~ p z G(z) represents the input image (including visible light sample image and stitched sample image) of the input generator, and G(z) represents the output image output by the generator based on the input image.

[0086] For example, the expression for the generator's first loss value is as follows:

[0087]

[0088] Specifically, in adversarial training, the generator's parameters can be fixed first, and the discriminator trained by maximizing the second loss value. Then, the discriminator's parameters are fixed, and the generator trained by minimizing the first loss value. These two processes are iteratively optimized alternately until the second discrimination result corresponding to the predicted fused image output by the generator meets the requirements. The iterative process can be represented by the following formula:

[0089]

[0090] In the above technical solution, the generator can generate an intermediate image that is highly similar to the real visible light image through adversarial learning, which is beneficial for the intermediate image to retain as much of the thermal radiation information and the texture information of the brightness component in the infrared image as possible at the same time.

[0091] Please see Figure 3 The diagram illustrates the processing flow of an image fusion algorithm according to an embodiment of the present invention. The visible image is decomposed into chroma and luminance components by the LCSM (Limited Channel Modeling) module. The luminance component and the infrared image are then concatenated by the channel stitching module to obtain a stitched image. This stitched image is input into a trained generator to obtain an intermediate image. The intermediate image and the chroma component are then fused by the LCMM (Limited Channel Modeling) module to obtain a fused image. During the training phase of the generator, it can be connected to a discriminator. The training process can be referred to the relevant descriptions in the foregoing embodiments, and will not be repeated here. Please refer to... Figure 4 The diagram shown illustrates the variation of the loss values ​​of the generator and discriminator with training steps according to an embodiment of the present invention. Figure 4 It can be seen that both the generator loss and the discriminator loss gradually decrease with the increase of training steps. A training step refers to one parameter update.

[0092] Please see Figure 5 As shown, it is a comparative diagram of the image segmentation results obtained by the embankment target object recognition method according to the embodiment of the present invention and by using related technologies. Figure 5 Examples include Image A and Image B. Figure 5 In this process, the images involved in image segmentation include an unfused visible light image, an unfused infrared image, a fused image obtained using the Denoising Diffusion Image Fusion Model (DDFM), and a fused image obtained using the image fusion algorithm (FGAN) of this invention. For details on the images involved in image segmentation, please refer to [link to relevant documentation]. Figure 5 The column labeled "Fused" contains the image segmentation results for each image involved in the segmentation process. (See also: [link to relevant documentation]). Figure 5 The mask images corresponding to the segmentation results of each image can be found in the column labeled "Segmented". Figure 5The column labeled "Label Mask" is located in the middle. In the mask image, the background area, properties area, piping area, and water area each have different grayscale values.

[0093] Please see Figure 6 The diagram shows the intersection-union ratio (IUGR) of the image segmentation results obtained using the embankment target object identification method of this invention and related technologies. First, the calculation method for the embankment target object is introduced. Each group of images used for segmentation can have corresponding annotation data, which indicates the object category and image location of the embankment target object in the corresponding image. In the annotation data, the set of image regions for each embankment target object can be denoted as G = {g1, g2, ..., g...}. m In the image segmentation results, the set of image regions for each target object on the dike can be denoted as Q = {q1, q2, ..., q}. n For the i-th levee target object, the formula for calculating the intersection-union ratio (IU / U) of the image region in the labeled data and the image region in the image segmentation result is as follows:

[0094]

[0095] In the formula, |g i ∩q i | represents the number of pixels at the intersection of the image region of the i-th embankment target object in the labeled data and the image region in the image segmentation result, |g i ∪q i | represents the number of pixels in the union of the image region of the i-th embankment target object in the labeled data and the image region in the image segmentation result. For each image region in set Q, a corresponding image region with the maximum IoU can be determined in set G. If the IoU between the two is greater than 0, a matching relationship is established between the two image regions, and the image regions with established matching relationships will no longer participate in other matching.

[0096] When evaluating image segmentation results, the Mean Intersection over Union (mIoU) can also be used. In this application, mIoU refers to the average IoU of the image regions of water bodies, property, and piping, and its expression is as follows:

[0097]

[0098] Figure 6 Includes the average intersection-union ratio (Mean) of each image segmentation result, and the image region of the piping / property.

[0099] The results of comparing the cross-union ratio of the property (Piping) and the cross-union ratio of the water image region. Specifically, in Figure 6 In the various comparison results, the unfused visible light image, the unfused infrared image, the fused image obtained using the denoising diffusion image fusion model, and the fused image obtained using the image fusion algorithm of this invention are respectively shown on the horizontal axis as Only IV, Only RI, DDFM, and FGAN. Figure 6 The vertical axis of each comparison result represents the magnitude of the corresponding cross-union ratio. Figure 6 It can be seen that the image segmentation results of the dike target object recognition method of the present invention not only have superior segmentation performance for large-scale dike target objects (the cross-union ratio of the image region of water reaches 0.93), but also significantly outperform the segmentation performance of small-scale dike target objects compared to related technologies. Specifically, using the method of the present invention, the cross-union ratio of the image region of property / piping reaches 0.33, while the cross-union ratios of the image regions of property / piping obtained by related technologies are 0.15, 0.19, and 0.22, respectively. In summary, the dike target object recognition method of the present invention has significantly higher segmentation performance for small-scale dike target objects than related technologies. Furthermore, the average cross-union ratio of the image segmentation results obtained by the dike target object recognition method of the present invention is the highest, at 0.84, which also indicates that the overall segmentation performance of the dike target object recognition method of the present invention is optimal.

[0100] According to another aspect of the present invention, an electronic device is also provided. Figure 7 A schematic block diagram of an electronic device 700 according to an embodiment of the present invention is shown. Figure 7 As shown, the electronic device 700 includes a processor 710 and a memory 720. The processor 710 stores computer program instructions, which, when executed by the processor 710, are used to perform the anti-abnormal target object detection method based on infrared and visible light image fusion according to embodiments of the present invention.

[0101] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored. When executed, the program instructions are used to perform the method for detecting abnormal target objects based on infrared and visible light image fusion according to embodiments of the present invention. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0102] According to another aspect of the present invention, a computer program product is provided, including computer program instructions, which, when executed, are used to perform the above-described method for detecting abnormal target objects on dikes based on infrared and visible light image fusion.

[0103] Those skilled in the art can understand the specific implementation schemes of the above-mentioned electronic devices, storage media, and computer program products by reading the above description of the method for detecting abnormal targets on dikes based on infrared and visible light image fusion. For the sake of brevity, they will not be described in detail here.

[0104] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0106] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0107] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting abnormal target objects on dikes based on infrared and visible light image fusion, characterized in that, The method for identifying target objects on a levee in a levee disaster scenario includes: Acquire infrared and visible light images of the area of ​​the dike to be tested; The infrared image and the visible light image are registered using a feature matching algorithm; An image fusion algorithm is used to fuse the registered infrared image and the visible light image to obtain a fused image; The following segmentation optimization operation is performed iteratively until the calculated value of the objective function of the mean shift algorithm is minimized, in order to obtain the image segmentation result of the fused image: The mean-shift algorithm is used to segment the fused image to obtain the current segmentation result. The calculated value of the objective function is determined based on the current segmentation result. The hyperparameters of the mean-shift algorithm are optimized based on the calculated value of the objective function. The current segmentation result is used to indicate the image location and object category of the embankment target object in the fused image. The objective function includes an entropy index and a penalty term. The entropy index is used to indicate the uncertainty of the object category in the current segmentation result, and the penalty term is used to indicate the degree to which the number of object categories in the current segmentation result exceeds the preset number of categories. The image segmentation result is the current segmentation result obtained when the segmentation optimization operation iteration is completed; the objective function is expressed by the following formula: Where J(C,L) represents the objective function, C represents the set of clusters in the current segmentation result, and C={c1, c2, c3, c4, c5, c6, c7, c8, c9 ... 2，…… c M }, M represents the number of clusters, and L represents the set of labels for each pixel in the current segmentation result, L={ l1, l2, ..., M} …… , l N }, where N represents the number of labels, EI(C,L) represents the entropy exponent, and H(C,L) represents the conditional entropy. It is an indicator function, k represents the k-th label, v represents any pixel value v in the fused image, C L p(v|C) represents the cluster center of the cluster corresponding to label L. L p(v|C) represents the frequency of occurrence of a pixel with pixel value v in the cluster corresponding to label L. L )>0 indicates that there is a pixel with pixel value v in the cluster corresponding to label L, β represents the preset coefficient, P(C) represents the penalty term, and α represents the preset number of categories; Each cluster represents a set of pixels corresponding to an object category.

2. The method according to claim 1, characterized in that, The objective function is the sum of the entropy exponent and the penalty term.

3. The method according to claim 1, characterized in that, The step of segmenting the fused image using the mean shift algorithm to obtain the current segmentation result includes: Repeat the following classification operation until all pixels in the fused image have been classified to obtain the current segmentation result: Randomly select any pixel from the fused image; The mean-shift algorithm is used to iteratively calculate the kernel density estimate of the pixel in order to determine the local density maxima of the pixel. If the local density maximum of the pixel belongs to the cluster center of any existing cluster, add the pixel to the existing cluster; if the local density maximum of the pixel does not belong to the cluster center of any existing cluster, create a new cluster and add the pixel to the new cluster. Each cluster represents a set of pixels corresponding to an object category.

4. The method according to claim 1, characterized in that, The process of registering the infrared image and the visible light image using a feature matching algorithm includes: The infrared image and the visible light image are respectively input into the coarse-grained feature extraction module of the feature matching algorithm to obtain the first coarse-grained feature and the second coarse-grained feature output by the coarse-grained feature extraction module, which correspond one-to-one with the infrared image and the visible light image. The infrared image and the visible light image are respectively input into the fine-grained feature extraction module of the feature matching algorithm to obtain the first fine-grained feature and the second fine-grained feature that correspond one-to-one with the infrared image and the visible light image; The first coarse-grained feature and the second coarse-grained feature are input into the encoding and decoding module of the feature matching algorithm to obtain a dense deformation field and a matching score output by the encoding and decoding module. The dense deformation field is used to represent the one-to-one mapping relationship between the pixels of the visible light image and the pixels of the infrared image, and the matching score is used to represent the confidence level of the matching between the corresponding pixels of the visible light image and the infrared image. Iteratively execute the feature refinement operation until the preset refinement condition is met: input the first fine-grained feature, the second fine-grained feature, the dense deformation field, and the matching score into the feature refinement module of the feature matching algorithm to obtain a new first fine-grained feature, a new second fine-grained feature, a new dense deformation field, and a new matching score output by the feature refinement module. Based on the latest dense deformation field and the latest matchability score at the end of the feature refinement operation iteration, a perspective transformation algorithm is used to align the image coordinates of the matching pixels in the visible light image and the infrared image.

5. The method according to claim 4, characterized in that, The step of aligning the image coordinates of matching pixels in the visible light image and the infrared image using a perspective transformation algorithm, based on the latest dense deformation field and the latest matchability score at the end of the feature refinement operation iteration, includes: Based on the latest dense deformation field, multiple matching pixel pairs are randomly selected. Each matching pixel pair includes a pixel in the visible light image and a pixel in the infrared image that has a mapping relationship with that pixel. Filter out matching pixel pairs whose matching scores do not meet the preset requirements from the plurality of matching pixel pairs; The perspective transformation matrix is ​​calculated using the filtered matching pixel pairs; Based on the perspective transformation matrix, the image coordinates of matching pixels in the visible light image and the infrared image are aligned.

6. The method according to any one of claims 1-5, characterized in that, The step of fusing the registered infrared image and the visible light image using an image fusion algorithm to obtain a fused image includes: The registered visible light image is input into the decomposition module of the image fusion algorithm to obtain the luminance component and chrominance component of the visible light image output by the decomposition module; The luminance component and the infrared image are input into the channel stitching module of the image fusion algorithm to obtain a stitched image output by the channel stitching module; The stitched image is input into the generator of the image fusion algorithm to obtain an intermediate image output by the generator; The intermediate image and the chroma components are input into the fusion module of the image fusion algorithm, so that the fusion module outputs the fused image.

7. The method according to claim 6, characterized in that, The training process of the generator includes: Acquire infrared and visible light sample images of the sample dike area; The brightness components of the infrared sample image and the visible light sample image are stitched together in the channel dimension to obtain a stitched sample image with channel stitching. The stitched sample image is input into the generator so that the generator outputs a predicted fused image; The visible light sample image and the predicted fused image are input into the discriminator, and the discriminator outputs a discrimination result. The discrimination result includes a first discrimination result corresponding to the visible light sample image and a second discrimination result corresponding to the predicted fused image. The discrimination result represents the confidence level that the corresponding image belongs to the real visible light image. Based on the discrimination results, calculate the first loss value of the generator and the second loss value of the discriminator respectively; The generator is tuned based on the first loss value, the discriminator is tuned based on the second loss value, and the operation of inputting the stitched sample image into the generator is returned until the first loss value of the generator and the second loss value of the discriminator meet the preset loss value requirements.

8. An electronic device, characterized in that, The device includes a processor and a memory, characterized in that the memory stores computer program instructions, which, when executed by the processor, are used to perform the method for detecting abnormal target objects on dikes based on infrared and visible light image fusion as described in any one of claims 1-7.

9. A storage medium storing a computer program / instructions, characterized in that, The computer program / instructions, when running, are used to execute the method for detecting abnormal target objects on dikes based on infrared and visible light image fusion as described in any one of claims 1-7.

10. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the method for detecting abnormal target objects on dikes based on infrared and visible light image fusion as described in any one of claims 1-7.

Images