An infrared and visible dual-light leakage target detection method, system and medium

Abstract

The application discloses an infrared and visible dual-light leakage target detection method, system and medium, and belongs to the technical field of computers. The method comprises the following steps: based on leakage characteristics, performing image registration processing on an initial infrared image and an initial visible light image of a dam, and performing image enhancement processing to obtain a second infrared image and a second visible light image; performing image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fusion image; then performing image labeling processing to obtain first training data; optimizing the fusion weight of an initial dam leakage detection model and the strategy of image enhancement processing to obtain a first dam leakage detection model; and performing dam leakage detection by using the first dam leakage detection model to obtain a leakage detection result. The application is conducive to coping with relatively complex or poor light conditions, reducing the influence of noise on the detection result, and making the detection result more accurate and efficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an infrared and visible dual-light leakage target detection method, system and medium. BACKGROUND

[0002] Image target recognition is an effective method for rapid detection of dam leakage. In the related art of dam leakage detection by single-source visual image, visible light images have high resolution and contain rich details. In an ideal state, visible light images can be used to intuitively determine whether leakage occurs. However, when the light condition is too poor or there is vegetation obstruction, it is impossible to observe the leakage condition based on visible light. Compared with visible light images, infrared images can easily represent the dam shallow leakage area that cannot be observed by visible light and are suitable for all-weather use. However, due to low resolution and insufficient detail information, infrared images are difficult to provide real spatial features, and the method of recognizing leakage based on infrared images may lead to misjudgment when the spatial environment is complex.

[0003] In addition, when using traditional image segmentation methods (such as threshold segmentation, region growing and edge detection, etc.) to segment the leakage image, the image segmentation parameters are difficult to have multi-scene adaptability, which may result in a segmentation result containing many noise points or even incorrect segmentation. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide an infrared and visible dual-light leakage target detection method, system and medium, which aims to realize efficient and accurate dam leakage detection.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides an infrared and visible dual-light leakage target detection method, which comprises:

[0006] Based on the leakage characteristics, the initial infrared image and the initial visible light image of the dam are subjected to image registration processing to obtain a first infrared image and a first visible light image;

[0007] The first infrared image and the first visible light image are subjected to image enhancement processing to obtain a second infrared image and a second visible light image;

[0008] The second infrared image and the second visible light image are subjected to image fusion processing according to a fusion weight to obtain a first fusion image;

[0009] The first fusion image is subjected to image labeling processing to obtain first training data;

[0010] According to the first training data, model training is performed on an initial dam leakage detection model, the fusion weight and the strategy of the image enhancement processing are optimized, and a first dam leakage detection model is obtained.

[0011] The infrared image to be detected and the corresponding visible light image to be detected are input into the first dam leakage detection model for detection, and a leakage detection result is obtained.

[0012] In some embodiments, based on the leakage feature, the initial infrared image and the initial visible light image of the dam are subjected to image registration processing to obtain a first infrared image and a first visible light image, including the following steps:

[0013] According to the leakage area and the contour of the initial infrared image and the initial visible light image, the apparent degree of the leakage feature is determined;

[0014] When the apparent degree is greater than a preset first condition, a double-light image registration algorithm based on scale-invariant feature transformation is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image;

[0015] When one of the apparent degrees is less than the first condition, a center registration algorithm is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image.

[0016] In some embodiments, the double-light image registration algorithm based on scale-invariant feature transformation is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image, including the following steps:

[0017] Performing scale-invariant feature transformation processing on the initial infrared image to obtain a visible light image target feature point set;

[0018] Performing scale-invariant feature transformation processing on the initial visible light image to obtain an infrared image target feature point set;

[0019] Performing feature point matching on the visible light image target feature point set and the infrared image target feature point set to obtain matching points;

[0020] Performing matching point correction on the matching points to obtain a first infrared image and a first visible light image.

[0021] In some embodiments, the step of scale-invariant feature transformation includes the following steps:

[0022] Performing scale space extreme value detection on the input image to obtain scale and rotation invariant interest points;

[0023] determining a key point according to a stability degree of the interest point;

[0024] assigning one or more directions to each of the key points based on gradient directions of local parts of the input image;

[0025] measuring gradient results of local parts of an image in a surrounding field of each of the key points to obtain a target feature point according to the directions.

[0026] In some embodiments, the image registration processing of the initial infrared image and the initial visible light image by using the center registration algorithm to obtain the first infrared image and the first visible light image comprises the following steps:

[0027] acquiring shooting distances and shooting angles of the initial infrared image and the visible light image;

[0028] determining center positions of the same objects according to the initial infrared image and the visible light image;

[0029] determining cropping parameters, scaling parameters and translation parameters according to the center positions;

[0030] when the shooting distances of the initial infrared image and the visible light image are consistent, performing image registration according to the cropping parameters, the scaling parameters and the translation parameters to obtain the first infrared image and the first visible light image.

[0031] In some embodiments, the model training of the initial dam leakage detection model according to the first training data, the optimization of the fusion weight and the strategy of the image enhancement processing, and the obtaining of the first dam leakage detection model comprise the following steps:

[0032] performing dam leakage detection on the first training data by the initial dam leakage detection model to obtain a first detection result;

[0033] calculating average precision and F1 values of the first detection result based on annotations of the first training data;

[0034] adjusting parameters of the fusion weight and the strategy of the image enhancement processing, and updating the initial dam leakage detection model;

[0035] repeating the step of performing dam leakage detection on the first training data by the initial dam leakage detection model to obtain a first detection result until the average precision and the F1 values of the fusion weight and the strategy of the image enhancement processing meet a preset second condition, and obtaining a first dam leakage detection model.

[0036] In some embodiments, the initial dam leakage detection model is constructed using a YOLOv8 network framework.

[0037] To achieve the above object, another aspect of the embodiments of the present application proposes an infrared and visible dual-light leakage target detection system, which comprises:

[0038] A first module is configured to perform image registration processing on initial infrared images and initial visible light images of a dam based on leakage features, to obtain first infrared images and first visible light images;

[0039] A second module is configured to perform image enhancement processing on the first infrared images and the first visible light images, to obtain second infrared images and second visible light images;

[0040] A third module is configured to perform image fusion processing on the second infrared images and the second visible light images according to a fusion weight, to obtain a first fusion image;

[0041] A fourth module is configured to perform image annotation processing on the first fusion image, to obtain first training data;

[0042] A fifth module is configured to perform model training on an initial dam leakage detection model according to the first training data, to optimize the fusion weight and a strategy of the image enhancement processing, to obtain a first dam leakage detection model;

[0043] A sixth module is configured to input infrared images of a dam to be detected and corresponding visible light images to be detected into the first dam leakage detection model for detection, to obtain a leakage detection result.

[0044] To achieve the above object, another aspect of the embodiments of the present application proposes an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0045] To achieve the above object, another aspect of the embodiments of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0046] The embodiments of the present application at least have the following beneficial effects: the present application provides an infrared and visible dual-light leakage target detection method, system and medium, which fuses the visible light image and the infrared image by setting the fusion parameter and inputs the dam leakage detection network model, can extract the fusion image containing spatial environment information and capable of representing the shallow leakage area, combines the advantages of the visible light image and the infrared image, and can cope with more complex or poor light conditions. Moreover, through the training and optimization of the model on the fusion parameter and the image enhancement strategy, it is beneficial to reduce the influence of noise on the detection result, and further makes the model detection more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0048] Figure 1 is a step diagram of the infrared and visible dual-light leakage target detection method provided by the embodiments of the present application;

[0049] Figure 2 is a flowchart of the infrared and visible dual-light leakage target detection method provided by the embodiments of the present application;

[0050] Figure 3 is a dual-light image registration flowchart based on the SIFT algorithm provided by the embodiments of the present application;

[0051] Figure 4 is a network structure diagram of the YOLOv8 model provided by the embodiments of the present application;

[0052] Figure 5 is a registered visible light image of lawn A provided by the embodiments of the present application;

[0053] Figure 6 is a registered infrared image of lawn A provided by the embodiments of the present application;

[0054] Figure 7 is a registered visible light image of lawn B provided by the embodiments of the present application;

[0055] Figure 8 is a registered infrared image of lawn B provided by the embodiments of the present application;

[0056] Figure 9 is a fusion image of lawn A provided by the embodiments of the present application;

[0057] Figure 10 is a fusion image of lawn B provided by the embodiments of the present application;

[0058] Figure 11is a verification set verification result graph provided by an embodiment of the present application;

[0059] Figure 12 is a module schematic diagram of an infrared and visible dual-light leakage target detection system provided by an embodiment of the present application;

[0060] Figure 13 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the following further describes the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0062] Although the functional modules are divided in the system schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the sequence in the flowchart. The terms "first / S100", "second / S200" and the like in the specification and claims and the above drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence.

[0063] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0064] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It will be explicitly understood that the embodiments described herein can be combined with each other in their entirety or with each other in part, without new combinations being explicitly listed.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

[0067] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0068] (1) Image registration: the process of matching and superimposing two or more images acquired at different times, different sensors (imaging devices), or different conditions (weather, illumination, camera position and angle, etc.), a method of optimally mapping one or more pictures (locally) to the target picture based on certain evaluation criteria.

[0069] (2) Image enhancement: processing of edges, contours, contrast, etc. to improve the visual effect of the image, improve the clarity of the image, or highlight useful information in the image and compress useless information, and convert the image into a form more suitable for computer analysis and processing.

[0070] In the related art, since the visible light image has high resolution and contains rich details, in an ideal state, it can be directly judged whether leakage occurs through the visible light image. However, when the light condition is too poor or there is vegetation obstruction, it is impossible to observe the leakage condition through the visible light. Compared with the visible light image, the infrared image can easily represent the dam shallow leakage area that cannot be observed by visible light and is suitable for all-weather use. However, due to low resolution and insufficient detail information, it is difficult for the infrared image to provide real spatial features, and the method of identifying leakage through the infrared image may lead to misjudgment when the spatial environment is complex.

[0071] In addition, when using traditional image segmentation methods (such as threshold segmentation, region growing, and edge detection, etc.) to segment the leakage image, since the image segmentation parameters are difficult to have multi-scene adaptability, the segmentation result may contain many noise points, or even false segmentation.

[0072] Therefore, the application provides an infrared and visible dual-light leakage target detection method, system and medium. The method fuses a visible light image and an infrared image by setting a fusion parameter, and inputs the fusion parameter into a dam leakage detection network model, so that a fusion image containing spatial environment information and capable of representing a shallow leakage area can be extracted, the advantages of the visible light image and the infrared image are combined, and the method can cope with a more complex or poor light condition scene. Moreover, the model is trained and optimized by a fusion parameter and an image enhancement strategy, so that the influence of noise on a detection result is reduced, and the model detection is more accurate.

[0073] The application provides an infrared and visible dual-light leakage target detection method, which relates to the technical field of computers. The infrared and visible dual-light leakage target detection method can be applied to a terminal, a server, or software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, or the like, but is not limited thereto. The server can be configured as a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the infrared and visible dual-light leakage target detection method, but is not limited to the above forms.

[0074] The application can be used in many general or specific computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0075] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0076] Figure 1 is an optional flowchart of an infrared and visible dual-light leakage target detection method provided by the embodiments of the present application, Figure 1 The method in the above step S600 can include but is not limited to steps S100-S600.

[0077] Step S100, based on a leakage feature, performing image registration processing on an initial infrared image and an initial visible light image of a dam to obtain a first infrared image and a first visible light image.

[0078] Step S200, performing image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image.

[0079] Step S300, performing image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fusion image.

[0080] Step S400, performing image annotation processing on the first fusion image to obtain first training data.

[0081] Step S500, according to the first training data, performing model training on an initial dam leakage detection model to optimize the fusion weight and the strategy of the image enhancement processing, and obtaining a first dam leakage detection model.

[0082] Step S600, inputting an infrared image to be detected and a corresponding visible light image to be detected into the first dam leakage detection model for detection to obtain a leakage detection result.

[0083] The steps S100 to S600 shown in the embodiments of the present application fuse the visible light image and the infrared image by setting the fusion parameters and input the dam leakage detection network model, can extract the fusion image containing the spatial environment information and capable of representing the shallow leakage area, combines the advantages of the visible light image and the infrared image, and can cope with more complex or poor light conditions. Moreover, by training and optimizing the model with the fusion parameters and the image enhancement strategy, it is beneficial to reduce the influence of noise on the detection result, and further makes the model detection more accurate.

[0084] In some embodiments, step S100 can include but is not limited to steps S110-S130:

[0085] Step S110, according to the leakage area and the contour of the initial infrared image and the initial visible light image respectively, determine the obvious degree of leakage feature;

[0086] Step S120, when the obvious degree is greater than the first condition, the double light image registration algorithm based on scale invariant feature transformation is used to perform image registration processing on the initial infrared image and the initial visible light image, to obtain the first infrared image and the first visible light image;

[0087] Step S130, when the obvious degree is less than the first condition, a center registration algorithm is used to perform image registration processing on the initial infrared image and the initial visible light image, to obtain the first infrared image and the first visible light image.

[0088] In some embodiments, step S120 can include but is not limited to steps S121-S124:

[0089] Step S121, performing scale invariant feature transformation processing on the initial infrared image to obtain a visible light image target feature point set;

[0090] Step S122, performing scale invariant feature transformation processing on the initial visible light image to obtain an infrared image target feature point set;

[0091] Step S123, performing feature point matching on the visible light image target feature point set and the infrared image target feature point set to obtain matching points;

[0092] Step S124, performing matching point correction on the matching points to obtain the first infrared image and the first visible light image.

[0093] In some embodiments, the step of scale invariant feature transformation of step S120 can include but is not limited to steps ①-④:

[0094] ①performing scale space extremal detection on the input image to obtain scale and rotation invariant interest points;

[0095] ②determining key points according to a stability degree of the interest points;

[0096] ③assigning one or more directions to each of the key points based on gradient directions of a local of the input image;

[0097] ④measuring gradient results of the local of the image in a surrounding field of each of the key points to obtain target feature points according to the directions.

[0098] In some embodiments, step S130 includes but is not limited to steps S131-S134:

[0099] Step S131, acquiring shooting distances and shooting angles of the initial infrared image and the visible light image;

[0100] Step S132, determining center positions of the same object according to the initial infrared image and the visible light image;

[0101] Step S133, determining cropping parameters, scaling parameters and translation parameters according to the center positions;

[0102] Step S134, when the shooting distances of the initial infrared image and the visible light image are consistent, performing image registration according to the cropping parameters, the scaling parameters and the translation parameters to obtain a first infrared image and a first visible light image.

[0103] In some embodiments, step S500 can include but is not limited to steps S510-S540:

[0104] Step S510, performing dam seepage detection on the first training data by the initial dam seepage detection model to obtain a first detection result;

[0105] Step S520, calculating average precision and F1 value of the first detection result based on a label of the first training data;

[0106] Step S530, adjusting parameters of the fusion weight and the image enhancement processing strategy, and updating the initial dam seepage detection model;

[0107] Step S540, repeatedly performing the step of performing dam seepage detection on the first training data by the initial dam seepage detection model to obtain a first detection result until the average precision and the F1 value of the fusion weight and the image enhancement processing strategy meet a preset second condition, and obtaining a first dam seepage detection model.

[0108] In some embodiments, the initial dam leakage detection model of step S500 can be constructed using a YOLO series model, for example, by using a YOLOv8 network framework.

[0109] Next, in conjunction with a specific application example of dam leakage detection, the scheme of the embodiments of the present application will be described and explained in detail:

[0110] In the embodiments of the present application, an infrared and visible dual-light leakage target detection method is provided, which can be applied to efficiently and accurately detect the leakage of a dam. Further, with reference to Figure 2 , the method of the embodiments of the present application mainly includes image registration, image enhancement, yolo model training and adjustment, and leakage target detection.

[0111] Specifically, a database of infrared images and visible light images can be established for storing relevant data. First, dam infrared images used for training and corresponding visible light images are obtained, and the corresponding infrared images and visible light images need to be registered. The selection of the image registration algorithm depends on the characteristics of the leakage image set, which can be divided into two cases: (1) when the leakage features (leakage area and leakage contour) of the visible light image and the infrared image are both obvious, the image registration algorithm combined with SIFT (Scale-invariant feature transform) is used for dual-light image registration; (2) when the leakage features of a single light image are not obvious, the center registration method is used for registration.

[0112] With reference to Figure 3 , in the embodiments of the present application, the basic flow of the SIFT algorithm is divided into four steps:

[0113] a. Scale space extreme value detection: search for image positions at all scales, and identify potential interest points that are invariant to scale and rotation through a Gaussian differential function;

[0114] b. Key point positioning: at each candidate position, a fine fitting model is used to determine the position and scale. The key points are selected according to the stability of the candidate points;

[0115] c. Direction determination: based on the gradient direction of the local image, one or more directions are assigned to each key point;

[0116] d. Key point description: measure the local gradient of the image in the selected scale within the field around each key point.

[0117] The SIFT algorithm is performed on the visible light image and the infrared image to obtain target feature point sets, and then feature point matching and matching point correction are performed according to the target feature point sets to obtain a first infrared image and a first visible light image after registration.

[0118] In the embodiment, the processing flow of the center registration algorithm is as follows: fixed double-light camera parameters (shooting distance and shooting angle) in a specific scene are acquired, and the cropping, scaling and translation parameters of image registration are determined according to the center positions of the same objects in the visible light image and the infrared image. Then, for the double-light images with consistent camera parameters, the image registration is performed by using the center registration parameters determined by the cosine.

[0119] Generally, the resolution of the visible light image is higher than that of the infrared image, and the field of view of the visible light image contains the field of view of the infrared image. Therefore, after the image registration, the size of the visible light image can be set to the size of the infrared image.

[0120] Further, in the method of the embodiment, an image enhancement operation is needed, which is mainly an image enhancement processing of the registered single-source visual image, aiming to improve the contrast of the infrared image and improve the definition of the visible light image.

[0121] The image registration adopts a linear enhancement method, and the gray image is represented as a two-dimensional function f(x, y). The gray image function s(x, y) after linear enhancement can be represented as:

[0122] s(x, y) = af(x, y) + b;

[0123] wherein a and b are adjustment factors, a is used to adjust the contrast of the image, and b is used to adjust the brightness of the image.

[0124] Further, after the image registration and the image enhancement are completed, an image fusion operation is performed based on a weighted average method. The visible light image after the image registration and the image enhancement is represented as a function I1(x, y), and the infrared image is represented as I2(x, y). The fusion image I(x, y) can be represented as:

[0125] I(x, y) = ωI1(x, y) + (1-ω)I2(x, y);

[0126] wherein ω represents the fusion weight of the visible light image.

[0127] When the image fusion operation is performed, the spatial information of the visible light image and the contrast information of the infrared image need to be preserved as much as possible.

[0128] Further, after the fusion is completed for image enhancement, the obtained data set is labeled to obtain first training data, and the initial dam leakage detection model is trained according to the first training data to optimize the fusion parameters and the first image enhancement strategy.

[0129] In some embodiments, the initial dam leakage detection model can be constructed by using a YOLOv8 network model, and the model structure of YOLOv8 is as shown in the following figure. Figure 4 In other embodiments, other YOLO series models, such as YOLOv5, can also be used for implementation. YOLOv8 is suitable for CPU parallel computing, and has good performance on data sets. By fusing the visible light image and the infrared image and inputting them into the YOLOv8 framework, a fusion image containing spatial environment information and capable of representing a shallow leakage area can be extracted, and a feature map containing a leakage position and a category can be extracted from a plurality of leakage images. The advantages of the visible light image and the infrared image are combined, which can cope with scenes with relatively complex environments or poor light conditions, and can reduce the influence of noise on the target detection result, further improving the accuracy of model detection.

[0130] The optimization process of the image enhancement strategy is as follows: keeping the variables of other modules unchanged, constantly adjusting the enhancement strategies of the single-source visual image and the fusion image in the training process, and automatically iterating the optimal linear enhancement parameters a and b of the image enhancement according to the training results of the YOLOv8 model. The optimal image enhancement result should meet the artificial evaluation requirements (i.e., experienced staff directly judge whether there is a leakage area by observing the enhanced fusion image) to prevent the image set from lacking basic reference value.

[0131] The optimization process of the image fusion parameter is as follows: keeping the variables of other modules unchanged, constantly modifying the fusion parameter ω of the weighted average method, and training the YOLOv8 model after obtaining the fusion image. The fusion parameter is evaluated according to the model training results, and the optimal image fusion parameter is determined.

[0132] When the image enhancement strategy (linear enhancement parameters a and b) and the image fusion parameter (fusion parameter ω of the weighted average method) are both optimal, the optimal target detection model is obtained. The judgment basis for reaching the best is that the sum of mAP@0.5 and F1 score of the YOLOv8 model training reaches the maximum value.

[0133] Further, the visible light image and the infrared image to be detected are input into the first dam leakage detection model for identification and detection, so that the leakage identification task is completed, and a leakage detection result is obtained.

[0134] The following takes the lawn surface leakage image set as an example to provide an explanation of the leakage detection effect of the embodiments of the present application.

[0135] As Figure 5 , Figure 6 , Figure 7 and Figure 8 shown, whether it is scene A or scene B, the position of the leakage in the double light image obtained after registration may exist in the position of the picture, accurate registration is achieved, and a basis is provided for subsequent double light image target detection. It should be noted that the box in the figure is to show the leakage position, and the actual processed image does not contain the box.

[0136] As Figure 9 and Figure 10 shown, the fusion image contains the spatial features of the visible light image and the contrast features of the infrared image, reflecting the leakage situation in the lawn.

[0137] The verification result of the method of the embodiment of the present application in the verification set is as shown in Figure 11 The method of the embodiment of the present application can accurately identify the position and size of the lawn surface leakage area.

[0138] In summary, the embodiment of the present application has at least the following beneficial effects:

[0139] 1. By fusing the visible light image and the infrared image and inputting them into the YOLOv8 framework, the fusion image containing spatial environment information and being able to represent the shallow leakage area can be extracted, the feature map containing the leakage position and category can be extracted from multiple leakage images, the advantages of the visible light image and the infrared image are combined, the scene with relatively complex environment or poor light condition can be coped with, and the influence of noise on the target detection result can be reduced, further making the model detection more accurate.

[0140] 2. The optimization scheme of the image enhancement strategy and the image fusion parameter is established, which can be compatible with the target detection of multiple leakage scenes.

[0141] 3. Combined with the YOLOv8 framework, the parallel computing of CPU can be adapted, so that the detection speed is faster and the efficiency is higher.

[0142] Please refer to Figure 12 , the embodiment of the present application also provides an infrared and visible double light leakage target detection system, which can realize the above-mentioned infrared and visible double light leakage target detection method, and the system comprises:

[0143] The first module 101 is used for performing image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage feature, to obtain a first infrared image and a first visible light image;

[0144] The second module 102 is used for performing image enhancement processing on the first infrared image and the first visible light image, to obtain a second infrared image and a second visible light image;

[0145] The third module 103 is configured to perform image fusion processing on the second infrared image and the second visible light image according to the fusion weight, to obtain a first fusion image.

[0146] The fourth module 104 is configured to perform image annotation processing on the first fusion image, to obtain first training data.

[0147] The fifth module 105 is configured to perform model training on an initial dam leakage detection model according to the first training data, to optimize the fusion weight and the strategy of the image enhancement processing, to obtain a first dam leakage detection model.

[0148] The sixth module 106 is configured to input an infrared image to be detected and a corresponding visible light image to be detected into the first dam leakage detection model for detection, to obtain a leakage detection result.

[0149] It can be understood that the content in the above method embodiments is applicable to the present system embodiments, the present system embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0150] The present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above infrared and visible dual-light leakage target detection method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0151] It can be understood that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0152] Please refer to Figure 13 , Figure 13 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0153] The processor 201 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the present application.

[0154] The memory 202 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), etc. The memory 202 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 202 and are called and executed by the processor 201 to implement the infrared and visible dual-light leakage target detection method according to an embodiment of the present application;

[0155] The input / output interface 203 is configured to realize information input and output.

[0156] The communication interface 204 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0157] The bus 205 is configured to transmit information between various components (for example, the processor 201, the memory 202, the input / output interface 203 and the communication interface 204) of the device.

[0158] The processor 201, the memory 202, the input / output interface 203 and the communication interface 204 are connected to each other through the bus 205 to realize the communication connection between the device.

[0159] The present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned infrared and visible dual-light leakage target detection method is realized.

[0160] It can be understood that the above-mentioned method embodiments are applicable to the present storage medium embodiments. The present storage medium embodiments specifically realize the same functions as the above-mentioned method embodiments and achieve the same beneficial effects as the above-mentioned method embodiments.

[0161] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state storage device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0162] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0163] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0164] The system embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0165] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system, and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0166] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0168] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0170] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0171] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0172] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for detecting leakage targets using both infrared and visible light, characterized in that, Includes the following steps: Based on the seepage characteristics, image registration processing is performed on the initial infrared image and the initial visible light image of the dam to obtain the first infrared image and the first visible light image; Image enhancement processing is performed on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image; The second infrared image and the second visible light image are fused according to the fusion weights to obtain the first fused image; The first fused image is subjected to image annotation processing to obtain the first training data; Based on the first training data, the initial dam seepage detection model is trained, and the fusion weights and the image enhancement processing strategy are optimized to obtain the first dam seepage detection model. The infrared image of the dam to be detected and the corresponding visible light image to be detected are input into the first dam seepage detection model for detection, and the seepage detection results are obtained. The process of image registration processing based on seepage characteristics of the initial infrared and visible light images of the dam to obtain a first infrared image and a first visible light image includes the following steps: The degree of prominence of the leakage features is determined based on the leakage area and contour of the initial infrared and initial visible light images, respectively. When the obviousness of the two images is greater than the preset first condition, the initial infrared image and the initial visible light image are registered using a dual-light image registration algorithm based on scale-invariant feature transformation to obtain the first infrared image and the first visible light image. When one of the obviousnesses is less than the first condition, the initial infrared image and the initial visible light image are image registered using a center registration algorithm to obtain a first infrared image and a first visible light image.

2. The method according to claim 1, characterized in that, The method of performing image registration processing on the initial infrared image and the initial visible light image using a scale-invariant feature transform-based dual-light image registration algorithm to obtain a first infrared image and a first visible light image includes the following steps: The initial infrared image is subjected to scale-invariant feature transformation to obtain a set of target feature points in the visible light image; The initial visible light image is subjected to scale-invariant feature transformation to obtain a set of target feature points in the infrared image; Feature point matching is performed on the visible light image target feature point set and the infrared image target feature point set to obtain matching points; Matching point correction is performed on the matching points to obtain a first infrared image and a first visible light image.

3. The method according to claim 1, characterized in that, The steps of scale-invariant feature transformation include the following: Scale-space extremum detection is performed on the input image to obtain scale- and rotation-invariant interest points; Key points are determined based on the stability of the points of interest. Based on the local gradient direction of the input image, assign one or more directions to each key point; Based on the stated direction, the gradient results of the local image are measured in the surrounding area of ​​each key point to obtain the target feature points.

4. The method according to claim 1, characterized in that, The step of performing image registration processing on the initial infrared image and the initial visible light image using a center registration algorithm to obtain a first infrared image and a first visible light image includes the following steps: Acquire the shooting distance and shooting angle of the initial infrared image and the visible light image; The center position of the same object is determined based on the initial infrared image and the visible light image; The clipping parameters, scaling parameters, and translation parameters are determined based on the center position. When the shooting distances of the initial infrared image and the visible light image are the same, image registration is performed according to the cropping parameters, the scaling parameters, and the translation parameters to obtain the first infrared image and the first visible light image.

5. The method according to claim 1, characterized in that, The step of training the initial dam seepage detection model based on the first training data, optimizing the fusion weights and the image enhancement processing strategy to obtain the first dam seepage detection model includes the following steps: The first training data is used to detect dam seepage using the initial dam seepage detection model to obtain the first detection result; Based on the annotation of the first training data, calculate the average precision and F1 score of the first detection result; The parameters of the fusion weights and the image enhancement processing strategy are adjusted to update the initial dam seepage detection model; Repeat the steps of performing dam seepage detection on the first training data using the initial dam seepage detection model to obtain the first detection result until the average accuracy and F1 value of the fusion weights and the image enhancement processing strategy meet the preset second condition, thereby obtaining the first dam seepage detection model.

6. The method according to claim 1, characterized in that, The initial dam seepage detection model was constructed using the YOLOv8 network framework.

7. A dual-light infrared and visible light leakage target detection system, characterized in that, include: The first module is used to perform image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics, so as to obtain the first infrared image and the first visible light image; The second module is used to perform image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image. The third module is used to perform image fusion processing on the second infrared image and the second visible light image according to the fusion weight to obtain the first fused image; The fourth module is used to perform image annotation processing on the first fused image to obtain the first training data; The fifth module is used to train the initial dam seepage detection model based on the first training data, optimize the fusion weights and the image enhancement processing strategy, and obtain the first dam seepage detection model. The sixth module is used to input the infrared image of the dam to be detected and the corresponding visible light image to be detected into the first dam seepage detection model for detection, and to obtain the seepage detection result; The first module is specifically used for: The degree of prominence of the leakage features is determined based on the leakage area and contour of the initial infrared and initial visible light images, respectively. When the obviousness of the two images is greater than the preset first condition, the initial infrared image and the initial visible light image are registered using a dual-light image registration algorithm based on scale-invariant feature transformation to obtain the first infrared image and the first visible light image. When one of the obviousnesses is less than the first condition, the initial infrared image and the initial visible light image are image registered using a center registration algorithm to obtain a first infrared image and a first visible light image.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

9. A computer storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1 to 6.

Images