Infrared and visible double-light leakage target detection method, system and medium

Abstract

The invention discloses an infrared and visible light leakage target detection method and system and a medium, and belongs to the technical field of computers, and the method comprises the steps: carrying out the image registration processing and image enhancement processing of an initial infrared image and an initial visible light image of a dam based on leakage characteristics, and obtaining a second infrared image and a second visible light image; performing image fusion processing on the second infrared image and the second visible light image according to the fusion weight to obtain a first fusion image; carrying out image annotation processing to obtain first training data; optimizing a fusion weight of the initial dam leakage detection model and an image enhancement processing strategy to obtain a first dam leakage detection model; and performing dam leakage detection by using the first dam leakage detection model to obtain a leakage detection result. The method is beneficial to coping with a relatively complex scene or a scene with a relatively poor light condition, the influence of noise on a detection result is reduced, and the detection result is more accurate and efficient.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, system and medium for detecting infrared and visible dual light leakage targets. Background Art

[0002] Image target recognition is an effective method for rapid detection of dam leakage. At present, in the field of dam leakage detection related technologies using single-source visual images, due to the high resolution and rich details of visible light images, under ideal conditions, it is possible to visually determine whether leakage has occurred through visible light images. However, when the lighting conditions are too poor or there are vegetation obstructions, it is impossible to observe leakage through visible light. Compared with visible light images, infrared images can easily characterize shallow leakage areas of dams that cannot be observed by visible light, and are applicable all day long. However, due to low resolution and insufficient detail information, infrared images are prone to difficulty in providing real spatial features. The method of identifying leakage through infrared images may lead to misjudgment when the spatial environment is complex.

[0003] In addition, when using traditional image segmentation methods (such as threshold segmentation, region growing and edge detection, etc.) to segment leakage images, the segmentation results may contain more noise or even incorrect segmentation because the image segmentation parameters are difficult to adapt to multiple scenes. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an infrared and visible dual-light leakage target detection method, system and medium, aiming to achieve high-efficiency and high-accuracy dam leakage detection.

[0005] To achieve the above objectives, an embodiment of the present application provides a method for detecting infrared and visible dual light leakage targets, the method comprising:

[0006] Based on the leakage characteristics, an initial infrared image and an initial visible light image of the dam are subjected to image registration processing to obtain a first infrared image and a first visible light image;

[0007] Performing image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image;

[0008] Performing image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image;

[0009] Performing image annotation processing on the first fused image to obtain first training data;

[0010] According to the first training data, the initial dam leakage detection model is trained, the fusion weight and the image enhancement processing strategy are optimized, and a first dam leakage detection model is obtained;

[0011] The infrared image of the dam to be detected and the corresponding visible light image to be detected are input into the first dam leakage detection model for detection to obtain a leakage detection result.

[0012] In some embodiments, the image registration processing is performed on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics to obtain the first infrared image and the first visible light image, including the following steps:

[0013] Determine the obviousness of the leakage feature according to the leakage area and contour of the initial infrared image and the initial visible light image;

[0014] When the obviousness is greater than a preset first condition, a bi-optical image registration algorithm based on scale-invariant feature transformation is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image;

[0015] When one of the conspicuity levels is less than the first condition, a center registration algorithm is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image.

[0016] In some embodiments, the using of a bi-optical image registration algorithm based on scale-invariant feature transformation to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image comprises the following steps:

[0017] Performing scale-invariant feature transformation processing on the initial infrared image to obtain a visible light image target feature point set;

[0018] Performing scale-invariant feature transformation processing on the initial visible light image to obtain an infrared image target feature point set;

[0019] Performing feature point matching on the visible light image target feature point set and the infrared image target feature point set to obtain matching points;

[0020] Matching point correction is performed on the matching points to obtain a first infrared image and a first visible light image.

[0021] In some embodiments, the step of scale-invariant feature transformation includes the following steps:

[0022] Perform scale space extremum detection on the input image to obtain scale and rotation invariant interest points;

[0023] Determining key points according to the stability of the points of interest;

[0024] Based on the local gradient direction of the input image, assign one or more directions to each key point;

[0025] According to the direction, in the surrounding area of ​​each key point, the local gradient result of the image is measured to obtain the target feature point.

[0026] In some embodiments, the using of a central registration algorithm to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image comprises the following steps:

[0027] Acquiring shooting distances and shooting angles of the initial infrared image and the visible light image;

[0028] Determine the center position of the same object according to the initial infrared image and the visible light image;

[0029] Determining cropping parameters, scaling parameters, and translation parameters according to the center position;

[0030] When the shooting distance and the shooting distance of the initial infrared image and the visible light image are consistent, image registration is performed according to the cropping parameter, the scaling parameter, and the translation parameter to obtain a first infrared image and a first visible light image.

[0031] In some embodiments, the initial dam leakage detection model is trained according to the first training data, the fusion weight and the image enhancement processing strategy are optimized, and the first dam leakage detection model is obtained, including the following steps:

[0032] Performing dam leakage detection on the first training data by using the initial dam leakage detection model to obtain a first detection result;

[0033] Calculate the average precision and F1 value of the first detection result based on the annotation of the first training data;

[0034] Adjusting the parameters of the fusion weight and the image enhancement processing strategy to update the initial dam leakage detection model;

[0035] Repeat the step of performing dam leakage detection on the first training data through the initial dam leakage detection model to obtain a first detection result, until the average accuracy and the F1 value of the fusion weight and the image enhancement processing strategy meet the preset second condition, and obtain the first dam leakage detection model.

[0036] In some embodiments, the initial dam leakage detection model is constructed using the YOLOv8 network framework.

[0037] To achieve the above purpose, another aspect of the embodiment of the present application provides an infrared and visible dual light leakage target detection system, the system comprising:

[0038] The first module is used to perform image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics to obtain a first infrared image and a first visible light image;

[0039] A second module is used to perform image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image;

[0040] A third module is used to perform image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image;

[0041] A fourth module is used to perform image annotation processing on the first fused image to obtain first training data;

[0042] A fifth module is used to perform model training on the initial dam leakage detection model according to the first training data, optimize the fusion weight and the image enhancement processing strategy, and obtain a first dam leakage detection model;

[0043] The sixth module is used to input the infrared image of the dam to be detected and the corresponding visible light image to be detected into the first dam leakage detection model for detection to obtain the leakage detection result.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0045] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0046] The embodiments of the present application include at least the following beneficial effects: The present application provides an infrared and visible dual light leakage target detection method, system and medium. The scheme fuses visible light images and infrared images by setting fusion parameters and inputs them into the dam leakage detection network model. It can extract fusion images containing spatial environmental information and capable of characterizing shallow leakage areas. It combines the advantages of visible light images and infrared images and can cope with more complex scenes or scenes with poor lighting conditions. In addition, by training and optimizing the fusion parameters and image enhancement strategies of the model, it is helpful to reduce the impact of noise on the detection results, and further make the model detection more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0048] Figure 1 It is a step diagram of the infrared and visible dual light leakage target detection method provided in an embodiment of the present application;

[0049] Figure 2 is a flow chart of the infrared and visible dual light leakage target detection method provided in an embodiment of the present application;

[0050] Figure 3 is a bi-optical image registration flow chart based on the SIFT algorithm provided in an embodiment of the present application;

[0051] Figure 4 It is a network structure diagram of the YOLOv8 model provided in an embodiment of the present application;

[0052] Figure 5 is a registered visible light image of lawn A provided in an embodiment of the present application;

[0053] Figure 6 is a registered infrared image of lawn A provided in an embodiment of the present application;

[0054] Figure 7 is a registered visible light image of lawn B provided in an embodiment of the present application;

[0055] Figure 8 is a registered infrared image of lawn B provided in an embodiment of the present application;

[0056] Fig. 9 is a fused image of lawn A provided in an embodiment of the present application;

[0057] Fig.10 is a fused image of lawn B provided in an embodiment of the present application;

[0058] Fig.11It is a verification result diagram of the verification set provided in the embodiment of the present application;

[0059] Fig.12 It is a module schematic diagram of an infrared and visible dual light leakage target detection system provided in an embodiment of the present application;

[0060] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0062] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0068] (1) Image registration: The process of matching and superimposing two or more images acquired at different times, using different sensors (imaging devices) or under different conditions (such as weather, illumination, camera position and angle). It is a method that can optimally map one or more images (parts) to a target image based on certain evaluation criteria.

[0069] (2) Image enhancement: Processing of edges, contours, contrast, etc. to improve the visual effect of the image, increase the clarity of the image, or highlight the useful information in the image, compress useless information, and convert the image into a form more suitable for computer analysis and processing.

[0070] In the related art, since visible light images have high resolution and contain rich details, under ideal conditions, it is possible to visually determine whether leakage has occurred through visible light images. However, when the lighting conditions are too poor or there are vegetation obstructions, it is impossible to observe leakage through visible light. Compared with visible light images, infrared images can easily characterize shallow leakage areas of dams that cannot be observed by visible light, and are applicable all day long. However, due to low resolution and insufficient detail information, infrared images are difficult to provide real spatial features. The method of identifying leakage through infrared images may lead to misjudgment when the spatial environment is complex.

[0071] In addition, when using traditional image segmentation methods (such as threshold segmentation, region growing and edge detection, etc.) to segment leakage images, the segmentation results may contain more noise or even incorrect segmentation because the image segmentation parameters are difficult to adapt to multiple scenes.

[0072] In view of this, an infrared and visible dual light leakage target detection method, system and medium are provided in the embodiment of the present application. The scheme fuses the visible light image and the infrared image by setting the fusion parameters and inputs them into the dam leakage detection network model. It can extract the fusion image containing the spatial environment information and the shallow leakage area. It combines the advantages of visible light images and infrared images and can cope with more complex scenes or scenes with poor lighting conditions. In addition, by training and optimizing the fusion parameters and image enhancement strategies of the model, it is helpful to reduce the impact of noise on the detection results, and further make the model detection more accurate. .

[0073] An infrared and visible dual light leakage target detection method provided in an embodiment of the present application relates to the field of computer technology. An infrared and visible dual light leakage target detection method provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an infrared and visible dual light leakage target detection method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Figure 1 is an optional flow chart of an infrared and visible dual light leakage target detection method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S600.

[0077] Step S100 : performing image registration processing on an initial infrared image and an initial visible light image of the dam based on leakage characteristics to obtain a first infrared image and a first visible light image.

[0078] Step S200: performing image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image.

[0079] Step S300: performing image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image.

[0080] Step S400: performing image annotation processing on the first fused image to obtain first training data.

[0081] Step S500: training the initial dam leakage detection model according to the first training data, optimizing the fusion weights and the image enhancement processing strategy, and obtaining a first dam leakage detection model.

[0082] Step S600: input the infrared image of the dam to be detected and the corresponding visible light image to be detected into the first dam leakage detection model for detection to obtain a leakage detection result.

[0083] In the steps S100 to S600 shown in the embodiment of the present application, by setting fusion parameters to fuse visible light images and infrared images, and inputting them into the dam leakage detection network model, it is possible to extract fusion images containing spatial environmental information and capable of characterizing shallow leakage areas, combining the advantages of visible light images and infrared images, and being able to cope with more complex or poor lighting conditions. In addition, by training and optimizing the fusion parameters and image enhancement strategies of the model, it is helpful to reduce the impact of noise on the detection results, and further make the model detection more accurate.

[0084] In some embodiments, step S100 may include but is not limited to steps S110 to S130:

[0085] Step S110, determining the degree of obviousness of the leakage feature according to the leakage area and contour of each of the initial infrared image and the initial visible light image;

[0086] Step S120, when the degree of conspicuity is greater than a preset first condition, performing image registration processing on the initial infrared image and the initial visible light image using a bi-optical image registration algorithm based on scale-invariant feature transformation to obtain a first infrared image and a first visible light image;

[0087] Step S130: When one of the conspicuity levels is less than the first condition, a center registration algorithm is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image.

[0088] In some embodiments, step S120 may include but is not limited to steps S121 to S124:

[0089] Step S121, performing scale-invariant feature transformation processing on the initial infrared image to obtain a visible light image target feature point set;

[0090] Step S122, performing scale-invariant feature transformation processing on the initial visible light image to obtain an infrared image target feature point set;

[0091] Step S123, performing feature point matching on the visible light image target feature point set and the infrared image target feature point set to obtain matching points;

[0092] Step S124, performing matching point correction on the matching points to obtain a first infrared image and a first visible light image.

[0093] In some embodiments, the scale-invariant feature transformation step of step S120 may include but is not limited to the following steps ① to ④:

[0094] ① Perform scale space extremum detection on the input image to obtain scale and rotation invariant interest points;

[0095] ② Determine the key points according to the stability of the points of interest;

[0096] ③ Based on the local gradient direction of the input image, assign one or more directions to each key point;

[0097] ④ According to the direction, in the surrounding area of ​​each key point, measure the local gradient result of the image to obtain the target feature point.

[0098] In some embodiments, step S130 includes but is not limited to the following steps S131 to S134:

[0099] Step S131, acquiring the shooting distance and shooting angle of the initial infrared image and the visible light image;

[0100] Step S132, determining the center position of the same object according to the initial infrared image and the visible light image;

[0101] Step S133, determining a cropping parameter, a scaling parameter, and a translation parameter according to the center position;

[0102] Step S134 , when the shooting distance and the shooting distance of the initial infrared image and the visible light image are consistent, image registration is performed according to the cropping parameter, the scaling parameter and the translation parameter to obtain a first infrared image and a first visible light image.

[0103] In some embodiments, step S500 may include but is not limited to the following steps S510 to S540:

[0104] Step S510, performing dam leakage detection on the first training data by using the initial dam leakage detection model to obtain a first detection result;

[0105] Step S520, calculating the average precision and F1 value of the first detection result based on the annotation of the first training data;

[0106] Step S530, adjusting the parameters of the fusion weight and the image enhancement processing strategy to update the initial dam leakage detection model;

[0107] Step S540, repeatedly executing the step of performing dam leakage detection on the first training data through the initial dam leakage detection model to obtain a first detection result, until the average accuracy and the F1 value of the fusion weight and the image enhancement processing strategy meet the preset second condition, and the first dam leakage detection model is obtained.

[0108] In some embodiments, the initial dam leakage detection model of step S500 can be constructed using a YOLO series model, for example, using a YOLOv8 network framework.

[0109] Below, in conjunction with a specific application example of a dam leakage detection scenario, the solution of the embodiment of the present application is described in detail and explained:

[0110] In the embodiment of the present application, a method for detecting leakage targets using infrared and visible dual light is provided, which can be applied to detect leakage of dams with high efficiency and accuracy. Figure 2 The method of the embodiment of the present application mainly includes image registration, image enhancement, YOLO model training adjustment and leakage target detection.

[0111] Specifically, a database of infrared images and visible light images can be established to store relevant data. First, the infrared image of the dam used for training and the corresponding visible light image are obtained, and the corresponding infrared image and the visible light image need to be registered. The selection of the image registration algorithm depends on the characteristics of the leakage image set, which can be divided into two cases: (1) When the leakage features (leakage area and leakage contour) of the visible light image and the infrared image are relatively obvious, the SIFT (Scale-invariant feature transform) image registration algorithm is combined for dual-light image registration; (2) When the leakage features of a single light image are not obvious, the central registration method is combined for registration.

[0112] Reference Figure 3 In the embodiment of the present application, the basic process of the SIFT algorithm is divided into four steps:

[0113] a. Scale space extremum detection: Search image locations at all scales and identify potential scale- and rotation-invariant points of interest through Gaussian derivative functions;

[0114] b. Key point positioning: At each candidate position, a fine-tuned model is used to determine the position and scale. Key points are selected based on the stability of the candidate points;

[0115] c. Direction determination: Based on the local gradient direction of the image, one or more directions are assigned to each key point;

[0116] d. Key point description: In the area around each key point, the local gradient of the image is measured at a selected scale.

[0117] By performing the above-mentioned SIFT algorithm processing on both the visible light image and the infrared image, respective target feature point sets are obtained, and then feature point matching and matching point correction processing are performed based on the target feature point sets to obtain the registered first infrared image and first visible light image.

[0118] In the embodiment of the present application, the processing flow of the center registration algorithm is as follows: obtain fixed bifocal camera parameters (shooting distance and shooting angle) in a specific scene, determine the cropping, scaling and translation parameters of image registration according to the center position of the same object in the visible light image and the infrared image. Then, for the bifocal images with the same camera parameters, the center registration parameters determined by cosine are used for image registration.

[0119] Usually, the resolution of a visible light image is higher than that of an infrared image, and the field of view of a visible light image includes the field of view of an infrared image. Therefore, after image registration, the size of the visible light image can be set to the size of the infrared image.

[0120] Furthermore, in the method of this embodiment, an image enhancement operation needs to be performed, which is mainly to perform image enhancement processing on the registered single-source visual image, with the purpose of increasing the contrast of the infrared image and improving the clarity of the visible light image.

[0121] Image registration uses a linear enhancement method to represent the grayscale image as a two-dimensional function f(x, y). The grayscale image function s(x, y) after linear enhancement can be expressed as:

[0122] s(x,y)=af(x,y)+b;

[0123] Among them, a and b are adjustment factors, a is used to adjust the contrast of the image, and b is used to adjust the brightness of the image.

[0124] Furthermore, after completing image registration and image enhancement, image fusion operation is performed based on the weighted average method. The visible light image after image registration and image enhancement is represented as function I1(x, y), and the infrared image is represented as I2(x, y). Then the fused image I(x, y) can be expressed as:

[0125] I(x,y)=ωI1(x,y)+(1-ω)I2(x,y);

[0126] Among them, ω represents the fusion weight of the visible light image.

[0127] When performing image fusion operations, it is necessary to preserve the spatial information of the visible light image and the contrast information of the infrared image as much as possible.

[0128] Furthermore, after completing the fusion and image enhancement, the obtained data set is annotated to obtain the first training data, and the initial dam leakage detection model is trained according to the first training data to optimize the fusion parameters and the first image enhancement strategy.

[0129] In some embodiments, the initial dam leakage detection model can be constructed using a YOLOv8 network model. The model structure of YOLOv8 is as follows: Figure 4 As shown. In other embodiments, other YOLO series models, such as YOLOv5, may also be used to implement it. YOLOv8 is suitable for CPU parallel computing and has good performance on data sets. By fusing visible light images and infrared images and inputting them into the YOLOv8 framework, it is possible to extract fused images containing spatial environmental information and capable of characterizing shallow leakage areas, and extract feature maps containing leakage locations and categories from a variety of leakage images. Combining the advantages of visible light images and infrared images, it can cope with scenes with more complex environments or poor lighting conditions, and can reduce the impact of noise on target detection results, further making model detection more accurate.

[0130] The optimization process of the image enhancement strategy is: keep the variables of other modules unchanged, and continuously adjust the strategies of single-source visual image and fused image enhancement during the training process. The image enhancement parameters are mainly the parameters a and b of linear enhancement. According to the training results of the YOLOv8 model, it is automatically iterated to determine the optimal linear enhancement parameters. Among them, the optimal image enhancement result should meet the needs of manual evaluation (that is, experienced staff can directly judge whether there is a leakage area by observing the enhanced fused image) to prevent the image set from lacking basic reference value.

[0131] The optimization process of image fusion parameters is as follows: keep other module variables unchanged, continuously modify the image fusion parameters, mainly the parameter ω of the weighted average method, and perform YOLOv8 model training after obtaining the fused image. According to the model training results, the advantages and disadvantages of the fusion parameters are evaluated, and then the optimal image fusion parameters are determined.

[0132] When the image enhancement strategy (linear enhancement parameters a and b) and the image fusion parameter (fusion parameter ω of the weighted average method) are both optimal, the optimal object detection model is obtained. The optimal judgment basis is that the sum of the mAP@0.5 and F1 scores of the YOLOv8 model training reaches the maximum value.

[0133] Furthermore, the visible light image and infrared image to be detected are input into the first dam leakage detection model for identification detection, thereby completing the leakage identification task and obtaining the leakage detection result.

[0134] The following is an example of a lawn surface leakage image set to provide a description of the leakage detection effect of the present application embodiment:

[0135] like Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown in the figure, no matter it is scene A or scene B, the position where leakage may exist in the bi-optical image obtained after registration is consistent with the position of the picture, which achieves accurate registration and provides a basis for subsequent bi-optical image target detection. It should be noted that the box in the figure is to show the leakage position, and the actual processed image does not contain the box.

[0136] like Fig. 9 and Fig.10 As shown, the fused image contains the spatial features of the visible light image and the contrast features of the infrared image, reflecting the leakage situation in the lawn.

[0137] The verification results of the method in the verification set of the present application embodiment are as follows: Fig.11 As shown, the method of the embodiment of the present application can accurately identify the location and size of the leakage area on the lawn surface.

[0138] In summary, the embodiments of the present application have at least the following beneficial effects:

[0139] 1. By fusing visible light images and infrared images and inputting them into the YOLOv8 framework, we can extract fused images containing spatial environmental information and representing shallow leakage areas, and extract feature maps containing leakage locations and categories from multiple leakage images. This method combines the advantages of visible light images and infrared images, can cope with scenes with more complex environments or poor lighting conditions, and can reduce the impact of noise on target detection results, further making model detection more accurate.

[0140] 2. An optimization scheme for image enhancement strategy and image fusion parameters is established, which is compatible with target detection in multiple leakage scenarios.

[0141] 3. Combined with the YOLOv8 framework, it can adapt to the parallel computing of the CPU, making the detection faster and more efficient.

[0142] See also Fig.12 The embodiment of the present application further provides an infrared and visible dual light leakage target detection system, which can implement the above-mentioned infrared and visible dual light leakage target detection method, and the system includes:

[0143] The first module 101 is used to perform image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics to obtain a first infrared image and a first visible light image;

[0144] The second module 102 is used to perform image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image;

[0145] The third module 103 is used to perform image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image;

[0146] The fourth module 104 is used to perform image annotation processing on the first fused image to obtain first training data;

[0147] The fifth module 105 is used to perform model training on the initial dam leakage detection model according to the first training data, optimize the fusion weight and the image enhancement processing strategy, and obtain a first dam leakage detection model;

[0148] The sixth module 106 is used to input the infrared image of the dam to be detected and the corresponding visible light image to be detected into the first dam leakage detection model for detection to obtain the leakage detection result.

[0149] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned infrared and visible dual light leakage target detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] See also Fig.13 , Fig.13 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0153] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0154] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes an infrared and visible dual light leakage target detection method of the embodiment of this application;

[0155] Input / output interface 203, used to implement information input and output;

[0156] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0157] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0158] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0159] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned infrared and visible dual-light leakage target detection method is implemented.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0163] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0164] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0166] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0172] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for detecting infrared and visible dual light leakage targets, characterized in that: The following steps are involved: Based on the leakage characteristics, an initial infrared image and an initial visible light image of the dam are subjected to image registration processing to obtain a first infrared image and a first visible light image; Performing image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image; Performing image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image; Performing image annotation processing on the first fused image to obtain first training data; According to the first training data, the initial dam leakage detection model is trained, the fusion weight and the image enhancement processing strategy are optimized, and a first dam leakage detection model is obtained; The infrared image of the dam to be detected and the corresponding visible light image to be detected are input into the first dam leakage detection model for detection to obtain a leakage detection result.

2. The method according to claim 1, characterized in that The method of performing image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics to obtain the first infrared image and the first visible light image includes the following steps: Determine the obviousness of the leakage feature according to the leakage area and contour of the initial infrared image and the initial visible light image; When the obviousness is greater than a preset first condition, a bi-optical image registration algorithm based on scale-invariant feature transformation is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image; When one of the conspicuity levels is less than the first condition, a center registration algorithm is used to perform image registration processing on the initial infrared image and the initial visible light image to obtain a first infrared image and a first visible light image.

3. The method according to claim 2, characterized in that The method of performing image registration processing on the initial infrared image and the initial visible light image using a bi-optical image registration algorithm based on scale-invariant feature transformation to obtain a first infrared image and a first visible light image comprises the following steps: Performing scale-invariant feature transformation processing on the initial infrared image to obtain a visible light image target feature point set; Performing scale-invariant feature transformation processing on the initial visible light image to obtain an infrared image target feature point set; Performing feature point matching on the visible light image target feature point set and the infrared image target feature point set to obtain matching points; Matching point correction is performed on the matching points to obtain a first infrared image and a first visible light image.

4. The method according to claim 2, characterized in that: The steps of scale-invariant feature transformation include the following steps: Perform scale space extremum detection on the input image to obtain scale and rotation invariant interest points; Determining key points according to the stability of the points of interest; Based on the local gradient direction of the input image, assign one or more directions to each key point; According to the direction, in the surrounding area of ​​each key point, the local gradient result of the image is measured to obtain the target feature point.

5. The method according to claim 2, characterized in that: The method of performing image registration processing on the initial infrared image and the initial visible light image using a central registration algorithm to obtain a first infrared image and a first visible light image includes the following steps: Acquiring shooting distances and shooting angles of the initial infrared image and the visible light image; Determine the center position of the same object according to the initial infrared image and the visible light image; Determining cropping parameters, scaling parameters and translation parameters according to the center position; When the shooting distance and the shooting distance of the initial infrared image and the visible light image are consistent, image registration is performed according to the cropping parameter, the scaling parameter, and the translation parameter to obtain a first infrared image and a first visible light image.

6. The method according to claim 1, characterized in that The method of training the initial dam leakage detection model according to the first training data, optimizing the fusion weight and the image enhancement processing strategy, and obtaining the first dam leakage detection model includes the following steps: Performing dam leakage detection on the first training data by using the initial dam leakage detection model to obtain a first detection result; Calculate the average precision and F1 value of the first detection result based on the annotation of the first training data; Adjusting the parameters of the fusion weight and the image enhancement processing strategy to update the initial dam leakage detection model; Repeat the step of performing dam leakage detection on the first training data through the initial dam leakage detection model to obtain a first detection result, until the average accuracy and the F1 value of the fusion weight and the image enhancement processing strategy meet the preset second condition, and obtain the first dam leakage detection model.

7. The method according to claim 1, characterized in that The initial dam leakage detection model is constructed using the YOLOv8 network framework.

8. An infrared and visible dual light leakage target detection system, characterized in that: include: The first module is used to perform image registration processing on the initial infrared image and the initial visible light image of the dam based on the leakage characteristics to obtain a first infrared image and a first visible light image; A second module is used to perform image enhancement processing on the first infrared image and the first visible light image to obtain a second infrared image and a second visible light image; A third module is used to perform image fusion processing on the second infrared image and the second visible light image according to a fusion weight to obtain a first fused image; A fourth module is used to perform image annotation processing on the first fused image to obtain first training data; A fifth module is used to perform model training on the initial dam leakage detection model according to the first training data, optimize the fusion weight and the image enhancement processing strategy, and obtain a first dam leakage detection model; The sixth module is used to input the infrared image of the dam to be detected and the corresponding visible light image to be detected into the first dam leakage detection model for detection to obtain the leakage detection result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images