Dual-light target detection enhancement method and system and electronic equipment
Through multi-scale Retinex enhancement and YOLO model training of visible light images, combined with LoFTR image registration and weighted box fusion, the accuracy problem of dual-light object detection under low illumination conditions is solved, and the goals of high accuracy and high robustness are achieved Test.
Patent Information
- Application Number
- CN202411067263.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to achieve high precision and high robust dual-light object detection under low illumination conditions, especially in complex backgrounds or target-intensive scenarios. The spatial resolution of infrared images is low and it is difficult to accurately distinguish between targets and backgrounds.
The multi-scale Retinex algorithm is used to enhance visible light images, and the infrared and visible light object detection models are trained using the YOLO algorithm, combined with the local feature network LoFTR for image registration and feature point matching, and the offset is calculated to map the visible light detection results into the infrared image. The detection results are fused through the weighted box fusion algorithm.
The accuracy of image registration and the accuracy of object detection results are improved, especially in the case of time difference between visible and infrared images or sensors are not aligned, and the object detection capability in low-illumination environments is enhanced.
Smart Images

Figure CN119090924A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and in particular to a dual-light target detection enhancement method, system and electronic equipment. Background Art
[0002] With the digital transformation of the global economy, the cultivation of new productivity and the development of the digital economy have become the key driving forces for social progress. Especially driven by artificial intelligence (AI) technology, all walks of life are undergoing unprecedented changes. In the field of low-altitude economy, the application of drone technology is becoming more and more extensive. Drones can carry various image sensors for various applications such as traffic reconnaissance, environmental monitoring and disaster relief. However, under low-light conditions such as night and cloudy days, drone aerial photography is limited by lighting conditions, and the quality of its aerial images is often difficult to meet the needs of high-precision target detection. This limitation not only limits the application of drones in low-altitude visual monitoring scenarios, but also affects the adaptability and application level of artificial intelligence technology in these scenarios.
[0003] In traditional AI visual monitoring scenarios, visible light images are widely used because of their high spatial resolution and rich color information, their ability to clearly display the details and appearance features of the target, and their ease of recognition and understanding by the human eye. However, under low-light conditions, the performance of visible light images is limited, and problems such as increased noise and reduced contrast may occur, resulting in reduced accuracy in target detection. However, infrared images are not restricted by lighting conditions by capturing the thermal radiation information of the target, so they can still effectively detect targets at night or in low light conditions. However, compared with visible light images, the spatial resolution of infrared images is usually lower, making it difficult to show the subtle structure and detailed features of the target, especially in scenes with complex backgrounds or dense targets, where it may not be possible to accurately distinguish between the target and the background.
[0004] Based on the above background, under low illumination conditions, dual-light imaging technology can significantly improve the accuracy of target detection by simultaneously acquiring visible light images and infrared images of the target and utilizing the complementary advantages of the two images. Existing dual-light target detection methods mainly focus on image registration fusion, deep feature fusion, and post-processing strategy fusion. Among them, the image feature registration algorithm uses the intrinsic feature matching relationship of the image to perform image registration and fusion, and has good adaptability to texture and shape changes in the image, but it may be affected by noise in the feature extraction and matching process, resulting in matching errors and increasing information loss in the fusion process. Deep feature fusion uses the feature extraction ability of deep learning to extract the features of the two images separately, and then fuses them at the feature level. However, due to the misalignment of the dual-light sensors, it is difficult for the fusion model to learn useful feature representations. Post-processing strategy fusion first learns the two data separately, and then formulates a fusion strategy for the output results based on the characteristics of the two images. Different fusion strategies can be flexibly selected according to the characteristics of the image, but post-processing fusion cannot solve the fundamental differences between images, and relies on a large amount of prior knowledge, which is limited by the design of the post-processing algorithm. It can be seen that how to effectively utilize the complementary characteristics of dual-light data and reduce information loss, so as to design a high-precision and high-robust dual-light target detection method under low-illumination conditions is still the research focus and difficulty at this stage. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a dual-light target detection enhancement method, system and electronic device, which solve the problem of the lack of high-precision and high-robust dual-light target detection under low illumination conditions in the prior art.
[0006] In a first aspect, according to an embodiment of the present invention, a dual-light target detection enhancement method is provided, which is used to enhance target detection when the illumination is below a threshold value. The dual-light target detection enhancement method comprises the following steps: S1, acquiring a visible light image and an infrared image, and performing image enhancement on the acquired visible light image; S2, acquiring an infrared target detection model and a visible light target detection model, wherein the infrared target detection model and the visible light target detection model are respectively trained on an infrared image training set and a visible light image training set, and are used to acquire targets of different categories and positions in the image; S3, inputting the enhanced visible light image into the visible light target detection model , detect the corresponding category of targets and positions in the visible light image; S4, input the acquired infrared image into the infrared target detection model, and detect the corresponding category of targets and positions in the infrared image; S5, use the local feature network LoFTR image feature point matching algorithm to perform image registration on the enhanced visible light image and the acquired infrared image to obtain the registered matching feature point set; S6, calculate the offset between the visible light image and the infrared image corresponding to the detection target based on the matching feature point set and the corresponding category of targets in the visible light image feature map, and then use the offset to map the detection result of the visible light image to the detection result in the infrared image and fuse the detection results.
[0007] According to an embodiment of the present invention, the image enhancement adopts a multi-scale Retinex algorithm.
[0008] According to an embodiment of the present invention, the infrared target detection model and the visible light target detection model are trained using the YOLO algorithm.
[0009] According to an embodiment of the present invention, the image registration includes the following steps: S51, extracting the coarse-grained feature map and the fine-grained feature map of the infrared image and the enhanced visible light image by a local feature convolutional neural network; S52, further extracting features from the coarse-grained feature map of the infrared image and the coarse-grained feature map of the visible light image by using a local feature network LoFTR to obtain a feature matrix; S53, obtaining a coarse-grained matching feature point set by using a mutual nearest neighbor algorithm based on the feature matrix of the infrared image and the feature matrix of the visible light image; S54, locating the obtained coarse-grained matching feature point set in the corresponding fine-grained feature map, intercepting a window of a preset size, and using a local feature network LoFTR of a size corresponding to the window to transform the features in the window to obtain a matched feature point set after registration.
[0010] According to an embodiment of the present invention, calculating the offset between the visible light image and the infrared image corresponding to the detected target based on the matching feature point set and the target of the corresponding category in the visible light image feature map includes the following steps: S61, constructing multiple target detection frames accommodating the detected targets of the corresponding category, and filtering the matching feature point set based on the multiple target detection frames to filter out objects outside the category to be detected, and obtaining matching feature point subsets in the multiple target detection frames in the infrared image and the visible light image respectively; S62, for each target detection frame, calculating the average value of the difference between all matching feature points in the matching feature point subset on the horizontal axis and the vertical axis as the offset.
[0011] According to an embodiment of the present invention, the matching feature point subset is further filtered based on a preset offset threshold.
[0012] According to an embodiment of the present invention, each matching feature point subset is further filtered based on a preset feature point quantity threshold.
[0013] According to an embodiment of the present invention, the detection result fusion is performed based on a weighted box fusion algorithm.
[0014] In a second aspect, according to another embodiment of the present invention, a dual-light target detection enhancement system is provided, which is configured to perform enhanced target detection using a dual-light target detection enhancement method as described in the first aspect, and is characterized in that the dual-light target detection enhancement system includes: an image acquisition module, including a visible light camera and an infrared camera, for respectively acquiring visible light images and infrared images; a data enhancement module, for performing data enhancement on visible light images; a target detection module, for detecting visible light images and infrared images to obtain the category and position information of each target in the corresponding image; an image registration module, for image registration of visible light images and infrared images, and for calculating the offset of the visible light image detection result corresponding to the infrared image, so as to map the visible light detection result to the infrared image; and a result fusion module, for post-processing fusion and filtering of the detection result of the visible light image mapped to the infrared image and the detection result of the infrared image to obtain the final detection result.
[0015] In the third aspect, according to another embodiment of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; and the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions.
[0016] The technical principle of the present invention is: in the local feature network LoFTR image feature point matching algorithm, the target detection result is additionally considered, and a separate target matching is performed for each detected target, and subsequent result fusion is performed.
[0017] Compared with the prior art, the above operation makes the registration of the target in the image more accurate, and the target detection result obtained in the end is more accurate. In addition, the method is particularly useful for the situation where there is a time difference between the visible light image and the infrared image, the dual light sensor is not aligned, and the target in the image is in relative motion. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 The figure is a flow chart of a dual-light target detection enhancement method according to an embodiment of the present invention.
[0019] Figure 2 Schematic diagram of image feature point matching results according to an embodiment of the present invention.
[0020] Figure 3 Schematic diagram of the image feature point filtering result according to an embodiment of the present invention.
[0021] Figure 4 Schematic diagram of target detection mapping results according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0023] As described in the background art, the prior art lacks high-precision and high-robust dual-light target detection under low-light conditions. Figure 1As shown, an embodiment of the present invention proposes a dual-light target detection enhancement method for enhancing target detection when the illumination is below a threshold value. The dual-light target detection enhancement method comprises the following steps: S1, acquiring a visible light image and an infrared image, and performing image enhancement on the acquired visible light image; S2, acquiring an infrared target detection model and a visible light target detection model, wherein the infrared target detection model and the visible light target detection model are respectively trained on an infrared image training set and a visible light image training set, and are used to acquire targets of different categories and positions in the image; S3, inputting the enhanced visible light image into the visible light target detection model, and detecting to the corresponding category of targets and positions in the visible light image; S4, input the acquired infrared image into the infrared target detection model, and detect the corresponding category of targets and positions in the infrared image; S5, use the local feature network LoFTR image feature point matching algorithm to perform image registration on the enhanced visible light image and the acquired infrared image to obtain the registered matching feature point set; S6, calculate the offset between the visible light image and the infrared image corresponding to the detection target according to the matching feature point set and the corresponding category of targets in the visible light image feature map, and then use the offset to map the detection result of the visible light image to the detection result in the infrared image and perform detection result fusion. For example, the illumination threshold can be 1 lux Lux. Through the above method, the present invention makes full use of the complementary characteristics of dual-light images, and uses the detection result of the infrared image as the reference result, which can effectively avoid the problem of poor detection results of the visible light image in a low-light environment; at the same time, the infrared image detection result is supplemented by the visible light detection result through feature matching and detection target calibration, which can better compensate for the missed detection phenomenon caused by the low spatial resolution, subtle structure and unclear detail features of the infrared image.
[0024] According to an embodiment of the present invention, the image enhancement adopts a multi-scale Retinex algorithm. The detailed steps are as follows: a multi-scale Retinex (MSR) algorithm is applied to the image to obtain an enhanced visible light data set. The algorithm first decomposes the original image into images of multiple scales, and the decomposition formula is:
[0025] I0=G(I0,σ0),
[0026] I1=G(I0,σ1),
[0027] ...,
[0028] I n =G(I0,σ n ),
[0029] Where G is a Gaussian filter, σ i(i∈(1,n)) represents the Gaussian kernel size of the i-th layer, I i represents the image at the i-th scale, where n represents the number of scales.
[0030] Then the enhanced image is obtained by the calculation formula of MSR, the formula is:
[0031]
[0032] Here R MSRi represents the enhanced image, w i represents the weight of the i-th layer scale, I i (x,y) represents the image of the i-th scale, x, y are the horizontal and vertical coordinates in the image, G i (x,y) represents Gaussian filtering of the i-th layer image.
[0033] According to an embodiment of the present invention, the infrared target detection model and the visible light target detection model are trained using the YOLO algorithm. In addition, the infrared target detection model and the visible light target detection model may also use algorithms such as OverFeat, SSD and RetinaNet.
[0034] According to an embodiment of the present invention, the image registration includes the following steps: S51, extracting the coarse-grained feature map and the fine-grained feature map of the infrared image and the enhanced visible light image by a local feature convolutional neural network; S52, further extracting features from the coarse-grained feature map of the infrared image and the coarse-grained feature map of the visible light image using the local feature network LoFTR to obtain a feature matrix; S53, obtaining a coarse-grained matching feature point set using a mutual nearest neighbor algorithm based on the feature matrix of the infrared image and the feature matrix of the visible light image; S54, locating the obtained coarse-grained matching feature point set in the corresponding fine-grained feature map, intercepting a window of a preset size, and using the local feature network LoFTR of a size corresponding to the window to transform the features in the window to obtain the matched feature point set after registration. In detail, the formula for extracting the coarse-grained feature map and the fine-grained feature map from the input image is expressed as:
[0035]
[0036] Here CNN (convolutional neural network) represents the local feature extraction network, I rgb ,I tir Respectively represent the input visible light image (enhanced) and infrared image, They represent the coarse-grained feature maps of visible light and infrared images, and the fine-grained feature maps of visible light and infrared images respectively.
[0037] Afterwards, get the confidence matrix:
[0038] P c (i,j)=softmax(S(i,·)·softmax(S(·,j),
[0039]
[0040] Here P c represents the confidence matrix, τ is the default preset parameter, S represents the score matrix, S(i,·) represents the i-th row, S(·,j) represents the j-th column, and Z k Represents the vector corresponding to a row or column of the score matrix, Z l Represents Z k The lth element in the vector, Respectively represent the feature matrices extracted by the LoFTR module from the coarse-grained feature matrices of the visible light image and infrared image.
[0041] Based on the above confidence matrix, the coarse-grained matching point set is obtained by the following formula:
[0042]
[0043] Here M c represents the coarse-grained matching point set, Indicates P c The set of feature points corresponding to the confidence matrix, MNN represents the mutual nearest neighbor algorithm, θ c Represents the confidence threshold.
[0044] Combined with the coarse-grained matching point set, for each coarse-grained paired point set, it is located on the fine-grained feature map, a w×w (usually 5x5) window is cut out from the fine-grained feature map, and a smaller LoFTR module is used to transform the features in the window to obtain the final matching feature point set M f .
[0045] According to an embodiment of the present invention, calculating the offset between the visible light image and the infrared image corresponding to the detected target based on the matching feature point set and the target of the corresponding category in the visible light image feature map includes the following steps: S61, constructing multiple target detection frames that accommodate the detected targets of the corresponding category, and filtering the matching feature point set based on the multiple target detection frames to filter objects outside the category to be detected, and obtaining matching feature point subsets in multiple target detection frames in the infrared image and the visible light image respectively; S62, for each target detection frame, calculating the average value of the difference between all matching feature points in the matching feature point subset on the horizontal axis and the vertical axis as the offset. Figure 3 As shown, after the matching feature point set is filtered, the second maple leaf target (not a category to be detected) is filtered out.
[0046] An example is Figure 2 As shown in the figure, 5 targets are shown. It can be seen that for each target, matching is performed in the corresponding target box (rectangle), and the feature points in all target boxes constitute a subset M. rgb (visible light) and M tir (infrared). In addition, the M corresponding to each detection box is calculated by the following formula tir To M rgb The average offset is:
[0047]
[0048] Here x ,O y They represent the offset on the x-axis and y-axis respectively, i represents any detection box, and N represents the number of feature points in the current box.
[0049] In addition, in order to further ensure the rationality of the mapping, according to an embodiment of the present invention, each matching feature point subset is further filtered based on a preset feature point quantity threshold.
[0050] In addition, in order to ensure the accuracy of the matching, the matching feature point set is also filtered based on a preset offset threshold. The offset threshold indicates the possible maximum offset value. Figure 3 ,In this example, the mismatched feature point pairs in the fourth and fifth targets are filtered out.
[0051] In addition, the mapping process is as follows Figure 4 As shown, the four targets in the visible light detection results are each mapped to the corresponding target in the infrared image.
[0052] According to an embodiment of the present invention, the detection result fusion is performed based on a weighted box fusion (WBF) algorithm. Specifically, WBF recalculates the confidence and coordinates using the following formula:
[0053]
[0054] Here X1,2,Y1,2 represent the x-coordinate and y-coordinate of the upper left corner and lower right corner of the mapped rectangular box, i represents any rectangular target box, C i represents the confidence of the i-th detection box, C represents the confidence of each corresponding detection result, and T represents the number of target boxes involved in the calculation.
[0055] According to another embodiment of the present invention, a dual-light target detection enhancement system is provided, which is configured to perform enhanced target detection using a dual-light target detection enhancement method as described above, and the dual-light target detection enhancement system includes: an image acquisition module, including a visible light camera and an infrared camera, for respectively acquiring visible light images and infrared images; a data enhancement module, for performing data enhancement on visible light images; a target detection module, for detecting visible light images and infrared images to obtain categories and position information of each target in the corresponding images; an image registration module, for image registration of visible light images and infrared images, and for calculating the offset of the visible light image detection result corresponding to the infrared image, so as to map the visible light detection result to the infrared image; and a result fusion module, for post-processing fusion and filtering of the detection result of the visible light image mapped to the infrared image and the detection result of the infrared image to obtain the final detection result.
[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
Claims
1. A dual light target detection enhancement method for enhancing target detection below a threshold illumination, characterized in that: The dual-light target detection enhancement method comprises the following steps: S1. Acquire a visible light image and an infrared image, and perform image enhancement on the acquired visible light image; S2. Obtain an infrared target detection model and a visible light target detection model, wherein the infrared target detection model and the visible light target detection model are trained on an infrared image training set and a visible light image training set, respectively, and are used to obtain targets of different categories and positions in an image; S3, inputting the enhanced visible light image into the visible light target detection model, and detecting targets and positions of corresponding categories in the visible light image; S4, inputting the acquired infrared image into the infrared target detection model, and detecting the target and position of the corresponding category in the infrared image; S5, using the local feature network LoFTR image feature point matching algorithm to perform image registration on the enhanced visible light image and the acquired infrared image to obtain a registered matching feature point set; S6. Calculate the offset between the visible light image and the infrared image corresponding to the detection target based on the matching feature point set and the corresponding category of targets in the visible light image, and then use the offset to map the detection result of the visible light image to the detection result in the infrared image and fuse the detection results.
2. A dual-light target detection enhancement method as claimed in claim 1, characterized in that: The image enhancement adopts a multi-scale Retinex algorithm.
3. A dual-light target detection enhancement method as claimed in claim 1, characterized in that: The infrared target detection model and the visible light target detection model are trained using the YOLO algorithm.
4. A dual-light target detection enhancement method as claimed in claim 1, characterized in that: The image registration comprises the following steps: S51, extracting a coarse-grained feature map and a fine-grained feature map of the infrared image and the enhanced visible light image by a local feature convolutional neural network; S52, further extracting features from the coarse-grained feature map of the infrared image and the coarse-grained feature map of the visible light image using a local feature network LoFTR to obtain a feature matrix; S53, obtaining a coarse-grained matching feature point set using a mutual nearest neighbor algorithm based on the feature matrix of the infrared image and the feature matrix of the visible light image; S54, positioning the obtained coarse-grained matching feature point set in the corresponding fine-grained feature map, intercepting a window of a preset size, and using a local feature network LoFTR of a size corresponding to the window to transform the features in the window to obtain the aligned matching feature point set.
5. A dual-light target detection enhancement method as claimed in claim 1, characterized in that: Calculating the offset between the visible light image and the infrared image corresponding to the detected target according to the matching feature point set and the target of the corresponding category in the visible light image feature map includes the following steps: S61, constructing a plurality of target detection frames accommodating the detected targets of the corresponding categories, and filtering the matching feature point set based on the plurality of target detection frames to filter out objects outside the category to be detected, and obtaining matching feature point subsets in the plurality of target detection frames in the infrared image and the visible light image respectively; S62: For each target detection frame, calculate an average value of the difference between the horizontal axis and the vertical axis of all matching feature points in the matching feature point subset as an offset.
6. A dual-light target detection enhancement method as claimed in claim 5, characterized in that: Each matching feature point subset is also filtered based on a preset feature point quantity threshold.
7. A dual-light target detection enhancement method as claimed in claim 5, characterized in that: The matching feature point subset is further filtered based on a preset offset threshold.
8. A dual-light target detection enhancement method as claimed in claim 1, characterized in that: The detection result fusion is performed based on a weighted box fusion algorithm.
9. A dual-light target detection enhancement system, configured to enhance target detection using a dual-light target detection enhancement method as claimed in any one of claims 1 to 8, characterized in that: The dual-light target detection enhancement system comprises: An image acquisition module, including a visible light camera and an infrared camera, for acquiring visible light images and infrared images respectively; A data enhancement module, used to perform data enhancement on visible light images; The target detection module is used to detect the visible light image and the infrared image to obtain the category and location information of each target in the corresponding image; An image registration module is used for image registration of visible light images and infrared images, and for calculating the offset of the visible light image detection result corresponding to the infrared image, so as to map the visible light detection result to the infrared image; The result fusion module is used to map the visible light image to the detection result of the infrared image and perform post-processing fusion and filtering on the detection result of the infrared image to obtain the final detection result.
10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.
