Target detection method and device based on multi-source data fusion, equipment and medium

Abstract

The invention relates to the technical field of target detection and artificial intelligence, and discloses a target detection method and device based on multi-source data fusion, equipment and a medium, and the method comprises the steps: carrying out the hard light interference evaluation of a visible light image, and obtaining a hard light interference index; filtering and denoising the visible light image based on the infrared image and the strong light interference index by adopting a guide filtering algorithm; carrying out multi-scale feature extraction on the de-noised visible light image and the de-noised infrared image; performing feature fusion on the multi-scale features based on the visible light fusion weight and the infrared fusion weight; and performing target detection based on the final fusion feature to obtain a target detection result. According to the method, a strong light evaluation mechanism is used, noise reduction is guided in combination with the infrared image, the problem of feature submerging under strong light can be solved, multi-scale feature extraction and fusion are performed on the infrared features and the visible light features, multi-level optimization can be performed for the strong light interference problem, accurate feature recognition under strong light is ensured, and the accuracy of feature recognition under strong light is improved. And the detection precision of the strong light area target is ensured.

Description

Technical Field

[0001] This application relates to the fields of target detection and artificial intelligence technology, and in particular to a target detection method, apparatus, equipment and medium based on multi-source data fusion. Background Technology

[0002] Existing technologies commonly employ visible-infrared dual-source fusion target detection schemes, which are mostly based on backbone networks such as ResNet to extract dual-modal features separately, and achieve fusion by summing and stitching fixed weights at the pixel level / feature level or by using a basic attention mechanism, and then combine them with mainstream detection heads such as YOLO and FCOS to output results.

[0003] While the above-mentioned solution can utilize the characteristics of infrared light to withstand harsh environments and adapt to some complex scenarios, if it is applied to strong light scenarios, problems such as a sharp drop in detection accuracy will occur due to the amplification of modal differences. Summary of the Invention

[0004] The main purpose of this application is to provide a target detection method, device, equipment and medium based on multi-source data fusion, which aims to solve the technical problem that the existing visible light-infrared light dual-source fusion target detection scheme cannot be applied to target detection in strong light scenarios.

[0005] The first aspect of this application provides a target detection method based on multi-source data fusion, which includes: Strong light interference is evaluated on the visible light image of the target scene to obtain a strong light interference index, which is used to indicate the degree of strong light interference in the visible light image. A guided filtering algorithm is used to filter and denoise the visible light image based on the edge features and strong light interference index of the infrared image in the target scene, so as to obtain the denoised visible light image. Multi-scale feature extraction is performed on the denoised visible light image and infrared image using the feature extraction layer in the target detection model; The visible light fusion weight and infrared fusion weight are determined based on the strong light interference index; Based on visible light fusion weights and infrared fusion weights, the feature fusion layer in the target detection model is used to perform feature fusion on the extracted multi-scale features to obtain the final fused features; The target detection head of the target detection model is used to perform target detection based on the final fused features, and the target detection results are obtained.

[0006] This application also provides a target detection device based on multi-source data fusion, which includes: The strong light interference assessment module is used to assess the strong light interference of the visible light image in the target scene and obtain a strong light interference index to indicate the degree of strong light interference in the visible light image. The strong light interference index includes a global strong light interference index to indicate the overall degree of strong light interference in the visible light image and a local strong light interference index to indicate the degree of strong light interference in each sub-region of the visible light image. The noise reduction module is used to filter and denoise the visible light image based on the edge features and strong light interference index of the infrared image in the target scene using a guided filtering algorithm, so as to obtain a denoised visible light image. The feature extraction module is used to extract multi-scale features from the denoised visible light image and infrared image using the feature extraction layer in the target detection model. The fusion weight determination module is used to determine the visible light fusion weight and infrared fusion weight based on the strong light interference index. The fusion module is used to perform feature fusion on the extracted multi-scale features based on visible light fusion weights and infrared fusion weights, and to obtain the final fused features by utilizing the feature fusion layer in the target detection model. The target detection module is used to perform target detection based on the final fused features using the detection head of the target detection model, and obtain the target detection results.

[0007] A third aspect of this application provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the computer device to execute the above-described target detection method based on multi-source data fusion.

[0008] The fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned target detection method based on multi-source data fusion.

[0009] This application employs a strong light evaluation mechanism combined with infrared image-guided noise reduction to address the issue of feature overshadowing under strong light, ensuring accurate feature recognition and enhancing target detection accuracy in strong light areas. By performing multi-scale feature extraction and fusion on infrared and visible light features, the characteristics of both light and infrared are fully utilized. Multi-level optimization is implemented to address strong light interference, further guaranteeing accurate feature recognition under strong light and enhancing target detection accuracy in strong light areas. This approach is suitable for target detection in various strong light scenarios. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the first embodiment of the target detection method based on multi-source data fusion in this application. Figure 2This is a flowchart illustrating the second embodiment of the target detection method based on multi-source data fusion in this application. Figure 3 This is a schematic diagram of the functional modules of one embodiment of the target detection device based on multi-source data fusion in this application. Figure 4 This is a schematic diagram of one embodiment of the computer device described in this application. Detailed Implementation

[0011] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0012] refer to Figure 1 This application provides a target detection method based on multi-source data fusion, which includes: S100: Evaluate the strong light interference in the visible light image of the target scene to obtain the strong light interference index, which is used to indicate the degree of strong light interference in the visible light image.

[0013] Specifically, a visible light sensor can be used to acquire visible light images of the target scene, and an infrared sensor can be used to acquire infrared images of the target scene. Visible light images and infrared images belong to two different modalities of images.

[0014] The acquisition of visible light images is affected by the ambient light level. Therefore, the acquired visible light images may contain one or more areas, such as bright light areas, dim light areas, and dark light areas. For example, when acquiring visible light images of certain areas within a tunnel, if the tunnel is equipped with bright lighting fixtures or prominently displayed billboards, the acquired visible light images may contain bright light areas due to interference from the ambient light. These bright light areas inevitably have a significant impact on target detection, reducing its accuracy. Therefore, to improve the accuracy of target detection, this embodiment will evaluate the degree of bright light interference in the visible light images.

[0015] The glare interference index comprises a global glare interference index, which indicates the overall glare interference level of a visible light image, and a local glare interference index, which indicates the glare interference level of individual sub-regions within a visible light image. The glare interference index quantifies the degree of glare interference in a visible light image.

[0016] The visible light image is divided into several sub-regions, and the average brightness of each sub-region is calculated. The average brightness is compared with the low intensity light threshold and with the high intensity light threshold. Based on the comparison results, the local intensity light interference index of the corresponding sub-region can be determined.

[0017] The global strong light interference index can be calculated based on the local strong light interference index of all sub-regions.

[0018] The higher the value of the global strong light interference index, the more severe the strong light interference is to the visible light image as a whole.

[0019] The higher the value of the local strong light interference index, the more severe the strong light interference in the corresponding sub-region.

[0020] This embodiment uses dynamic strong light interference assessment, which can solve the problem of feature submersion under strong light.

[0021] S200: The guided filtering algorithm is used to filter and denoise the visible light image based on the edge features and strong light interference index of the infrared image in the target scene, so as to obtain the denoised visible light image.

[0022] Specifically, guided image filtering is a linear filter that balances image smoothing and edge preservation using a guide image. Its core principle lies in adjusting the smoothness of the output image based on the gradient information of the guide image, while avoiding gradient inversion. Through an adaptive weighting mechanism, more details are preserved in boundary regions, and a stronger smoothing effect is achieved in flat areas. When the guide image is the same as the input image, the filter is equivalent to mean filtering; when the guide image is a noisy image, it can achieve denoising.

[0023] Based on the local strong light interference index, it can be determined which sub-regions belong to strong light interference areas and which sub-regions belong to non-strong light interference areas.

[0024] More specifically, the local strong light interference index of each sub-region is compared with the interference index threshold. If the local strong light interference index exceeds the interference index threshold, the corresponding sub-region is determined to be a strong light interference region; if the local strong light interference index does not exceed the interference index threshold, the corresponding sub-region is determined to be a non-strong light interference region.

[0025] A strong light mask can be generated based on the local strong light interference index. If the local strong light interference index of sub-region i... Then the strong light mask of sub-region i Marked as 1, indicating that sub-region i belongs to the strong light interference region; if the local strong light interference index of sub-region i is... Then the strong light mask of sub-region i Marked as 0, it is used to indicate that sub-region i belongs to a non-strong light interference region.

[0026] The threshold value of the interference index is greater than 0.5 and less than 1. For example, it can be any one of the values ​​such as 0.6, 0.7, 0.8, 0.85, etc. The specific configuration depends on the actual application scenario, and this application does not impose any restrictions on it.

[0027] Guided filtering algorithm based on infrared image Edge features are used to filter and denoise areas with strong light interference in visible light images:

[0028] Formula 4 in, The guiding filter window represents the area surrounding the (x,y) position of a pixel, and is usually a square window (e.g., 3*3 pixels, 5*5 pixels).

[0029] This is the luminance value of pixel (x,y) after denoising.

[0030] This is a Gaussian kernel function used to assign "weights" to neighboring pixels at different positions within the guided filter window.

[0031] This represents the original brightness value of a neighboring pixel (x', y') in a visible light image.

[0032] This is the strong light mask for the neighboring pixels (x', y') in the visible light image.

[0033] Formula 4 above means that a Gaussian filter is applied to the pixel and the strong light interference area is suppressed. The neighboring pixel is located in the strong light interference area. The neighboring pixel does not contribute to the molecule and is equivalent to being "filtered out".

[0034] This embodiment combines infrared images to guide the noise reduction of visible light images.

[0035] S300: The feature extraction layer in the target detection model is used to perform multi-scale feature extraction on the denoised visible light image and infrared image respectively.

[0036] Specifically, the feature extraction layer in the target detection model is used to extract multi-scale features from the denoised visible light image and infrared image, respectively.

[0037] Multi-scale features contain features at least two scales. For example, multi-scale features contain features at least two scales from low-level features, mid-level features, and high-level features.

[0038] S400: Determine the visible light fusion weight and infrared fusion weight based on the strong light interference index.

[0039] Specifically, the strong light interference index can provide dynamic weights for subsequent fusion strategies. Based on the strong light interference index, the degree to which a visible light image is affected by strong light interference can be determined, as well as the importance of the visible light image in the feature fusion process.

[0040] The more severe the strong light interference in a visible light image, the lower the importance of visible light features during feature fusion, and correspondingly, the lower the visible light fusion weight. Conversely, the less severe the strong light interference in a visible light image, the higher the importance of visible light features during feature fusion, and correspondingly, the higher the visible light fusion weight.

[0041] Based on this, the visible light fusion weight and infrared fusion weight can be dynamically adjusted according to the strong light interference index to ensure that infrared features dominate the fusion under strong light and visible light features are highlighted under weak light, thus ensuring that the final fused features can reflect the true features of the target scene.

[0042] S500: Based on visible light fusion weights and infrared fusion weights, the feature fusion layer in the target detection model is used to perform feature fusion on the extracted multi-scale features to obtain the final fused features.

[0043] Specifically, the feature fusion process can be multi-level feature fusion, with each level being a local feature fusion used to generate intermediate fused features. These intermediate fused features can be used for lower-level feature fusion, and the final fused feature is obtained through the last level of feature fusion.

[0044] S600: Target detection is performed using the detection head of the target detection model based on the final fused features to obtain the target detection result.

[0045] Specifically, the object detection model can be derived from models such as YOLO and includes a feature extraction layer, a feature fusion layer, and a detection head. The feature extraction layer can be built based on the backbone network of the YOLO model, and the detection head can be built based on the detection head of the YOLO model. The object detection model is a pre-trained model.

[0046] The target detection method based on multi-source data fusion in this embodiment can accurately detect targets in scenarios such as subway stations, tunnels, and bus stops where there are brightly lit billboards and lights.

[0047] This embodiment employs a strong light evaluation mechanism combined with infrared image-guided noise reduction to address the issue of feature overwhelming under strong light, ensuring accurate feature recognition and guaranteeing target detection accuracy in strong light areas. By performing multi-scale feature extraction and fusion on infrared and visible light features, the characteristics of both light and infrared can be fully utilized. Multi-level optimization is performed to address strong light interference, further ensuring accurate feature recognition under strong light and guaranteeing target detection accuracy in strong light areas. This approach is suitable for target detection in various strong light scenarios.

[0048] In one embodiment, the feature extraction layer of the object detection model includes a low-level feature extraction layer, a cross-attention layer, and a high-level feature extraction layer connected in sequence. In step S300, the feature extraction layer in the target detection model is used to perform multi-scale feature extraction on the denoised visible light image and infrared image, including: The low-level features of the denoised visible light image and infrared image are extracted using a low-level feature extraction layer and used as the first low-level feature and the second low-level feature, respectively. By using a cross-attention layer to enhance the low-level features of the denoised visible light image and infrared image through channel attention, the first mid-level features corresponding to the denoised visible light image and the second mid-level features corresponding to the infrared image are obtained. The high-level feature extraction layer is used to extract the first high-level feature and the second high-level feature from the first middle-level feature and the second middle-level feature, respectively.

[0049] Specifically, the low-level feature extraction layer includes a first low-level feature extraction module and a second low-level feature extraction module. After the denoised visible light image is input to the first low-level feature extraction module, the first low-level feature extraction module extracts the low-level features of the denoised visible light image as the first low-level feature. After the infrared image is input to the second low-level feature extraction module, the second low-level feature extraction module extracts the low-level features of the infrared image as the second low-level feature.

[0050] The cross-attention layer comprises a first cross-attention module and a second cross-attention module. The inputs to both the first and second cross-attention modules include first and second low-level features.

[0051] Both the first and second cross-attention modules obtain the mid-layer features using the following formula 9: Formula 9 in, For global average pooling, To provide spatial attention for the parameters of the convolutional layer, multi-channel information is fused. By learning the channel weights of the semantic features of the target, the target-related channels (e.g., the thermal radiation channel of obstacles, the color channel of signs, etc.) are strengthened.

[0052] In formula 9, and These are characteristics from two different sources. This represents the mid-level features obtained after channel attention enhancement. When the cross-attention module is used in the visible light channel (i.e., the first cross-attention module), It is an infrared feature (e.g., a second low-level feature). These are visible light features (e.g., first low-level features). This refers to the mid-level features (denoised as the first mid-level features) of the denoised visible light image obtained after channel attention enhancement. When the cross-attention module is used in the infrared channel (i.e., the second cross-attention module), These are visible light features (e.g., first low-level features). It is an infrared feature (e.g., a second low-level feature). This refers to the mid-level features (denoted as the second mid-level features) of the infrared image obtained after channel attention enhancement.

[0053] Among them, Sigmoid The function is the activation function of the neural network, which can map variables to the range [0,1].

[0054] In one embodiment, the feature fusion layer of the target detection model includes a dual-source feature fusion layer and a multi-level feature fusion layer; In step S500, based on visible light fusion weights and infrared fusion weights, the extracted multi-scale features are fused using the feature fusion layer in the target detection model to obtain the final fused features, including: Using a dual-source feature fusion layer, the first target scale features are fused based on visible light fusion weights and infrared fusion weights to obtain intermediate fused features. The first target scale features include some scale features from the multi-scale features of the denoised visible light image and infrared image. By using a multi-level feature fusion layer, the second target scale feature and the intermediate fusion feature are fused to obtain the final fused feature. The second target scale feature includes features of at least one scale from the multi-scale features of the denoised visible light image and infrared image.

[0055] Specifically, both the denoised visible light image and the infrared image contain corresponding multi-scale features, and these multi-scale features are extracted through the feature extraction layer.

[0056] For example, the multi-scale features of a denoised visible light image include a first low-level feature, a first mid-level feature, and a first high-level feature. The multi-scale features of an infrared image include a second low-level feature, a second mid-level feature, and a second high-level feature.

[0057] The dual-source feature fusion layer is used to fuse some scale features from the multi-scale features of the denoised visible light image and some scale features from the multi-scale features of the infrared image.

[0058] More specifically, for example, the dual-source feature fusion layer is used to fuse the first high-level feature and the second high-level feature using visible light fusion weights and infrared fusion weights to obtain the first intermediate fused feature.

[0059] And / or, The dual-source feature fusion layer is used to fuse the first and second intermediate features using visible light fusion weights and infrared fusion weights to obtain the second intermediate fused feature.

[0060] The multi-level feature fusion layer further fuses the intermediate fused features and the second target scale features to obtain the final fused features.

[0061] For example, the first target scale feature includes a first high-level feature and a second high-level feature, and the second target scale feature includes a first low-level feature, a second low-level feature, a first mid-level feature, and a second mid-level feature.

[0062] Of course, the fusion of features is not limited to the examples above and can be configured according to the actual situation. This application does not impose any restrictions on this.

[0063] In one embodiment, multi-scale features include low-level features, mid-level features, and high-level features; The first target scale feature includes the first high-level feature in the multi-scale features of the denoised visible light image and the second high-level feature in the multi-scale features of the infrared image; Using a dual-source feature fusion layer, feature fusion is performed on the first target-scale features based on visible light fusion weights and infrared fusion weights to obtain intermediate fused features, including: The cross-modal attention matrix is ​​calculated using a normalization function based on the first and second high-level features; Based on visible light fusion weights, infrared fusion weights, and cross-modal attention matrices, feature fusion is performed on the first high-level features and the second high-level features to obtain intermediate fused features.

[0064] Specifically, this embodiment introduces a cross-modal attention matrix. In the process of dual-source feature fusion, complementary regions of the dual-source features can be enhanced. For example, the orbital edges of infrared images and non-strong light interference regions of visible light images can be enhanced.

[0065] The normalization function, also known as the probability transformation function, is used to transform real values ​​into probability values ​​between [0,1], or to transform a set of real values ​​into a probability distribution between [0,1].

[0066] The normalization function can be, for example, the normalization exponential function, the Softmax() function, or other non-linear activation functions that can transform numbers to probability values ​​between [0,1].

[0067] In one specific embodiment, the cross-modal attention matrix A is calculated using the following formula 5: Formula 5 Where A is the cross-modal attention matrix. This represents the first high-level feature of the denoised visible light image. is the second high-level feature of the infrared image; D is the dimension of the feature.

[0068] by Taking a dimension of C×H×W as an example, the value of D is H×W.

[0069] Equation 5 can be used to calculate the similarity of the two-source features, and attention weights can be assigned to each pixel. Based on the cross-modal attention matrix, adaptive fusion of local features can be achieved.

[0070] The intermediate fusion features can be calculated using the following formula 6: Formula 6 in, As an intermediate fusion feature, This represents the first high-level feature of the denoised visible light image. Here, A represents the second high-level feature of the infrared image, and A is the cross-modal attention matrix. For visible light fusion weights, For infrared fusion weights.

[0071] The dimensions of A are H×W×H×W.

[0072] In one embodiment, the visible light fusion weight and the infrared fusion weight are calculated using the following formulas 1-2: Formula 1 Formula 2 in, For visible light fusion weights, For infrared fusion weights, This represents the global strong light interference index.

[0073] Specifically, using Formulas 1 and 2, the global strong light interference index can be used as a basis. Dynamically adjust visible light fusion weights Infrared fusion weights .

[0074] For example, when the global strong light interference index When =0, it indicates that there is no strong light in the visible light image. =0.8、 =0.2, at which point visible light features dominate the fusion.

[0075] When the global strong light interference index When =1, it indicates that there is extreme light in the visible light image. =0.1、 =0.9, at which point the infrared features dominate the fusion under strong light.

[0076] In one embodiment, a strong light interference assessment is performed on a visible light image of a target scene to obtain a strong light interference index that indicates the degree of strong light interference in the visible light image, including: Divide the visible light image into multiple sub-regions; Calculate the local strong light interference index for each sub-region; Based on the infrared image, a preliminary detection is performed to determine whether each sub-region contains infrared target candidate regions, and the sub-weight of each sub-region is determined based on the detection results. The global strong light interference index of the visible light image is calculated based on the local strong light interference index and sub-weight of the sub-region.

[0077] Specifically, visible light images The image is divided into N sub-regions, which can be the same size or different sizes. For example, a visible light image... Divide the local area into 16×16 pixel blocks to obtain N local blocks B1, B2, ..., BN; each local block is a sub-region.

[0078] Get the average brightness of any sub-region i Compare average brightness With low intensity light threshold Size and comparison of average brightness With high intensity light threshold Based on the size comparison results, the local strong light interference index of sub-region i can be determined. Where i takes values ​​from 1, 2, ..., N.

[0079] The global strong light interference index is calculated using the following formula 7: Formula 7 in, The global strong light interference index. Let i be the sub-weight of sub-region i. Let be the local strong light interference index of sub-region i, and N be the total number of sub-regions.

[0080] The sub-weight of sub-region i can be determined based on whether it contains infrared target candidate regions. Sub-regions of the first type, which contain infrared target candidate regions, are more important than those of the second type, which do not. Therefore, the sub-weight of sub-regions of the first type, which contain infrared target candidate regions, is higher than that of sub-regions of the second type, which do not.

[0081] In one specific embodiment, if sub-region i contains an infrared target candidate region, then If sub-region i does not contain infrared target candidate regions, then .

[0082] Of course, the above is just an example. The specific values ​​of the sub-weights are set according to the actual application scenario, and this application does not impose any restrictions on them.

[0083] Global strong light interference index It can be used to dynamically and adaptively adjust the fusion weights of dual-source features, that is, to adjust the fusion weights of visible light and infrared.

[0084] In one embodiment, calculating the local strong light interference index for each sub-region includes: Calculate the average brightness of each sub-region; The local strong light interference index of sub-regions with average brightness not higher than the low strong light threshold is set to 0. The local strong light interference index of the sub-region with an average brightness not lower than the high intensity light threshold is defined as 1; If the average brightness is greater than the low intensity light threshold but less than the high intensity light threshold, the brightness standard deviation of the corresponding sub-region is calculated. Based on the average brightness, brightness standard deviation, low intensity light threshold, high intensity light threshold and preset maximum brightness standard deviation, the local intensity light interference index of the sub-region is calculated.

[0085] Specifically, the local strong light interference index is calculated using the following formula 8: Formula 8 in, Let be the local strong light interference index of sub-region i. Let i be the average brightness of sub-region i. Let be the standard deviation of the brightness of sub-region i. Low intensity light threshold, For high intensity light threshold, The preset maximum brightness standard deviation, ∈[0,1].

[0086] In one specific embodiment, =180, =220, =50.

[0087] Of course, the above is only an example. The values ​​of low intensity light threshold, high intensity light threshold and maximum brightness standard deviation are set according to the actual application scenario, and this application does not limit them.

[0088] As shown in Formula 8, if the average brightness of sub-region i... Not higher than the low intensity light threshold Then the local strong light interference index of sub-region i will be... Set to 0.

[0089] If the average brightness of sub-region i Not lower than the high intensity light threshold Then the local strong light interference index of sub-region i will be... Set to 1.

[0090] If the average brightness of sub-region i Above the low intensity light threshold And below the high intensity light threshold Then calculate the standard deviation of brightness in sub-region i. According to average brightness luminance standard deviation Low intensity light threshold High intensity light threshold and the preset maximum brightness standard deviation The local strong light interference index of sub-region i is calculated. .

[0091] In one embodiment, the total loss function of the object detection model during training is shown in Equation 3 below:

[0092] Formula 3 in, Total loss function , For bounding box regression loss, For category loss, For confidence loss, q is a preset weight value greater than 1. This indicates that the corresponding sub-region is a region of strong light interference. This indicates pixels located in areas of strong light interference. This indicates a pixel located in an area free from strong light interference.

[0093] Specifically, bounding box regression loss The Complete Intersection over Union (CIoU) can be used as a metric in object detection to evaluate the degree of overlap between the predicted bounding box and the ground truth bounding box.

[0094] Category loss Cross-entropy loss can be used to measure the difference between the predicted class and the true class.

[0095] The confidence loss in this embodiment includes the confidence loss in areas with strong light interference and the confidence loss in areas without strong light interference.

[0096] This indicates that the corresponding sub-region is a non-strong light interference region.

[0097] The value of q can be any one of the following, such as 1.2, 1.4, or 1.5.

[0098] This embodiment introduces a weighted loss for both strong light and non-strong light regions, assigning higher weight to the confidence loss in areas with strong light interference. This allows the target detection model to focus on learning the features of these areas, ensuring the accuracy of target detection in such regions. By introducing higher weights for strong light interference regions, this embodiment is suitable for target detection in high-brightness tunnel scenarios with billboards, searchlights, etc., solving the problem of feature overshadowing under strong light and ensuring the accuracy of target detection in these areas.

[0099] Figure 2 This is a flowchart illustrating the second embodiment of the target detection method based on multi-source data fusion in this application. (See reference) Figure 2The visible light image is evaluated for strong light interference to obtain a strong light interference index. A guided filtering algorithm is then used to filter and denoise the visible light image based on the edge features and strong light interference index of the infrared image in the target scene, obtaining a denoised visible light image. Low-level feature extraction is performed on the denoised visible light image to obtain the first low-level feature, and low-level feature extraction is performed on the infrared image to obtain the second low-level feature. A cross-attention module used in the visible light channel is then used to extract the first mid-level feature of the visible light image based on the first and second low-level features. Finally, a cross-attention module used in the infrared channel is used to extract the first mid-level feature of the visible light image based on the first and second low-level features. The second middle-layer feature of the infrared image; the first high-layer feature of the visible light image is extracted based on the first middle-layer feature; the second high-layer feature of the infrared image is extracted based on the second middle-layer feature; the visible light fusion weight and infrared fusion weight in the dual-source feature fusion are determined based on the strong light interference index; based on the visible light fusion weight and infrared fusion weight, the first high-layer feature and the second high-layer feature are fused to obtain the intermediate fused feature; multi-level feature fusion is performed on the intermediate fused feature, the first low-layer feature, the second low-layer feature, the first middle-layer feature and the second middle-layer feature to obtain the final fused feature; the target detection is performed using the detection head based on the final fused feature to obtain the target detection result.

[0100] This application uses a dynamic strong light evaluation mechanism to solve the problem of feature submersion under strong light. It combines infrared to guide visible light image noise reduction, and combines low-level attention with high-level feature fusion. It makes full use of the characteristics of visible light and infrared, and performs multi-level optimization for strong light problems. It achieves strong light interference suppression while improving target detection accuracy.

[0101] refer to Figure 3 This application also provides a target detection device based on multi-source data fusion, which includes: Strong light interference assessment module 100 is used to assess strong light interference in a visible light image of a target scene and obtain a strong light interference index to indicate the degree of strong light interference in the visible light image. The strong light interference index includes a global strong light interference index to indicate the overall degree of strong light interference in the visible light image and a local strong light interference index to indicate the degree of strong light interference in each sub-region of the visible light image. The noise reduction module 200 is used to filter and denoise the visible light image based on the edge features and strong light interference index of the infrared image in the target scene using a guided filtering algorithm, so as to obtain a denoised visible light image. The feature extraction module 300 is used to perform multi-scale feature extraction on the denoised visible light image and infrared image respectively using the feature extraction layer in the target detection model; The fusion weight determination module 400 is used to determine the visible light fusion weight and infrared fusion weight based on the strong light interference index. The fusion module 500 is used to perform feature fusion on the extracted multi-scale features based on visible light fusion weights and infrared fusion weights, using the feature fusion layer in the target detection model to obtain the final fused features. The target detection module 600 is used to perform target detection based on the final fused features using the detection head of the target detection model to obtain the target detection result.

[0102] In one embodiment, the feature fusion layer of the target detection model includes a dual-source feature fusion layer and a multi-level feature fusion layer; Fusion module 500 includes: The dual-source feature fusion module is used to perform feature fusion on the first target scale features based on visible light fusion weights and infrared fusion weights using the dual-source feature fusion layer to obtain intermediate fused features. The first target scale features include some scale features from the multi-scale features of the denoised visible light image and infrared image. The multi-level feature fusion module is used to fuse the second target scale features and intermediate fusion features using a multi-level feature fusion layer to obtain the final fused features. The second target scale features include features of at least one scale from the multi-scale features of the denoised visible light image and infrared image.

[0103] In one embodiment, the first target scale feature includes a first high-level feature in the multi-scale features of the denoised visible light image and a second high-level feature in the multi-scale features of the infrared image. The dual-source feature fusion module includes: The cross-modal attention matrix calculation unit is used to calculate the cross-modal attention matrix based on the first high-level feature and the second high-level feature using a normalization function; The dual-source feature fusion unit is used to fuse the first high-level features and the second high-level features based on visible light fusion weights, infrared fusion weights, and cross-modal attention matrix to obtain intermediate fused features.

[0104] In one embodiment, the glare interference index includes a global glare interference index used to indicate the overall glare interference level of a visible light image; The visible light fusion weight and the infrared fusion weight are calculated using the following formulas 1 and 2: Formula 1 Formula 2 in, For visible light fusion weights, For infrared fusion weights, This represents the global strong light interference index.

[0105] In one embodiment, the strong light interference index includes a global strong light interference index for indicating the overall strong light interference level of a visible light image and a local strong light interference index for indicating the strong light interference level of each sub-region in the visible light image. Strong light interference evaluation module 100 includes: The segmentation module is used to divide a visible light image into multiple sub-regions; The local interference index calculation module is used to calculate the local strong light interference index of each sub-region. The sub-weight determination module is used to initially detect whether each sub-region contains infrared target candidate regions based on infrared images, and to determine the sub-weight of each sub-region based on the detection results. The global interference index calculation module is used to calculate the global strong light interference index of a visible light image based on the local strong light interference index and sub-weight of the sub-region.

[0106] In one embodiment, the local disturbance index calculation module is specifically used for: Calculate the average brightness of each sub-region; The local strong light interference index of sub-regions with average brightness not higher than the low strong light threshold is set to 0. The local strong light interference index of the sub-region with an average brightness not lower than the high intensity light threshold is defined as 1; If the average brightness is greater than the low intensity light threshold but less than the high intensity light threshold, the brightness standard deviation of the corresponding sub-region is calculated. Based on the average brightness, brightness standard deviation, low intensity light threshold, high intensity light threshold and preset maximum brightness standard deviation, the local intensity light interference index of the sub-region is calculated.

[0107] In one embodiment, the total loss function of the object detection model during training is shown in Equation 3 below:

[0108] Formula 3 in, For the total loss function, For bounding box regression loss, For category loss, For confidence loss, q is a preset weight value greater than 1. This indicates that the corresponding sub-region is a region of strong light interference. This indicates pixels located in areas of strong light interference. This indicates a pixel located in an area free from strong light interference.

[0109] Figure 4This is a schematic diagram of the structure of a computer device 7000 provided in an embodiment of this application. The computer device 7000 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 7000. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the computer device 7000.

[0110] The computer device 7000 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input / output interfaces 760, and / or one or more operating systems 731, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0111] This application also provides a computer device, which includes a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the target detection method based on multi-source data fusion described in the above embodiments. This application also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the steps of the target detection method based on multi-source data fusion.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A target detection method based on multi-source data fusion, characterized in that, The target detection method based on multi-source data fusion includes: Strong light interference is evaluated on the visible light image of the target scene to obtain a strong light interference index, which is used to indicate the degree of strong light interference in the visible light image. A guided filtering algorithm is used to filter and denoise the visible light image based on the edge features of the infrared image in the target scene and the strong light interference index, so as to obtain a denoised visible light image. Multi-scale feature extraction was performed on the denoised visible light and infrared images using the feature extraction layer in the target detection model. The visible light fusion weight and infrared fusion weight are determined based on the strong light interference index. Based on the visible light fusion weight and infrared fusion weight, the extracted multi-scale features are fused using the feature fusion layer in the target detection model to obtain the final fused features. The target detection head of the target detection model is used to perform target detection based on the final fused features to obtain the target detection result.

2. The target detection method based on multi-source data fusion according to claim 1, characterized in that, The feature fusion layer of the target detection model includes a dual-source feature fusion layer and a multi-level feature fusion layer; Based on the visible light fusion weights and infrared fusion weights, the extracted multi-scale features are fused using the feature fusion layer in the target detection model to obtain the final fused features, including: Using the dual-source feature fusion layer, feature fusion is performed on the first target scale feature based on the visible light fusion weight and the infrared fusion weight to obtain intermediate fused features, wherein the first target scale feature includes some scale features from the multi-scale features of the denoised visible light image and the infrared image. The second target scale feature and the intermediate fusion feature are fused using the multi-level feature fusion layer to obtain the final fused feature. The second target scale feature includes features of at least one scale from the multi-scale features of the denoised visible light image and infrared image.

3. The target detection method based on multi-source data fusion according to claim 2, characterized in that, The first target scale feature includes the first high-level feature in the multi-scale features of the denoised visible light image and the second high-level feature in the multi-scale features of the infrared image; The method of using the dual-source feature fusion layer to perform feature fusion on the first target scale features based on the visible light fusion weight and the infrared fusion weight to obtain intermediate fused features includes: The cross-modal attention matrix is ​​calculated using a normalization function based on the first and second high-level features; Based on the visible light fusion weight, infrared fusion weight, and cross-modal attention matrix, the first high-level feature and the second high-level feature are fused to obtain the intermediate fused feature.

4. The target detection method based on multi-source data fusion according to any one of claims 1-3, characterized in that, The strong light interference index includes a global strong light interference index used to indicate the overall level of strong light interference in a visible light image; The visible light fusion weight and the infrared fusion weight are calculated using the following formulas 1 and 2: Official 1 Official 2 in, For visible light fusion weights, For infrared fusion weights, This represents the global strong light interference index.

5. The target detection method based on multi-source data fusion according to any one of claims 1-3, characterized in that, The strong light interference index includes a global strong light interference index, which indicates the overall strong light interference level of a visible light image, and a local strong light interference index, which indicates the strong light interference level of each sub-region in a visible light image. The assessment of strong light interference in the visible light image of the target scene to obtain a strong light interference index, which indicates the degree of strong light interference in the visible light image, includes: Divide the visible light image into multiple sub-regions; Calculate the local strong light interference index for each sub-region; Based on the infrared image, a preliminary detection is performed to determine whether each sub-region contains an infrared target candidate region, and the sub-weight of each sub-region is determined based on the detection results. The global strong light interference index of the visible light image is calculated based on the local strong light interference index and sub-weight of the sub-region.

6. The target detection method based on multi-source data fusion according to claim 5, characterized in that, The calculation of the local strong light interference index for each sub-region includes: Calculate the average brightness of each sub-region; The local strong light interference index of sub-regions with average brightness not higher than the low strong light threshold is set to 0. The local strong light interference index of the sub-region with an average brightness not lower than the high intensity light threshold is defined as 1; If the average brightness is greater than the low intensity light threshold and less than the high intensity light threshold, the brightness standard deviation of the corresponding sub-region is calculated. Based on the average brightness, the brightness standard deviation, the low intensity light threshold, the high intensity light threshold, and the preset maximum brightness standard deviation, the local intensity light interference index of the sub-region is calculated.

7. The target detection method based on multi-source data fusion according to any one of claims 1-3 and 6, characterized in that, The total loss function of the object detection model during training is shown in Formula 3 below: Formula 3 in, For the total loss function, For bounding box regression loss, For category loss, For confidence loss, q is a preset weight value greater than 1. This indicates that the corresponding sub-region is a region of strong light interference. This indicates pixels located in areas of strong light interference. This indicates a pixel located in an area free from strong light interference.

8. A target detection device based on multi-source data fusion, characterized in that, The target detection device based on multi-source data fusion includes: A strong light interference assessment module is used to assess the strong light interference of a visible light image in a target scene and obtain a strong light interference index to indicate the degree of strong light interference in the visible light image. The strong light interference index includes a global strong light interference index to indicate the overall degree of strong light interference in the visible light image and a local strong light interference index to indicate the degree of strong light interference in each sub-region of the visible light image. The noise reduction module is used to filter and denoise the visible light image based on the edge features of the infrared image in the target scene and the strong light interference index using a guided filtering algorithm, so as to obtain a denoised visible light image. The feature extraction module is used to extract multi-scale features from the denoised visible light image and infrared image using the feature extraction layer in the target detection model. A fusion weight determination module is used to determine the visible light fusion weight and the infrared fusion weight based on the strong light interference index; The fusion module is used to perform feature fusion on the extracted multi-scale features based on the visible light fusion weight and the infrared fusion weight, using the feature fusion layer in the target detection model to obtain the final fused features. The target detection module is used to perform target detection based on the final fused features using the detection head of the target detection model, and obtain the target detection result.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the computer device to execute the target detection method based on multi-source data fusion as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the target detection method based on multi-source data fusion as described in any one of claims 1-7.

Images