Cross-modal enhanced object detection method suitable for complex lighted road scene

Abstract

The present application relates to a kind of complex illumination road scene is suitable for cross-modal enhancement target detection method.It includes: obtaining the multiple modal image group to be detected under road scene;The multiple modal image group to be detected is loaded to the multiple modal target detection model of pre-construction, and the multiple modal target detection information corresponding to the multiple modal image group to be detected is generated by the multiple modal target detection model to the multiple modal image group to be detected with target detection, wherein, when the multiple modal image group to be detected is carried out target detection, at least the visible light target scale feature, infrared light target scale feature obtained by feature extraction is cross-modal fusion.The present application reduces the influence of background redundancy interaction and local mismatch on recognition while suppressing the modal characteristic noise transmission and irrelevant thermal response in complex illumination road scene, improves the target detection precision in complex illumination road scene, strengthens the robustness of target detection in night, weak light, shielding and complex background.

Description

Technical Field

[0001] This invention relates to a target detection method, and more particularly to a cross-modal enhanced target detection method suitable for complex lighting road scenes. Background Technology

[0002] Road object detection is a fundamental visual task in autonomous driving assistance, intelligent transportation, and road monitoring. With the increasing popularity of autonomous driving technology, road object detection, as a core component of this technology, has become a focus of research in complex lighting scenarios such as nighttime, backlighting, strong headlight illumination, local overexposure, and alternating light and dark conditions.

[0003] In target detection, visible light images, based on the reflection of ambient light, can provide the target's color, texture, and complete outline. However, in complex road lighting scenarios such as nighttime, strong headlights, road surface reflections, tunnel entrances and exits, local backlighting, and alternating light and dark conditions, the information obtained from a single image is often insufficient to reliably support road target detection.

[0004] The impact of complex lighting on visible light images is not simply a reduction in brightness, but rather an imaging perturbation with significant spatial structure. For example, low illumination at night weakens the texture response of distant pedestrians, cyclists, and vehicle edges; oncoming headlights and streetlight reflections can create local overexposure, bright patches, and glare areas in the image; building shadows, vehicle occlusion, and abrupt changes in road brightness can lead to unstable contrast between target outlines and background. The presence of these phenomena can cause problems such as local missing evidence of target appearance, blurred boundaries, or enhanced interference from background highlights in visible light images.

[0005] Unlike visible light images, infrared images are based on the thermal radiation of the target, and can still highlight the thermal response of road targets such as pedestrians and vehicles at night and under low light conditions. However, the infrared response is not a direct representation of the complete appearance of the target. It is greatly affected by the target's temperature distribution, material emissivity, distance, and ambient heat sources. For example, the infrared response of pedestrians is usually concentrated in local areas such as the head and torso, and the boundaries of clothing and limbs may not be clear. The prominent infrared areas of vehicles may be concentrated in the engine compartment, tires, windows, or exhaust, rather than the complete outline of the vehicle body. Streetlights, building facades, heat-reflecting areas, and other unrelated heat sources may also produce strong local responses.

[0006] In complex lighting road scenes, the effective detection evidence for the same pedestrian, vehicle, or bicycle target in infrared and visible light images does not completely correspond. Instead, it exhibits scale correlation, inconsistent response regions, and local asymmetric distribution. Specifically, visible light images are more likely to preserve target contours, textures, and appearance boundaries, but are susceptible to interference from low illumination, glare, shadows, and reflections; infrared images are more likely to highlight the target's thermal response, but the thermal response region is often concentrated locally and may be affected by unrelated heat sources. If the detection network cannot effectively model the response differences, common responses, and contextual relationships between the two modalities, it is prone to problems such as missed detection of small targets at long distances, false detections in bright light areas, and detection boxes biased towards local thermal responses or highlighted areas.

[0007] To improve the reliability of target detection, current methods often employ the fusion of infrared and visible light. However, existing methods for fusing infrared and visible light still have certain shortcomings, specifically: 1) Simple addition, splicing, or fixed-weight fusion makes it difficult to determine which mode is more reliable in different regions, and easily introduces interference such as headlight glare, road surface reflection, building shadows, or irrelevant heat sources into the fusion features at the same time.

[0008] 2) Although the cross-attention method can establish the correlation between two modalities, if there is a lack of cross-modal response relationship constraints for complex lighting scenes, it is easy to perform indiscriminate interaction on a large number of road background locations, resulting in large computational overhead, and may use modality-specific interference as effective complementary information in the aggregation.

[0009] Furthermore, the target scale varies significantly in complex lighting road scenes. The receptive fields required for distant pedestrians and bicycles, medium-distance vehicles, and large targets at close range are not the same. If high-resolution features without scale enhancement and spatial filtering are directly used to construct cross-modal relation queries, road textures, building edges, road surface reflections, and irrelevant thermal responses will occupy a large amount of representation space, making subsequent cross-modal fusion more susceptible to background interference. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a cross-modal enhanced target detection method suitable for complex lighting road scenes. While suppressing modal-specific noise transmission and irrelevant thermal response in complex lighting road scenes, it reduces the impact of background redundant interaction and local mismatch recognition, improves the target detection accuracy in complex lighting road scenes, and enhances the robustness of target detection in nighttime, low light, occlusion and complex background.

[0011] According to the technical solution provided by the present invention, a cross-modal enhanced target detection method suitable for complex lighting road scenes is provided, the target detection method comprising: Acquire a multimodal image set to be inspected in a road scene, wherein the multimodal image set to be inspected includes a visible light image to be inspected and an infrared light image to be inspected that is synchronously acquired and generated with the visible light image to be inspected; A set of multimodal images to be detected is loaded into a pre-constructed multimodal target detection model, and the multimodal target detection model performs target detection on the set of images to be detected to generate multimodal target detection information corresponding to the set of images to be detected. When performing target detection on the multimodal image group to be inspected, at least the visible light target scale features and infrared light target scale features obtained by feature extraction should be fused across modes to generate target scale cross-modal fusion features, and multimodal target detection information should be generated based on the cross-modal fusion features. Cross-modal fusion includes at least sequentially performing multi-scale prior-guided spatial feature screening and relation modeling processing, and relation query-guided deformable cross-modal fusion processing, and generating target-scale cross-modal fusion features through deformable cross-modal fusion processing.

[0012] The spatial feature filtering and relationship modeling process includes sequential prior augmentation spatial filtering and spatial relationship modeling. When performing prior enhancement spatial filtering, prior scale-guided enhancement filtering is performed on the visible light target scale features and infrared light target scale features to generate visible light enhancement filtering features and infrared light enhancement filtering features with significant scale enhancement position response. Subsequently, corresponding position codes are added to the visible light enhancement filtering features and infrared light enhancement filtering features to generate visible light feature sequences and infrared light feature sequences. When performing spatial relationship modeling, relationship query guidance is performed based on visible light feature sequences and infrared light feature sequences to generate visible light guided queries and infrared light guided queries.

[0013] The deformable cross-modal fusion process includes sequential deformable aggregation and feature fusion processing, wherein... When performing deformable aggregation processing, infrared light guided query is used to perform deformable aggregation operation on visible light feature sequence, and visible light guided query is used to perform variable aggregation operation on infrared light feature sequence, so as to generate visible light sampling aggregation feature and infrared light sampling aggregation feature respectively. During feature fusion processing, the residual of visible light sampling aggregation features is injected into the visible light target scale features to generate visible light enhanced features; and the residual of infrared light sampling aggregation features is injected into the infrared light target scale features to generate infrared light enhanced features. Visible light enhancement features are fused with infrared light enhancement features to generate cross-modal fused features.

[0014] When performing prior enhancement spatial filtering on visible light target-scale features, the process includes sequential multi-scale prior enhancement processing, coordinate attention mechanism processing, and spatial feature filtering processing. When performing multi-scale prior enhancement, the visible light target scale features are segmented into multiple visible light channel sub-features along the channel dimension. Then, the target structure is extracted under different receptive fields for the segmented visible light channel sub-features to generate visible light target structure features corresponding to each visible light channel sub-feature. All visible light target structural features and visible light target scale features are sequentially spliced ​​and integrated through channels to generate visible light multi-scale fused features; When performing coordinate attention mechanism processing, coordinate attention recalibration operation is performed on visible light multi-scale fusion features to generate visible light multi-scale prior enhancement features; When performing spatial feature filtering, the visible light multi-scale prior enhancement features are subjected to average pooling and max pooling respectively to generate visible light average pooling features and visible light max pooling features respectively. Then, the visible light average pooling features and visible light max pooling features are fused through learnable weights to generate visible light enhanced fused features. The visible light enhanced fusion feature is flattened to generate a visible light enhanced screening feature.

[0015] When performing coordinate attention recalibration on visible light multi-scale fused features, the following steps are included: Horizontal pooling and vertical pooling are performed on the visible light multi-scale fusion features to generate visible light horizontal pooling features and visible light vertical pooling features, respectively. The visible light horizontal pooling feature and the visible light vertical pooling feature are fused to generate the visible light horizontal and vertical pooling fused feature; Standard nonlinear mapping is applied to the visible light horizontal and vertical pooling fusion features to generate visible light nonlinear mapping features; The nonlinear mapping features of visible light are segmented according to spatial dimensions and attention weights are calculated to generate horizontal attention weights and vertical attention weights for visible light, respectively. The visible light horizontal attention weights, visible light vertical attention weights, and visible light multi-scale fusion features are multiplied element-wise to generate visible light multi-scale prior enhancement features.

[0016] When generating visible light-guided queries and infrared light-guided queries, the following are included: Based on visible light and infrared light feature sequences, multidimensional cross-modal response features of visible light and infrared light are constructed. When constructing multi-dimensional cross-modal response features of visible light, at least the visible light feature sequence, the infrared light feature sequence, the response difference sequence between visible light and infrared light, and the common response sequence between visible light and infrared light should be spliced ​​along the channel dimension. When constructing the multidimensional cross-modal response characteristics of infrared light, at least the infrared light feature sequence, the visible light feature sequence, the response difference sequence between infrared light and visible light, and the common response sequence between infrared light and visible light should be spliced ​​along the channel dimension. The query correction amount and gating weight are calculated for the visible light multidimensional cross-modal response features to obtain the visible light query correction amount and visible light gating weight respectively. Based on the visible light query correction amount and visible light gating weight, the visible light corrected gating feature is generated. Then, the visible light feature sequence and the visible light corrected gating feature are added element by element to generate the infrared light guided query. The query correction amount and gating weight are calculated for the multidimensional cross-modal response features of infrared light to generate the infrared light query correction amount and the infrared light gating weight respectively. Based on the infrared light query correction and the infrared light gating weight, the infrared light corrected gating feature is calculated and generated. Then, the infrared light feature sequence and the infrared light corrected gating feature are added element by element to generate the visible light guided query.

[0017] The visible light feature sequence is subtracted from the infrared light feature sequence element by element to generate the response difference sequence between visible light and infrared light; The visible light feature sequence and the infrared light feature sequence are multiplied element-wise to generate a common response sequence for visible and infrared light.

[0018] The visible light multidimensional cross-modal response characteristics are processed using a visible light second multilayer perceptron to calculate the visible light query correction. The visible light multidimensional cross-modal response features are processed using a visible light first-layer perceptron and a visible light sigmoid function to calculate the visible light gating weights.

[0019] When generating visible light sampling aggregation features, the following are included: Based on infrared light-guided queries, the infrared-guided sampling offset and infrared-guided sampling weight are calculated and generated. The visible light feature sequence is sampled and aggregated using infrared guided sampling offset and infrared guided sampling weight to generate visible light relationship enhancement features; The visible light relationship enhancement features are reconstructed into a two-dimensional visible light feature map, and then upsampling interpolation and channel projection are used to generate visible light sampling aggregate features. The visible light sampling aggregate features and the visible light target scale features have the same spatial size and number of channels.

[0020] When performing target detection on the multimodal image group to be detected using the multimodal target detection model, the feature extraction network within the multimodal target detection model is first used to extract features from the visible light image and the infrared light image to be detected, respectively, to generate visible light P3 scale features, visible light P4 scale features, visible light P5 scale features, infrared light P3 scale features, infrared light P4 scale features, and infrared light P5 scale features. When performing cross-modal fusion, at least the visible light P5 scale features and the infrared light P5 scale features should be used as visible light target scale features and infrared light target scale features, respectively. The multimodal target detection model further includes a neck network and a detection head network adapted and connected to the neck network, wherein... The target-scale cross-modal fusion features generated by cross-modal fusion are loaded into the neck network, and the detection head network generates multimodal target detection information. The multimodal target detection information includes visible light target detection information and / or infrared light target detection information.

[0021] The advantages of this invention are as follows: During cross-modal fusion, at least the following steps are performed sequentially: multi-scale prior-guided spatial feature screening and relationship modeling processing, and relationship query-guided deformable cross-modal fusion processing. After deformable cross-modal fusion processing, target-scale cross-modal fusion features are generated. After multi-scale prior-guided spatial feature screening and relationship modeling processing, and relationship query-guided deformable cross-modal fusion processing, the multi-modal target detection model can complete adaptive deformable aggregation in the target-related region, thereby achieving feature enhancement under multi-dimensional cross-modal modeling guidance. This can improve the target detection accuracy in complex lighting road scenes and enhance the robustness of target detection in nighttime, low light, occlusion, and complex backgrounds. Attached Figure Description

[0022] Figure 1 This is a schematic flowchart of one embodiment of the target detection method of the present invention.

[0023] Figure 2 This is a structural block diagram of one embodiment of the multimodal target detection model of the present invention.

[0024] Figure 3 This is a structural block diagram of one embodiment of the multi-scale relation prior fusion module of the present invention.

[0025] Figure 4 This is a structural block diagram of one embodiment of the cross-modal relationship modeling and query module of the present invention.

[0026] Figure 5 This is a schematic diagram of an embodiment of the multi-scale prior-guided spatial feature module of the present invention.

[0027] Figure 6 This is a structural block diagram of one embodiment of the coordinate attention module of the present invention.

[0028] Figure 7 This is a schematic diagram of one embodiment of the present invention for generating multimodal target detection information. Detailed Implementation

[0029] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0030] To suppress modal-specific noise propagation and irrelevant thermal responses in complex lighting road scenes, reduce the impact of background redundancy interactions and local mismatch recognition, improve target detection accuracy in complex lighting road scenes, and enhance robustness of target detection in nighttime, low-light, occluded, and complex background conditions, this invention provides a cross-modal enhanced target detection method suitable for complex lighting road scenes. Specifically, the target detection method includes: Acquire a multimodal image set to be inspected in a road scene, wherein the multimodal image set to be inspected includes a visible light image to be inspected and an infrared light image to be inspected that is synchronously acquired and generated with the visible light image to be inspected; A set of multimodal images to be detected is loaded into a pre-constructed multimodal target detection model, and the multimodal target detection model performs target detection on the set of images to be detected to generate multimodal target detection information corresponding to the set of images to be detected. When performing target detection on the multimodal image group to be inspected, at least the visible light target scale features and infrared light target scale features obtained by feature extraction should be fused across modes to generate target scale cross-modal fusion features, and multimodal target detection information should be generated based on the cross-modal fusion features. Cross-modal fusion includes at least sequentially performing multi-scale prior-guided spatial feature screening and relation modeling processing, and relation query-guided deformable cross-modal fusion processing, and generating target-scale cross-modal fusion features through deformable cross-modal fusion processing.

[0031] Depend on Figure 1 As can be seen, when performing target detection, a set of multimodal images to be detected should be acquired. This set of images can be acquired under complex lighting conditions, such as in a road scene. Alternatively, it can be acquired under simple lighting conditions. Generally, the set of images to be detected should include one visible light image and one infrared image. The visible light and infrared images should be acquired synchronously. The method of acquiring these images can be consistent with existing technologies and will not be elaborated here.

[0032] After acquiring the multimodal image group to be inspected, the multimodal image group to be inspected should be loaded into the multimodal target detection model so that the multimodal target detection model can perform target detection on the multimodal image group to be inspected. Here, the meaning of target detection on the multimodal image group to be inspected is consistent with the meaning of target detection in the general sense of this technical field, that is, to detect and determine the target objects contained in the visible light image and the infrared light image to be inspected. Therefore, after target detection, the multimodal target detection model can output the corresponding multimodal target detection information.

[0033] It should be understood that the multimodal target detection information includes visible light target detection information and / or infrared light target detection information. Similar to existing technologies, the visible light target detection information may include the location, type, and confidence level of each target object within the visible light image to be detected. Similarly, the infrared light target detection information may include the location, type, and confidence level of each target object within the infrared light image to be detected. Here, location refers to its position within the image, and type refers to the type of target object; for example, in a road scene, the type could be pedestrian, car, or bicycle.

[0034] To suppress modal-specific noise propagation and irrelevant thermal response in complex lighting road scenes, reduce the impact of background redundancy and local mismatch recognition, and improve target detection accuracy in complex lighting road scenes, in one embodiment of the present invention, cross-modal fusion should be performed when using a multimodal target detection model for target detection. Specifically, before performing cross-modal fusion, feature extraction should be performed by the multimodal target detection model, and the visible light target scale features and infrared light target scale features obtained from the feature extraction should be fused cross-modally to generate target scale cross-modal fusion features. Subsequently, within the multimodal target detection model, multimodal target detection information can be generated and output using the target scale cross-modal fusion features. The method of generating multimodal target detection information using target scale cross-modal fusion features will be described in detail below.

[0035] In one embodiment of the present invention, cross-modal fusion includes at least sequentially performing multi-scale prior-guided spatial feature screening relationship modeling processing and relationship query-guided deformable cross-modal fusion processing, and generating target-scale cross-modal fusion features through deformable cross-modal fusion processing. It should be noted that after multi-scale prior-guided spatial feature screening relationship modeling processing and relationship query-guided deformable cross-modal fusion processing, the multi-modal target detection model can complete adaptive deformable aggregation in the target-related region, thereby achieving feature enhancement under multi-dimensional cross-modal modeling guidance, which can improve the target detection accuracy in complex lighting road scenes and enhance the robustness of target detection in nighttime, low light, occlusion, and complex backgrounds.

[0036] In one embodiment of the present invention, when performing target detection on the multimodal image group to be detected using a multimodal target detection model, the feature extraction network within the multimodal target detection model is first used to extract features from the visible light image and the infrared light image to be detected, respectively, so as to generate visible light P3 scale features, visible light P4 scale features, visible light P5 scale features, infrared light P3 scale features, infrared light P4 scale features, and infrared light P5 scale features, respectively. When performing cross-modal fusion, at least the visible light P5 scale features and the infrared light P5 scale features should be used as visible light target scale features and infrared light target scale features, respectively. The multimodal target detection model further includes a neck network and a detection head network adapted and connected to the neck network, wherein... The target-scale cross-modal fusion features generated by cross-modal fusion are loaded into the neck network, and the detection head network generates multimodal target detection information. Figure 2 The figure illustrates an embodiment of the multimodal target detection model of the present invention. As shown in the figure, the multimodal target detection model may include a visible light feature extraction module, an infrared light feature extraction module, a neck network, and a detection head network. The visible light feature extraction module and the infrared light feature extraction module may constitute a feature extraction network. Figure 2 In this context, NECK stands for neck network, and Detect1, Detect2, and Detect3 are three detectors contained within the detector head network. These three detectors enable the detection of targets of different sizes. In practice, the visible light feature extraction module and the infrared light feature extraction module can use the dual-stream backbone network in YOLOv5, and the neck network here can also be the neck network of YOLOv5.

[0037] During target detection, the visible light image to be detected should be loaded into the visible light feature extraction module, and the infrared light image to be detected should be loaded into the infrared light feature extraction module. The visible light feature extraction module will then extract features from the visible light image to be detected, generating visible light P3 scale features, visible light P4 scale features, and visible light P5 scale features. Similarly, after the infrared light feature extraction module extracts features from the infrared light image to be detected, it can obtain infrared light P3 scale features, infrared light P4 scale features, and infrared light P5 scale features. The meanings of the P3, P4, and P5 scales are consistent with existing definitions and will not be elaborated here.

[0038] In specific implementation, at least the visible light P5 scale features and the infrared light P5 scale features can be used as visible light target scale features and infrared light target scale features respectively, that is, at least the visible light P5 scale features and infrared light P5 scale features can be fused across modes. Figure 2The document illustrates an embodiment that simultaneously uses visible light P4 scale features, visible light P3 scale features, infrared light P4 scale features, and infrared light P3 scale features as visible light target scale features and infrared light target scale features, respectively. Figure 2 The MDPF in the present invention is the deformable cross-modal fusion network that performs the above-mentioned cross-modal fusion. Each deformable cross-modal fusion network should be connected to the neck network.

[0039] It should be noted that if the visible light P4 scale features and infrared light P4 scale features are not used as the visible light target scale features and infrared light target scale features of this invention, the visible light P4 scale features and infrared light P4 scale features should generally be fused. In this case, other cross-modal fusion forms besides the deformable cross-modal fusion of this invention can be used. Similarly, the situation where the visible light P3 scale features and infrared light P3 scale features are not used as the corresponding target scale features can be obtained. For details, please refer to the description here.

[0040] As can be seen from the above description, the principle of target detection by the multimodal target detection model of the present invention is consistent with the prior art. For example, features are first extracted from the visible light image and the infrared light image to be detected. Then, the fused cross-modal features are loaded into the neck network. After processing by the neck network, the results are finally output through the detection head network. Figure 7 The diagram illustrates an embodiment of using a detector head network to display multimodal target detection information. It should be noted that the fused cross-modal features here can be the target-scale cross-modal fusion features mentioned above, or cross-modal fusion features generated using other cross-modal fusion methods.

[0041] Figure 7 In this document, the original image (RGB) specifically refers to the acquired visible light image to be inspected; the RGB before improvement specifically refers to the result of target detection on the visible light image to be inspected using existing target detection methods; the IR before improvement specifically refers to the result of target detection on the infrared light image to be inspected using existing target detection methods; the RGB after improvement specifically refers to the result of target detection on the visible light image to be inspected using the target detection method of this invention; and the IR after improvement specifically refers to the result of target detection on the infrared light image to be inspected using the target detection method of this invention.

[0042] In one embodiment of the present invention, the spatial feature filtering and relationship modeling process includes sequentially performing prior augmentation spatial filtering and spatial relationship modeling processes, wherein... When performing prior enhancement spatial filtering, prior scale-guided enhancement filtering is performed on the visible light target scale features and infrared light target scale features to generate visible light enhancement filtering features and infrared light enhancement filtering features with significant scale enhancement position response. Subsequently, corresponding position codes are added to the visible light enhancement filtering features and infrared light enhancement filtering features to generate visible light feature sequences and infrared light feature sequences. When performing spatial relationship modeling, relationship query guidance is performed based on visible light feature sequences and infrared light feature sequences to generate visible light guided queries and infrared light guided queries.

[0043] As explained above, when performing spatial feature screening and relationship modeling processing within cross-modal fusion, it is necessary to... Figure 2 Within the deformable cross-modal fusion network mentioned in the text, Figure 3 An embodiment of a deformable cross-modal fusion network is shown in the figure. Figure 3 In the diagram, MPSF1 and MPSF2 are both multi-scale prior-guided spatial feature modules, and ADD1 and ADD2 are adders that perform element-wise addition. V For visible light target scale characteristics, F i For infrared targets, the feature dimension is C. For visible targets, the feature dimension is C. V ×H×W, the feature dimension of the infrared target scale feature is C. i ×H×W,C V To represent the number of channels for visible light target scale characteristics, C i To represent the number of channels for infrared target scale features, H represents the height of the visible light target scale features and the infrared light target scale features, and W represents the width of the visible light target scale features and the infrared light target scale features.

[0044] During operation, the spatial feature module MPSF1 performs prior enhancement spatial filtering on the scale features of visible light targets, generating visible light enhancement filtering features with significant scale enhancement position responses. Then, adder ADD1 adds these visible light enhancement filtering features element-wise to their corresponding position codes to generate a visible light feature sequence. Simultaneously, the spatial feature module MPSF2 performs prior enhancement spatial filtering on the scale features of infrared light targets, generating infrared light enhancement filtering features with significant scale enhancement position responses. Then, adder ADD2 adds these infrared light enhancement filtering features element-wise to their corresponding position codes to generate an infrared light feature sequence.

[0045] Figure 3The CRQ module is a cross-modal relationship modeling query module. Spatial relationship modeling processing can be performed using the cross-modal relationship modeling query module. During operation, the visible light feature sequence and infrared light feature sequence generated above should be loaded into the cross-modal relationship modeling query module, and the cross-modal relationship modeling query module will output visible light guided query and infrared light guided query.

[0046] In one embodiment of the present invention, the prior enhancement spatial filtering process for visible light target scale features includes sequentially performing multi-scale prior enhancement processing, coordinate attention mechanism processing, and spatial feature filtering processing, wherein... When performing multi-scale prior enhancement, the visible light target scale features are segmented into multiple visible light channel sub-features along the channel dimension. Then, the target structure is extracted under different receptive fields for the segmented visible light channel sub-features to generate visible light target structure features corresponding to each visible light channel sub-feature. All visible light target structural features and visible light target scale features are sequentially spliced ​​and integrated through channels to generate visible light multi-scale fused features; When performing coordinate attention mechanism processing, coordinate attention recalibration operation is performed on visible light multi-scale fusion features to generate visible light multi-scale prior enhancement features; When performing spatial feature filtering, the visible light multi-scale prior enhancement features are subjected to average pooling and max pooling respectively to generate visible light average pooling features and visible light max pooling features respectively. Then, the visible light average pooling features and visible light max pooling features are fused through learnable weights to generate visible light enhanced fused features. The visible light enhanced fusion feature is flattened to generate a visible light enhanced screening feature.

[0047] In practice, spatial feature modules MPSF1 and MPSF2 can generally adopt the same form, although they can also adopt different forms. Preferably, spatial feature modules MPSF1 and MPSF2 adopt the same form. Figure 5 The diagram shows one possible structural form for the spatial feature modules MPSF1 and MPSF2. The working principle of the spatial feature module MPSF1 of the present invention will be explained below using the example of prior enhancement spatial screening processing of visible light target scale features. The method and process of prior enhancement spatial screening processing of infrared light target scale features by the spatial feature module MPSF2 can be referred to here.

[0048] Figure 5In this process, the channel segmentation module (split) is used to segment the visible light target scale features into multiple visible light channel sub-features along the channel dimensions. For example, if the visible light target scale features have 128 channel dimensions, four visible light channel sub-features can be obtained. Each visible light channel sub-feature includes features corresponding to 32 channels. The number of visible light channel sub-features obtained can be selected as needed, and will not be illustrated here. After obtaining the visible light channel sub-features, target structure extraction should be performed under different receptive fields to generate visible light target structure features respectively.

[0049] Figure 5 The document illustrates one embodiment of dividing the visible light target scale features into four visible light channel sub-features, which can be: When extracting target structural features under different receptive fields, the visible light channel sub-features are mainly processed by convolutional kernels of different receptive fields. This addresses the problem of significant differences in target scale in complex lighting road scenes by extracting target structural information of different ranges. In this way, a scale prior containing small target details, medium-scale structure, and a large-scale context can be established for each modality before entering cross-modal interaction.

[0050] When convolutional kernels with different receptive fields are used to process the sub-features of the visible light channels, then:

[0051] in, Indicates the first The kernel size of each convolutional branch. This represents the multi-scale prior features of the corresponding branch output, which is also the visible light target structure feature of the output.

[0052] As explained above, after the input features are divided into multiple sub-features along the channel dimension, the kernel sizes of the corresponding convolutional branches should not be exactly the same. Figure 5 The figure shows one embodiment with four convolutional branches. In the figure, Conv3, Conv4, Conv5 and Conv6 are four convolutional branches. The corresponding convolutional kernels of the four convolutional branches can be 3×3, 5×5, 7×7 and 7×7 respectively. The size of the convolutional kernel can be selected according to actual needs and will not be elaborated here.

[0053] Depend on Figure 5It can be seen that after extracting the target structure from the visible light channel sub-features using convolution kernels with different receptive fields, the channel stitcher Concat4 should be used to stitch the visible light target scale features and all visible light target structural features. Then, convolution is performed using a 1×1 convolution kernel Conv7 to achieve channel integration. After the convolution operation of convolution kernel Conv7, visible light multi-scale fusion features can be generated. Specifically, the visible light multi-scale fusion features can be represented as:

[0054] in, This represents the multi-scale fusion characteristics of visible light. This indicates the channel integration operation performed using the Conv7 convolution kernel. This indicates channel splicing performed by the channel splicer Concat4. This represents the scale characteristics of visible light targets.

[0055] Figure 5 The CA in the text refers to the coordinate attention mechanism unit. The coordinate attention mechanism unit can be used to perform coordinate attention mechanism processing. When performing coordinate attention mechanism processing, the visible light multi-scale fusion features generated by the convolution kernel Conv7 should be received. The coordinate attention mechanism unit can be used to recalibrate the target-related regions to obtain visible light multi-scale prior enhancement features. Coordinate attention enables target regions such as pedestrians, vehicles and bicycles to obtain higher responses and suppresses the interference of road texture, building edges and irrelevant thermal responses on subsequent cross-modal interactions.

[0056] In one embodiment of the present invention, when performing coordinate attention recalibration on visible light multi-scale fusion features, the method includes: Horizontal pooling and vertical pooling are performed on the visible light multi-scale fusion features to generate visible light horizontal pooling features and visible light vertical pooling features, respectively. The visible light horizontal pooling feature and the visible light vertical pooling feature are fused to generate the visible light horizontal and vertical pooling fused feature; Standard nonlinear mapping is applied to the visible light horizontal and vertical pooling fusion features to generate visible light nonlinear mapping features; The nonlinear mapping features of visible light are segmented according to spatial dimensions and attention weights are calculated to generate horizontal attention weights and vertical attention weights for visible light, respectively. The visible light horizontal attention weights, visible light vertical attention weights, and visible light multi-scale fusion features are multiplied element-wise to generate visible light multi-scale prior enhancement features.

[0057] Figure 6The figure illustrates one embodiment of the coordinate attention mechanism unit. As shown in the figure, when performing coordinate attention recalibration, horizontal pooling and vertical pooling should be performed on the visible light multi-scale fusion features respectively, such as... Figure 6 In this process, the visible light multi-scale fusion features are subjected to horizontal pooling and vertical pooling respectively through horizontal pooling and vertical pooling modules. For example, when the feature dimension of the visible light multi-scale fusion features is C×H×W, the feature dimension of the visible light horizontal pooling features generated after horizontal pooling is C×H×1, while the feature dimension of the visible light vertical pooling features generated after vertical pooling is C×1×W.

[0058] Figure 6 In this process, the visible light horizontal pooling feature and the visible light vertical pooling feature are spliced ​​along the spatial / positional dimension using the channel splicer Concat5 to form a visible light horizontal and vertical pooling fusion feature, wherein the feature dimension of the visible light horizontal and vertical pooling fusion feature is C×(H+W)×1. Figure 6 In this context, CBR stands for Standard Nonlinear Mapping Module. It is used to perform standard nonlinear mapping on the visible light horizontal and vertical pooling fusion features to generate visible light nonlinear mapping features. In specific implementation, the standard nonlinear mapping module can be composed of a convolutional module (Conv) + batch normalization (BN) + ReLU activation function. Of course, the standard nonlinear mapping module can also take other forms, which will not be illustrated here.

[0059] Depend on Figure 6 It can be seen that after generating the visible light nonlinear mapping features, the first step is to segment according to the spatial dimension, that is... Figure 6 The `split` representation in the code divides the data along the spatial dimension; the spatial division can be found in the explanations above. After spatial division, attention weights should be calculated separately for each segment. Figure 6 After spatial dimension segmentation, two visible light nonlinear mapping channel sub-features can be obtained. Attention weights are calculated for the two visible light nonlinear mapping channel sub-features respectively to generate visible light horizontal attention weights and visible light vertical attention weights.

[0060] In practice, attention weights can be calculated through sequential convolution and sigmoid computation. Figure 6 In the diagram, Conv8 and Conv9 represent convolutional layers that perform convolutional processing, and Sg3 and Sg4 represent sigmoid functions that can perform sigmoid calculations. For example, after processing through convolutional layer Conv8 and sigmoid function Sg3 in sequence, visible light horizontal attention weights can be generated, while after processing through convolutional layer Conv9 and sigmoid function Sg4, visible light vertical attention weights can be generated.

[0061] Depend on Figure 6 It can be seen that after generating the visible light horizontal attention weights and visible light vertical attention weights, the visible light horizontal attention weights and visible light vertical attention weights should be multiplied element-wise with the visible light multi-scale fusion features to generate visible light multi-scale prior enhancement features. For the generated visible light multi-scale prior enhancement features, we have:

[0062] in, This refers to the visible light multi-scale prior enhancement features output by the coordinate attention mechanism unit. This indicates the coordinate attention recalibration operation being performed. This indicates element-wise multiplication. For visible light level attention weights, For visible light vertical attention weights.

[0063] Figure 6 In this context, DC3 is a multiplier capable of performing element-wise multiplication. That is, using the multiplier DC3, the visible light multi-scale fusion features can be multiplied element-wise with the visible light horizontal attention weights and the visible light vertical attention weights.

[0064] After coordinate attention recalibration and generation of visible light multi-scale prior enhancement features, spatial feature filtering should be performed. Figure 5 It can be seen that during the spatial feature filtering process, average pooling and max pooling are performed on the visible light multi-scale prior enhancement features to generate visible light average pooling features and visible light max pooling features, respectively. Then, the visible light average pooling features and visible light max pooling features are fused using learnable weights to generate visible light enhanced fused features. Therefore, the generated visible light enhanced fused features can be expressed as:

[0065] in, For visible light enhanced fusion features, , For learnable weights, This is an average pooling process performed on the multi-scale prior enhancement features of visible light. This is a max pooling process performed on the multi-scale prior enhancement features of visible light.

[0066] Figure 5 In the process, the visible light average pooling features and visible light max pooling features are fused using learnable weights through the adder ADD7. Furthermore... Figure 5The flattening operation in this context refers to the flattening of visible light enhanced fusion features, which generates visible light enhanced filtering features. It should be noted that the flattening method can be consistent with existing technologies, and will not be elaborated upon here. As can be seen from the generation method of visible light enhanced fusion features, it can reduce road background redundancy while retaining key responses used to determine target location and scale. This achieves prior processing of target scale, target coordinates, and significant responses for each modality before cross-modal interaction, providing more reliable input for subsequent deformable cross-modal sampling aggregation.

[0067] As explained above, the visible light enhancement screening feature is the result of the spatial feature module MPSF1 performing prior enhancement spatial screening processing on the visible light target scale features. Figure 3 It is evident that after obtaining the visible light enhanced screening features, positional encoding should also be added. Figure 3 In this process, the adder ADD1 performs element-wise addition of the position code and the visible light enhancement screening features to generate a visible light feature sequence. The generated visible light feature sequence can be:

[0068] in, It is a visible light characteristic sequence. Flattening operation performed by visible light enhancement fusion features This is a positional encoding corresponding to visible light.

[0069] It should be understood that, referring to the above-described method for generating visible light feature sequences, infrared light feature sequences can be generated; the specific generation process will not be elaborated here. Figure 1 It is known that the generated visible light feature sequence and infrared light feature sequence should be loaded into the cross-modal relationship modeling query module so that the cross-modal relationship modeling query module can generate visible light guided query and infrared light guided query.

[0070] Complex lighting road scenes not only cause overall image darkening or local overexposure, but more importantly, they can split the effective detection evidence for the same target in visible and infrared images: the contour, texture, and boundary information preserved in the visible light branch may be interfered with by glare, shadows, and road surface reflections, while the thermal response in the infrared branch may be concentrated in local areas of the target, accompanied by irrelevant heat source responses. After generating visible and infrared guided queries, these queries can be used to guide subsequent deformable sampling, enabling selective cross-modal aggregation of corresponding features in infrared and visible light at target-related locations. This effectively suppresses modal-specific noise propagation and irrelevant thermal responses in complex lighting road scenes, reducing the impact of background redundancy and local mismatch recognition.

[0071] In one embodiment of the present invention, generating visible light guided queries and infrared light guided queries includes: Based on visible light and infrared light feature sequences, multidimensional cross-modal response features of visible light and infrared light are constructed. When constructing multi-dimensional cross-modal response features of visible light, at least the visible light feature sequence, the infrared light feature sequence, the response difference sequence between visible light and infrared light, and the common response sequence between visible light and infrared light should be spliced ​​along the channel dimension. When constructing the multidimensional cross-modal response characteristics of infrared light, at least the infrared light feature sequence, the visible light feature sequence, the response difference sequence between infrared light and visible light, and the common response sequence between infrared light and visible light should be spliced ​​along the channel dimension. The query correction amount and gating weight are calculated for the visible light multidimensional cross-modal response features to obtain the visible light query correction amount and visible light gating weight respectively. Based on the visible light query correction amount and visible light gating weight, the visible light corrected gating feature is generated. Then, the visible light feature sequence and the visible light corrected gating feature are added element by element to generate the infrared light guided query. The query correction amount and gating weight are calculated for the multidimensional cross-modal response features of infrared light to generate the infrared light query correction amount and the infrared light gating weight respectively. Based on the infrared light query correction and the infrared light gating weight, the infrared light corrected gating feature is calculated and generated. Then, the infrared light feature sequence and the infrared light corrected gating feature are added element by element to generate the visible light guided query.

[0072] Figure 4 The image illustrates one embodiment of a cross-modal relationship modeling query module, which can generate visible light-guided queries and infrared light-guided queries. Figure 4 It is evident that when generating visible light-guided queries and infrared light-guided queries, the visible light multidimensional cross-modal response features and infrared light multidimensional cross-modal response features should be constructed first. Figure 4 The image illustrates one embodiment for constructing visible light multidimensional cross-modal response characteristics and infrared light multidimensional cross-modal response characteristics. Figure 4 In constructing the multidimensional cross-modal response characteristics of visible light, a sequence combination consisting of visible light feature sequence, infrared light feature sequence, response difference sequence between visible and infrared light, and common response sequence between visible and infrared light should be constructed first. The response difference sequence between visible and infrared light can be generated by subtracting the visible light feature sequence from the infrared light feature sequence using the subtractor JF1. The common response sequence between visible and infrared light can be generated by multiplying the visible light feature sequence from the infrared light feature sequence using the element-wise multiplier CF1.

[0073] Depend on Figure 4As can be seen from the above description, when constructing the multidimensional cross-modal response characteristics of infrared light, a sequence combination consisting of infrared light feature sequence, visible light feature sequence, response difference sequence between infrared and visible light, and common response sequence between infrared and visible light should be constructed first. Among them, the subtractor JF2 can be used to perform element-wise subtraction between the infrared light feature sequence and the visible light feature sequence to generate the response difference sequence between infrared and visible light; the element-wise multiplier CF2 can be used to perform element-wise multiplication between the infrared light feature sequence and the visible light feature sequence to generate the common response sequence between infrared and visible light.

[0074] As can be seen from the above calculation method of response difference sequence, response difference sequence can highlight the position and channel where the infrared light and visible light response are inconsistent, while common response sequence can be used to preserve target areas or stable structural areas where both modes respond strongly at the same time.

[0075] Depend on Figure 4 It can be seen that by using the channel splicer Concat2 to splice together sequences formed by visible light feature sequences, infrared light feature sequences, response difference sequences between visible and infrared light, and common response sequences between visible and infrared light, multidimensional cross-modal response features of visible light can be generated. Furthermore, the channel splicer Concat3 can generate multidimensional cross-modal response features of infrared light. Figure 4 middle, This refers to the multi-dimensional cross-modal response characteristics of visible light. For the multidimensional cross-modal response characteristics of infrared light, we have:

[0076] in, The response difference sequence is generated by subtracting the visible light feature sequence from the infrared light feature sequence element by element. This is the common response sequence generated by performing element-wise multiplication of the visible light feature sequence and the infrared light feature sequence. The characterization is achieved by splicing together the visible light feature sequence, the infrared light feature sequence, the response difference sequence between visible and infrared light, and the common response sequence between visible and infrared light along the channel dimension.

[0077] As can be seen from the above description, the constructed visible light multidimensional cross-modal response features and infrared light multidimensional cross-modal response features not only include modal difference information, but also the context of the two modes and the common response sequence, which provides favorable support for subsequent guidance.

[0078] When generating infrared light-guided queries, the query correction amount and gating weight should be calculated based on the visible light multidimensional cross-modal response characteristics. Specifically, the visible light second multilayer perceptron is used to process the visible light multidimensional cross-modal response characteristics to calculate the visible light query correction amount; the visible light first multilayer perceptron and the visible light Sigmoid function are used to process the visible light first multilayer perceptron to calculate the visible light gating weight.

[0079] Figure 4 In the diagram, MLP2 is the second multilayer perceptron for visible light, MLP1 is the first multilayer perceptron for visible light, and Sg1 is the visible light sigmoid function. As shown in the figure, by processing the multidimensional cross-modal response characteristics of visible light using MLP2, the visible light query correction can be obtained. The expression for calculating the visible light query correction is as follows: ,in, This represents the calculation of the query correction amount performed by the multilayer perceptron. This is the visible light query correction. Similarly, for the calculated visible light gating weights, we have: ,in, For visible light gating weights, This indicates the gating weight calculation performed by the multilayer perceptron.

[0080] Referring to the above calculations of visible light query correction and visible light gating weight, it can be seen that... Figure 4 In this diagram, MLP4 is the second multilayer perceptron for infrared light, MLP3 is the first multilayer perceptron for infrared light, and Sg2 is the sigmoid function for infrared light. For the calculation of the infrared light query correction, we have: Similarly, for infrared light gating weights, we have: .

[0081] It should be noted that, as can be seen from the above calculation of query correction and gating weight, the Sigmoid function can limit the gating weight to between 0 and 1. The query correction and gating weight are used to selectively calibrate the two modal relationships, so that the subsequently generated visible light guided query and infrared light guided query pay more attention to the target-related cross-modal response relationship.

[0082] Depend on Figure 4 It can be seen that after calculating the visible light query correction and the visible light gating weight, the visible light query correction and the visible light gating weight are multiplied element by element using multiplier DC1 to generate the visible light correction gating feature. Similarly, the infrared light query correction and the infrared light gating weight can be multiplied element by element using multiplier DC2 to generate the infrared light correction gating feature.

[0083] Figure 4 middle, Infrared light guides the query. When calculating and generating infrared-guided queries and visible-light-guided queries for visible light-guided queries, the following applies:

[0084] in, , These are learnable scaling coefficients. Figure 4 In this process, the element-by-element addition in the above correction is achieved through adders ADD5 and ADD6.

[0085] In one embodiment of the present invention, the deformable cross-modal fusion processing includes sequentially performing deformable aggregation processing and feature fusion processing, wherein... When performing deformable aggregation processing, infrared light guided query is used to perform deformable aggregation operation on visible light feature sequence, and visible light guided query is used to perform variable aggregation operation on infrared light feature sequence, so as to generate visible light sampling aggregation feature and infrared light sampling aggregation feature respectively. When performing feature fusion processing, residual injection is performed on visible light target-scale features using visible light sampling aggregation features to generate visible light enhanced features; and residual injection is performed on infrared light target-scale features using infrared light sampling aggregation features to generate infrared light enhanced features. Visible light enhancement features are fused with infrared light enhancement features to generate cross-modal fused features.

[0086] Depend on Figure 3 It is understood that after generating infrared light-guided queries and visible light-guided queries, the visible light-guided queries and infrared light feature sequences should be loaded into a deformable cross-modal sampling module. In this case, the infrared light feature sequence serves as the Value, and the visible light-guided query serves as the Query. Simultaneously, the infrared light-guided query and visible light feature sequences are loaded into another deformable cross-modal sampling module. In this case, the visible light feature sequence serves as the Value, and the infrared light-guided query serves as the Query. Each deformable cross-modal sampling module performs deformable cross-modal fusion processing. The deformable cross-modal sampling module of this invention can adopt a form commonly used in this technical field, and can be selected according to needs, prioritizing the ability to perform deformable cross-modal fusion processing. That is, each deformable cross-modal sampling module can form a branch for performing deformable cross-modal fusion processing.

[0087] For a deformable cross-modal sampling module that receives visible light guided queries and infrared light feature sequences, infrared light sampling aggregated features are generated after performing deformable aggregation processing. Similarly, for a deformable cross-modal sampling module that receives infrared light guided queries and visible light feature sequences, corresponding visible light sampling aggregated features are generated after performing deformable aggregation processing.

[0088] In one embodiment of the present invention, generating visible light sampling aggregation features includes: Based on infrared light-guided queries, the infrared-guided sampling offset and infrared-guided sampling weight are calculated and generated. The visible light feature sequence is sampled and aggregated using infrared guided sampling offset and infrared guided sampling weight to generate visible light relationship enhancement features; The visible light relationship enhancement features are reconstructed into a two-dimensional visible light feature map, and then upsampling interpolation and channel projection are used to generate visible light sampling aggregate features. The visible light sampling aggregate features and the visible light target scale features have the same spatial size and number of channels.

[0089] Specifically, when generating the infrared guided sampling offset and the infrared guided sampling weight, we have:

[0090] in, This is the infrared-guided sampling offset. For infrared guided sampling weights, This indicates the calculation of the sampling offset. This indicates the calculation of the sampling guidance weights.

[0091] It should be noted that the position and aggregation intensity of subsequent cross-modal feature sampling can be controlled by the infrared guided sampling offset and infrared guided sampling weight. When sampling and aggregating visible light feature sequences using the infrared guided sampling offset and infrared guided sampling weight to generate visible light relationship enhancement features, then:

[0092] in, Enhanced features for visible light relationships, Indicates the current sampling reference position. Indicates the number of sampling points. Indicates the first The offset of each sampling point This indicates the weight of the corresponding sampling point.

[0093] It should be noted that the vector dimension of the visible light enhancement screening feature generated after flattening is N×d, where N is H×W, d is the feature dimension at each position, and each spatial position within the visible light enhancement screening feature corresponds to a reference sampling position. The number of sampling points can generally be set as needed. This indicates that the k-th sampling point is based on the sampling reference position. Similarly, the predicted offset can be obtained. The corresponding meaning.

[0094] As can be seen from the above description, when using infrared light to guide queries for deformable sampling, the source of the sampled features is not directly changed. Instead, the sampling offset and aggregation weight are generated by using the multidimensional cross-modal response relationship between infrared and visible light. This guides the corresponding modal features to perform adaptive deformable aggregation in the target-related region and generates corresponding relationship enhancement features.

[0095] Furthermore, in complex lighting road scenes, visible light and infrared features exhibit both common responses and local asymmetric responses to the same target. Simple addition or splicing cannot distinguish complementary differences and modality-specific interference. Ordinary cross-attention, lacking relational constraints and spatial filtering, easily performs indiscriminate interactions on a large number of background areas and may include overexposed textures, shadow responses, or irrelevant thermal responses as effective complementary information in the aggregation. Therefore, by constraining deformable cross-modal sampling processes through infrared-guided queries and visible light-guided queries, cross-modal interactions are more concentrated on target-related regions. This reduces the interference of redundant background regions and modality-specific noise on fused features, enhances the effective features of target boundaries, thermally significant regions, and weakly textured targets, and ultimately improves the positioning accuracy and small target detection stability under complex lighting conditions.

[0096] When generating visible light sampling aggregation features, we have:

[0097] in, For the generated visible light sampling aggregation features, This indicates that the visible light relation enhancement features are rearranged into a two-dimensional feature map. This indicates an upsampling interpolation operation used to restore the spatial resolution to the scale features of a visible light target. This represents a channel projection operation used to align the number of channels to the number of channels in a visible light target-scale feature. Through this reconstruction projection operation, deformable sampling aggregation results can be residually fused with the visible light target-scale feature.

[0098] Referring to the above explanation, we can obtain Figure 3 In This refers to infrared light sampling aggregation features. The method for generating infrared light sampling aggregation features can be referred to the above explanation, and will not be repeated here. As can be seen from the explanation of the generation methods of infrared light sampling aggregation features and visible light sampling aggregation features: for the visible light enhancement direction, infrared light guiding query can guide visible light features to perform deformable sampling aggregation around the target-related region; for the infrared light enhancement direction, visible light guiding query guides infrared features to perform adaptive spatial aggregation in the same way, thereby forming feature enhancement guided by cross-modal relationships, avoiding background interference caused by fixed-position fusion and global indiscriminate interaction.

[0099] Depend on Figure 3It can be seen that after generating the visible light sampling aggregation feature, the visible light target scale feature can be added element-wise using adder ADD3 to obtain the visible light enhancement correction feature. For the visible light enhancement correction feature, we have:

[0100] in, For visible light enhancement correction features, These are learnable correction parameters.

[0101] Similarly, the adder ADD4 can be used to perform element-wise addition between the infrared target scale features and the infrared sampling aggregation features to obtain the infrared enhancement correction features. For the infrared enhancement correction features, we have:

[0102] in, For infrared light enhancement correction features, These are learnable correction parameters.

[0103] Depend on Figure 3 It can be seen that the visible light enhancement correction features are convolved by a 1×1 convolutional layer Conv1, and the infrared light enhancement correction features are convolved by a 1×1 convolutional layer Conv2. Then, the channel stitcher Concat1 concatenates the convolutional results to generate cross-modal fusion features at the target scale. Figure 3 middle, This refers to the target-scale cross-modal fusion feature, and the feature dimensions of the target-scale cross-modal fusion feature are: .

[0104] As can be seen from the above method of generating target-scale cross-modal fusion features, the generated target-scale cross-modal fusion features not only retain the target edge, contour and texture details in the visible light image to be inspected, but also enhance the thermal response and target saliency information in the infrared light image to be inspected, thereby providing more stable and discriminative multimodal features for subsequent detection heads.

[0105] For the multimodal target detection model described above, a feasible construction method is as follows: A basic model for multimodal object detection is constructed, corresponding to the multimodal object detection model. A model training dataset is also constructed for training the basic model. The model training dataset includes several training samples and sample labels. Each training sample includes one frame of training visible light image and one frame of training infrared light image. The sample labels are used to label the information of the target present in the training visible light image and the training infrared light image. The target information may include the type and location of the target. The meaning of type, location, etc. can be referred to the corresponding explanation above.

[0106] Configure the model training parameters, and train the basic multimodal object detection model using the model training dataset under the configured model training parameters. When the training reaches the target state, the basic multimodal object detection model is configured as the multimodal object detection model.

[0107] It is understandable that the basic multimodal object detection model is a multimodal object detection model that has not been trained, while the multimodal object detection model is a target model formed after training the basic multimodal object detection model. Therefore, the basic multimodal object detection model can be constructed by referring to the above description of the multimodal object detection model.

[0108] The model training dataset can be constructed using the publicly available FLIR dataset, which consists of real-world images collected under various weather and lighting conditions. This dataset emphasizes diverse traffic environments (urban areas, highways) and a wide variety of traffic scenarios, including a training set of 4129 pairs of infrared-visible images and a validation set of 1013 pairs of infrared-visible images. The method for constructing the model training dataset using the FLIR dataset will not be detailed here.

[0109] The configured model training parameters may include: mosaic probability set to 1.0, horizontal flip probability set to 0.5, input image size set to 640×640, number of training epochs set to 60, and batch size set to 8. The optimizer uses the SGD stochastic gradient descent algorithm, with an initial learning rate set to 0.01, momentum factor set to 0.937, and weight decay coefficient set to 0.0005. Furthermore, the configured model training parameters should also include a training loss function, which can adopt the loss function commonly used in object detection in YOLOv5 models, and will not be elaborated here. During model training, the training is considered to have reached the target state when the training loss function stabilizes or when the set number of training epochs is reached.

[0110] To verify the target detection performance of the multimodal target detection model of this invention, the multimodal target detection model of this invention was compared with CFT and EI. 2 The infrared-visible target detection methods such as Det, TarDAL, and ICA Fusion were compared, with mAP50, mAP50:95, and AP values ​​for pedestrians, bicycles, and cars as the comparison metrics. Under consistent experimental conditions, the comparison results are shown in Table 1.

[0111] Table 1

[0112] Among the comparative models mentioned above, CFT (Cross-Modality Fusion Transformer for Multispectral Object Detection) fuses bimodal global context features through Transformer self-attention; EI²Det (Edge-Guided Illumination-Aware Interactive Learning for Visible-Infrared Object Detection) achieves bimodal feature fusion through cross-modal interaction, illumination-aware weighting, and edge guidance; MMI-Det (Exploring Multi-Modal Integration for Visible and Infrared Object Detection) enhances the complementary representation of visible light and infrared features through multimodal information interaction and fusion guidance mechanisms; and ICAFSion (Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection) gradually enhances bimodal complementary features through iterative interaction methods involving bidirectional cross-attention and parameter sharing.

[0113] It should be noted that in the above comparison, "-" indicates that the corresponding model did not detect the corresponding target or did not have the corresponding subdivision results.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-modal augmented target detection method suitable for complex lighting road scenes, characterized in that, The target detection method includes: Acquire a multimodal image set to be inspected in a road scene, wherein the multimodal image set to be inspected includes a visible light image to be inspected and an infrared light image to be inspected that is synchronously acquired and generated with the visible light image to be inspected; A set of multimodal images to be detected is loaded into a pre-constructed multimodal target detection model, and the multimodal target detection model performs target detection on the set of images to be detected to generate multimodal target detection information corresponding to the set of images to be detected. When performing target detection on the multimodal image group to be inspected, at least the visible light target scale features and infrared light target scale features obtained by feature extraction should be fused across modes to generate target scale cross-modal fusion features, and multimodal target detection information should be generated based on the cross-modal fusion features. When performing cross-modal fusion, it includes at least sequentially performing multi-scale prior-guided spatial feature screening relationship modeling processing and relationship query-guided deformable cross-modal fusion processing, so that the multi-modal target detection model can complete adaptive deformable aggregation in the target-related region and generate target-scale cross-modal fusion features through deformable cross-modal fusion processing; When performing spatial feature filtering relationship modeling, it includes prior augmented spatial filtering processing for prior organization and achieving significant scale-enhanced location response, and spatial relationship modeling processing for generating guided queries, and the prior augmented spatial filtering processing and spatial relationship modeling processing are performed sequentially. When performing deformable cross-modal fusion processing, it includes deformable aggregation processing and feature fusion processing for cross-modal feature fusion. Specifically, when performing deformable aggregation processing, deformable sampling aggregation is performed for the visible light enhancement direction, and adaptive spatial aggregation is performed for the infrared light enhancement direction.

2. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 1, characterized in that, When performing prior enhancement spatial filtering, prior scale-guided enhancement filtering is performed on the visible light target scale features and infrared light target scale features to generate visible light enhancement filtering features and infrared light enhancement filtering features with significant scale enhancement position response. Subsequently, corresponding position codes are added to the visible light enhancement filtering features and infrared light enhancement filtering features to generate visible light feature sequences and infrared light feature sequences. When performing spatial relationship modeling, relationship query guidance is performed based on visible light feature sequences and infrared light feature sequences to generate visible light guided queries and infrared light guided queries.

3. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 2, characterized in that, When performing deformable aggregation processing, infrared light guided query is used to perform deformable aggregation operation on visible light feature sequence, and visible light guided query is used to perform variable aggregation operation on infrared light feature sequence, so as to generate visible light sampling aggregation feature and infrared light sampling aggregation feature respectively. During feature fusion processing, the residual of visible light sampling aggregation features is injected into the visible light target scale features to generate visible light enhanced features; and the residual of infrared light sampling aggregation features is injected into the infrared light target scale features to generate infrared light enhanced features. Visible light enhancement features are fused with infrared light enhancement features to generate cross-modal fused features.

4. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 2, characterized in that, When performing prior enhancement spatial filtering on visible light target-scale features, the process includes sequential multi-scale prior enhancement processing, coordinate attention mechanism processing, and spatial feature filtering processing. When performing multi-scale prior enhancement, the visible light target scale features are segmented into multiple visible light channel sub-features along the channel dimension. Then, the target structure is extracted under different receptive fields for the segmented visible light channel sub-features to generate visible light target structure features corresponding to each visible light channel sub-feature. All visible light target structural features and visible light target scale features are sequentially spliced ​​and integrated through channels to generate visible light multi-scale fused features; When performing coordinate attention mechanism processing, coordinate attention recalibration operation is performed on visible light multi-scale fusion features to generate visible light multi-scale prior enhancement features; When performing spatial feature filtering, the visible light multi-scale prior enhancement features are subjected to average pooling and max pooling respectively to generate visible light average pooling features and visible light max pooling features respectively. Then, the visible light average pooling features and visible light max pooling features are fused through learnable weights to generate visible light enhanced fused features. The visible light enhanced fusion feature is flattened to generate a visible light enhanced screening feature.

5. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 4, characterized in that, When performing coordinate attention recalibration on visible light multi-scale fused features, the following steps are included: Horizontal pooling and vertical pooling are performed on the visible light multi-scale fusion features to generate visible light horizontal pooling features and visible light vertical pooling features, respectively. The visible light horizontal pooling feature and the visible light vertical pooling feature are fused to generate the visible light horizontal and vertical pooling fused feature; Standard nonlinear mapping is applied to the visible light horizontal and vertical pooling fusion features to generate visible light nonlinear mapping features; The nonlinear mapping features of visible light are segmented according to spatial dimensions and attention weights are calculated to generate horizontal attention weights and vertical attention weights for visible light, respectively. The visible light horizontal attention weights, visible light vertical attention weights, and visible light multi-scale fusion features are multiplied element-wise to generate visible light multi-scale prior enhancement features.

6. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 2, characterized in that, When generating visible light-guided queries and infrared light-guided queries, the following are included: Based on visible light and infrared light feature sequences, multidimensional cross-modal response features of visible light and infrared light are constructed. When constructing multi-dimensional cross-modal response features of visible light, at least the visible light feature sequence, the infrared light feature sequence, the response difference sequence between visible light and infrared light, and the common response sequence between visible light and infrared light should be spliced ​​along the channel dimension. When constructing the multidimensional cross-modal response characteristics of infrared light, at least the infrared light feature sequence, the visible light feature sequence, the response difference sequence between infrared light and visible light, and the common response sequence between infrared light and visible light should be spliced ​​along the channel dimension. The query correction amount and gating weight are calculated for the visible light multidimensional cross-modal response features to obtain the visible light query correction amount and visible light gating weight respectively. Based on the visible light query correction amount and visible light gating weight, the visible light corrected gating feature is generated. Then, the visible light feature sequence and the visible light corrected gating feature are added element by element to generate the infrared light guided query. The query correction amount and gating weight are calculated for the multidimensional cross-modal response features of infrared light to generate the infrared light query correction amount and the infrared light gating weight respectively. Based on the infrared light query correction and the infrared light gating weight, the infrared light corrected gating feature is calculated and generated. Then, the infrared light feature sequence and the infrared light corrected gating feature are added element by element to generate the visible light guided query.

7. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 6, characterized in that, The visible light feature sequence is subtracted from the infrared light feature sequence element by element to generate the response difference sequence between visible light and infrared light; The visible light feature sequence and the infrared light feature sequence are multiplied element-wise to generate a common response sequence for visible and infrared light.

8. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 6, characterized in that, The visible light multidimensional cross-modal response characteristics are processed using a visible light second multilayer perceptron to calculate the visible light query correction. The visible light multidimensional cross-modal response features are processed using a visible light first-layer perceptron and a visible light sigmoid function to calculate the visible light gating weights.

9. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to claim 3, characterized in that, When generating visible light sampling aggregation features, the following are included: Based on infrared light-guided queries, the infrared-guided sampling offset and infrared-guided sampling weight are calculated and generated. The visible light feature sequence is sampled and aggregated using infrared guided sampling offset and infrared guided sampling weight to generate visible light relationship enhancement features; The visible light relationship enhancement features are reconstructed into a two-dimensional visible light feature map, and then upsampling interpolation and channel projection are used to generate visible light sampling aggregate features. The visible light sampling aggregate features and the visible light target scale features have the same spatial size and number of channels.

10. The cross-modal enhanced target detection method suitable for complex lighting road scenes according to any one of claims 1 to 9, characterized in that, When performing target detection on the multimodal image group to be detected using the multimodal target detection model, the feature extraction network within the multimodal target detection model is first used to extract features from the visible light image and the infrared light image to be detected, respectively, to generate visible light P3 scale features, visible light P4 scale features, visible light P5 scale features, infrared light P3 scale features, infrared light P4 scale features, and infrared light P5 scale features. When performing cross-modal fusion, at least the visible light P5 scale features and the infrared light P5 scale features should be used as visible light target scale features and infrared light target scale features, respectively. The multimodal target detection model further includes a neck network and a detection head network adapted and connected to the neck network, wherein... The target-scale cross-modal fusion features generated by cross-modal fusion are loaded into the neck network, and the detection head network generates multimodal target detection information. The multimodal target detection information includes visible light target detection information and / or infrared light target detection information.

Images