An RGB-infrared multi-modal adaptive fusion target detection method and system

Abstract

The application belongs to the field of target detection, and provides an RGB-infrared multi-modal adaptive fusion target detection method and system, which synchronously acquires RGB image data and infrared image data in a traffic scene, and performs preprocessing; multi-scale feature information in the RGB image and the infrared image is extracted respectively; modal contribution degree evaluation is performed on the extracted RGB multi-scale feature information and infrared multi-scale feature information respectively, the contribution degree of the two modes to the current detection task is obtained, and adaptive fusion weights are generated according to the contribution degree; based on the adaptive fusion weights, the feature information of different scales of the two modes is fused respectively, and the fused multi-modal feature is obtained; based on the multi-modal feature, a pre-trained target detection network is used to obtain a target detection result. The application improves the stability of the detection result in a complex traffic environment.

Description

Technical Field

[0001] This invention belongs to the field of target detection, specifically relating to an RGB-infrared multimodal adaptive fusion target detection method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of intelligent transportation systems and autonomous driving technology, accurate perception of road targets such as vehicles and pedestrians has become a crucial foundation for understanding the intelligent transportation environment. Currently, visual information-based target detection methods mainly rely on RGB-visible light images for target recognition. RGB images can provide rich color, texture, and structural information, and have good detection performance under good lighting conditions. However, in complex traffic environments such as nighttime, low light, backlight, rain, and fog, visible light imaging is easily affected by changes in external lighting, leading to reduced image brightness, blurred target edges, and loss of key texture information, thus causing a decrease in target detection accuracy.

[0004] Infrared thermal imaging technology can image targets using their own thermal radiation information, maintaining good target saliency even in low-light or no-light environments, effectively compensating for the shortcomings of RGB images in low-light conditions. However, infrared images primarily reflect the target's thermal distribution information, lacking detailed features such as color and texture, thus limiting their ability to distinguish target categories, handle complex backgrounds, and accurately locate targets. Therefore, relying solely on either RGB or infrared images is insufficient to meet the requirements for stable detection of pedestrians, vehicles, and other targets in complex traffic environments.

[0005] Existing RGB-infrared multimodal fusion detection methods typically improve target detection capabilities in complex environments by fusing RGB and infrared information. However, most existing methods employ a fixed fusion approach, where RGB and infrared features are fused according to a fixed structure or ratio after model training. This fails to dynamically adjust the contribution of each type of information based on different targets, scales, and modal feature quality. When the information quality of a particular modality deteriorates due to environmental factors, the fixed fusion approach can easily introduce invalid features or even cause intermodal information conflicts, reducing the fusion effect.

[0006] Furthermore, in complex traffic scenarios, the scale of targets varies significantly. Small targets such as distant pedestrians and small vehicles are easily affected by background interference, leading to the loss of detailed information during feature extraction and fusion. Meanwhile, RGB and infrared images contain significantly different feature information due to their different imaging mechanisms. Existing methods effectively extract complementary information from these two modalities and achieve efficient fusion at different scales. Summary of the Invention

[0007] To address the aforementioned problems, this invention proposes an RGB-infrared multimodal adaptive fusion target detection method and system. This invention can achieve effective synergy between the two modal features, thereby improving the stability of detection results in complex traffic environments.

[0008] According to some embodiments, the present invention adopts the following technical solution: An RGB-infrared multimodal adaptive fusion target detection method includes the following steps: Simultaneously acquire RGB and infrared image data from traffic scenes and perform preprocessing; Multi-scale feature information is extracted from RGB and infrared images respectively; The modal contribution of the extracted RGB multi-scale feature information and infrared multi-scale feature information is evaluated to obtain the contribution of the two modes to the current detection task, and an adaptive fusion weight is generated based on the contribution. Based on the adaptive fusion weights, feature information of different scales of the two modalities is fused to obtain fused multimodal features; Based on the aforementioned multimodal features, a pre-trained target detection network is used to obtain target detection results.

[0009] As an alternative implementation method, the process of simultaneously acquiring RGB image data and infrared image data in a traffic scene includes: simultaneously acquiring the same traffic scene using RGB camera equipment and infrared thermal imaging equipment installed on vehicles, roadside, or intelligent monitoring equipment, and obtaining RGB images containing color, texture, and structural information, as well as infrared images containing target thermal radiation information.

[0010] As an optional implementation, the preprocessing process includes: sequentially performing image size adjustment, image noise removal, spatial registration of the RGB image and infrared image on the acquired RGB image data and infrared image data, and image normalization processing to ensure that the target position in the RGB image data is consistent with the corresponding target position in the infrared image data.

[0011] As an alternative implementation, the process of extracting multi-scale feature information from RGB images and infrared images respectively includes: performing feature extraction using RGB feature extraction branches and infrared feature extraction branches respectively, wherein the RGB feature extraction branch is used to extract the target's color, texture, edge and spatial structure information, and the infrared feature extraction branch is used to extract the target's thermal radiation distribution, contour information and salient region features.

[0012] As an alternative implementation method, the process of extracting multi-scale feature information from RGB images and infrared images respectively includes: the multi-scale feature information includes shallow features, medium features and deep features, wherein shallow features are used to describe target edges and texture details, medium features are used to describe target structure and local semantic information, and deep features are used to describe the overall semantic information of the target.

[0013] As an alternative implementation method, the process of evaluating the modal contribution of the extracted RGB multi-scale feature information includes: calculating the contribution of the texture, color and structural information contained in the RGB features to the target detection results.

[0014] As an alternative implementation method, the process of evaluating the modal contribution of the extracted infrared multi-scale feature information includes: calculating the contribution of the thermal radiation information and target contour information contained in the infrared features to the target detection results.

[0015] As an alternative implementation, the process of generating adaptive fusion weights based on the contribution level includes: the adaptive fusion weights are: ; ; in, Indicates the RGB feature fusion weights; Indicates the weights for infrared feature fusion; Indicates the contribution of RGB modes. This indicates the contribution of the infrared mode.

[0016] As an alternative implementation, based on the adaptive fusion weight, the process of fusing feature information of different scales of the two modalities to obtain fused multimodal features includes: for shallow features, fusing edge and texture detail information in RGB image data to improve the spatial localization capability of small targets; For mid-level features, enhance the information interaction between RGB features and infrared features to improve the ability to express the target structure; For deep features, high-level semantic information from both modalities is fused to improve target classification capabilities; Features at each scale are weighted and fused using adaptive fusion weights.

[0017] An RGB-infrared multimodal adaptive fusion target detection system includes: The dual-modal data acquisition and preprocessing module is used to simultaneously acquire RGB image data and infrared image data in traffic scenarios and perform preprocessing. A dual-branch multi-scale feature extraction module is used to extract multi-scale feature information from RGB images and infrared images respectively; The modal contribution evaluation and adaptive fusion weight generation module is used to evaluate the modal contribution of the extracted RGB multi-scale feature information and infrared multi-scale feature information respectively, to obtain the contribution degree of the two modes to the current detection task, and to generate adaptive fusion weights based on the contribution degree. The multi-scale modality contribution adaptive fusion module is used to fuse feature information of two modalities at different scales based on the adaptive fusion weights to obtain fused multi-modal features. The target detection output module is used to obtain target detection results based on the multimodal features and a pre-trained target detection network.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention establishes a dynamic feature fusion mechanism between RGB and infrared images, automatically adjusting the fusion ratio based on the expressive power of different modal features, so that the texture, color, and structural information in the visible light image and the thermal target information in the infrared image can be fully complementary. At the same time, it enhances the feature expression capability of targets of different sizes through multi-scale feature fusion, thereby improving the accuracy and stability of pedestrian, vehicle and other target detection in complex traffic scenarios.

[0019] This invention analyzes the features of RGB and infrared images and dynamically adjusts the fusion weights based on the effective information content of different modal features. This allows the texture, color, and structural information in the RGB image to be fully combined with the thermal radiation and target contour information in the infrared image, avoiding the problem of insufficient utilization of single-modal information in traditional fixed fusion methods and improving the multimodal feature fusion effect.

[0020] This invention employs a dual-modal input method of RGB and infrared, utilizing the complementary characteristics between the two imaging modes. When the RGB image is affected by factors such as low illumination or changes in lighting, resulting in weakened target texture information, the infrared image can provide additional target saliency information. At the same time, when the infrared image lacks color and detail information, the RGB image can provide auxiliary features, thereby improving the stability of target detection in complex scenes.

[0021] This invention uses a multi-scale feature fusion method to fuse RGB and infrared information in features at different levels. This allows shallow features to retain target edge and detail information, mid-level features to enhance target structural information, and deep features to improve the target's semantic expression ability, thereby improving the detection effect of targets such as pedestrians at a distance and small vehicles.

[0022] Because RGB images and infrared images have different imaging characteristics, they contain complementary information but may also have feature differences. This invention dynamically adjusts the fusion ratio of different modal features, allowing highly effective modal features to make a greater contribution to the fusion, reducing the interference of invalid information on the detection results, and improving the quality of fused features.

[0023] This invention combines a dynamic fusion strategy with the target detection process, enhancing the ability of fused features to describe traffic targets by fully utilizing RGB and infrared dual-modal information. Compared to single-modal detection methods and fixed fusion methods, this invention can obtain richer target feature information, improving the accuracy and robustness of pedestrian, vehicle, and other target detection.

[0024] This invention employs an RGB-infrared dual-modal adaptive fusion method. Its technical solution is not limited to a single traffic target detection scenario, but can also be applied to road monitoring, autonomous driving visual perception, intelligent robots, and other target detection scenarios that require multimodal visual information fusion.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a schematic diagram of a method flow according to one embodiment; Figure 2 This is a schematic diagram of the system structure of one embodiment; Figure 3 This is a schematic diagram illustrating the modal contribution evaluation and adaptive fusion weight generation process of one embodiment. Figure 4 This is a schematic diagram of a multi-scale dynamic fusion process in one embodiment; Figure 5 This is a schematic diagram of the target detection output process in one embodiment. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0032] Example 1 An RGB-infrared multimodal adaptive fusion target detection method, such as Figure 1 As shown, it includes the following steps: Simultaneously acquire RGB and infrared image data from traffic scenes and perform preprocessing; Multi-scale feature information is extracted from RGB and infrared images respectively; The modal contribution of the extracted RGB multi-scale feature information and infrared multi-scale feature information is evaluated to obtain the contribution of the two modes to the current detection task, and an adaptive fusion weight is generated based on the contribution. Based on the adaptive fusion weights, feature information of different scales of the two modalities is fused to obtain fused multimodal features; Based on the aforementioned multimodal features, a pre-trained target detection network is used to obtain target detection results.

[0033] In this embodiment, the process of simultaneously acquiring RGB image data and infrared image data in a traffic scene includes: simultaneously acquiring the same traffic scene using RGB camera equipment and infrared thermal imaging equipment installed on vehicles, roadside, or intelligent monitoring equipment, and obtaining RGB images containing color, texture, and structural information, as well as infrared images containing target thermal radiation information.

[0034] In this embodiment, the preprocessing process includes: sequentially adjusting the image size, removing image noise, spatially registering the RGB image and the infrared image, and normalizing the image to ensure that the target position in the RGB image data is consistent with the corresponding target position in the infrared image data.

[0035] In this embodiment, the process of extracting multi-scale feature information from RGB images and infrared images respectively includes: performing feature extraction using RGB feature extraction branches and infrared feature extraction branches respectively. The RGB feature extraction branch is used to extract the target's color, texture, edge, and spatial structure information, while the infrared feature extraction branch is used to extract the target's thermal radiation distribution, contour information, and salient region features.

[0036] Of course, in some embodiments, the extraction methods for RGB image features and infrared image features are not limited to a specific network structure. The dual-branch feature extraction module can be implemented using convolutional neural networks, residual networks, Transformer networks, visual attention networks, or other network structures with feature extraction capabilities.

[0037] In different application scenarios, different feature extraction methods can be selected according to computing resources, detection accuracy and real-time requirements. As long as RGB modal features and infrared modal features can be obtained respectively, they are all within the protection scope of this invention.

[0038] In this embodiment, the process of extracting multi-scale feature information from RGB images and infrared images respectively includes: multi-scale feature information includes shallow features, mid-level features and deep features, wherein shallow features are used to describe target edges and texture details, mid-level features are used to describe target structure and local semantic information, and deep features are used to describe the overall semantic information of the target.

[0039] In this embodiment, the process of evaluating the modal contribution of the extracted RGB multi-scale feature information includes: calculating the contribution of the texture, color, and structural information contained in the RGB features to the target detection result.

[0040] In this embodiment, the process of evaluating the modal contribution of the extracted infrared multi-scale feature information includes: calculating the contribution of the thermal radiation information and target contour information contained in the infrared features to the target detection results.

[0041] In this embodiment, the process of generating adaptive fusion weights based on the contribution level includes: the adaptive fusion weights are: ; ; in, Indicates the RGB feature fusion weights; Indicates the weights for infrared feature fusion; Indicates the contribution of RGB modes. This indicates the contribution of the infrared mode.

[0042] Of course, in other implementations, the fusion weights can be obtained through attention mechanisms, gating networks, neural network prediction models, modal credibility evaluation models, or other adaptive weight calculation methods.

[0043] For example, by analyzing the consistency, complementarity, and effective information content between RGB features and infrared features, corresponding fusion weights can be automatically generated to achieve dynamic adjustment of different modal information.

[0044] In this embodiment, based on the adaptive fusion weight, the process of fusing feature information of different scales of the two modalities to obtain the fused multimodal features includes: for shallow features, fusing edge and texture detail information in RGB image data to improve the spatial localization capability of small targets; For mid-level features, enhance the information interaction between RGB features and infrared features to improve the ability to express the target structure; For deep features, high-level semantic information from both modalities is fused to improve target classification capabilities; Features at each scale are weighted and fused using adaptive fusion weights.

[0045] In other implementations, feature splicing, feature weighted summation, feature mapping fusion, attention fusion, cross-scale feature interaction, or other multi-scale fusion methods can be used to achieve information fusion of RGB features and infrared features at different scales.

[0046] The method of dividing shallow, medium and deep features can also be adjusted according to the actual detection network structure.

[0047] Example 2 An RGB-infrared multimodal adaptive fusion target detection system, such as Figure 2 As shown, it includes: an RGB-infrared dual-modal data acquisition module, an image preprocessing module, a dual-branch feature extraction module, a dynamic fusion weight generation module, a multi-scale feature fusion module, and a target detection output module.

[0048] Through the collaborative work of the above modules, the effective fusion of RGB and infrared image information is achieved, enabling the system to dynamically adjust the fusion method according to different target features and different modal information quality, thereby improving the accuracy and stability of pedestrian, vehicle and other target detection in complex traffic environments.

[0049] RGB-Infrared Dual-Mode Data Acquisition Module This module is used to acquire RGB image data and infrared image data in traffic scenarios.

[0050] Specifically, RGB cameras and infrared thermal imaging devices installed on vehicles, roadside, or intelligent monitoring equipment are used to simultaneously capture data on the same traffic scene, obtaining RGB images containing color, texture, and structural information, as well as infrared images containing target thermal radiation information.

[0051] Among them, RGB images can provide rich target appearance information to describe vehicle color, target edges and road environment details; infrared images can provide stable target contour information at night, in low light or complex weather conditions to enhance target salience.

[0052] By simultaneously acquiring data from two modalities, a data foundation is provided for subsequent dual-modal fusion detection. The preprocessed RGB and infrared images maintain spatial correspondence, enabling effective matching of corresponding regions for different modalities during subsequent feature extraction. This provides a basis for subsequent modality contribution evaluation and adaptive fusion.

[0053] In other implementations, vehicle-mounted camera equipment, road monitoring equipment, drone vision equipment, robot vision equipment, or other equipment capable of simultaneously acquiring visible light and infrared information can be used.

[0054] Meanwhile, RGB and infrared images can be acquired synchronously or asynchronously after time calibration, as long as the RGB and infrared image information for the corresponding scene can be obtained.

[0055] Image preprocessing module: This module is used to preprocess the acquired RGB and infrared images to ensure that the two modal images meet the requirements for subsequent feature extraction and fusion.

[0056] Specifically, it includes: Image resizing; Image noise removal; Spatial registration of RGB and infrared images; Image normalization processing.

[0057] The image resizing, noise removal, spatial registration of RGB and infrared images, and image normalization processes described above are all implemented using existing image processing techniques, and this invention does not limit these techniques. The aim is to eliminate size differences, noise interference, and spatial position deviations between RGB and infrared images, ensuring that both modalities meet the requirements for subsequent feature extraction and fusion.

[0058] Since RGB camera devices and infrared thermal imaging devices use different imaging methods, the two images may have viewing angle deviations and spatial position offsets. Therefore, this module uses an image registration method to ensure that the target position in the RGB image is consistent with the corresponding position in the infrared image, so as to ensure that different modal features can be accurately matched in the subsequent fusion process.

[0059] Dual-branch multi-scale feature extraction module: This module is used to extract multi-level feature information from RGB and infrared images respectively, providing a foundation for subsequent modality contribution evaluation and adaptive fusion.

[0060] Specifically, the system is configured with RGB feature extraction branches and infrared feature extraction branches.

[0061] in: The RGB feature extraction branch is used to extract the target's color, texture, edge, and spatial structure information; The infrared feature extraction branch is used to extract the target's thermal radiation distribution, contour information, and salient region features.

[0062] Specifically, in this embodiment, for the RGB image feature extraction branch, while RGB images can provide rich information on target texture, color, and spatial structure, they are easily affected by environmental factors under conditions such as nighttime, strong light, shadow occlusion, and complex backgrounds, resulting in the loss of target detail information. During the RGB feature extraction process, a multi-scale feature enhancement structure is constructed to extract and fuse target feature information at different scales. This enables the network to simultaneously acquire shallow texture information and deep semantic information of the target, improving the RGB branch's feature extraction capability for targets of different sizes in complex traffic scenes.

[0063] For the infrared image feature extraction branch, since infrared images mainly rely on target thermal radiation information for imaging, they have good target perception capabilities in low-light environments, but suffer from insufficient texture details and background thermal noise interference. This embodiment addresses the characteristics of infrared images by introducing a target region enhancement mechanism during the infrared feature extraction process. By increasing the feature response intensity of the target region and reducing background information interference from non-target regions, the infrared branch can more accurately extract effective features of traffic targets such as vehicles and pedestrians.

[0064] Furthermore, to enhance the information complementarity between the RGB and infrared modes, this invention establishes cross-modal feature association during the dual-branch feature extraction process, and jointly constrains the feature information at corresponding positions of the two modes, so that the texture structure information in the RGB branch and the thermal radiation information in the infrared branch can complement each other, reducing the target missed detection and false detection problems caused by insufficient information in a single mode.

[0065] By using the above-mentioned RGB and infrared dual-branch feature extraction optimization method, this embodiment can enhance the ability of different modal images to express target features in complex traffic environments, improve the effectiveness of subsequent multimodal feature fusion, and provide more accurate feature information for traffic target detection in complex environments.

[0066] After dual-branch feature extraction, RGB modal multi-scale features and infrared modal multi-scale features were obtained, including: Shallow features: used to describe detailed information such as target edges and textures; Mid-level features: used to describe the target structure and local semantic information; Deep features: used to describe the overall semantic information of the target.

[0067] By acquiring dual-modal features at different scales, the system can evaluate the modal contribution of features at different scales and generate corresponding fusion weights based on the evaluation results, thereby improving the utilization efficiency of RGB and infrared information at different scales.

[0068] The modal contribution evaluation and adaptive fusion weight generation module is the core technology module of this embodiment. It is used to evaluate the contribution of RGB modal features and infrared modal features to the target detection task, and generate corresponding adaptive fusion weights based on the evaluation results.

[0069] Existing RGB-infrared fusion methods typically employ a fixed fusion approach, meaning that the same fusion ratio is used regardless of the scene or target conditions. This fails to adjust the information contribution based on changes in the quality of the RGB and infrared modal features, resulting in some effective information not being fully utilized.

[0070] This embodiment first evaluates the modal contribution of RGB features and infrared features respectively, to obtain the degree of contribution of the two modes to the current detection task, such as... Figure 3 As shown.

[0071] Among them, the RGB modality contribution represents the degree to which the texture, color, and structural information contained in the RGB features contribute to the target detection result; Infrared modal contribution indicates the degree to which the thermal radiation information and target contour information contained in the infrared features contribute to the target detection results.

[0072] We obtained the following respectively: RGB modal contribution: ; Infrared mode contribution: ; Subsequently, adaptive fusion weights are generated based on the contributions of the two modalities: ; ; in: Indicates the RGB feature fusion weights; This represents the weighting of infrared feature fusion.

[0073] After RGB-infrared dual-modal feature extraction, the effective target information contained in the RGB and infrared images differs under different environmental conditions. If a fixed ratio is used for feature fusion, some modal information may be weakened. Therefore, this embodiment designs a dynamic modal contribution calculation method that adaptively adjusts the fusion ratio according to the expressive power of different modal features in the current input image.

[0074] Specifically, for RGB modal features and infrared modal characteristics First, the characteristic response intensities of the two modes are calculated separately to evaluate the amount of effective information contained in the current mode.

[0075] For the A mode, whose characteristic response intensity is defined as: ; in, Indicates the first The first modal feature diagram line, number Column, No. Characteristic response values ​​of each channel, , , These represent the feature map height, width, and number of channels, respectively.

[0076] This indicator reflects the activation level of the current modality features. The stronger the feature response, the richer the target information contained in the modality.

[0077] Furthermore, to avoid the amplification of some noise features due to relying solely on the intensity of the feature response, this invention further introduces a detection feedback evaluation index.

[0078] The contribution of different modalities to the target detection task is calculated using the output results of the fusion detection network.

[0079] Define the effectiveness index of the test: ; in: This indicates that only the first one is used. The average accuracy of target detection using various modalities; Indicates the RGB single-mode detection accuracy; This indicates the accuracy of infrared single-mode detection.

[0080] This metric can measure the actual contribution of different modalities to the detection task.

[0081] Subsequently, the feature response index and the detection effectiveness index are fused to obtain the final modality contribution: ; in: Indicates the first Modal contribution; , The weighting adjustment parameter satisfies: ; in, Used to adjust the importance of the current modality's feature representation capability. Used to adjust the importance of historical test results.

[0082] For RGB mode: ; For infrared modes: ; To ensure that the contributions of the two modes satisfy the fusion constraints, they are normalized: ; ; The final dynamic fusion feature is obtained: ; Increase the RGB mode fusion weight when RGB features contain richer target texture and structural information; increase the infrared mode fusion weight when infrared features can provide more obvious target contours and thermal information.

[0083] By employing the above methods, the RGB and infrared modes can be adaptively adjusted according to their respective feature contributions, thereby improving the efficiency of dual-modal information utilization.

[0084] Multi-scale modal contribution adaptive fusion module: The system evaluates the contribution of RGB and infrared features at different scales and generates corresponding fusion weights based on the information contributions of the two modalities at different scales, achieving adaptive fusion in feature spaces at different scales. Because target sizes vary significantly in traffic environments—for example, distant pedestrians and small vehicles are smaller, while nearby vehicles are larger—single-scale fusion can easily lead to the loss of some target detail information.

[0085] In this embodiment, as Figure 4 As shown, a multi-scale dynamic fusion method is adopted: For shallow features: focus on fusing detailed information such as edges and textures in RGB images to improve the spatial localization capability of small targets; For mid-level features: enhance the information interaction between RGB and infrared features to improve the ability to express the target structure; For deep features: integrate high-level semantic information from two modalities to improve target classification capabilities.

[0086] For the first output of the feature extraction network The layer RGB features and infrared features are represented as follows: and ,in, This represents feature layers at different scales.

[0087] First, the feature response intensities of the RGB and infrared modes are calculated separately for features at different scales to obtain the modal contribution at the corresponding scales: ; ;in, Indicates the first RGB modal contribution at different scales Indicates the first Infrared modal contribution at the scale This represents the contribution calculation function based on feature response information.

[0088] Furthermore, based on the contributions of the two modalities at different scales, dynamic fusion weights for the corresponding scales are generated: , ; in, Indicates the first RGB modal fusion weights at different scales Indicates the first Infrared mode fusion weights at different scales.

[0089] Then, dynamic fusion is performed separately for features at different scales: ; in, Indicates the first Multimodal features after fusion at different scales.

[0090] The above method enables features at different scales to select more suitable modal information for fusion based on their own information characteristics. In the shallow feature fusion stage, the RGB modality contains richer texture, edge, and spatial detail information, thus providing more effective target structural features; in the deep feature fusion stage, the infrared modality can provide more stable target response information, which helps to enhance the target semantic feature expression capability under complex lighting conditions.

[0091] Finally, the multimodal features fused from different scales are input into the target detection module to achieve the classification and localization of targets such as vehicles and pedestrians in complex traffic environments.

[0092] By using multi-scale dynamic fusion, the RGB and infrared modes can play their maximum role at different feature levels.

[0093] The target detection output module is used to complete traffic target detection using fused multimodal features.

[0094] like Figure 5 As shown, the fused features are input into the target detection network to classify and locate targets in traffic scenes, and output: target category, target bounding box location, and detection confidence score. Detected targets include, but are not limited to: pedestrians, cars, non-motorized vehicles, and other road traffic targets.

[0095] Meanwhile, the system can adjust the fusion weights in the next frame image based on the feedback of the current detection results, so that the dynamic fusion process can be continuously optimized and the detection stability in continuous traffic scenarios can be improved.

[0096] In other embodiments, this system can also be applied to scenarios requiring multimodal visual perception, such as road monitoring, autonomous driving perception, intelligent robots, unmanned inspection, and industrial inspection.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An RGB-infrared multimodal adaptive fusion target detection method, characterized in that, Includes the following steps: Simultaneously acquire RGB and infrared image data from traffic scenes and perform preprocessing; Multi-scale feature information is extracted from RGB and infrared images respectively; The modal contribution of the extracted RGB multi-scale feature information and infrared multi-scale feature information is evaluated to obtain the contribution of the two modes to the current detection task, and an adaptive fusion weight is generated based on the contribution. Based on the adaptive fusion weights, feature information of different scales of the two modalities is fused to obtain fused multimodal features; Based on the multimodal features, a pre-trained target detection network is used to obtain target detection results.

2. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of simultaneously acquiring RGB image data and infrared image data in a traffic scene includes: simultaneously acquiring the same traffic scene using RGB camera equipment and infrared thermal imaging equipment installed on vehicles, roadside, or intelligent monitoring equipment, and obtaining RGB images containing color, texture, and structural information, as well as infrared images containing target thermal radiation information.

3. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The preprocessing process includes: sequentially adjusting the image size, removing image noise, spatially registering the RGB and infrared images, and normalizing the image data to ensure that the target position in the RGB image data is consistent with the corresponding target position in the infrared image data.

4. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of extracting multi-scale feature information from RGB and infrared images respectively includes: performing feature extraction using RGB feature extraction branches and infrared feature extraction branches respectively. The RGB feature extraction branch is used to extract the target's color, texture, edge and spatial structure information, while the infrared feature extraction branch is used to extract the target's thermal radiation distribution, contour information and salient region features.

5. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of extracting multi-scale feature information from RGB and infrared images respectively includes: multi-scale feature information includes shallow features, mid-level features and deep features. Shallow features are used to describe target edges and texture details, mid-level features are used to describe target structure and local semantic information, and deep features are used to describe the overall semantic information of the target.

6. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of evaluating the modal contribution of the extracted RGB multi-scale feature information includes: calculating the contribution of the texture, color and structural information contained in the RGB features to the target detection results.

7. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of evaluating the modal contribution of the extracted infrared multi-scale feature information includes: calculating the contribution of the thermal radiation information and target contour information contained in the infrared features to the target detection results.

8. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, The process of generating adaptive fusion weights based on the contribution level includes: the adaptive fusion weights are: ； ； in, Indicates the RGB feature fusion weights; Indicates the weights for infrared feature fusion; Indicates the contribution of RGB modes. This indicates the contribution of the infrared mode.

9. The RGB-infrared multimodal adaptive fusion target detection method as described in claim 1, characterized in that, Based on the adaptive fusion weights, the process of fusing feature information of different scales of the two modalities to obtain fused multimodal features includes: for shallow features, fusing edge and texture detail information in RGB image data to improve the spatial localization capability of small targets; For mid-level features, enhance the information interaction between RGB features and infrared features to improve the ability to express the target structure; For deep features, high-level semantic information from both modalities is fused to improve target classification capabilities; Features at each scale are weighted and fused using adaptive fusion weights.

10. An RGB-infrared multimodal adaptive fusion target detection system, characterized in that, include: The dual-modal data acquisition and preprocessing module is used to simultaneously acquire RGB image data and infrared image data in traffic scenarios and perform preprocessing. A dual-branch multi-scale feature extraction module is used to extract multi-scale feature information from RGB images and infrared images respectively; The modal contribution evaluation and adaptive fusion weight generation module is used to evaluate the modal contribution of the extracted RGB multi-scale feature information and infrared multi-scale feature information respectively, to obtain the contribution degree of the two modes to the current detection task, and to generate adaptive fusion weights based on the contribution degree. The multi-scale modality contribution adaptive fusion module is used to fuse feature information of two modalities at different scales based on the adaptive fusion weights to obtain fused multi-modal features. The target detection output module is used to obtain target detection results based on the multimodal features and a pre-trained target detection network.

Images