A multi-modal data fusion pedestrian target recognition method in a weak light environment

Abstract

The application discloses a pedestrian target recognition method in a weak light environment based on multi-modal data fusion, and relates to the technical field of intelligent vehicle automatic driving, to solve the problem that the target detection effect of the prior art is poor in a weak light environment. The method comprises the following steps: a traffic scene database containing pedestrians and vehicles is constructed, the database comprising infrared images and visible light images; the infrared images and the visible light images are subjected to feature registration and space-time alignment, and are divided into a training set, a verification set and a test set in proportion; a double-branch target detection network based on multi-modal data fusion is provided, the double-branch target detection network is trained with the training set as input, a Dark RGBT -YOLO network model suitable for a weak light environment is obtained; the test set is input into the Dark RGBT -YOLO network model to detect targets and obtain detection results; and position risk scores and distance risk scores of each pedestrian target in the images are analyzed according to the detection results, the position risk scores and the distance risk scores are added together to obtain a total risk assessment result.

Description

Technical Field

[0001] This invention relates to the fields of image processing and autonomous driving technology for intelligent vehicles, and in particular to a method for pedestrian target recognition in low-light environments based on multimodal data fusion. Background Technology

[0002] With the continuous increase in car ownership, traffic accidents occur frequently, with a higher incidence rate in low-light environments such as nighttime. As a result, this has gradually become an important research direction for intelligent driving systems.

[0003] Several scholars have proposed solutions. For example, patent application CN116934627A discloses a target detection method in low-light environments. This method uses traditional methods based on convolution and adversarial generative networks to enhance images in low-light environments and points out the use of knowledge distillation to improve the target detection model, thus obtaining a model suitable for target detection in low-light environments. Patent application CN116612153A discloses an RGBT pedestrian multi-target tracking method based on graph neural networks. This method uses DensNet to construct an infrared and visible light image fusion network, designs a graph data association network based on graph neural networks to achieve cross-frame data association, and combines CenterNet to achieve multi-target tracking of pedestrians.

[0004] The multimodal low-light target detection methods proposed in the above-mentioned prior art each have their own focus and have made their own innovations in feature fusion or target detection methods. However, there is still room for improvement in the detection effect, and the lightweighting problem of the model in the actual deployment process of intelligent system has not been considered. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting and warning of pedestrian targets in low-light traffic environments, which can improve the effect of target detection and issue warning prompts in dangerous situations.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for pedestrian target recognition in low-light environments based on multimodal data fusion includes:

[0008] Construct a traffic scene database containing pedestrians and vehicles, the database including infrared images and visible light images;

[0009] The infrared and visible light images are feature registered and spatiotemporally aligned, and then divided into training, validation, and test sets according to a ratio.

[0010] A dual-branch target detection network based on multimodal data fusion is provided. The training set is used as input to train the dual-branch target detection network to obtain a Dark target detection network adapted to low-light environments. RGBT -YOLO network model;

[0011] The test set is input into Dark RGBT - In the YOLO network model, pedestrian targets in the image are detected, and the detection results are obtained;

[0012] Based on the detection results, a location risk score and a distance risk score are analyzed for each pedestrian target in the image. The location risk score and the distance risk score are added together to obtain the total risk assessment result.

[0013] A further technical solution is that the Dark RGBT The YOLO network includes a multimodal fusion module (MFM), a dual-branch backbone module, a multi-scale feature aggregation pyramid module, and a lightweight model design.

[0014] A further technical solution is that the multimodal fusion module (MFM) includes:

[0015] The input image information is quickly compressed to 50%;

[0016] The compressed image is subjected to main path convolution and mask operation to generate a mask image with the same shape as the image information, and then multiplied pixel by pixel with the image information to extract the first feature;

[0017] The scaled image is subjected to branch convolution and CBAM attention module to extract the second feature;

[0018] The first feature is added to the second feature to obtain the fused feature;

[0019] The fused features are concatenated with the original image information along the channel direction, and channel attention enhancement is performed through the SE Block module to obtain the fused multimodal feature map.

[0020] A further technical solution is that the dual-branch backbone module includes a CBLiner submodule and a CBFuse submodule, which are used to integrate the features extracted by the feature extraction branch into multi-dimensional and multi-level feature information and use it as the input of the feature fusion branch;

[0021] The CBLiner submodule is used to map the input feature map to multiple output channels using convolutional layers to reflect different feature dimensions.

[0022] The CBFuse submodule is used to align feature maps of different sizes through interpolation operations, and then merge feature maps of different sizes through summation and fusion operations.

[0023] A further technical solution is that the multi-scale feature aggregation pyramid module includes a CARAFE sub-module and a C2f-SWC sub-module, which are used to randomly fuse feature information at different scales;

[0024] The CARAFE submodule is used to optimize the upsampling process by dynamically generating an adaptive recombination kernel for each pixel;

[0025] The C2f-SWC submodule is used to capture long-distance feature relationships through displacement convolution.

[0026] A further technical solution lies in the lightweight design of the model, specifically including:

[0027] GhostConv is used as the downsampling convolutional layer of the DarkRGBT-YOLO network model;

[0028] C2f-Faster is used as the feature extraction and feature fusion module of the dual-branch backbone module.

[0029] A further technical solution involves using the training set as input to Dark RGBT - The YOLO network is trained, and its training parameters are specifically set as follows:

[0030] The initial learning rate was 0.01, the optimizer was set to auto, the batch size was set to 8, the number of training epochs was 200, and an early stopping strategy was adopted during model training.

[0031] A further technical solution involves analyzing the location risk score and distance risk score for each pedestrian target in the image based on the detection results, and then adding the location risk score and distance risk score to obtain the total risk assessment result, specifically including:

[0032] Based on the detection results, the detection box of the pedestrian target and the distance from the center point of the detection box to the center line of the image are marked. Based on the detection box and the distance from the detection box to the center line of the image, the location risk score is calculated.

[0033] The actual distance from the center point of the detection frame of the pedestrian target to the vehicle is calculated based on the detection results. The distance risk score is calculated based on the actual distance from the center point of the detection frame to the vehicle and the current vehicle speed.

[0034] The location risk score is added to the distance risk score to obtain the total risk assessment result.

[0035] A further technical solution is that the location risk score is calculated by the following formula:

[0036] score1 = (1-2d / W) × 30%;

[0037] In the formula, score1 represents the location risk score, d represents the distance from the center point of each detection box to the center line of the image, and W represents the width of the image;

[0038] The distance risk score is calculated using the following formula:

[0039] score2 = (1-D / Dv) × 70%;

[0040] In the formula, score2 represents the distance risk score, D represents the actual distance from the center point of the pedestrian target to the vehicle, and Dv is the evaluation parameter that varies with the vehicle speed.

[0041] Compared with the prior art, the method provided by the present invention has the following beneficial effects:

[0042] Dark in this invention RGBT The YOLO network model is a multimodal, dual-branch fusion target detection model, primarily designed for detecting people, vehicles, and other targets in low-light traffic scenes. (The last sentence appears to be incomplete and possibly refers to a different model.) RGBT The multimodal fusion module in the YOLO model effectively fuses infrared (IR) and visible light (RGB) images. A dual-branch backbone makes the fusion of IR and visible light images more efficient, while segmenting the features of the input image to prevent feature redundancy. A multi-scale aggregation pyramid overcomes the limitation of unidirectional information flow in traditional feature pyramid networks, improving feature extraction capabilities and the efficiency of multi-scale feature interaction. Lightweight model design maintains target detection performance while reducing the number of model parameters and computational overhead. Compared with existing target detection algorithms, the proposed method exhibits more accurate detection, robustness, and ease of deployment in low-light traffic target detection. Furthermore, to improve the effectiveness of safety prompts in intelligent driving systems, targeted pedestrian risk assessment rules and corresponding early warning methods have been developed. Attached Figure Description

[0043] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0044] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0045] Figure 2This is a flowchart of the image registration process provided in an embodiment of the present invention;

[0046] Figure 3 Dark provided for embodiments of the present invention RGBT - YOLO network model structure diagram;

[0047] Figure 4 Dark, the subject of this invention RGBT - A simplified diagram of the MFM module structure in the YOLO network model;

[0048] Figure 5 This is a simplified structural diagram of the CBLiner submodule and CBFuse submodule in the dual-branch backbone module of this invention;

[0049] Figure 6 This is a simplified structural diagram of the C2f-SWC submodule in the multi-scale aggregation pyramid module of this invention;

[0050] Figure 7 This is a simplified structural diagram of the C2f-Faster submodule in the lightweight design of this invention;

[0051] Figure 8 A flowchart for pedestrian target risk assessment provided in an embodiment of the present invention. Detailed Implementation

[0052] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0053] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] This invention provides a method for pedestrian target recognition in low-light environments using multimodal data fusion, mainly comprising: continuously acquiring traffic scene images in low-light environments using infrared and visible light cameras to construct a multimodal traffic scene dataset; registering the acquired infrared and visible light image pairs and fusing the two modalities; and inputting the acquired and fused images into a pre-designed and trained Dark... RGBT- The YOLO network model is used to obtain pedestrian target detection results in traffic scenarios. Based on the detection results and distance prediction of pedestrian targets, a risk assessment is performed on each pedestrian target. When the risk exceeds the threshold, the intelligent driving system is warned to take safety precautions.

[0055] In this embodiment of the invention, the proposed Dark RGBT The YOLO network module is a multimodal, dual-branch fusion target detection model, primarily designed for detecting people, vehicles, and other targets in low-light traffic scenarios. It features high detection accuracy and lightweight design. Compared to existing advanced target detection algorithms, the method proposed in this invention achieves more accurate detection results and a more lightweight model design in low-light traffic target detection. Furthermore, to improve the effectiveness of safety warnings for driving risks ahead in intelligent driving systems, targeted pedestrian risk assessment rules and corresponding early warning methods have been specifically developed.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0058] This invention provides a method for pedestrian target recognition in low-light environments based on multimodal data fusion, applicable to automobiles. The main process of the method is described below. Figure 1 As shown:

[0059] Step S1: Construct a traffic scene database containing pedestrians and vehicles, the database including infrared images and visible light images.

[0060] The image acquisition process utilizes both infrared and visible light cameras. Continuous image acquisition is performed in traffic scenes at night, with each infrared and visible light image corresponding to the previous one. In low-light conditions, insufficient light leads to insufficient image clarity from visible light cameras, and images captured under low light often contain excessive noise, resulting in blurred target edges. Infrared cameras overcome this deficiency, providing clear edge information for targets such as people and vehicles even in low-light conditions. The simultaneous acquisition of infrared and visible light images forms infrared and visible light image pairs, providing a foundation for subsequent target detection.

[0061] Step S2: Perform feature registration on the infrared image and the visible light image, and divide them into training set, validation set and test set according to the ratio.

[0062] Because infrared and visible light images differ in resolution, field of view, lens distortion, etc., the infrared and visible light image pairs acquired in the previous step cannot be used directly. They must be registered before the infrared and visible light image pairs can be accurately matched.

[0063] This embodiment employs a coarse-to-fine image registration method based on the contour angle direction. The registration process is as follows: Figure 2 As shown, after inputting infrared and visible light image pairs, the images are first converted to grayscale and normalized. Then, edge detection and curvature size space corner point detection are used to extract image contour feature points. Next, the rotation-invariant principal direction is calculated for each feature point using the contour angle direction method. This method involves constructing a feature triangle and selecting its angle bisector direction as the principal direction, then extracting an improved feature descriptor, and performing preliminary feature matching using multi-scale sampling and a bidirectional matching strategy. A two-stage matching strategy from coarse to fine is then employed: first, obvious mismatches are removed using rotation angle estimation and connector consistency; then, the matching point positions are precisely adjusted using global iterative optimization. Finally, an affine transformation matrix is ​​calculated based on the optimized matching points to transform the infrared image to the visible light image coordinate system, achieving accurate registration. The registered dataset is divided into a training set, a validation set, and a test set, with a ratio of 3:1:1.

[0064] Step S3: Provide a dual-branch target detection network based on multimodal data fusion, and train the dual-branch target detection network using the training set as input to obtain a Dark target detection network adapted to low-light environments. RGBT -YOLO network model.

[0065] Dark RGBT The specific structural innovations of the YOLO network include: a front-end multimodal fusion module (MFM) for fusing infrared and visible light image pairs, resulting in a fused image that incorporates information from both infrared and visible light sources, facilitating subsequent target detection; a dual-branch backbone module to optimize the feature extraction process; and the introduction of CBLiner and CBFuse modules to enhance the interaction of feature information at different levels, enabling Dark... RGBTThe YOLO network model can efficiently extract multi-scale target information in complex scenes. The multi-scale feature fusion part of the neck section was modified, and a multi-scale feature aggregation pyramid module was independently developed by combining the structural characteristics of CARAFE, C2f-SWC, and RepGFPN. The model was designed to be lightweight, replacing all convolutional layers with GhostConv and replacing the C2f module in the backbone with C2f-Faster. The specific structure is as follows: Figure 3 As shown.

[0066] Specifically, the main purpose of the multimodal fusion module (MFM) is to effectively fuse infrared (IR) and visible light (RGB) images before feature extraction, with a design structure as follows: Figure 4 As shown. This module significantly reduces computational costs and improves model performance through a dual-branch structure and attention mechanism, combined with a pixel-level fusion strategy. The overall architecture of MFM includes multiple components such as input splitting, mask operation, additional branches, CBAM attention enhancement, convolution, and SE Block. The specific process is as follows:

[0067] The infrared and visible light images are compressed to 50% to reduce computation. A main-path convolution and mask operation are performed on the compressed infrared and visible light images to generate a mask image with the same shape as the infrared and visible light images. This mask image is then multiplied pixel-by-pixel with the infrared and visible light images to extract the first feature. Specifically, the following formula is used:

[0068]

[0069] In the formula, and Infrared and visible light images, respectively. For convolution operations, This is a pixel-by-pixel multiplication.

[0070] Two additional branches were designed outside the main branch. On these branches, branch convolution operations were performed on the scaled infrared and visible light images, and a CBAM attention module was applied to extract the second feature. The first feature and the second feature were added together to obtain the fused feature. The fused feature was then stitched together with the original infrared and visible light images in the channel direction, and channel attention enhancement was performed through the SE Block module to obtain the fused multimodal feature map.

[0071] The dual-branch backbone module uses dual-branch convolutional layers to build the basic backbone network structure, undertaking the initial feature extraction work. It includes the CBLiner and CBFuse sub-modules, which integrate the features extracted by the feature extraction branches into multi-dimensional and multi-level feature information, using this information as input to the feature fusion branch. This enhances the model's feature extraction capability. For details, please refer to [link / reference]. Figure 5 .

[0072] The CBLiner submodule is applied to different layers of the two main branches. Its function is to map the input feature map to multiple output channels through convolutional layers to represent different feature dimensions. Each convolutional operation can extract different feature patterns in the image. In this way, the CBLiner submodule improves the diversity of features and enhances the model's ability to capture details, making the feature representation richer and more accurate.

[0073] The CBFuse submodule can perform feature fusion from different backbone networks. It aligns feature maps of different sizes through interpolation and then merges them through summation. This fusion method can effectively integrate multi-level features while preserving key information and improving detection accuracy.

[0074] The dual-branch backbone module offers advantages in several aspects: Firstly, the dual-branch structure optimizes the feature extraction process through parallel branch networks, allowing Dark... RGBT - The YOLO network model balances accuracy and speed in environments with limited computing resources, which is crucial for real-time detection tasks. Secondly, the dual-backbone network can extract features at different levels for the fused image separately, leveraging a parallel structure to extract more deep features and improve the model's adaptability to complex scenes. The addition of the CBLiner and CBFuse submodules makes feature fusion more efficient. The CBLiner submodule effectively segments the features of the input image, preventing excessive feature redundancy, and also improves the diversity of feature representations through convolution operations. The CBFuse submodule can align feature maps from different branches through interpolation and achieve feature fusion through summation operations. This approach ensures feature alignment while avoiding redundant computation, resulting in richer fused features and ultimately improving the accuracy and effectiveness of object detection. The dual-backbone architecture offers greater flexibility, allowing adjustment of computational complexity and feature extraction strategies according to different task requirements. For example, when processing specific modal images, certain features can be selectively enhanced to optimize task performance. This flexibility and scalability make the architecture widely applicable, meeting the needs of various multimodal tasks.

[0075] The multi-scale feature aggregation pyramid module includes the CARAFE sub-module and the C2f-SWC sub-module, and integrates the advantages of CARAFE, ShiftwiseConv and RepGFPN. By fusing feature information at different scales through the CARAFE and C2f-SWC modules, the feature extraction and fusion capabilities of the target detection model are improved.

[0076] The CARAFE submodule optimizes the upsampling process by dynamically generating adaptive recombination kernels for each pixel. Unlike traditional interpolation methods, CARAFE can aggregate contextual information over a large field of view, effectively recovering details in low-resolution feature maps and preventing feature loss due to fixed kernels.

[0077] Reference Figure 6 The C2f-SWC submodule employs a ShiftwiseConv convolutional design in its network neck layer to capture long-distance feature relationships through displacement convolution. While traditional large convolutional kernels can cover a wider area, they are computationally expensive and prone to introducing redundant information. In contrast, C2f-SWC effectively captures long-distance feature relationships through local displacement, improving the model's ability to recognize targets in complex scenes. When processing multimodal images, it can more fully mine the feature information in the fused image, thereby improving detection accuracy.

[0078] The multi-scale aggregation connection method optimizes the feature fusion process, breaking through the limitation of unidirectional information flow in traditional feature pyramid networks and improving the interaction efficiency of multi-scale features. This connection method draws on the characteristics of RepGFPN's multi-path feature aggregation and adopts a lightweight design to obtain a multi-scale feature aggregation pyramid structure.

[0079] Lightweight Model Design: In dual-branch networks, each branch performs independent convolution operations, resulting in a large number of parameters and computational overhead, increasing model complexity and computational burden. To reduce the number of parameters and computational overhead while maintaining object detection performance, a lightweight model design is implemented. Specifically, this includes using GhostConv as the Dark... RGBT - The downsampling convolutional layer of the YOLO network model; C2f-Faster is used as the feature extraction and feature fusion module of the dual-branch backbone module.

[0080] GhostConv primarily reduces the computational burden of convolutional neural networks by generating lightweight feature maps. Unlike traditional convolution operations, GhostConv first generates basic feature maps with minimal computation, then generates more feature maps through simple linear transformations. These "ghost feature maps" are closely related to the original feature maps, significantly reducing feature map generation redundancy while maintaining high feature representation capabilities. In low-light object detection tasks, GhostConv allows the model to operate efficiently even when processing large amounts of input data. The C2f-Faster module, for example... Figure 7 As shown, it is responsible for efficient feature extraction and fusion. By introducing the FasterNet Block, the input feature map is segmented and stitched, reducing redundant computation and ensuring sufficient information transfer and accurate feature capture. In C2f-Faster, due to the addition of partial convolution, each convolution operation only processes a portion of the feature map, leaving the other part unchanged. This segmentation strategy effectively reduces computational overhead and improves the efficiency of feature processing. Based on this, C2f-Faster can maintain efficient extraction and fusion of multimodal image features, especially when dealing with complex backgrounds and multi-scale targets, effectively improving detection accuracy.

[0081] In step S3, the training set is used as input to Dark. RGBT The YOLO network was trained with the following parameters: initial learning rate of 0.01, optimizer mode set to auto, batch size set to 8, training epochs set to 200, and an early stopping strategy was adopted during model training to reduce overfitting.

[0082] Step S4: Input the test set into Dark, a suitable low-light environment. RGBT In the YOLO network model, pedestrian targets in images are detected, and detection results are obtained.

[0083] Dark in this invention RGBT The detection results of the YOLO network compared with other models are shown in the table below:

[0084] Dark, a system suitable for traffic target detection in low-light environments, has been trained. RGBT The Dark-YOLO network model was tested using a test set. To evaluate its performance, it was compared with several existing state-of-the-art object detection models, including CFT, ICAfusion, DEYOLO, and Double-YOLO. The test results in the table above show that the Dark-YOLO network model in this invention… RGBT The YOLO model has the best object detection performance. In terms of mAP metric, Dark... RGBTThe Dark-YOLO network model shows significant improvements compared to CFT, ICAFSion, and Double-YOLOv4. This indicates that Dark... RGBT The YOLO network model maintains good detection performance across different IoU thresholds. Regarding recall, Dark... RGBT The Dark YOLO network model shows varying degrees of improvement compared to CFT, ICAFSion, and DEYOLO, indicating that Dark... RGBT The YOLO network model can maintain high accuracy while also possessing a high target detection capability. Although its accuracy is slightly lower than CFT and DEYOLO, it remains the optimal choice overall.

[0085] Step S5: Based on the detection results, analyze the location risk score and distance risk score for each pedestrian target in the image, and add the location risk score and distance risk score to obtain the total risk assessment result.

[0086] The detection result can be Dark RGBT The detection results of the YOLO network model on the test set in step S2 or S3 can also be image information acquired at any time and detected by Dark. RGBT - Results of YOLO network model detection. Risk assessment includes the following steps, please refer to... Figure 8 :

[0087] Step S51: Based on the detection results, mark the detection box of the pedestrian target and the distance d from the center point of the detection box to the center line of the image. Based on the detection box and the distance from the detection box to the center line of the image, calculate the location risk score score1. The location risk score is calculated by the following formula:

[0088] score1 = (1 - 2d / W) × 30%;

[0089] In the formula, score1 represents the location risk score, d represents the distance from the center point of each detection box to the center line of the image, and W represents the width of the image.

[0090] Step S52: Calculate the actual distance D from the center point of the detection box of the pedestrian target to the vehicle based on the detection results. Calculate the distance risk score score2 based on the actual distance from the center point of the detection box to the vehicle and the current vehicle speed. The distance risk score is calculated using the following formula:

[0091] score2 = (1-D / Dv) × 70%;

[0092] In the formula, score2 represents the distance risk score; D represents the actual distance from the center point of the pedestrian target to the vehicle, which is calculated from the parallax images of the two cameras; Dv is an evaluation parameter that varies with vehicle speed. When the vehicle speed is in the low speed range (0-30km / h), Dv=50m, otherwise let Dv=100m.

[0093] Step S53: Add the location risk score to the distance risk score to obtain the total risk assessment result.

[0094] When the overall risk assessment score is greater than 0.75, it is identified as a high-risk target and a warning message will be issued: on the one hand, a voice reminder will be given of the actual distance between the target and the vehicle, and on the other hand, a high-risk target ahead will be visually marked on the screen.

[0095] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0096] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely exemplary descriptions of the invention as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A pedestrian target recognition method in a weak light environment based on multi-modal data fusion, characterized in that, Comprise: A traffic scene database containing pedestrians and vehicles is constructed, which includes infrared images and visible light images; The infrared images and visible light images are feature-registered and spatio-temporally aligned, and are proportionally divided into a training set, a validation set and a test set; A dual-branch target detection network based on multi-modal data fusion is provided, and the dual-branch target detection network is trained with the training set as input to obtain a Dark RGBT YOLO network model; the Dark RGBT The YOLO network model comprises a multi-modal fusion module MFM, a dual-branch backbone module, a multi-scale feature aggregation pyramid module, and a model lightweight design. The multi-modal fusion module MFM is used to compress the infrared images and visible light images by 50%, perform main branch convolution operations and mask operations on the compressed infrared images and visible light images, generate mask images with the same shape as the infrared images and visible light images, and multiply the mask images with the infrared images and visible light images pixel by pixel to extract first features; The scaled infrared images and visible light images are subjected to branch convolution operations and CBAM attention modules to extract second features; the first features and the second features are added to obtain fusion features, the fusion features and the original infrared images and visible light images are spliced in the channel direction, and channel attention enhancement is performed through an SE Block module to obtain a fused multi-modal feature map; The double-branch backbone module includes a CBLiner sub-module and a CBFuse sub-module, which are used to integrate the features extracted by the feature extraction branch in multiple dimensions and multiple levels, and serve as inputs of the feature fusion branch; the CBLiner sub-module is used to map the input feature map to multiple output channels through a convolution layer to represent different feature dimensions; The CBFuse sub-module is used to align feature maps of different sizes through interpolation operation, and then merge the feature maps of different sizes through summation fusion operation; The multi-scale feature aggregation pyramid module includes a CARAFE sub-module and a C2f-SWC sub-module, which are used to randomly fuse feature information of different scales; the CARAFE sub-module is used to optimize the up-sampling process by dynamically generating adaptive reorganization kernels for each pixel; the C2f-SWC sub-module is used to capture long-distance feature relationships through displacement convolution; The model lightweight design adopts GhostConv as the Dark RGBT -YOLO network model down-sampling convolution layer; adopt C2f-Faster as the feature extraction and feature fusion module of the double-branch backbone module; inputting the test set into the Dark RGBT In the YOLO network model, the pedestrian target in the image is detected to obtain a detection result. According to the detection result, the position risk score and the distance risk score of each pedestrian target in the image are analyzed, and the total risk assessment result is obtained by adding the position risk score and the distance risk score.

2. The method as claimed in claim 1, wherein the method for identifying the pedestrian target in the weak light environment using the multi-modal data fusion is characterized by, Dark RGBT The YOLO network is trained with the training set as input, and the training parameters are specifically set as: The initial learning rate is 0.01, the optimizer is set to auto, the Batch Size is set to 8, the number of training rounds is 200, and the early stopping strategy is used during model training.

3. The method as claimed in claim 1, wherein the method of pedestrian target recognition in a multi-modal data fusion in a low light environment is characterized by, According to the detection result, the position risk score and the distance risk score of each pedestrian target in the image are analyzed, and the total risk assessment result is obtained by adding the position risk score and the distance risk score. Specifically, it includes: According to the detection result, the detection frame of the pedestrian target and the distance from the center point of the detection frame to the image center line are marked, and the position risk score is calculated according to the detection frame and the distance from the detection frame to the image center line; According to the detection result, the actual distance from the center point of the detection frame of the pedestrian target to the vehicle is calculated, and the distance risk score is calculated according to the actual distance from the center point of the detection frame to the vehicle and the current vehicle speed. The position risk score and the distance risk score are added to obtain a total risk assessment result.

4. The method of claim 3, wherein the method is a multi-modal data fusion based pedestrian target recognition method in a low-light environment. The position risk score is calculated by the following formula: score1 = (1-2d / W) × 30%; In the formula, score1 represents the position risk score, d represents the distance from the center point of each detection frame to the center line of the image, and W represents the width of the image. The distance risk score is calculated by the following formula: score2 = (1-D / Dv) × 70%; In the formula, score2 represents the distance risk score, D represents the actual distance from the center point of the pedestrian target to the vehicle, and Dv is an evaluation parameter that varies with the vehicle speed.

Images