Pedestrian target detection method based on bimodal feature level fusion in automatic driving environment

Abstract

The invention belongs to the field of target detection of images, and particularly relates to a pedestrian target detection method based on bimodal feature level fusion in an automatic driving environment. The method comprises the following implementation steps: firstly, acquiring infrared and visible light bimodal pedestrian target information in an automatic driving scene from a KAIST data set, and carrying out data preprocessing; secondly, registration is carried out on the bimodal image based on an SURF algorithm, and a better training sample is provided for subsequent target detection network training. And then, improving and training the YOLOv7 network by adopting a double-branch feature extraction strategy and a multi-stage feature interaction fusion strategy. And finally, inputting a to-be-detected image into the improved YOLOv7 target detection network for detection, extracting features under different scales, fusing the features of each stage, and inputting the fused features into a detection head for target detection. Experiments prove that the method can effectively overcome the problem of difficulty in pedestrian detection in a complex scene, keeps stable detection precision, has higher recognition precision under complex conditions, and is high in detection speed; the performance of the YOLOv7 network in processing the bimodal pedestrian target is improved, and the accuracy and efficiency of target detection are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of image target detection, and specifically relates to a pedestrian target detection method based on dual-modal feature-level fusion in an autonomous driving environment. Background Art

[0002] With the rapid development of autonomous driving technology in modern urban traffic environments, the requirements for autonomous vehicles' environmental perception capabilities are increasing. Pedestrian detection in urban road environments, a key research area for autonomous driving, relies heavily on vision systems. Vision systems utilize various sensors to acquire visual information about the road and vehicles. Through image processing and computer vision algorithms, they interpret and analyze information to facilitate vehicle decision-making and path planning. However, significant challenges exist. Single-modal vision systems struggle to accurately capture road and object information in adverse weather conditions, such as rain, snow, and strong sunlight. They are also prone to processing delays and misjudgments when dealing with fast-moving objects or complex traffic scenarios, potentially leading to accidents. To overcome these challenges, the need for multimodal fusion is becoming increasingly important. Multimodal fusion is the process of integrating information from multiple sensors and data sources for analysis and decision-making. In autonomous driving or assisted driving technologies, leveraging the strengths of different sensors can improve system robustness and accuracy, thereby reducing the risk of accidents.

[0003] Existing algorithms for pedestrian detection in autonomous driving scenarios have limitations, including sensitivity to lighting changes, difficulty handling occlusion and dynamic environments, inadequate multimodal data fusion, limited computing resources, insufficient generalization, difficulty meeting real-time requirements, poor adaptability in complex scenarios, hardware compatibility issues, and high rates of false detection and missed detection, which reduce the safety and reliability of autonomous driving systems. Patent CN116012882A, developed by Huazhong University of Science and Technology, proposes a pedestrian detection system based on HOG-SVM. This pedestrian detection algorithm comprises four components: image preprocessing, HOG feature extraction, SVM classification, and post-processing of the detection frame. However, this algorithm also suffers from insufficient detection accuracy, large size, high power consumption, high cost, and poor real-time performance.

[0004] Patent CN202310251456.4, developed by Suzhou Inspur Intelligent Technology Co., Ltd., proposes a pedestrian detection method that fuses visible light and infrared images. This method uses a pedestrian detection model to analyze and extract pedestrian contour features, improving detection accuracy through feature matching and mitigating the effects of ambient light variations and background similarity on detection results. However, this method suffers from slow detection speed and requires supplemental lighting in low-light conditions.

[0005] Patent CN115564691A, developed by the Beijing Institute of Electronic Engineering, proposes a target detection method that combines infrared and visible light. This method obtains infrared and visible light images of the target, performs image alignment, uses the infrared image to determine the target's position, then determines image blocks in the visible light image based on this position. Finally, target detection is performed using a deep convolutional neural network. However, this method requires high computing resources, resulting in a slow network operation rate that cannot meet real-time requirements. Furthermore, the dual-mode target detection method suffers from poor interaction between the two-mode image information, resulting in low detection accuracy. Summary of the Invention

[0006] The present invention proposes a pedestrian target detection method based on dual-modal feature-level fusion in an autonomous driving environment to solve the problems of low pedestrian detection accuracy, poor stability and slow speed in complex scenes in the target detection algorithms in the existing technology.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: A pedestrian target detection method based on bimodal feature-level fusion in an autonomous driving environment, comprising the following steps:

[0008] Step 1: Obtain infrared and visible light dual-modal pedestrian target image data in autonomous driving scenarios from the KAIST dataset, and perform data cleaning, data conversion, and optimization;

[0009] Step 2: Perform feature-level registration on the dual-mode images based on the SURF algorithm:

[0010] 2.1. Complete feature extraction and description of the bimodal image based on the SURF algorithm, obtain stable edge points in the bimodal image for subsequent feature extraction; and perform key point detection and direction assignment in the two bimodal feature scale spaces;

[0011] 2.2. Extract feature descriptors from images based on local gradient histogram descriptors;

[0012] 2.3. Match the extracted feature points and perform straight line fitting using the FSC algorithm to eliminate incorrectly matched feature points;

[0013] Step 3: Based on the improved dual-branch YOLOv7 feature extraction network, perform multi-stage feature interaction fusion on the dual-mode feature fusion part, and finally perform target detection:

[0014] 3.1. Improve and train the YOLOv7 network using a dual-branch feature extraction strategy and a multi-stage feature interaction fusion strategy;

[0015] 3.2. Feature extraction is performed on infrared and visible light images respectively to obtain feature maps at different scales.

[0016] 3.3. Input the fused features into the detection head for target detection.

[0017] Furthermore, in step 2.1 above, the SURF algorithm uses the integral image to approximate the determinant of the Hessian matrix as follows:

[0018] D(x,y,σ)=λ1(x,y,σ)·λ2(x,y,σ) (2)

[0019] Among them, λ1 and λ2 are the largest and second largest eigenvalues ​​of the Hessian matrix at that point, respectively.

[0020] Furthermore, in step 2.2 above, a local gradient histogram descriptor is added to the SURF algorithm and calculated using the Sobel operator, which consists of two convolution kernels, used to estimate the gradient components in the horizontal and vertical directions, respectively, as shown in formula (5):

[0021]

[0022] The gradient amplitude g can be calculated by the following formula, as shown in formula (6):

[0023]

[0024] Where * represents the convolution operation and I(x,y) is the pixel value of the image in the local neighborhood of the point (x,y).

[0025] Furthermore, in step 2.3 above, the transformation parameters are estimated using FSC to initialize the algorithm, the best features are selected for matching, and finally the model is calculated through affine transformation, and then the fit of the model to all point pairs is evaluated; through iteration, RANSAC can identify and remove outliers, and the affine transformation is expressed as formula (7):

[0026] Ax+b=y (7)

[0027] Where A is the transformation matrix, b is the offset vector, x and y are the points before and after the transformation, respectively. The fitted model is used to perform consistency checks on all point pairs, and the error between each point pair and the model is calculated.

[0028] Furthermore, in step 3.1 above, the single-branch backbone of the YOLOv7 network is expanded to a dual-branch one to simultaneously extract features from infrared and visible light images and perform feature fusion.

[0029] Furthermore, in step 3.2 above, a multi-modal multi-stage feature interaction method is adopted. The specific steps are as follows: first, the features of the infrared and visible light modalities are fused to obtain fused features; then the fused features are added to the visible light modality and the infrared modality respectively to jointly guide the generation of the next layer of visible light and infrared features; at the same time, the fused features are also input to the fusion module of the next stage, thereby completing the interaction between the modalities. The generation method of the visible light feature can be expressed as formula (1 1):

[0031]

[0032] In the above formula, τ(x,y) represents the interactive mode of feature addition; It represents the fusion feature obtained after multi-stage multi-feature fusion of visible light features and infrared features, as shown in the formula:

[0033]

[0034] According to the chain rule, the corresponding back propagation is obtained as formula (13):

[0035]

[0036] The infrared feature generation method is the same as that of the visible light feature generation method, as shown in formula (14):

[0037]

[0038] Furthermore, in the above step 3.3, the image to be detected is subjected to feature-level registration in step 2 and then input into the improved YOLOv7 target detection network for detection. The features at different scales are extracted through the dual-branch expansion structure designed in step 3.1. Then, the dual-modal multi-feature interaction module involved in step 3.2 is used to fuse the dual-modal information. Finally, the fused features are input into the detection head for target detection.

[0039] Compared with the existing technology, the beneficial effects of the present invention are:

[0040] (1) To address the problem that the target detection algorithm under a single modality cannot achieve accurate detection in autonomous driving scenarios, a dual-modal pedestrian image dataset was adopted, and the image enhancement algorithm was used to preprocess the dataset to enhance the richness of the target detection data samples and achieve higher recognition accuracy under complex conditions.

[0041] (2) In view of the misalignment problem that exists in the image acquisition process of dual-modal image sensors, the present invention proposes a dual-modal image registration algorithm based on the SURF algorithm. By constructing a scale space, feature extraction and matching are performed on the dual-modal images to solve the image acquisition misalignment problem, so that the dual-modal images have a better degree of registration and provide better training samples for subsequent target detection network training.

[0042] (3) To address the problem of simple single-modal features and difficult feature information interaction, the present invention designs a multi-stage multi-feature fusion pedestrian target detection algorithm based on an improved YOLOv7 network. This algorithm not only enhances the interaction between bimodal feature information, but also maintains a comprehensive understanding of the overall features, making the network perform better in image processing tasks. This makes the network more accurate in capturing key information, thereby improving the accuracy and efficiency of target detection.

[0043] (4) The above three improvements work together to improve the performance of the YOLOv7 network when processing bimodal pedestrian targets. By better preserving bimodal information, enhancing feature diversity, and improving feature expression capabilities, the improved target detection network performs well in target recognition, feature expression, and capture in complex environments, significantly improving the accuracy and efficiency of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the overall flow chart of the network model of the present invention;

[0045] Figure 2 This is a diagram of the network model structure of the present invention;

[0046] Figure 3 Schematic diagram of image registration algorithm and result diagram;

[0047] Figure 4 This is the structure diagram of the multi-stage multi-feature fusion algorithm;

[0048] Figure 5 This is the structure diagram of the multi-stage multi-feature interactive fusion model;

[0049] Figure 6 This is a visualization comparison of the pedestrian target detection results of the present invention and the comparison algorithm in complex scenes at night, where: (a) is the original annotation box diagram, (b) is the detection result of the ACF+T+THOG algorithm, (c) is the detection result of the Halfway-Fusion algorithm, and (d) is the detection result of the algorithm of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and main points of the present invention clearer, the present invention will be described in further detail below in conjunction with the accompanying examples. Obviously, the described embodiments are only part of the embodiments of the present invention and are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0051] like Figure 1 The figure shows the overall flow chart of the present invention. First, the pedestrian target information of the dual-modality of infrared and visible light in the autonomous driving scene is obtained from the KAIST dataset, and data preprocessing is performed. Secondly, the dual-modal images are aligned based on the SURF algorithm to provide better training samples for subsequent target detection network training. Then, the YOLOv7 network is improved and trained using a dual-branch feature extraction strategy and a multi-stage feature interactive fusion strategy. Finally, the image to be detected is input into the improved YOLOv7 target detection network for detection, features at different scales are extracted, and features at each stage are fused, input into the detection head and target detection is performed through the detection head, effectively fusing multi-source information and improving recognition performance in complex environments.

[0052] Based on the above basic ideas, the present invention provides a pedestrian target detection algorithm based on the improved YOLOv7 network multi-stage multi-feature fusion, see Figure 2 , the specific implementation steps are as follows.

[0053] Step 1: Obtain infrared and visible light dual-modal pedestrian target information in autonomous driving scenarios from the KAIST dataset, and perform data cleaning, data conversion, and optimization;

[0054] Step 1.1: Clean the dataset.

[0055] The dataset contains images from different time periods (daytime and nighttime) and various environmental conditions. These images were captured using infrared and visible light cameras mounted on top of a vehicle. Laser beam splitters and camera calibration procedures were used to ensure that each pair of images was aligned in spatial locations, such as a campus, street, and rural area. The image pairs in the dataset consist of visible light color images and corresponding infrared images. A total of 95,328 images were collected and extensively annotated.

[0056] The raw data needs to be cleaned to improve data quality. This includes removing duplicate or low-quality images, standardizing the annotation format, and uniformly annotating difficult-to-distinguish individuals (labeled as "people") and cyclists (labeled as "cyclist") as pedestrians ("pedestrian"). Furthermore, to enhance the model's generalization and avoid overfitting, a sampling strategy is implemented to reduce redundancy between images.

[0057] Step 1.2: Perform data transformation and optimization.

[0058] The cleaned data needs to be converted into a format suitable for training the improved YOLOv7 object detection model. This involves partitioning the dataset into training, validation, and test sets, and using a specific script to convert the images and annotations into the YOLO format, a data format widely used for object detection tasks. The converted dataset is optimized to ensure efficient and effective training of the object detection network model.

[0059] Step 2: See Figure 3 ,An image registration algorithm based on the SURF algorithm is used to perform feature-level registration on the dual-mode images, and correct the misalignment between the visible light and infrared images so that they can have a good effect when fused, thereby improving the recognition rate of image targets.

[0060] Step 2.1: Extract features of the dual-mode image based on the Speeded Up Robust Features (SURF) algorithm.

[0061] The SURF algorithm is a feature detection method based on scale and rotation invariance. It achieves rapid feature point detection by finding local extreme points in the multi-scale Gaussian scale space of the image. First, it is necessary to construct a scale space. By applying Gaussian blur to the image at different scales, images at a series of scales are obtained. The filter formula for constructing the scale space is:

[0062]

[0063] Here, G represents the Gaussian function, σ represents the standard deviation of the filter, and x and y represent spatial coordinates.

[0064] In order to speed up the detection process of feature points, the SURF algorithm introduces the concept of integral image. The integral image can quickly calculate the pixel sum of the local area of ​​the image, thereby avoiding repeated calculations and improving the efficiency of the algorithm. The Gaussian difference image in the scale space is used to detect key points. These key points usually correspond to corners and edges in the image. Next, for each key point, a SURF matrix needs to be constructed. The SURF matrix is ​​a vector that describes the local features around the key point, including the position, scale and direction of the key point, as well as the gradient information of the area around the key point. The detection of feature points depends on the determinant of the Hessian matrix. The SURF algorithm uses the integral image to approximate the determinant of the Hessian matrix as shown in formula (2):

[0065] D(x,y,σ)=λ1(x,y,σ)·λ2(x,y,σ) (2)

[0066] Where λ1 and λ2 are the maximum and second-largest eigenvalues ​​of the Hessian matrix at that point, respectively. Once the feature points are detected, the SURF algorithm further assigns a principal direction to each feature point to ensure rotation invariance. The principal direction of the feature point can be calculated using the following formula (3):

[0067]

[0068] Here, N represents the neighborhood around the feature point, w(x,y) is a weight function, is the gradient vector of the image at point (x, y), θ represents a candidate direction, Represents the gradient of this direction vector with respect to the spatial variable (such as image coordinates).

[0069] Finally, the SURF algorithm generates a descriptor for each feature point based on the gradient magnitude and direction around the feature point. The feature description d is as follows:

[0070]

[0071] Among them, g ij is the gradient magnitude at the grid point (i, j), w ij is the corresponding weight, xij is the coordinate of the grid point, and y ij is the unit vector of the main direction of the feature point, and the value range of i and j depends on the size of the local grid.

[0072] Step 2.2: Extract feature descriptors using local gradient histogram descriptor (LGHD).

[0073] In order to further enhance the robustness of feature description, the local gradient histogram descriptor (Log-GaborHistogramDescri ptor) is used in the SURF algorithm to improve the expressiveness of its feature points. LGHD constructs a feature vector by calculating the gradient histogram in the neighborhood of the feature point, and constructs a feature descriptor by calculating the gradient magnitude and direction of each pixel in the image. It combines the information of local gradient and histogram direction, and can use the Sobel operator to calculate the gradient magnitude and direction of each pixel. The gradient magnitude is usually calculated using the Sobel operator, which consists of two convolution kernels, used to estimate the gradient components in the horizontal and vertical directions, respectively, as shown in Equation (5):

[0074]

[0075] The gradient amplitude g can be calculated by the following formula, as shown in formula (6):

[0076]

[0077] Here, * represents the convolution operation, and I(x,y) is the pixel value of the image in the local neighborhood of the point (x,y). The gradient directions are grouped according to a certain angle range, and the gradient amplitude of each direction is counted. Finally, the values ​​of these gradient histograms are combined into a feature vector, which is normalized to improve its invariance to scale and rotation changes. The normalized histogram is used as the feature descriptor of the local area. The feature descriptors of multiple local areas can be combined into a global feature vector to represent the features of the entire image. This method can not only quickly detect key points in the image, but also provide an accurate description of these key points, making it perform well in applications such as image matching, object recognition, and scene understanding.

[0078] Step 2.3: Use the FSC algorithm to match the nearest neighbor points and remove mismatched points.

[0079] The core idea of ​​the FSC algorithm is to quickly find approximate nearest neighbor matches using a greedy strategy. The algorithm steps are as follows: First, the first descriptor is used as the initial matching set. Then, for each of the remaining descriptors, the distance to each element in the current matching set is calculated. Finally, the element with the closest distance is selected as a candidate match, and the matching set is updated.

[0080] The FSC algorithm has a lower time complexity than traditional nearest neighbor search algorithms because it avoids a comprehensive search of all candidate matches. After feature selection and consistency constraints, the final matching result is obtained, which removes some mismatched points and improves the matching accuracy.

[0081] To further improve the accuracy of the matching, the model is calculated through affine transformation, and then the fit of the model to all point pairs is evaluated. Through iteration, RANSAC can identify and remove outliers, that is, mismatched points. The selected seed point pairs are used to fit an affine transformation model. The affine transformation can be expressed as Equation (7):

[0082]

[0083] Where A is the transformation matrix, b is the offset vector, and x and y are the points before and after the transformation, respectively. Use the fitted model to perform a consistency check on all point pairs, calculating the error between each point pair and the model.

[0084] Step 3: Based on the improved dual-branch YOLOv7 feature extraction network, the YOLOv7 network backbone structure is expanded to a dual-branch structure, and the dual-mode feature fusion part is subjected to multi-stage feature interaction fusion, and finally detection is performed:

[0085] Step 3.1: Improve and train the YOLOv7 network using a dual-branch feature extraction strategy and a multi-stage feature interaction fusion strategy.

[0086] For all-weather urban road scenarios, this paper designs a dual-branch feature extraction network based on YOLOv7, expanding the single-branch backbone of the YO LOv7 network to a dual-branch one to simultaneously extract features from infrared and visible light images, and performing multi-stage feature interaction fusion of the two modal features to achieve better feature modal interaction, so as to process infrared and visible light images and enhance the robustness and stability of pedestrian target detection.

[0087] Step 3.2: Extract features from infrared and visible light images respectively, and fuse the features obtained at different scales.

[0088] In order to strengthen the interaction between the two modal feature generation modules, narrow the differences between the modalities, obtain more discriminative optimized features, and establish long-term dependencies to reduce differences, the present invention adopts a multimodal multi-stage feature interaction method in the multimodal feature generation method to enhance the interactivity between the two modalities.

[0089] In the feature generation process of the dual-mode feature independent generation module, the high-level features of the visible light branch and the infrared light branch are generated through a series of convolution and pooling operations using the low-level features of each branch. The infrared image and the visible light image will first be extracted through the convolution layer, and then the features extracted from the corresponding layers will be extracted. These features from different modalities are then processed through a specific fusion mechanism to generate fused features for subsequent detection tasks. To more intuitively represent the feature generation process, it can be expressed as follows (8):

[0090]

[0091] in represents the lth layer feature of the visible light modality, represents the L+1th layer feature of the visible light modality. f(x,l) represents the convolution pooling and activation operation of the lth layer. According to the chain rule, the corresponding loss function can be deduced as the back propagation formula on the feature as shown in the following formula (9):

[0092]

[0093] The infrared modal features The corresponding back propagation is shown in the following formula (10):

[0094]

[0095] The multimodal multi-stage feature interactive generation module is different from the above multimodal feature independent generation module, see Figure 4The specific steps are as follows: first, the features of the infrared and visible light modalities are fused to obtain fused features; then the fused features are added to the visible light modality and infrared modality respectively to jointly guide the generation of the next layer of visible light and infrared features; at the same time, the fused features are also input into the fusion module of the next stage, thus completing the interaction between the modalities. The generation method of the visible light feature can be expressed as formula (11):

[0096]

[0097] In the above formula, τ(x,y) represents the interactive mode of feature addition; It represents the fusion feature obtained after multi-stage multi-feature fusion of visible light features and infrared features, as shown in the formula:

[0098]

[0099] According to the chain rule, the corresponding back propagation is obtained as formula (13):

[0100]

[0101] The infrared feature generation method is the same as that of the visible light feature generation method, as shown in formula (14):

[0102]

[0103] In the feature interaction generation module, the generation of visible light features involves the participation of both visible light features and fused features from the previous layer. This is based on the joint interaction of visible light and fused features. The same is true for the generation of infrared features. Therefore, the feature interaction generation module can deepen the connections between multimodal networks and establish deeper and more accurate modal associations, thereby obtaining more discriminative enhanced features and improving target recognition performance. Furthermore, unlike the independent multimodal feature generation module, where the fused features obtained through the fusion mechanism are directly input into the detection layer, the feature interaction generation module inputs the fused features into the next-stage modal perception fusion mechanism to participate in the generation of the next-stage fused features, thus strengthening the connection between the features.

[0104] Step 3.3: After the image to be detected has been feature-level registered in step 2, it is input into the improved YOLOv7 target detection network for detection. The dual-branch expansion structure designed in step 3.1 is used to extract features at different scales. Then, the dual-modal multi-feature interaction module involved in step 3.2 is used to fuse the dual-modal information. Finally, the fused features are input into the detection head for target detection.

[0105] The final visual detection result of the present invention is as follows Figure 6As shown, in order to more intuitively verify the effectiveness of the algorithm in this chapter, the algorithm of the present invention is compared with the detection examples of the two algorithms ACF+T+THOG and Halfway-Fusion on the KAIST multispectral pedestrian detection dataset. From left to right, the first column is (a), representing the original annotation image, and the second to fourth columns are (b), (c), and (d), representing the detection results of ACF+T+THOG, Halfway-Fusion algorithm and the algorithm of the present invention, respectively. The red box is the original annotation, and the green box is the actual recognition effect of different algorithms. The accuracy of the algorithm detection effect can be intuitively seen by the superposition and overlap of the red and green boxes. The present invention shows the result graph of pedestrian target detection in complex scenes at night. Complex scenes under night conditions are more challenging for target detection algorithms. Complex scenes refer to scenes in which there are interference factors such as complex lighting conditions, poor weather conditions, frequent occlusions (object occlusion, pedestrians occluding each other), and cluttered background environment (interference from multiple complex elements) in autonomous driving pedestrian detection, which pose challenges to the target detection of the visual system. From Figure 6 The experimental comparison results of different algorithms in complex night scenes show that since infrared images can clearly display pedestrian targets in night environments, both the existing image fusion algorithm and the image fusion algorithm proposed in this section can detect pedestrian instances in the image. However, the Halfway-Fusion algorithm still has some false detections when pedestrians are close together and the street background is complex. The detection frame generated by the algorithm in this section is more accurate than that generated by the ACF+T+THOG algorithm. The experimental results show that the algorithm of the present invention has a lower missed detection rate than other existing advanced algorithms in the KAIST dataset, and has good detection performance for pedestrian targets in different lighting scenarios under all weather conditions, while ensuring a high detection speed. This verifies that the proposed algorithm has good performance and can meet the requirements of pedestrian detection tasks.

[0106] As shown in Table 1, the existing methods ACF+T+THOG, Halfway-fusion, and IAF-RCNN were compared on the KAIST multispectral pedestrian detection test set. Among them, ACF+T+THOG is the baseline method provided by the dataset. Halfway-fusion, as a multispectral pedestrian detection algorithm that uses deep learning in recent years, has outstanding performance among single-stage fusion algorithms. Compared with other similar single-stage fusion algorithms, its mid-term fusion method has the best performance. The IAF-RCNN algorithm is a target detection algorithm that integrates attention mechanism and transfer learning and has good detection accuracy. Since the full time period includes all situations during the day and night, the performance under the full time period is mainly compared. The algorithm based on the multimodal multi-stage fusion strategy proposed in this invention is compared with the existing single-stage fusion algorithm. The present invention demonstrates its advanced nature by comparing three data: Miss Rate (MR), Average Precision (AP), and Frames Per Second (FPS). Among them, MR is the missed detection rate, which refers to the proportion of actual targets that the algorithm fails to detect and is an important indicator for measuring the performance of the target detection system. AP stands for Average Precision, a metric that comprehensively considers the model's performance at different confidence thresholds. It provides a more comprehensive assessment of the performance of object detection models. FPS stands for Frames Per Second, representing the number of frames an algorithm can process per second and directly impacting the system's smoothness and responsiveness. As shown in the table below, our method achieves the highest AP while maintaining a relatively good MR value, and its detection rate is significantly higher than that of other algorithms, demonstrating its advanced nature.

[0107] Table 1 Performance comparison of different algorithms on the KAIST pedestrian dataset

[0108]

[0109] The above description is an explanation of the specific implementation of the present invention, rather than a limitation of the present invention. Those skilled in the relevant technical field can also make various equivalent technical solutions without departing from the scope of the present invention, so all equivalent technical solutions should be included in the scope of protection of the present invention.

Claims

1. A pedestrian detection method based on bimodal feature-level fusion in an autonomous driving environment, characterized by: The following steps are involved: Step 1: Obtain infrared and visible light dual-modal pedestrian target image data in autonomous driving scenarios from the KAIST dataset, and perform data cleaning, data conversion, and optimization; Step 2: Perform feature-level registration on the dual-mode images based on the SURF algorithm: 2.

1. Complete feature extraction and description of the bimodal image based on the SURF algorithm, obtain stable edge points in the bimodal image for subsequent feature extraction; and perform key point detection and direction assignment in the two bimodal feature scale spaces; 2.

2. Extract feature descriptors from images based on local gradient histogram descriptors; 2.

3. Match the extracted feature points and perform straight line fitting using the FSC algorithm to eliminate incorrectly matched feature points; Step 3: Based on the improved dual-branch YOLOv7 feature extraction network, perform multi-stage feature interaction fusion on the dual-mode feature fusion part, and finally perform target detection: 3.

1. Improve and train the YOLOv7 network using a dual-branch feature extraction strategy and a multi-stage feature interaction fusion strategy; 3.

2. Feature extraction is performed on infrared and visible light images respectively to obtain feature maps at different scales. 3.

3. Input the fused features into the detection head for target detection.

2. The method for pedestrian detection based on bimodal feature-level fusion in an autonomous driving environment according to claim 1, characterized in that: In step 2.1, the SURF algorithm uses the integral image to approximate the determinant of the Hessian matrix as follows: D(x,y,σ)=λ1(x,y,σ)·λ2(x,y,σ) (2) Among them, λ1 and λ2 are the largest and second largest eigenvalues ​​of the Hessian matrix at that point, respectively.

3. The pedestrian target detection method based on bimodal feature-level fusion in an autonomous driving environment according to claim 2, characterized in that: In step 2.2, a local gradient histogram descriptor is added to the SURF algorithm and calculated using the Sobel operator, which consists of two convolution kernels, used to estimate the gradient components in the horizontal and vertical directions, respectively, as shown in formula (5): The gradient amplitude g can be calculated by the following formula, as shown in formula (6): Where * represents the convolution operation and I(x,y) is the pixel value of the image in the local neighborhood of the point (x,y).

4. The method for pedestrian detection based on bimodal feature-level fusion in an autonomous driving environment according to claim 3, characterized in that: In step 2.3, the transformation parameters are estimated using FSC to initialize the algorithm, the best features are selected for matching, and finally the model is calculated through affine transformation, and then the fit of the model to all point pairs is evaluated; through iteration, RANSAC can identify and remove outliers, and the affine transformation is expressed as formula (7): Ax+b=y (7) Where A is the transformation matrix, b is the offset vector, x and y are the points before and after the transformation, respectively. The fitted model is used to perform consistency checks on all point pairs, and the error between each point pair and the model is calculated.

5. The pedestrian target detection method based on bimodal feature-level fusion in an autonomous driving environment according to claim 4, characterized in that: In step 3.1, the single-branch backbone of the YOLOv7 network is expanded to a dual-branch backbone to simultaneously extract features from infrared and visible light images and perform feature fusion.

6. The pedestrian target detection method based on bimodal feature-level fusion in an autonomous driving environment according to claim 5, characterized in that: In step 3.2, a multi-modal multi-stage feature interaction method is adopted. The specific steps are as follows: first, the features of the infrared and visible light modalities are fused to obtain fused features; then the fused features are added to the visible light modality and the infrared modality respectively to jointly guide the generation of the next layer of visible light and infrared features; at the same time, the fused features are also input into the fusion module of the next stage, thereby completing the interaction between the modalities. The generation method of the visible light feature can be expressed as formula (11): In the above formula, τ(x,u) represents the interactive mode of feature addition; It represents the fusion feature obtained after multi-stage multi-feature fusion of visible light features and infrared features, as shown in the formula: According to the chain rule, the corresponding back propagation is obtained as formula (13): The infrared feature generation method is the same as that of the visible light feature generation method, as shown in formula (14):

7. The method for pedestrian detection based on bimodal feature-level fusion in an autonomous driving environment according to claim 6, characterized in that: In step 3.3, the image to be detected is subjected to feature-level registration in step 2 and then input into the improved YOLOv7 target detection network for detection. Features at different scales are extracted through the dual-branch expansion structure designed in step 3.

1. Then, the dual-modal multi-feature interaction module involved in step 3.2 is used to fuse the dual-modal information. Finally, the fused features are input into the detection head for target detection.

Images