A multi-model pedestrian recognition method for highway videos based on convolutional neural networks

Abstract

The present invention provides a multi-model pedestrian recognition method for highway videos based on convolutional neural networks, including obtaining a video stream through a surveillance camera and intercepting continuous frame images; performing multi-environment compensation preprocessing on the images to generate a standardized feature map group; inputting the feature map group into three parallel convolutional neural network models to extract spatial, motion and context feature matrices respectively; fusing the feature matrices through a dynamic feature fusion module to obtain a comprehensive feature tensor; adaptively weighting the comprehensive feature tensor based on the real-time environmental parameters output by the environmental perception module to generate an environmental compensation feature tensor; performing multi-level analysis on the feature tensor using a cascade classifier to output an initial recognition result; and finally performing spatiotemporal consistency verification on the recognition results of continuous frames through a time series verification module to generate a final recognition result. The present invention can improve the accuracy and robustness of highway video pedestrian recognition in complex environments and improve system reliability.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and intelligent transportation, and more specifically, to a highway video multi-model pedestrian recognition method based on a convolutional neural network. Background Art

[0002] In intelligent transportation systems, highway video surveillance technology has been widely used, mainly for traffic flow monitoring, accident warning and pedestrian detection. Existing pedestrian recognition methods are mostly based on a single model, relying on traditional image processing technology or simple neural network architecture. These methods can achieve certain results under ideal conditions, but in complex and changeable highway environments, such as rainy and foggy weather, lighting changes or background interference, the recognition accuracy drops significantly. In addition, traditional methods often ignore the spatiotemporal continuity and environmental adaptability of pedestrian features, which leads to false alarms or missed detections in dynamic scenes.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: a single model is difficult to cope with complex and changeable environmental conditions, lacks spatiotemporal consistency analysis of dynamic scenes, and has insufficient adaptability to environmental interference, resulting in the accuracy and robustness of pedestrian recognition being difficult to meet actual application requirements. Summary of the invention

[0004] The present invention provides a highway video multi-model pedestrian recognition method based on convolutional neural network, comprising: S1. acquiring highway video stream data in real time through a monitoring camera, and intercepting continuous The frame images constitute the input sequence, where .

[0005] S2. Perform multi-environment compensation preprocessing on the input sequence to generate a standardized feature map group.

[0006] S3. Input the standardized feature map group into three parallel convolutional neural network models to obtain spatial feature matrices respectively. , motion feature matrix , context association matrix .

[0007] S4. The dynamic feature fusion module performs multi-model feature fusion on the three feature matrices and calculates the comprehensive feature tensor , whose dimensions are .

[0008] S5. Based on the real-time environmental parameters output by the environmental perception module, perform adaptive feature weighting on the comprehensive feature tensor to generate the environmental compensation feature tensor .

[0009] S6. Using cascade classifiers Perform multi-level feature analysis and output the initial recognition result set .

[0010] S7. Through the time series verification module, continuous Frame Perform spatiotemporal consistency verification to generate the final recognition result ,in .

[0011] Furthermore, the S2 specifically includes: S21. Using an adaptive histogram equalization algorithm to process each frame image, the enhancement function is: ;in, is the gray level, For the The probability of grayscale occurrence, is the pixel coordinate.

[0012] S22. Identify and eliminate rain streak interference through a raindrop detection network, which contains 5 residual blocks and 3 attention modules.

[0013] S23. Perform image dehazing based on dark channel prior, and the restoration model is: ;in To observe the image, To restore the image, is the atmospheric light value, is the transmittance.

[0014] Furthermore, the dynamic feature fusion module in S4 performs the following operations: ;in represents the channel dimension concatenation operation, is the dynamic weight coefficient, and the calculation formula is: ; In the formula is the Sigmoid function, is the learnable parameter matrix, is the bias term.

[0015] Furthermore, the adaptive feature weighting process in S5 includes: ;in represents element-wise multiplication, is the environmental compensation matrix, and its calculation formula is: ; In the formula is the real-time rainfall intensity coefficient, is the fog concentration coefficient, is the light intensity coefficient, is the compensation coefficient.

[0016] Furthermore, the cascade classifier of S6 comprises: S61. The first-level classifier adopts The depth-wise separable convolutional layer extracts local features and outputs preliminary classification confidence .

[0017] S62. The second-level classifier expands the receptive field through the hole convolution layer and calculates the global association confidence .

[0018] S63. The final classification decision function is: ;in is the confidence of temporal association, is a fixed weight parameter.

[0019] Furthermore, the time series correlation confidence The calculation method is: ;in is the temporal consistency function, defined as follows: ; In the formula is the target center coordinate, is the scale parameter.

[0020] Furthermore, the time series verification module of S7 performs: ;in, For the goal In the The verification score of the frame, is the verification threshold, and the verification score calculation formula is: ; In the formula To test the confidence level, is the trajectory continuity score, is the appearance similarity, .

[0021] Furthermore, it also includes: S8. Targets in the filter are subject to false positive filtering, and the filtering conditions are: ;in is the video frame height. When the target size meets both conditions, it is considered a false positive.

[0022] Furthermore, the environmental perception module calculates environmental parameters by the following formula: Rainfall intensity coefficient: ; Fog concentration coefficient: ; In the formula is the gradient magnitude map, is the historical average gradient value, is the mean value of the red channel of the current frame, is the maximum reference value.

[0023] Furthermore, the three convolutional neural network models adopt differentiated structures: the spatial feature model adopts the ResNet50 architecture; the motion feature model includes a 3D convolution layer and an LSTM layer; and the context model adopts a Nonlocal network structure, and its associated calculation formula is: ;in is the similarity function, is the feature transformation function, is the normalization factor.

[0024] The above-mentioned embodiments according to the present invention have at least the following beneficial effects: The present invention can effectively improve the accuracy and robustness of pedestrian recognition in complex environments through multi-model feature fusion and dynamic feature weighting. Multi-environment compensation preprocessing can enhance image quality, eliminate rain and fog interference, and adapt to lighting changes, thereby providing high-quality input data for subsequent recognition. The dynamic feature fusion module combines spatial, motion and context-related features to make full use of information in different dimensions and further improve the richness and discrimination of feature expression. Adaptive feature weighting adjusts the feature tensor according to real-time environmental parameters, enhances the model's adaptability to environmental changes, and ensures stable recognition performance under different weather and lighting conditions.

[0025] In addition, the present invention uses a cascade classifier and a time series verification module to perform multi-level analysis and spatiotemporal consistency verification on pedestrian features. The cascade classifier can accurately distinguish pedestrians from non-pedestrian targets through multi-level feature extraction and confidence fusion, reducing false alarms and missed detections. The time series verification module further verifies the recognition results of consecutive frames to ensure the continuity and appearance consistency of the target trajectory, thereby further improving the reliability and accuracy of pedestrian recognition. Finally, through the false alarm filtering mechanism, falsely detected targets that do not meet the size characteristics of pedestrians can be eliminated, further optimizing the recognition results and meeting the actual application needs of highway video surveillance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation.

[0027] in: Figure 1 A schematic flow chart of a method for multi-model pedestrian recognition in highway videos based on a convolutional neural network provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0029] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0030] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0031] Reference below Figure 1 , Figure 1 The flowchart of the multi-model pedestrian recognition method for highway video based on convolutional neural network is provided in one embodiment of the present invention. Figure 1 As shown, a highway video multi-model pedestrian recognition method 100 based on convolutional neural network includes: S1. Real-time acquisition of highway video stream data by monitoring camera, and interception of continuous The frame images constitute the input sequence, where .

[0032] S2. Perform multi-environment compensation preprocessing on the input sequence to generate a standardized feature map group.

[0033] S3. Input the standardized feature map group into three parallel convolutional neural network models to obtain spatial feature matrices respectively. , motion feature matrix , context association matrix .

[0034] S4. The dynamic feature fusion module performs multi-model feature fusion on the three feature matrices and calculates the comprehensive feature tensor , whose dimensions are .

[0035] S5. Based on the real-time environmental parameters output by the environmental perception module, perform adaptive feature weighting on the comprehensive feature tensor to generate the environmental compensation feature tensor .

[0036] S6. Using cascade classifiers Perform multi-level feature analysis and output the initial recognition result set .

[0037] S7. Through the time series verification module, continuous Frame Perform spatiotemporal consistency verification to generate the final recognition result ,in .

[0038] It should be noted that the present invention obtains highway video stream data in real time through a monitoring camera, and intercepts continuous frame images to form an input sequence. A monitoring camera refers to a video monitoring device installed on a highway, which is used to capture road conditions in real time. An input sequence refers to continuous image frames intercepted from a video stream, which are usually used for subsequent image processing and analysis, wherein the number of frames is not less than 5 frames.

[0039] Specifically, multi-environment compensation preprocessing is performed on the input sequence to generate a standardized feature map group. Multi-environment compensation preprocessing includes adaptive histogram equalization, raindrop detection and removal, and image defogging based on dark channel prior. Adaptive histogram equalization is used to enhance image contrast, raindrop detection and removal uses residual blocks and attention modules to identify and remove rain streak interference, and image defogging uses a restoration model to remove the effects of haze. A standardized feature map group refers to a group of image data with uniform features generated after preprocessing.

[0040] Preferably, the standardized feature map group is input into three parallel convolutional neural network models to obtain the spatial feature matrix, motion feature matrix and context association matrix respectively. The spatial feature model adopts the ResNet50 architecture, the motion feature model includes a 3D convolution layer and an LSTM layer, and the context model adopts a Nonlocal network structure. The three feature matrices are multi-model feature fused by the dynamic feature fusion module to calculate the comprehensive feature tensor. The dynamic feature fusion module realizes the fusion of different features through channel dimension splicing operation and dynamic weight coefficient calculation. Based on the real-time environmental parameters output by the environmental perception module, the comprehensive feature tensor is adaptively weighted to generate an environmental compensation feature tensor. The environmental perception module adjusts the weight of the feature tensor by calculating the rainfall intensity coefficient, the fog concentration coefficient and the light intensity coefficient. The cascade classifier is used to perform multi-level feature analysis on the environmental compensation feature tensor and output the initial recognition result set. The cascade classifier extracts features and calculates the classification confidence through a 3×3 depth-separable convolution layer and a hole convolution layer. The initial recognition results of the continuous frames are verified for spatiotemporal consistency by the time series verification module to generate the final recognition result. The time series verification module ensures the accuracy and reliability of the recognition results by verifying the score calculation and object size filtering.

[0041] In some embodiments, S2 specifically includes: S21. Using an adaptive histogram equalization algorithm to process each frame image, the enhancement function is: ;in is the gray level, For the The probability of grayscale occurrence, is the pixel coordinate.

[0042] S22. Identify and eliminate rain streak interference through a raindrop detection network, which contains 5 residual blocks and 3 attention modules.

[0043] S23. Perform image dehazing based on dark channel prior, and the restoration model is: ;in To observe the image, To restore the image, is the atmospheric light value, is the transmittance.

[0044] It should be noted that the multi-environment compensation preprocessing mentioned in the present invention is one of the key steps to achieve pedestrian recognition accuracy. Specifically, this step includes adaptive histogram equalization, raindrop detection and elimination, and image defogging. Adaptive histogram equalization is an image enhancement technology that adjusts the grayscale distribution of the image to enhance the contrast of the image, making it more suitable for subsequent feature extraction and analysis. The raindrop detection network is a deep learning model that is used to identify and eliminate rain streak interference in the image, thereby reducing the impact on pedestrian recognition in rainy environments. Image defogging is based on the dark channel prior theory. The fog component in the image is removed through the restoration model to improve the clarity and visibility of the image.

[0045] Specifically, the enhancement function of the adaptive histogram equalization algorithm is: Where L is the gray level, For the The probability of grayscale occurrence, and is the pixel coordinate. The structure of the raindrop detection network contains 5 residual blocks and 3 attention modules. The residual blocks are used to learn the residual features in the image, thereby improving the network's ability to detect raindrops; the attention modules are used to enhance the network's attention to the raindrop area and improve the elimination effect. The restoration model of image dehazing is: ;in, To observe the image, To restore the image, is the atmospheric light value, The model restores the clarity of the image by estimating the atmospheric light value and transmittance.

[0046] Preferably, in order to further optimize the effect of multi-environment compensation preprocessing, the relevant parameters can be adjusted. For example, in adaptive histogram equalization, the grayscale level L can be adjusted according to the grayscale distribution of the actual image to better adapt to the image enhancement requirements under different lighting conditions. For the raindrop detection network, the detection capability of complex raindrop patterns can be further improved by increasing the number of residual blocks or adjusting the weight of the attention module. In the image defogging process, more prior knowledge, such as prior information of the sky area, can be introduced to more accurately estimate the atmospheric light value and transmittance. In addition, other defogging algorithms, such as the defogging method based on the physical model, can be combined with the dark channel prior method to further improve the defogging effect.

[0047] In some embodiments, the dynamic feature fusion module in S4 performs the following operations: ;in represents the channel dimension concatenation operation, is the dynamic weight coefficient, and the calculation formula is: ; In the formula is the Sigmoid function, is the learnable parameter matrix, is the bias term.

[0048] It should be noted that the dynamic feature fusion module is a key link for fusing the feature matrices extracted by different convolutional neural network models in the present invention. This module integrates the spatial feature matrix, the motion feature matrix and the context association matrix through a specific fusion strategy to generate a comprehensive feature tensor. This process not only takes into account the complementarity of different feature matrices, but also enables the fusion process to adaptively adjust the weight distribution according to the characteristics of the input features through the calculation of dynamic weight coefficients. Among them, the calculation of the dynamic weight coefficient is based on the learnable parameter matrix and the bias term, and is normalized by the Sigmoid function to ensure the rationality of the weight.

[0049] Specifically, the operation formula of the dynamic feature fusion module is: ;in, represents the channel dimension concatenation operation, and are dynamic weight coefficients. The calculation formula for these weight coefficients is: ;in, is the Sigmoid function, which is used to normalize the weight value to the (0,1) interval; and is the learnable parameter matrix, and is the bias term. These parameters are optimized through back propagation during training to adapt to different input features and scenarios.

[0050] Preferably, in order to further optimize the performance of the dynamic feature fusion module, it can be refined or replaced in the following aspects. First, a more complex weight calculation mechanism can be introduced, such as introducing a multi-layer perceptron (MLP) to replace the linear transformation to better capture the complex relationship between feature matrices. Secondly, you can consider introducing an attention mechanism so that the model can focus more on important features, thereby improving the fusion effect. In addition, for the channel dimension splicing operation, other fusion strategies such as weighted summation or feature crossover can be tried to explore a better fusion method. Finally, during the training process, regularization techniques (such as Dropout or L2 regularization) can be used to prevent overfitting and further improve the generalization ability of the model.

[0051] In some embodiments, the adaptive feature weighting process in S5 includes: ;in represents element-wise multiplication, is the environmental compensation matrix, and its calculation formula is: ; In the formula is the real-time rainfall intensity coefficient, is the fog concentration coefficient, is the light intensity coefficient, is the compensation coefficient.

[0052] It should be noted that the adaptive feature weighting process is the key link in the present invention to dynamically adjust the feature tensor weight according to environmental parameters. This process weights the comprehensive feature tensor element by element through the environmental compensation matrix to enhance the model's adaptability to different environmental conditions. Among them, the calculation of the environmental compensation matrix is ​​based on the real-time rainfall intensity coefficient, fog concentration coefficient and light intensity coefficient. These coefficients reflect the degree of influence of the current environment on the image quality. By multiplying and adding the compensation coefficients, an environmental compensation matrix for weighting is generated.

[0053] Specifically, the formula for adaptive feature weighting is: ;in, represents an element-wise multiplication operation, is the environmental compensation matrix, and the calculation formula is: ;in, is the real-time rainfall intensity coefficient, is the fog concentration coefficient, is the light intensity coefficient. Compensation coefficient and It is set based on experimental experience to balance the impact of different environmental factors on the feature tensor.

[0054] Preferably, in order to further optimize the effect of adaptive feature weighting, the following aspects can be refined or replaced. First, a dynamic adjustment mechanism can be introduced to adaptively adjust the compensation coefficient according to the changes in real-time environmental parameters. and , rather than a fixed value, to better adapt to extreme environmental conditions. Secondly, it is possible to consider introducing nonlinear functions (such as ReLU or Tanh) to replace linear weighting to enhance the expressive power of the environmental compensation matrix. In addition, for the acquisition of environmental parameters, a variety of sensor data (such as weather station data or the output of the environmental perception module) can be combined to improve the accuracy and reliability of environmental parameters. Finally, the optimal combination of compensation coefficients under different environmental conditions can be verified through experiments and optimized as a hyperparameter of the model to further improve the robustness of the model in complex environments.

[0055] In some embodiments, the cascade classifier of S6 comprises: S61. The first-level classifier adopts The depth-wise separable convolutional layer extracts local features and outputs preliminary classification confidence .

[0056] S62. The second-level classifier expands the receptive field through the hole convolution layer and calculates the global association confidence .

[0057] S63. The final classification decision function is: ;in is the confidence of temporal association, is a fixed weight parameter.

[0058] It should be noted that the cascade classifier is the key link for multi-level feature analysis and pedestrian recognition in the present invention. It gradually extracts features through multi-level convolutional layers and makes classification decisions based on confidence levels at different levels, thereby improving the accuracy and reliability of recognition. Among them, the first-level classifier uses a depth-separable convolutional layer to extract local features, the second-level classifier uses a hole convolutional layer to expand the receptive field to calculate the global correlation confidence, and the final classification decision function combines local features, global correlation, and temporal correlation confidence to output pedestrian recognition results.

[0059] Specifically, the first-level classifier uses a 3×3 depth-separable convolutional layer, which can reduce the amount of computation while maintaining the effectiveness of feature extraction by separating the standard convolution operation into channel-by-channel convolution and point-by-point convolution. Its output is the preliminary classification confidence , which is used to evaluate the local feature similarity of whether the target is a pedestrian. The second-level classifier uses a dilated convolution layer to expand the receptive field of the convolution kernel by introducing a dilation rate, thereby capturing a wider range of contextual information and calculating the global association confidence. The final classification decision function is: ;in, is the time series association confidence, which is used to evaluate the continuity of the target in the time series. Fixed weight parameter and It is set according to experimental results to balance the contribution of different confidence levels to classification decisions.

[0060] Preferably, in order to further optimize the performance of the cascade classifier, refinement or substitution can be performed in the following aspects. First, more convolutional layers or different types of convolution operations (such as deformable convolution) can be introduced to enhance feature extraction capabilities. Second, the weight parameters can be dynamically adjusted. and , so that it changes adaptively according to the quality of input features and environmental conditions, rather than being a fixed value.

[0061] Furthermore, an attention mechanism can be introduced to enable the classifier to focus more on the key areas of pedestrian features, thereby improving recognition accuracy. Finally, for the calculation of time-series correlation confidence, it is possible to consider introducing a more complex time series model (such as Transformer) to better capture the dynamic changes of the target in the time series and further improve the accuracy and robustness of pedestrian recognition.

[0062] In some embodiments, the timing correlation confidence The calculation method is: ;in, is the temporal consistency function, defined as follows: ; In the formula is the target center coordinate, is the scale parameter.

[0063] It should be noted that the calculation of the temporal correlation confidence is an important step in the present invention for evaluating the consistency and continuity of pedestrian targets in the time series. It ensures the temporal and spatial consistency of the recognition results by calculating the similarity scores of the targets in consecutive frames, thereby improving the accuracy and reliability of pedestrian recognition. The calculation of the temporal correlation confidence is based on the temporal consistency function, which evaluates the stability of the target in the time series by measuring the distance between the target center coordinates.

[0064] Specifically, the confidence of temporal correlation The calculation formula is: ; Where N is the number of frames of the target in the time series, is a temporal consistency function, which is used to measure the similarity of the target in two consecutive frames. The temporal consistency function is defined as: ;in, and are the center coordinates of the target in the i-th frame and the i+1-th frame, respectively. is a scale parameter used to control the sensitivity of distance. is set to 5, a parameter value obtained through experimental verification, which aims to balance the distance difference between the target center coordinates and the stability of the similarity score.

[0065] Preferably, in order to further optimize the calculation effect of the temporal correlation confidence, the following aspects can be refined or replaced. First, more feature information (such as the appearance characteristics or motion trajectory of the target) can be introduced to enhance the expression ability of the temporal consistency function, rather than just relying on the target center coordinates. Second, the scale parameter can be dynamically adjusted. , so that it changes adaptively according to the target's movement speed or scene complexity, rather than a fixed value. In addition, it is possible to consider introducing a more complex timing model (such as a recurrent neural network or a Transformer architecture) to replace the timing consistency function to better capture the dynamic changes of the target in the time series. Finally, regularization techniques (such as Dropout) can be introduced to prevent overfitting and further improve the robustness of the model in complex scenarios.

[0066] In some embodiments, the time series verification module of S7 performs: ;in For the goal In the The verification score of the frame, is the verification threshold, and the verification score calculation formula is: ; In the formula To test the confidence level, is the trajectory continuity score, is the appearance similarity, .

[0067] It should be noted that what needs to be done here is the final confirmation of the pedestrian detection results, using the spatial correlation confidence obtained previously , feature association confidence Confidence associated with time series , through the formula Calculate the comprehensive association confidence P. Among them, is the corresponding weighting coefficient, which is used to adjust the importance of different associated confidences in the comprehensive results. If P is greater than the set threshold T, the detected target is confirmed as a pedestrian.

[0068] Specifically, the weighting coefficient satisfy ,and This set of coefficients is obtained through a large number of experiments and can better balance the impact of spatial, feature and temporal correlation information on the final result. The threshold T is set to 0.7, which is an empirical value used to determine whether the target is a pedestrian. Spatial association confidence Measures the degree of association between targets in spatial position and feature association confidence Reflects the similarity of target features and the confidence of temporal association Reflects the consistency of goals in time series.

[0069] Preferably, in practical applications, the weighting coefficient can be dynamically adjusted according to different scenarios. For example, in a monitoring scenario, if the target moves faster, you can increase the , to pay more attention to the temporal consistency of the target; if the characteristics of the targets in the scene are very different, increase For the threshold T, an adaptive adjustment method can also be used to dynamically adjust the threshold using a machine learning algorithm based on the accuracy of the detection results over a period of time to improve the accuracy of detection. In addition, when calculating the comprehensive association confidence P, some penalty items can be added. For example, when a certain association confidence value is too low, an additional penalty is imposed on it to further improve the reliability of detection.

[0070] In some embodiments, it also includes: S8. Targets in the filter are subject to false positive filtering, and the filtering conditions are: ;in is the video frame height. When the target size meets both conditions, it is considered a false positive.

[0071] It should be noted that this is feature fusion of multimodal information in a multimodal pedestrian detection system. The multimodal information used includes visual image information and millimeter wave radar information. The feature fusion operation is to integrate these two different types of information to obtain a unified feature representation. The feature fusion method used is bilinear pooling, the purpose of which is to enhance the expressiveness of the features in this way so that pedestrians can be detected more accurately in the future.

[0072] Specifically, visual image information is scene information collected by the camera, which contains rich visual features such as the appearance and posture of pedestrians. Millimeter-wave radar information is the distance, speed, angle and other information of the target obtained by the millimeter-wave radar after transmitting and receiving millimeter-wave signals. When the bilinear pooling method performs feature fusion, it will first extract feature vectors from the visual image information and the millimeter-wave radar information respectively, and then perform outer product operations on the two feature vectors, and finally perform some normalization operations to obtain the fused features. In this process, normalization can use L2 normalization, which normalizes the modulus of the feature vector to 1 to avoid the influence of eigenvalues ​​that are too large or too small on subsequent detection.

[0073] Preferably, different deep learning network structures can be used when extracting feature vectors of visual image information and millimeter-wave radar information. For visual image information, the ResNet network can be used, which has a very deep network structure and can extract more advanced semantic features. For millimeter-wave radar information, a convolutional neural network specially designed for radar data can be used to better adapt to the characteristics of radar data. When performing outer product operations, in order to reduce the amount of calculation, a low-rank approximation method can be used, such as projecting the feature vector into a low-dimensional subspace and then performing an outer product operation. In addition, in addition to L2 normalization, you can also try the Batch Normalization method, which can make the feature distribution more stable during training and speed up the convergence of the model.

[0074] In some embodiments, the environment perception module calculates the environmental parameters by the following formula: Rainfall intensity coefficient: ; Fog concentration coefficient: ; In the formula is the gradient magnitude map, is the historical average gradient value, is the mean value of the red channel of the current frame, is the maximum reference value.

[0075] It should be noted that what is described here is the process of using the fused multimodal features to perform pedestrian detection and classification in a multimodal pedestrian detection system. The support vector machine (SVM) classifier is used, which will judge whether the target is a pedestrian based on the fused features obtained by bilinear pooling. SVM is a supervised machine learning algorithm, the core of which is to find an optimal hyperplane to separate samples of different categories, which is to distinguish between pedestrians and non-pedestrians in this case.

[0076] Specifically, the fused multimodal features are unified feature representations obtained after bilinear pooling operations, which integrate the features of visual image information and millimeter-wave radar information. When training a support vector machine classifier, a large amount of sample data with labels (pedestrians or non-pedestrians) is required. The kernel function selects the radial basis function (RBF), which can map data in a low-dimensional space to a high-dimensional space, making it easier to find hyperplanes that separate samples of different categories. For the radial basis function, there is an important parameter \gamma, which is set to 0.1 here. It controls the width of the kernel function and affects the generalization ability and classification accuracy of the classifier.

[0077] Preferably, when selecting a kernel function, in addition to the radial basis function, a polynomial kernel function can also be tried. The form of the polynomial kernel function is ; where r and d are parameters that need to be adjusted. When the data has a certain polynomial distribution law in a low-dimensional space, the polynomial kernel function may achieve better classification results. When training a support vector machine classifier, a cross-validation method can be used to select the optimal parameters. For example, the training data is divided into several parts, one of which is used as a validation set and the rest as a training set in turn, and the optimal parameter setting is found by comparing the classification accuracy on the validation set under different parameter combinations. In addition, the fused multimodal features can also be reduced in dimensionality, such as using the principal component analysis (PCA) method to reduce the dimension of the features, reduce the amount of calculation, and may also improve the performance of the classifier.

[0078] In some embodiments, the three convolutional neural network models adopt a differentiated structure: The spatial feature model uses the ResNet50 architecture; the motion feature model includes a 3D convolutional layer and an LSTM layer; the context model uses a Nonlocal network structure, and its associated calculation formula is: ;in is the similarity function, is the feature transformation function, is the normalization factor.

[0079] It should be noted that this step is to evaluate the multimodal pedestrian detection system, and the evaluation indicator used is mean average precision (mAP). Mean average precision is an important indicator for measuring the performance of target detection algorithms, which comprehensively considers the accuracy and recall rate of the detection results. By calculating this indicator, the performance of the entire multimodal pedestrian detection system can be quantitatively evaluated in order to judge the pros and cons of the system and make improvements.

[0080] Specifically, when calculating the mean average precision (mAP), you need to first calculate the average precision (AP) of each category. For pedestrian detection, the category is pedestrian. The calculation of average precision is based on the precision-recall curve (PR curve), which describes the change in precision under different recall rates. Integrating the area under the PR curve gives the average precision (AP). The mean average precision (mAP) is the average of the average precision of all categories. Although there is only one category of pedestrians here, the calculation method is the same. During the calculation process, the detection results need to be compared with the actual annotations to determine whether the detected pedestrians are correct, so as to calculate the precision and recall.

[0081] Preferably, when calculating the PR curve, an interpolation method can be used to smooth the curve so that the calculation of the average precision is more accurate. For example, for each recall value, the maximum value of all subsequent precision rates is taken as the precision rate of that point, which can avoid the influence of the fluctuation of the curve on the calculation of the average precision. In addition, in addition to the mean average precision (mAP), other evaluation indicators such as the F1 value can also be introduced. The F1 value is the harmonic mean of the precision rate and the recall rate, and the formula is: .

[0082] It can more comprehensively reflect the balance between the accuracy and recall of the detection system. At the same time, when comparing the detection results and the real annotations, different IoU (intersection over union) thresholds can be set to calculate the mAP under different thresholds, so as to more carefully evaluate the performance of the system under different matching accuracy requirements.

[0083] The above-mentioned embodiments of the present invention have the following beneficial effects: The present invention can significantly improve the accuracy and robustness of pedestrian recognition in complex environments through multi-model feature fusion and dynamic feature weighting. Multi-environment compensation preprocessing can enhance image quality, eliminate rain and fog interference, and adapt to lighting changes, thereby providing high-quality input data for subsequent recognition. The dynamic feature fusion module combines spatial, motion and context-related features to make full use of information in different dimensions and further improve the richness and discrimination of feature expression. Adaptive feature weighting adjusts the feature tensor according to real-time environmental parameters, enhances the model's adaptability to environmental changes, and ensures stable recognition performance under different weather and lighting conditions.

[0084] In addition, the present invention uses a cascade classifier and a time series verification module to perform multi-level analysis and spatiotemporal consistency verification on pedestrian features. The cascade classifier can accurately distinguish pedestrians from non-pedestrian targets through multi-level feature extraction and confidence fusion, reducing false alarms and missed detections. The time series verification module further verifies the recognition results of consecutive frames to ensure the continuity and appearance consistency of the target trajectory, thereby further improving the reliability and accuracy of pedestrian recognition. Finally, through the false alarm filtering mechanism, falsely detected targets that do not meet the size characteristics of pedestrians can be eliminated, further optimizing the recognition results and meeting the actual application needs of highway video surveillance.

[0085] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0086] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A multi-model pedestrian recognition method for highway videos based on convolutional neural networks, characterized in that The following steps are involved: S1. Obtain highway video stream data in real time through surveillance cameras and capture continuous The frame images constitute the input sequence, where ; S2. Perform multi-environment compensation preprocessing on the input sequence to generate a standardized feature map group; S3. Input the standardized feature map group into three parallel convolutional neural network models to obtain spatial feature matrices respectively , motion feature matrix , context association matrix ; S4. The dynamic feature fusion module performs multi-model feature fusion on the three feature matrices and calculates the comprehensive feature tensor , whose dimensions are ; S5. Based on the real-time environmental parameters output by the environmental perception module, perform adaptive feature weighting on the comprehensive feature tensor to generate an environmental compensation feature tensor ; S6. Using cascade classifiers Perform multi-level feature analysis and output the initial recognition result set ; S7. Through the time series verification module, continuous Frame Perform spatiotemporal consistency verification to generate the final recognition result ,in .

2. The method according to claim 1, characterized in that The S2 specifically includes: S21. Using an adaptive histogram equalization algorithm to process each frame image, the enhancement function is: ;in, is the gray level, For the The probability of grayscale occurrence, is the pixel coordinate; S22. Identify and eliminate rain streak interference through the raindrop detection network, the network structure contains 5 residual blocks and 3 attention modules; S23. Perform image defogging based on dark channel prior, the restoration model is: ;in, To observe the image, To restore the image, is the atmospheric light value, is the transmittance.

3. The method according to claim 1, characterized in that The dynamic feature fusion module in S4 performs the following operations: ;in, represents the channel dimension concatenation operation, is the dynamic weight coefficient, and the calculation formula is: ; In the formula, is the Sigmoid function, is the learnable parameter matrix, is the bias term.

4. The method according to claim 1, characterized in that The adaptive feature weighting process in S5 includes: ;in, represents element-wise multiplication, is the environmental compensation matrix, and its calculation formula is: ; In the formula is the real-time rainfall intensity coefficient, is the fog concentration coefficient, is the light intensity coefficient, is the compensation coefficient.

5. The method according to claim 1, characterized in that The cascade classifier of S6 comprises: S61. The first level classifier adopts The depth-wise separable convolutional layer extracts local features and outputs preliminary classification confidence ; S62. The second-level classifier expands the receptive field through the hole convolution layer and calculates the global association confidence ; S63. The final classification decision function is: ;in, is the confidence of temporal association, is a fixed weight parameter.

6. The method according to claim 5, characterized in that The timing correlation confidence The calculation method is: ;in, is the temporal consistency function, defined as follows: ; In the formula, is the target center coordinate, is the scale parameter.

7. The method according to claim 1, characterized in that The S7 time series validation module performs: ;in, For the goal In the The verification score of the frame, is the verification threshold, and the verification score calculation formula is: ; In the formula, To test the confidence level, is the trajectory continuity score, is the appearance similarity, .

8. The method according to claim 1, characterized in that Also includes: S8.Yes Targets in the filter are subject to false positive filtering, and the filtering conditions are: ;in, is the video frame height. When the target size meets both conditions, it is considered a false positive.

9. The method according to claim 1, characterized in that The environmental perception module calculates environmental parameters through the following formula: Rainfall intensity coefficient: ; Fog concentration coefficient: ; In the formula is the gradient magnitude map, is the historical average gradient value, is the mean value of the red channel of the current frame, is the maximum reference value.

10. The method according to claim 1, characterized in that The three convolutional neural network models adopt differentiated structures: the spatial feature model adopts the ResNet50 architecture; The motion feature model includes a 3D convolution layer and an LSTM layer; the context model adopts a Nonlocal network structure, and its associated calculation formula is: ;in is the similarity function, is the feature transformation function, is the normalization factor.

Images