Self-vehicle traffic police command gesture recognition method and system based on multi-modal fusion

Abstract

The invention discloses a gesture recognition method and system for self-vehicle traffic police command based on multi-modal fusion. According to the method, firstly, the scale of an intersection is judged by integrating multi-source radar sensing data, lane information is positioned by using a double-branch model, a theoretical attention angle geometric priori area is constructed, the relation between the body orientation of a traffic police and the space is mapped into the attention intensity of the traffic police, and whether a command gesture faces a vehicle or not is determined; detecting the output traffic police identity confidence through YOLOv8; extracting skeleton key points by using OpenPose and modeling an action time sequence to obtain a preliminary gesture probability; and finally, carrying out multi-modal joint coding on lane information, traffic police attention intensity, traffic police identity confidence and traffic police command gesture probability values, and inputting the coded information into a gesture recognition model for global feature fusion and decision making. According to the invention, illumination change, fuzzy traffic police command intention affiliation, traffic police height difference and complex background interference can be effectively overcome, and command gesture instructions facing the vehicle can be accurately distinguished.

Description

A method and system for recognizing traffic police gestures for autonomous vehicles based on multimodal fusion Technical Field

[0001] This invention relates to the field of interactive perception for autonomous vehicles, particularly for applications in complex traffic scenarios, and to a method and system for recognizing traffic police hand gestures for autonomous vehicles based on multimodal fusion. Background Technology

[0002] Currently, autonomous driving technology is developing rapidly in general scenarios, but due to constraints imposed by complex traffic environments, the realization and widespread adoption of Level 4 / Level 5 systems will still take time. During the transition phase in technology and regulations, traffic police gesture recognition has become a crucial link in the driving and interaction of autonomous vehicles with the outside world. In real-world traffic scenarios, traffic police hand signals have higher priority than traffic lights and signs. In special circumstances such as traffic control and traffic accidents, autonomous vehicles must obey traffic police commands. Therefore, developing a dynamic recognition method for traffic police gestures is a core technology for achieving human-machine collaborative traffic management and is of great significance for ensuring road safety during the transition period.

[0003] Current research on traffic police gesture recognition mainly falls into two categories: depth image-based methods and gesture feature extraction-based methods. Traditional vision-based traffic police gesture recognition methods typically involve two steps: first, extracting spatial features of the gesture from video or depth images; and second, using a classifier to classify and recognize the extracted features. However, traditional technologies still face significant challenges in practical applications, with one of the core technical difficulties being the effective resolution of the "direction determination of traffic police commands directed towards vehicles." Specifically, most existing methods rely on a single visual modality for analysis, which struggles to provide sufficient depth perception and spatial geometric information in complex three-dimensional road spaces. At real traffic intersections, traffic police commands are directed at vehicles in specific directions, involving complex spatial interactions between the traffic police, the environment, and autonomous vehicles (including both the driver and other vehicles). Due to the lack of an effective fusion mechanism for multimodal data such as visual features, depth information, and spatial location, existing technologies cannot construct accurate relative spatial vector relationships between the traffic police and the vehicle, making it difficult to capture the deep correlation between the traffic police's body orientation and the vehicle's trajectory. This limitation of perception based on a single modality makes it difficult for traditional methods to accurately determine whether a traffic police officer's hand gestures are directed towards the vehicle. In complex traffic flows, this lack of directional judgment can easily lead to misjudgments by autonomous vehicles. Furthermore, traditional methods have poor robustness and are unable to cope with interference such as changes in lighting or obstructed vision, failing to meet the high reliability and accurate perception requirements of autonomous vehicles. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method and system for recognizing traffic police hand gestures directed towards vehicles, based on multimodal fusion. This invention utilizes the traffic police officer's attention intensity to measure the officer's intent and determine whether the hand gesture is directed towards the vehicle. It overcomes interference from factors such as the officer's height, clothing, and the ambiguity of the officer's intent, achieving unambiguous and efficient instruction transmission in complex traffic environments. It also boasts high recognition accuracy and real-time performance, meeting the needs of practical applications.

[0005] In a first aspect, the present invention provides a method for recognizing traffic police hand gestures for vehicles based on multimodal fusion, comprising the following steps:

[0006] Step 1) Obtain the fused sensing data of lidar and millimeter-wave radar, and determine the scale parameters of the target intersection accordingly.

[0007] Step 2) Construct a dual-branch forward road perception model to locate and track lane lines at intersections and identify current lane information;

[0008] Step 3) Based on the intersection scale parameters obtained in Step 1), calculate the theoretical attention angle of each lane relative to the traffic police station position, and generate the geometric prior probability distribution of the theoretical attention angle.

[0009] Simultaneously, OpenPose is used to obtain the 2D coordinates of the traffic police's core key points, reconstruct the traffic police's local 3D coordinate system, calculate the traffic police's attention angle in real time, and map the real-time attention angle to the geometric prior probability distribution to obtain the traffic police's attention intensity facing the vehicle.

[0010] Step 4) Use the trained YOLOv8 model to detect the traffic police officer's identity in the image and output the category confidence of the detection box;

[0011] Step 5) Input the 2D coordinate sequence of the core joints obtained in Step 3) into the 3D pose reconstruction network to obtain the 3D joint coordinate sequence, and further input it into the LSTM temporal network to output the classification probability of the traffic police command gesture.

[0012] Step 6) Jointly encode the current lane information from Step 2), the traffic police attention intensity from Step 3), the traffic police identity confidence from Step 4), and the command gesture classification probability from Step 5) to form a multimodal feature vector.

[0013] Step 7) Input the multimodal feature vector into the Transformer-based multimodal traffic police command gesture recognition model, generate and fuse the information of each modality through global context representation, and finally output the traffic police command gesture category that the vehicle should obey.

[0014] Secondly, the present invention provides a gesture recognition system for traffic police commands based on multimodal fusion, comprising:

[0015] The intersection scale perception module is used to fuse LiDAR and millimeter-wave radar data and output the scale parameters of the drivable area of ​​the intersection.

[0016] The dual-branch forward road perception module is connected to the intersection scale perception module and is used to perform lane line fitting and traffic sign detection on the forward road image and output the current lane information.

[0017] The traffic police attention intensity estimation module is connected to the intersection scale perception module and the dual-branch forward road perception module. It is used to calculate the theoretical attention angle prior probability distribution of each lane based on the intersection scale parameters, and combine it with the traffic police attention angle calculated in real time by the local 3D coordinate system reconstructed from the traffic police core key points obtained by OpenPose to map and generate the traffic police attention intensity facing the vehicle.

[0018] The traffic police identity detection module is used to detect traffic police targets in the input image using a trained YOLOv8 model and output the confidence score of the traffic police identity.

[0019] The 3D pose and gesture probability module is connected to the traffic police attention intensity estimation module. It is used to input the 2D coordinate sequence of the core key points into the 3D pose reconstruction network to obtain the 3D coordinate sequence, and further process it through the LSTM temporal network to output the classification probability of the traffic police command gesture.

[0020] The multimodal fusion decision module is connected to the dual-branch forward road perception module, the traffic police attention intensity estimation module, the traffic police identity detection module, and the 3D pose and gesture probability module. It is used to jointly encode the current lane information, traffic police attention intensity, traffic police identity confidence, and command gesture classification probability into a feature vector, and input it into the Transformer-based multimodal fusion model. Finally, it outputs the traffic police command gesture category signal that the vehicle should obey.

[0021] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for recognizing traffic police hand gestures for autonomous vehicles based on multimodal fusion.

[0022] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method for recognizing traffic police gestures for autonomous vehicles based on multimodal fusion.

[0023] This invention has the following advantages:

[0024] 1. To address the problem of ambiguous command recipient identification in existing technologies using single features, this invention cascades constraints on macroscopic intersection topology, traffic police attention intensity, traffic police identity recognition, and traffic police gesture recognition. Through a multimodal collaborative reasoning mechanism, it achieves accurate matching between traffic police command gestures and controlled vehicles in complex mixed traffic flow scenarios, realizing unambiguous and highly deterministic command transmission.

[0025] 2. To address the limitation of robustness caused by the sensitivity of traditional visual recognition algorithms to viewpoint changes, this invention proposes introducing the traffic police attention angle as a key spatial decision parameter. This invention achieves quantitative analysis of the focus of attention and attention intensity by calculating the three-dimensional spatial mapping of the torso posture. Regarding the anti-interference mechanism, a continuous probability density field is constructed to suppress random high-frequency noise and small perturbations, and the long-tail effect of the probability distribution is utilized to achieve robust inclusion of marginal outliers. Experiments show that this invention can effectively eliminate background and viewpoint coupling interference, significantly improving the accuracy of command directionality determination in uncontrolled viewpoint scenarios.

[0026] 3. Compared with traditional spatial modeling methods based on manual features, this invention adopts basic pose processing units to explicitly model joint spatial dependence and temporal continuity, and effectively overcomes the interference of traffic police height differences and complex lighting through a parallelized spatiotemporal feature learning mechanism.

[0027] 4. To address the problem that traditional methods ignore human kinematic chain constraints, this invention encodes the skeletal chain anatomical structure into a multi-stage key point detection network, and achieves progressive learning from local features to global constraints through cascaded residual modules.

[0028] 5. Compared with traditional methods such as convolutional neural networks, the multimodal traffic police command gesture recognition model based on Transformer adopted in this invention has stronger multimodal data fusion and discrimination capabilities, effectively solving the recognition error problem caused by limited perspective and lack of contextual information in complex intersection scenarios. It has significantly improved recognition accuracy and response speed, and has high practical application value. Attached Figure Description

[0029] Figure 1 is an overall flowchart of the present invention.

[0030] Figure 2 is a schematic diagram of the theoretical attention angle geometric prior region based on Monte Carlo simulation in this invention.

[0031] Figure 3 is a schematic diagram of the traffic police attention angle in this invention.

[0032] Figure 4 is a schematic diagram of traffic police identification in this invention.

[0033] Figure 5 is a schematic diagram of the 3D pose reconstruction network in this invention.

[0034] Figure 6 is a schematic diagram of the structure of the Transformer-based multimodal traffic police command gesture recognition model in this invention. Detailed Implementation

[0035] To make the technical solution of the present invention clearer and more explicit, the implementation methods of the present invention will be comprehensively and systematically described below in conjunction with the accompanying drawings and specific embodiments.

[0036] As shown in Figure 1, this application embodiment provides a method for recognizing traffic police gestures for autonomous vehicles based on multimodal fusion, the specific steps of which are as follows:

[0037] 1) By integrating the perception data of lidar and millimeter-wave radar, the size of the intersection can be determined.

[0038] 2) Construct a forward road perception model for dual-branch roads, locate driving lane lines, and identify current lane information.

[0039] 3) Combine the intersection scale data obtained in 1) to construct the geometric prior region of the traffic police attention angle, and match it with the real-time estimated traffic police attention angle. Map their spatial relationship to the traffic police attention intensity to measure the traffic police intention attribution and determine whether the traffic police command is directed towards the vehicle.

[0040] 4) Use the trained YOLOv8 model to detect the identity of traffic police officers and output the category confidence of the corresponding detection box.

[0041] 5) Obtain the 2D coordinate sequence of 14 core joints using OpenPose, and then... Pose reconstruction networks map it to The gestures are identified and their temporal dynamics are modeled using an LSTM network to output the classification probability of the traffic police's hand gestures.

[0042] 6) Jointly encode lane information, traffic police attention intensity, traffic police identity confidence level and traffic police command gesture probability value, and input them into the Transformer-based multimodal traffic police command gesture recognition model to realize the comprehensive recognition and decision-making of traffic police command gestures.

[0043] In a preferred embodiment, the specific construction steps of the dual-branch forward road perception model in step 2) are as follows:

[0044] 2.1) Enhance lane line features;

[0045] 2.2) Locate and draw lane lines using a sliding window search;

[0046] 2.3) Use YOLOv8 to identify traffic signs in the lanes;

[0047] Furthermore, in step 2.1), the Sobel operator is used to perform horizontal gradient detection, gradient magnitude calculation, and direction filtering. Combined with threshold segmentation of the S channel in the HLS color space, a binary image is generated to highlight the lane line candidate area. Subsequently, perspective transformation is used to convert the road area into a bird's-eye view. Projection relationships are established through hard-coded source and target points to effectively eliminate perspective distortion and focus on key areas.

[0048] Furthermore, in step 2.2), the initial search position is determined by calculating the peak value of the histogram in the lower half of the image. Pixel distribution is tracked layer by layer along the vertical direction, and the left and right lane lines are fitted using a quadratic polynomial. To improve processing efficiency, subsequent frames establish marginal regions around the previously fitted curves for local searching. Simultaneously, a sliding window method is used to smooth the polynomial coefficients of the most recent 5 frames to enhance robustness.

[0049] Furthermore, in step 2.3), by integrating public datasets and real-vehicle data, a diverse traffic sign sample library covering multiple scenarios, weather conditions, and lighting conditions is constructed. Based on the YOLOv8 algorithm, real-time and accurate identification and positioning of traffic signs in lane areas are achieved in complex road environments.

[0050] As shown in Figures 2 and 3, in a preferred embodiment, the specific steps for constructing the geometric prior region of the theoretical attention angle in step 3) and matching it with the real-time estimated traffic police attention angle to map their spatial relationship to the traffic police attention intensity are as follows:

[0051] 3.1) Calculate the theoretical attention angle: Establish a two-dimensional Cartesian coordinate system with the geometric center of the traffic police command post area as the origin, calculate the angles of the left and right boundaries of the target lane in the traffic police's field of vision, and define the arithmetic mean of the two as the theoretical attention angle of the traffic police for the lane.

[0052] 3.2) Construct a theoretical attention angle geometric prior region based on Monte Carlo simulation, and perform multidimensional random sampling and statistical inference using the Monte Carlo simulation method: randomly select target lanes based on the Poisson distribution of traffic flow data, randomly sample vehicle positions on the centerline of the selected lanes, and randomly sample traffic police station positions within the area of ​​the traffic police command post; calculate the corresponding traffic police attention angle sample set based on the above sampling data, and construct a probability density function characterizing the intensity of traffic police attention using the kernel density estimation method;

[0053] 3.3) Constructing a Local Coordinate System for Traffic Police: Using a human posture recognition algorithm, the coordinates of key points on the traffic police officer's torso are obtained. These key points include the left shoulder, right shoulder, left hip, and right hip. In a right-handed world coordinate system with the Y-axis representing the "up" direction, the Z-axis representing the "forward" direction, and the X-axis representing the "right" direction, the horizontal vectors of the shoulder (from left shoulder to right shoulder) and hip (from left hip to right hip) are calculated. The average of these two vectors is taken and normalized to obtain the unit "right" direction vector of the traffic police officer's torso. A cross product is performed between the unit "up" direction vector of the world coordinate system and the unit "right" direction vector of the traffic police officer's torso, and this is normalized to obtain the unit "forward" direction vector representing the direction of the traffic police officer's face. Finally, a cross product is performed between the unit "forward" direction vector and the unit "right" direction vector of the traffic police officer's torso to obtain the unit "up" direction vector perpendicular to the torso plane, thus completing the construction of the local coordinate system for the traffic police officer.

[0054] 3.4) Calculate the traffic police attention angle and convert it into traffic police attention intensity: Construct a rotation matrix based on the three unit direction vectors of the traffic police local coordinate system, and convert the rotation matrix into a unit quaternion containing a scalar part and a vector part; Calculate the traffic police's orientation angle relative to the camera in the horizontal plane using the bivariate arctangent function by using the combination relationship of the components of the quaternion; Substitute the orientation angle into the probability density function to calculate the traffic police attention intensity.

[0055] Furthermore, in step 3.1), the aim is to construct a standardized spatial reference system, transforming static road geometry features and traffic police station location information into computable vector constraints by solving for the angle between the line-of-sight vector and the reference axis. This quantifies the geometric range of the traffic police's theoretical attention angle. The specific basic data required for calculating the theoretical attention angle are as follows:

[0056] (1) Coordinate system: Establish a two-dimensional Cartesian coordinate system Traffic police command post area The geometric center is the origin of the coordinate system. .

[0057] (2) Lane geometry information: For any lane ( Its centerline is determined by the parametric equation. express, lane The first one relative to the stop line Vehicles, lane width is .

[0058] (3) Traffic police station area: defined as .

[0059] (4) Line of sight vector: from the traffic police position To the target point The vector is .

[0060] Furthermore, in step 3.1), the specific steps for calculating the theoretical attention angle are as follows:

[0061] For lane The left and right boundary points corresponding to them in the traffic police's field of vision are respectively and These two points can be calculated from the intersection size obtained in 1).

[0062] Let the traffic policeman's orientation be the reference axis, and the line-of-sight vector be... Angle with reference axis for:

[0063]

[0064] Lane Theoretical attention angle Defined as:

[0065]

[0066] Among them, the left and right boundary angles , for:

[0067]

[0068]

[0069] This step separates static road geometry features from dynamic traffic police positioning information, providing a geometric truth value unaffected by environmental interference for subsequent calculations of traffic police attention intensity. This significantly suppresses environmental interference and improves its robustness and anti-interference capability in complex scenarios. Simultaneously, it uses... The function avoids quadrant ambiguity and ensures the uniqueness and accuracy of angle calculation values.

[0070] Furthermore, step 3.2) aims to address the problem of unstructured uncertainty in real-world scenarios. Since traffic police positions are subject to random disturbances and vehicle positions within lanes are dynamically changing, a single theoretical perspective cannot cover the real situation. By constructing a probability density function, the system transforms "yes / no judgments" into "probability measures." The specific steps for constructing the theoretical attention angle geometric prior region based on Monte Carlo simulation are as follows:

[0071] 3.2.1) Random Sampling Based on Monte Carlo Simulation

[0072] 3.2.2) Statistical Inference and Interval Estimation of Traffic Police Attention Intensity

[0073] Furthermore, in step 3.2.1), the specific steps of random sampling based on the Monte Carlo simulation method are as follows:

[0074] 1) Target lane selection: In each iteration ( In this process, a target lane is randomly selected based on a Poisson distribution of traffic flow. .

[0075] 2) Target vehicle location sampling: in the selected lane Above, along its center line Randomly sample a vehicle location .

[0076] 3) Sampling at traffic police stations: If the area of ​​the traffic police station... If it is not a single point, then in Randomly sample a traffic police station location .

[0077] Furthermore, in step 3.2.2), the specific steps for statistical inference and interval estimation of traffic police attention intensity are as follows:

[0078] For each sample Combine the above steps and use the theoretical attention angle calculation formula from step 3.1) to calculate the corresponding traffic police attention angle. :

[0079]

[0080] All traffic police attention angle samples According to its lane Classify and obtain lane-specific results Traffic police attention angle sample set .

[0081] The probability density function of traffic police attention intensity is constructed using the kernel density estimation method, and its formula is as follows:

[0082]

[0083] in, Indicates the traffic police at the angle The higher the value of the traffic police's attention intensity in a given direction, the more likely the traffic police are to be directing the current vehicle. lane The number of observation angle samples; Indicates bandwidth parameter; The kernel function is expressed as follows: ; Indicates the first One observation angle sample.

[0084] The construction of a probability density function for traffic police attention intensity significantly enhances the robustness and fault tolerance mechanism of the embodiments of this application. By constructing a continuous probability density field, it effectively suppresses random high-frequency interference from noise and minute perturbations of the target, and also utilizes the long-tail characteristic of the probability distribution to effectively encompass marginal outlier samples. This paradigm shift from "discrete hard threshold truncation" to "continuous fuzzy soft decision" makes the reasoning logic of the embodiments of this application closer to the judgment logic of ordinary drivers.

[0085] Furthermore, in step 3.3), the specific steps for constructing the traffic police local coordinate system are as follows:

[0086] 3.3.1) Through Data on the coordinates of key points for traffic police

[0087] 3.3.2) Calculate the "right" direction vector ( ).

[0088] 3.3.3) Calculate the "forward" direction vector ( ).

[0089] 3.3.4) Calculate the "up" direction vector ( ).

[0090] Furthermore, in step 3.3.1), through The coordinates of the key points obtained from the traffic police are as follows:

[0091] left shoulder right shoulder Left hip Right hip .

[0092] in, Represents a Points in space This represents the coordinate components of a point in a three-dimensional coordinate system. For ease of standardized calculation, a right-handed coordinate system is used, and it is agreed that the Y-axis represents the world "up" direction, denoted as . The Z-axis represents the world's "forward" direction, denoted as... The X-axis represents the "right" direction of the world, denoted as... .

[0093] Furthermore, in step 3.3.2), the "right" direction vector is calculated ( The specific steps are as follows:

[0094]

[0095]

[0096]

[0097]

[0098] in, It is the horizontal vector of the shoulder. It is the lateral vector of the hip. It is the average horizontal vector. Represents a three-dimensional vector. Representative vector The length of the mold, Represents a unit vector.

[0099] Furthermore, in step 3.3.3), the "forward" direction vector is calculated. The specific steps are as follows:

[0100] This direction vector represents the direction from the left side of the traffic police officer's body to the right side, and its calculation process is as follows:

[0101]

[0102]

[0103] Furthermore, in step 3.3.4), the "up" direction vector is calculated. The specific steps are as follows:

[0104]

[0105]

[0106] In computer vision, changes in camera viewpoint can lead to nonlinear distortion of the observed data. This application's embodiment achieves viewpoint invariance by constructing a local orthogonal basis based on the human skeleton, meaning that regardless of the camera's location at the intersection, the calculated pose is always relative to the traffic police officer. Furthermore, this application's embodiment uses continuous vector normalization and cross product operations to enforce the orthogonality of the coordinate axes, effectively correcting measurement errors caused by natural non-rigid deformations of the human body (such as twisting or tilting the shoulders), ensuring the mathematical rigor of the pose description.

[0107] Furthermore, the specific steps for calculating the traffic police attention angle in step 3.4) are as follows:

[0108] Combine the three direction vectors calculated in 3.3) into a 3x3 rotation matrix. The calculation formula is as follows:

[0109]

[0110] Then calculate its yaw angle. This refers to the traffic police's attention angle, which is calculated using the following formula:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] in, It is a unit quaternion. It is the scalar part of the unit quaternion. , , It is the vector part of the unit quaternion.

[0118] The calculated traffic police attention angle Substitute the probability density function from step 3.2) This allows traffic police to gain an understanding of the intensity of their attention.

[0119] This application employs quaternions instead of traditional rotation matrices for attitude calculation, effectively avoiding the potential gimbal lock singularity problem in Euler angle representation and significantly enhancing the numerical stability of this application when handling high-dynamic actions such as large turns by traffic police. Furthermore, by mapping the traffic police attention angle obtained in real time to the probability density function, a paradigm shift from "discrete hard threshold truncation" to "continuous fuzzy soft decision" is achieved. Compared to the deterministic Euclidean distance threshold determination in traditional research, this probabilistic matching strategy can more objectively and quantitatively characterize the confidence of traffic police in directing traffic in a specific lane, providing highly robust data support for the decision-making layer of intelligent transportation systems.

[0120] The specific steps for verifying the traffic police officer's identity in step 4) are as follows:

[0121] As shown in Figure 4, a diverse traffic police scenario library is constructed through on-site shooting, simulated scene recording, and public datasets. Based on the YOLOv8 algorithm, image preprocessing, multi-scale feature extraction, and dynamic sample allocation strategy are used to train the model to achieve real-time identification of traffic police and output the probability value of being identified as a traffic police.

[0122] In a preferred embodiment, the specific steps for outputting the classification probability of the traffic police's hand gestures in step 5) are as follows:

[0123] 5.1) Collect video sequences of traffic police officers' command actions, perform human skeleton key node detection on the video sequences, and select and retain 14 preset core joint point data from the detection results; thereby constructing an original feature dataset to characterize the dynamic structure of traffic police officers;

[0124] 5.2) Construct a 3D pose reconstruction network model and use the model to perform deep pose evaluation and three-dimensional coordinate inference on the core joint data obtained in step 5.1): Deconstruct the local spatial features of different limb parts using a specific joint sub-encoder, and perform deep fusion of the original features of the input stream and the modified features inferred by the network through the residual feature aggregation module. After applying kinematic constraints and the 3D pose reconstruction network model, output a high-fidelity pose sequence that conforms to the laws of human movement.

[0125] 5.3) Input the 3D pose sequence information generated in step 5.2) into the temporal analysis network to complete the dynamic feature discrimination of traffic police command gestures, and finally calculate and output the classification confidence of each type of gesture: Receive the corrected pose sequence, use temporal convolution and recurrent processing units to capture the evolution manifold of skeleton motion in the time dimension, and transform the underlying joint trajectory features into semantic labels of traffic police command gestures through nonlinear mapping, and finally complete the recognition of traffic police gestures.

[0126] Furthermore, the specific steps for detecting key human skeletal nodes in step 5.1) involve using the OpenPose technology framework to analyze the traffic police's hand gesture video stream, locating the human skeletal nodes, and extracting the coordinates of 14 key points. The 14 retained core joint points correspond to the following anatomical locations: 1: tip of the nose; 2: center of the neck; 3: right shoulder joint; 4: right elbow joint; 5: right wrist joint; 6: left shoulder joint; 7: left elbow joint; 8: left wrist joint; 9: right hip joint; 10: right knee joint; 11: right ankle joint; 12: left hip joint; 13: left knee joint; 14: left ankle joint.

[0127] Furthermore, the specific steps for constructing the 3D pose reconstruction network model and performing pose evaluation in step 5.2) are as follows:

[0128] 5.2.1) Constructing the basic attitude processing unit

[0129] 5.2.2) Convert the coordinates of key points on the two-dimensional plane into 2D feature vector representations of traffic police hand gestures;

[0130] 5.2.3) Construct a 3D pose reconstruction network model based on a cascaded structure;

[0131] 5.2.4) Construct a mapping function from 2D feature vectors to 3D pose.

[0132] Furthermore, the specific steps for constructing the basic pose processing unit in step 5.2.1) involve an overall network architecture comprising nine cascaded processing units, nine corresponding deep feature fusion layers, and necessary connection layers. The internal structure of a single processing unit includes a recurrent processing core configured with 100 hidden neurons and a fully connected processing layer configured with 150 hidden neurons.

[0133] Furthermore, the specific steps for constructing the 2D feature vector representation in step 5.2.2) are as follows:

[0134] (1) Define the coordinates of the 14 core joints obtained in step 5.1) as a 2D key point set. ;

[0135] (2) The skeleton is constructed by calculating the connection between key points and these vectors are divided into trunk core joint vector group (body seed joint vector) and limb extension joint vector group (limb joint vector).

[0136] Among them, the core joint vector group of the torso Defined as:

[0137]

[0138] Limb extension joint vector group Defined as:

[0139]

[0140] In the above definition, the symbol Indicates starting from the number The key point points to the number The directed vectors of the key points. For simplicity in the following description, the notation is used. This represents the set of vectors constructed above.

[0141] Furthermore, the specific steps for constructing the 3D pose reconstruction network model in step 5.2.3) are as follows:

[0142] To provide a more comprehensive description of Figure 5, each module is explained using letter labels: (A) represents the original image input of the traffic police hand gesture to be processed; (B) represents the feature extraction module used to parse the 2D pose from the image; and (C) represents the 2D pose estimation result output by module (B). (D) represents the process of using the basic pose processing unit to infer depth information from the 2D pose in (C); (E) represents a basic pose processing unit in the network; (F) represents the step of performing 3D pose reconstruction through cascaded basic pose processing units based on the structural connectivity between human body parts; (G) represents the final output 3D pose estimation result. .

[0143] The 3D pose reconstruction network model consists of nine basic pose processing units. At the beginning of the processing flow, the first basic pose processing unit focuses on constructing the joints of the core body region; these joints are designated as seed joints. Each subsequent basic pose processing unit is responsible for reconstructing the pose of its corresponding body part, and information transfer and connection between units are achieved through fully connected layers.

[0144] The overall operation mechanism of the network is described as follows: First, the input 2D pose data is integrated with the preliminary estimated 3D joint features through a deep feature fusion layer; then, the fused features are input into a cascaded structure consisting of 9 basic pose processing units for progressive refinement; finally, the network outputs a complete full-body 3D pose estimation result.

[0145] Furthermore, for ease of description, the basic pose processing units in this application are named p-LSTMs, and the specific body parts reconstructed by each basic pose processing unit correspond to the following:

[0146] p-LSTMs_1 is used to construct the 3D joint coordinates of the nose, neck, left shoulder, right shoulder, left hip, and right hip;

[0147] p-LSTMs_2 is used to construct the 3D joint coordinates of the right elbow;

[0148] p-LSTMs_3 is used to construct the 3D joint coordinates of the left elbow;

[0149] p-LSTMs_4 is used to construct the 3D joint coordinates of the right knee;

[0150] p-LSTMs_5 is used to construct the 3D joint coordinates of the left knee;

[0151] p-LSTMs_6 is used to construct the 3D joint coordinates of the right hand;

[0152] p-LSTMs_7 is used to construct the 3D joint coordinates of the left hand;

[0153] p-LSTMs_8 is used to construct the 3D joint coordinates of the right foot;

[0154] p-LSTMs_9 is used to construct the 3D joint coordinates of the left foot;

[0155] Finally, all the above components are integrated into a unified end-to-end network model to ensure the efficiency and high accuracy of the 3D pose extraction process.

[0156] It is important to note that from the second basic pose processing unit to the last basic pose processing unit, the input strictly depends on the output of the preceding unit. Furthermore, to prevent the attenuation of initial 2D pose information during depth-cascade propagation, each basic pose processing unit incorporates the original 2D pose data. As auxiliary information, this auxiliary information is merged with its own processing results in the deep fusion layer inside it.

[0157] Furthermore, in step 5.2.4), a mapping function is established. The specific steps are as follows: Set the input as a 2D attitude vector. The output is a 3D pose vector. The algorithmic logic of this mapping process can be represented by the following pseudocode:

[0158] for arrive :

[0159] if :

[0160]

[0161]

[0162] otherwise:

[0163]

[0164]

[0165]

[0166] return

[0167] In the above algorithm, Indicates the first The 3D pose portion of the p-LSTM output; Indicates fully connected layer operations; Indicates deep fusion layer operations; This represents the output of the deep fusion layer; Represents the input 2D pose vector; Indicates the index number of the processing unit.

[0168] Furthermore, the specific steps for outputting the classification probability of traffic police hand gestures in step 5.3) are as follows:

[0169] 5.3.1) Construct a memory retention and gating adjustment mechanism for 3D pose temporal features;

[0170] 5.3.2) Calculate the output feature vector of the temporal network at each time step;

[0171] 5.3.3) Perform the final classification prediction and calculate the probability distribution.

[0172] Furthermore, in step 5.3.1) The specific steps of attitude temporal feature preservation and gating mechanism are as follows:

[0173] The timing data flow is controlled by employing "gating units" in an LSTM network. At each time step... The input contains two elements: the pose feature vector at the current time step. ; and the previous time step Passing hidden state LSTM memory cells It can dynamically accumulate posture evolution trends and selectively forget non-critical information, thereby capturing the temporal features of complete gesture movements. The mathematical expression of its gating mechanism is as follows:

[0174]

[0175]

[0176]

[0177]

[0178] in, Indicates time step Traffic police 3D posture input; This represents the temporal characteristics of the output at the previous time step; This represents the concatenation and fusion of feature vectors; and These represent the corresponding weight matrix and bias term, respectively; This represents the Sigmoid activation function; Represents the hyperbolic tangent activation function; This represents the element-wise product of vectors (Hadamard product). Indicates time step The cellular state is used to store long-term memories.

[0179] Specifically: the Gate of Forgetting The forget gate is responsible for determining which historical information should be removed. For example, when a traffic police officer's hand gesture changes from a "stop" signal to a "left turn" signal, the forget gate reduces the memory weight of the previous "stop" action feature; the input gate... The memory gate is responsible for deciding which new pose information needs to be included in the memory unit; the memory update process combines the calculation results of the forget gate and the input gate to update the cell state of the LSTM network. The state is updated to maintain long-term dependencies in the sequence of gesture actions.

[0180] Furthermore, the specific method for calculating the output feature vector in step 5.3.2) is as follows:

[0181] The output vector generated at each time step encodes the action information of the current frame and incorporates historical temporal features. Updated memory unit. It needs to be transformed into the output features at the current time. This is used by subsequent classifiers. Its calculation formula is defined as follows:

[0182]

[0183]

[0184] in, Indicates at time step The output feature (hidden state) is also passed to the next time step. and These represent the weight parameters and bias parameters related to the output gate, respectively.

[0185] Specifically: Output gate Its function is to filter the information in the memory unit, retaining only the features that are discriminative for the current gesture classification. For example, the output gate might suppress the influence of the traffic police officer's slight body sway, focusing instead on the key displacement movements of the arm; the nonlinear transformation stage... Function enhancement models improve their ability to represent complex nonlinear action patterns.

[0186] Furthermore, the specific steps for calculating the prediction probability in step 5.3.3) are as follows:

[0187] The temporal feature vector output by the LSTM network at the last time step The input is fed into a fully connected layer, and then passed through a Softmax classifier to calculate the probability value of the action sequence corresponding to each preset traffic police hand gesture category. The calculation formula is as follows:

[0188]

[0189] The Softmax function is used to calculate the probability distribution. , representing the input sequence Category The probability of. and These represent the weight matrix and bias vector of the fully connected layer, respectively. This represents the timing characteristics of the final traffic police command gesture output by the LSTM network.

[0190] Specifically, the fully connected layer is responsible for mapping high-dimensional temporal features to the target classification space. For example, if the system pre-defines five gesture categories (such as stop, go straight, turn left, turn right, and no gesture), the fully connected layer outputs a 5-dimensional vector. The Softmax function then normalizes the values ​​of this output vector, generating a probability distribution that intuitively represents the confidence level of the currently recognized action sequence belonging to each category.

[0191] As shown in Figure 6, in a preferred embodiment, the specific steps of using the Transformer-based multimodal traffic police hand gesture recognition model in step 6) are as follows:

[0192] 6.1) Implement multimodal information encoding: Use a convolutional neural network to extract features from the acquired lane information to generate a lane context feature vector that represents the constraint relationship between the lane spatial structure and the directional direction of traffic police gestures; map the calculated traffic police command direction probability into an attention vector to represent the probability distribution of the traffic police's attention area in the spatial dimension; convert the traffic police's identity confidence into an identity feature vector as a weighting factor to suppress interference from non-target objects; and encode the initially identified probability distribution of traffic police command gestures into a traffic police command gesture feature vector as a basic visual feature.

[0193] 6.2) Joint Feature Fusion and Location Encoding: The lane context feature vector, the attention vector, the identity feature vector, and the traffic police command gesture feature vector are concatenated along the feature dimension to construct an aggregated feature vector; the aggregated feature vector is projected onto a latent space of a preset dimension through a linear mapping layer to obtain fused features; and the fused features are added element-wise with the learnable location encoding to generate the input sequence of the Transformer encoder.

[0194] A classification identifier is embedded at the beginning of the input sequence. After forward propagation calculation is performed by a multi-layer Transformer encoder, the final hidden state vector corresponding to the classification identifier is extracted as a global context representation of the aggregated intersection environment topology, multimodal interaction information and semantic features.

[0195] 6.3) Recognition and decision-making of the multimodal traffic police command gesture recognition model based on Transformer: The global context representation is input into the classifier and subjected to Softmax normalization to calculate the predicted probability distribution of traffic police command gestures; a cross-entropy loss function is constructed to measure the difference between the predicted probability distribution and the true label, and the model parameters are jointly optimized through the backpropagation algorithm.

[0196] Furthermore, the specific steps for implementing multimodal information encoding in step 6.1) are as follows:

[0197] 6.1.1) Constructing the lane information coding submodule

[0198] 6.1.2) Construct a traffic police attention intensity coding submodule

[0199] 6.1.3) Construct a traffic police identification confidence coding submodule

[0200] 6.1.4) Construct a traffic police hand signal probability encoding submodule

[0201] Furthermore, the specific steps for constructing the lane information encoding submodule in step 6.1.1) involve encoding the lane information identified in step 2) into a lane context feature vector using a convolutional neural network. This is used to enhance the model's understanding of the directionality of traffic police hand gestures.

[0202] Furthermore, the specific steps for constructing the traffic police attention intensity encoding submodule in step 6.1.2) are as follows: mapping the traffic police attention intensity calculated in step 3) into an attention vector representing the probability distribution of the traffic police's attention area. It helps determine the effective range of a gesture.

[0203] Furthermore, the specific steps for constructing the traffic police identity confidence encoding submodule in step 6.1.3) are as follows: mapping the traffic police identity confidence score output in step 4) into an identity feature vector. This is to suppress misidentification by non-traffic police personnel.

[0204] Furthermore, the specific steps of constructing the traffic police command gesture probability encoding submodule in step 6.1.4) are as follows: encoding the traffic police command gesture probability value output in step 5) into a traffic police command gesture feature vector. This serves as the core visual basis for subsequent integration.

[0205] Furthermore, the specific steps of joint feature fusion and location encoding in step 6.2) are as follows:

[0206] Concatenate the four feature vectors into an aggregate feature vector. :

[0207]

[0208] It is then projected onto a unified latent space dimension through a linear transformation. , obtain fusion features To preserve modal order information, learnable positional encoding is introduced. This is then added to the fused features to obtain the Transformer encoder input. :

[0209]

[0210] Furthermore, the specific steps of the recognition and decision-making of the Transformer-based multimodal traffic police command gesture recognition model in step 6.3) are as follows:

[0211] 6.3.1) Global Context Representation Generation

[0212] 6.3.2) Integration and recognition of traffic police hand gestures

[0213] 6.3.3) Model Training and Optimization

[0214] Furthermore, the specific steps for generating the global context representation in step 6.3.1) are as follows:

[0215] A special traffic police hand gesture classification token is pre-set at the beginning of the input sequence ([ This token acts as a global information aggregator in the sequence. After forward propagation through 12 layers of Transformer encoders, its corresponding final output hidden state is... It gathers interactive information between intersection size, lane information, traffic police attention intensity, traffic police identification probability value and traffic police command gesture recognition probability value, and is a global context representation of the entire sequence.

[0216] Furthermore, the specific steps for calculating the probability distribution of traffic police command gestures in step 6.3.2) are as follows:

[0217] The global representation obtained in step 6.3.1) The input is fed into a Softmax classifier to calculate the probability distribution of all candidate traffic police hand gestures. The calculation formula is as follows:

[0218]

[0219] in, and These are the trainable weight matrix and bias term of the classifier, respectively; It is a probability vector, where each dimension represents the confidence level of the model's prediction that the input sample belongs to the corresponding traffic police command gesture. Finally, the traffic police command gesture with the highest confidence level is selected as the final judgment output.

[0220] Furthermore, the specific steps for model training and optimization in step 6.3.3) are as follows:

[0221] Using cross-entropy loss function The optimization objective of the model is defined as predicting the probability distribution. One-hot encoded tags for real gestures The difference between them is calculated using the following formula:

[0222]

[0223] in, This represents the total number of traffic police hand signals. The model minimizes this loss function end-to-end through backpropagation, thereby jointly optimizing all parameters of the Transformer encoder and classifier.

[0224] Based on the methods provided in the above embodiments, this application also provides a multimodal fusion-based gesture recognition system for traffic police commands of vehicles, comprising:

[0225] The intersection scale perception module is used to fuse LiDAR and millimeter-wave radar data and output the scale parameters of the drivable area of ​​the intersection.

[0226] The dual-branch forward road perception module is connected to the intersection scale perception module and is used to perform lane line fitting and traffic sign detection on the forward road image and output the current lane information.

[0227] The traffic police attention intensity estimation module is connected to the intersection scale perception module and the dual-branch forward road perception module. It is used to calculate the theoretical attention angle prior probability distribution of each lane based on the intersection scale parameters, and combine it with the traffic police attention angle calculated in real time by the local 3D coordinate system reconstructed from the traffic police core key points obtained by OpenPose to map and generate the traffic police attention intensity facing the vehicle.

[0228] The traffic police identity detection module is used to detect traffic police targets in the input image using a trained YOLOv8 model and output the confidence score of the traffic police identity.

[0229] The 3D pose and gesture probability module is connected to the traffic police attention intensity estimation module. It is used to input the 2D coordinate sequence of the core key points into the 3D pose reconstruction network to obtain the 3D coordinate sequence, and further process it through the LSTM temporal network to output the classification probability of the traffic police command gesture.

[0230] The multimodal fusion decision module is connected to the dual-branch forward road perception module, the traffic police attention intensity estimation module, the traffic police identity detection module, and the 3D pose and gesture probability module. It is used to jointly encode the current lane information, traffic police attention intensity, traffic police identity confidence, and command gesture classification probability into a feature vector, and input it into the Transformer-based multimodal fusion model. Finally, it outputs the traffic police command gesture category signal that the vehicle should obey.

[0231] Based on the methods provided in the above embodiments, this application also provides a computer-readable storage medium storing a computer program that implements the above methods when executed by a processor.

[0232] Based on the methods provided in the above embodiments, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above methods.

[0233] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0235] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0236] It should be noted that the above specific embodiments enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the patent of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for recognizing traffic police gestures for autonomous vehicles based on multimodal fusion, characterized in that, Includes the following steps: Step 1) Obtain fused perception data from LiDAR and millimeter-wave radar, and determine the scale parameters of the target intersection accordingly; Step 2) Construct a dual-branch forward road perception model to locate and track lane lines at the intersection and identify current lane information; Step 3) Based on the intersection scale parameters obtained in Step 1), calculate the theoretical attention angle of each lane relative to the traffic police officer's position, and generate the geometric prior probability distribution of the theoretical attention angle; simultaneously, use OpenPose to obtain the 2D coordinates of the traffic police officer's core key points, reconstruct the traffic police officer's local 3D coordinate system, calculate the traffic police officer's attention angle in real time, and map the real-time attention angle to the geometric prior probability distribution to obtain the traffic police officer's attention intensity facing the vehicle; Step 4) Use a trained YOLOv8 model to analyze the image... Step 5) Detect the identity of the traffic police officer and output the category confidence of the detection box; Step 6) Input the 2D coordinate sequence of the core key points obtained in Step 3) into the 3D pose reconstruction network to obtain the 3D key point coordinate sequence, and further input it into the LSTM temporal network to output the classification probability of the traffic police officer's hand gesture; Step 7) Jointly encode the current lane information in Step 2), the traffic police officer's attention intensity in Step 3), the traffic police officer's identity confidence in Step 4), and the hand gesture classification probability in Step 5) to form a multimodal feature vector; Step 8) Input the multimodal feature vector into the Transformer-based multimodal traffic police hand gesture recognition model, generate and fuse the information of each modality through global context representation, and finally output the category of the traffic police hand gesture that the vehicle should obey.

2. The method according to claim 1, characterized in that, Step 2) The first branch of the "dual-branch forward road perception model" is: performing Sobel horizontal gradient and HLS-S channel threshold segmentation on the forward view image, obtaining a bird's-eye view through perspective transformation, and obtaining the lane line equation through sliding window polynomial fitting. The second approach involves using YOLOv8 to detect traffic signs on the same bird's-eye view and outputting the sign category and location. The results from the two approaches are then merged to obtain current lane information that includes lane lines and traffic signs.

3. The method according to claim 1, characterized in that, Step 3) The theoretical attention angle is calculated as follows: a two-dimensional Cartesian coordinate system is established with the center of the traffic police station as the origin. For each lane, its left and right boundary points are taken, and the azimuth angles of the two boundary points relative to the traffic police orientation reference axis are calculated respectively to obtain the theoretical attention angle range of the lane.

4. The method according to claim 3, characterized in that, The geometric prior probability distribution of the theoretical attention angle is generated by Monte Carlo random sampling. The Monte Carlo random sampling includes: randomly selecting the target lane according to the Poisson distribution of traffic flow, uniformly and randomly sampling vehicle positions along the lane centerline, and randomly sampling traffic police positions within the allowed traffic police station area. Attention angle samples are calculated for each sampling combination, and finally the attention angle probability density function of each lane is obtained by kernel density estimation.

5. The method according to claim 3 or 4, characterized in that, Step 3) The reconstruction of the traffic police's local 3D coordinate system includes: using the three-dimensional coordinates of the left and right shoulders and left and right hips obtained by OpenPose, calculating the average lateral vector of the shoulder-hip and normalizing it to obtain the right direction vector, taking the upper direction of the world coordinate system as a reference, obtaining the forward direction vector by cross product, and obtaining the upper direction vector by cross product of the forward direction vector and the right direction vector, and constructing an orthogonal normalized rotation matrix.

6. The method according to claim 5, characterized in that, The rotation matrix R is converted into a unit quaternion. And further calculate the yaw angle using a formula. ： in It is the scalar part of the unit quaternion. 、 、 It is the vector part of the unit quaternion; As the real-time traffic police attention angle, the corresponding probability value is obtained by substituting it into the attention angle probability density function, which serves as the traffic police attention intensity towards the vehicle.

7. The method according to claim 1, characterized in that, Step 7) The multimodal traffic police command gesture recognition model based on Transformer includes: encoding lane information, traffic police attention intensity, identity confidence and gesture probability into feature vectors of the same dimension and concatenating them, superimposing learnable position encoding, and inputting them into a 12-layer Transformer encoder; introducing a traffic police command gesture classification token at the beginning of the sequence, and using its output hidden state to obtain the final command gesture category probability distribution through a Softmax classifier.

8. A gesture recognition system for traffic police command based on multimodal fusion, characterized in that, include: The intersection scale perception module is used to fuse LiDAR and millimeter-wave radar data and output the scale parameters of the drivable area of ​​the intersection; the dual-branch forward road perception module is connected to the intersection scale perception module and is used to perform lane line fitting and traffic sign detection on the forward road image and output the current lane information. The traffic police attention intensity estimation module, connected to the intersection scale perception module and the dual-branch forward road perception module, is used to calculate the theoretical attention angle prior probability distribution of each lane based on the intersection scale parameters, and combine it with the traffic police attention angle calculated in real time by the local 3D coordinate system reconstructed from the traffic police core key points obtained by OpenPose, to map and generate the traffic police attention intensity facing the vehicle; the traffic police identity detection module is used to perform traffic police target detection on the input image using a trained YOLOv8 model and output the traffic police identity confidence score; the 3D pose and gesture probability module, connected to the traffic police attention intensity estimation module, is used for The 2D coordinate sequence of the core key points is input into the 3D pose reconstruction network to obtain the 3D coordinate sequence, which is further processed by the LSTM temporal network to output the classification probability of traffic police command gestures. The multimodal fusion decision module is connected to the dual-branch forward road perception module, the traffic police attention intensity estimation module, the traffic police identity detection module and the 3D pose and gesture probability module. It is used to jointly encode the current lane information, traffic police attention intensity, traffic police identity confidence and command gesture classification probability into a feature vector, and input it into the Transformer-based multimodal fusion model. Finally, it outputs the traffic police command gesture category signal that the vehicle should obey.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images