Multimodal pedestrian directing gesture recognition method fusing large model

Abstract

The application relates to a multi-modal pedestrian command gesture recognition method of a fusion large model and belongs to the field of automatic driving. The method comprises the following steps: collecting a pedestrian data set and performing data set division; adopting a pre-trained pedestrian detection model to extract human body key points from the data set; processing the extracted human body key point information through a target tracking network to obtain dynamic key point trajectory information; establishing a multi-modal action recognition model for identifying gestures, which at least comprises a skeleton encoder and a text encoder; training and optimizing the multi-modal gesture recognition model through multi-part contrast learning loss to obtain a recognition result. The trained model is installed on an automatic driving vehicle to distinguish pedestrian gestures and accurately judge the intention of the pedestrians. The method adopts a multi-modal framework, has high recognition precision and fast recognition speed.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving and relates to a multimodal pedestrian command gesture recognition method integrating a large model. Background Art

[0002] Autonomous vehicles often exhibit overly cautious behavior when interacting with pedestrians. This phenomenon is particularly noticeable in popular self-driving taxi services, such as "Robot Run." These vehicles often stop immediately upon seeing pedestrians, regardless of whether they actually intend to cross the road. This overly courteous behavior not only increases passenger wait times but can also reduce traffic flow and even cause traffic jams. This phenomenon reflects the numerous challenges that autonomous driving technology still faces in handling complex road environments.

[0003] A major reason why autonomous vehicles are overly cautious around pedestrians is the inadequacy of current pedestrian gesture recognition technology. Existing pedestrian gesture recognition systems still have significant room for improvement in accuracy, response speed, and model lightweighting. Many autonomous driving systems rely on pre-set rules and behavioral patterns to make decisions. These rules are often based on the most conservative assumptions to ensure absolute pedestrian safety. However, overly conservative algorithm design also has the side effect of causing vehicles to stop frequently and unnecessarily to let pedestrians pass, even when pedestrians have no clear intention to cross the road. This situation clearly has a negative impact on the efficiency of the entire transportation system.

[0004] Current autonomous vehicle decision-making systems rely primarily on a series of rule-based algorithms, which often struggle to cope with complex and dynamically changing road scenarios. For example, on busy roads or at intersections, pedestrian movements can be highly erratic, sometimes even abruptly changing direction or lingering near crosswalks. Traditional algorithms struggle to effectively distinguish whether pedestrians truly intend to cross the road. Therefore, autonomous vehicles often choose to stop when there is any uncertainty to minimize potential accidents. While this safety-first approach reduces the probability of accidents, it also reduces autonomous vehicle efficiency and increases travel time and costs. Summary of the Invention

[0005] In view of this, the object of the present invention is to provide a multimodal pedestrian command gesture recognition method that integrates a large model.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal pedestrian command gesture recognition method integrating a large model, the method comprising the following steps:

[0008] S1. Collect pedestrian datasets from various locations and time periods and divide the datasets;

[0009] S2. Use the pre-trained pedestrian detection model to extract human key points from the dataset and generate corresponding text prompts in the dataset;

[0010] S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information;

[0011] S4. establishing a multimodal action recognition model for recognizing gestures, which includes at least a skeleton encoder and a text encoder;

[0012] S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain the recognition results.

[0013] Furthermore, in step S1, different visual data sources are collected, which are divided into at least videos and images containing pedestrian activities; the visual data sources cover videos and images of pedestrian activities in different scenes, different time periods, and different weather conditions; and pedestrian activities cover at least multiple pedestrian intentions, including: pedestrians walking towards the road, pedestrians waiting to cross the road, pedestrians signaling to stop, pedestrians signaling vehicles to pass, and pedestrians signaling to turn around; all collected visual data sources are divided into a training set, a validation set, and a test set according to a preset ratio.

[0014] Furthermore, in step S2, the pedestrian detection model uses the OpenPose deep learning model, which identifies the coordinate information of the key points of the human body in the video or image and connects the key points with line segments according to the corresponding coordinate information to obtain the skeleton structure of the human body. The process includes:

[0015] S201, input image: input a static image or video frame, the input image contains one or more human bodies;

[0016] S202, Feature Extraction Network: Use the pre-trained deep convolutional neural network VGG to extract high-level features of the image;

[0017] S203, dual-branch architecture: Two branches are used to process the information in the image in parallel, including a joint heat map branch and a skeletal association map branch, wherein:

[0018] The joint heatmap branch predicts the probability heatmap of each joint in the image, i.e., the probability distribution of joint locations;

[0019] The skeleton association graph branch is used to predict the lines or connections between joints, helping the model understand the relationship between different joints;

[0020] S204, multi-stage processing: progressively optimizing the predictions through a multi-stage approach, where each stage utilizes the output of the previous stage to further adjust and optimize the current joint detection and connection predictions;

[0021] S205, non-maximum suppression: After multiple stages, non-maximum suppression is applied to retain the best detection results of each joint;

[0022] S206, post-processing stage: after obtaining the joint heat map and the bone association map, the joint points are connected to form a complete skeleton;

[0023] S207, output: finally output the skeleton diagram, in which the key joints of the human body and their connections are marked;

[0024] Secondly, the GPT-4 model is used to identify the dataset to generate text prompts of pedestrian actions. The GPT-4 model generates text descriptions in the form of paragraph descriptions, synonym descriptions, and part descriptions.

[0025] Furthermore, in step S3, the target tracking network adopts the ByteTrack network model, which converts the extracted skeleton structure data into time series data. The steps are as follows:

[0026] S301, detecting input data, i.e., skeleton key points in an image frame;

[0027] S302, calculate the confidence value based on the detected key point information. If the confidence value is greater than the predefined threshold track_thresh, proceed to S303; otherwise, jump to S308;

[0028] S303, try to match the key point to the existing trajectory, and determine whether the parameter is_activated is true: if the current trajectory has been activated, go to S304; otherwise jump to S306;

[0029] S304, predict the next position of the current trajectory, and compare whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, go to S305; if the match succeeds, jump to S309;

[0030] S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307;

[0031] S306. Match the key points that are not matched with the detection box with the inactive existing track again. If the match is successful, the existing track is converted to the active state and the key points are marked as tracked. If the match fails again, the tracking box is removed and the confidence score of the detection box is again determined to be greater than the threshold high_treshs. If it is greater than the threshold, a new track is created, and the key points are marked as tracked and the track is marked as inactive.

[0032] S307: Check whether the key points that do not match the tracking frame are in the tracked state. If so, proceed to S308; if not, delete the tracks whose continuous loss time exceeds the maximum time max_time_lost;

[0033] S308, matching the key points with a confidence score lower than track_thresh with the track of the unmatched tracking box. If the match is successful, proceed to S309, otherwise proceed to S310;

[0034] S309: Matching is successful, and the status is marked as tracked;

[0035] S310: Matching fails, and the status is marked as discarded.

[0036] Furthermore, in step S4, the multimodal gesture recognition model includes at least a skeleton encoder E s and text encoder E t , which are used to generate skeleton features and text features respectively, where

[0037] Skeleton Encoder E s A graph convolutional network is used as the backbone network, which includes multiple graph convolution layers and multiple multi-scale temporal convolution modules.

[0038] Text Encoder E t A language model based on the Transformer architecture is used, which first performs standard word segmentation and then processes it through the Transformer module to output a feature vector used to represent the text description.

[0039] Furthermore, in the skeleton encoder E s In the graph layer, the human skeleton is represented as a graph G = {V, ε}, where V is the set of human joints, |V| = N, and ε is the set of edges; H∈R represents the features of the l-th layer of human joints with feature dimension F, then the graph convolution is expressed as follows:

[0040]

[0041] Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W lis the learnable parameter of layer l, and δ is the activation function;

[0042] The multi-scale temporal convolution module is then used for temporal modeling: the multi-scale temporal convolution module consists of four different branches, each of which contains 1×1 convolution. The first and second branches are temporal convolution branches with variable dilation branches, the third branch is the maximum pooling layer branch, and the fourth branch only contains 1×1 convolution. The outputs of the four branches are connected to generate the final result.

[0043] The skeleton-based action recognition method maps human skeleton data to one-hot encoding of action labels for classification, and the labels are trained using cross entropy loss:

[0044] L cts =-ylogp θ (x)

[0045] Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

[0046] Furthermore, the text encoder E t The data processing process includes:

[0047] S401, preprocessing the input text, wherein the preprocessing process first performs word segmentation to divide the text into basic units, each basic unit is a token, and each token is mapped to a unique integer index;

[0048] S402, word embedding process: In the embedding layer, the Word2Vec method is used to convert the input token index into a dense vector representation. The embedded vector represents the semantic information of each word or subword in a high-dimensional space;

[0049] S403, Positional encoding: Positional encoding is introduced to represent the order information of the input tokens. Positional encoding refers to adding the codes generated by sine and cosine functions to the word embedding, so that the model can identify the relative position of the tokens in the sequence;

[0050] S404, Contextual Information Modeling: Using a multi-head self-attention mechanism, each token interacts with other tokens in the input sequence, and each token is weighted and summed based on its relevance to other tokens. Through multiple text encoder layers, each word embedding vector in the input is gradually converted into a high-dimensional representation that includes context.

[0051] S405, global semantic representation generation: use the last token in the sentence, i.e., the [EOS] token, as the global representation vector of the entire sentence;

[0052] S406. The final output representation is the semantic representation of the entire sentence.

[0053] Further, in step S5, E s Cross entropy loss and multi-part contrastive loss are used for training, and some text descriptions are used as additional guidance. The text guidance content is generated by GPT-4. The fully connected layer connects the aligned skeleton features and text features. The following loss function is used to train the model:

[0054]

[0055] Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM.

[0056] Action description is done with a skeleton encoder E s and a text encoder E t The dual encoder encodes the skeleton data and operation description respectively, where the dual encoder jointly optimizes by comparing the skeleton-text pairs in two directions in the batch:

[0057]

[0058] Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size;

[0059] When calculating the loss, we choose to use KL divergence as the contrast loss between skeleton and text:

[0060]

[0061] Where D is the entire dataset, y s2t and y t2s Ground truth similarity score, with probability 0 for negative pairs and probability 1 for positive pairs.

[0062] Further, step S5 includes the following steps:

[0063] S51. Multi-part contrast learning: The skeleton is divided into several regions, and multi-part contrast loss is applied to the local features and overall features of each region. Part features are obtained by aggregating joint features within the same group through joint pooling. The regions after skeleton division include: head, arms, torso, legs and feet. The loss function of multi-part contrast loss is expressed as:

[0064]

[0065] Where K is the total number of groups;

[0066] S52, Pedestrian Gesture Recognition: N pedestrian gestures are pre-entered into the model, which contain the pedestrian intention information corresponding to the gestures. Then, Transformer is used as the decoder, and the activation function adopts the Softmax function to output the probabilities of N command action categories at the current moment. The current command action has the highest probability.

[0067] Further, in step S52, the following process is included:

[0068] S521, Input Embedding: The input data containing the time series feature vector is passed to the Transformer decoder. The decoder gradually extracts and integrates the context information of the input data through the multi-head self-attention layer and the feedforward network layer, and the decoder outputs a series of hidden layer vectors.

[0069] S522, Linear Transformation: Use a linear layer to map the decoder's hidden vector to the output space. The linear layer converts the hidden state of each time step into N numerical values.

[0070] S523, Softmax activation function: The Softmax activation function converts N values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1;

[0071] For the prediction at each time step, the Softmax formula is:

[0072]

[0073] Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i;

[0074] S524, Action Prediction: After obtaining the probabilities of N categories through Softmax output, the category with the highest probability is selected as the command action A at the current moment:

[0075] A=argmax(P(y1),P(y2),…,P(y N ))

[0076] A represents the final recognized action type, and the pedestrian intention is obtained based on the recognized action type.

[0077] The beneficial effects of the present invention are:

[0078] The present invention is applicable to the field of autonomous driving. It uses an on-board camera to collect real-time images of the surrounding environment, and then uses a lightweight multimodal model to distinguish pedestrian gestures, which can accurately determine the pedestrian's intention. The method of the present invention adopts a multimodal framework with high recognition accuracy and fast recognition speed. It can be well deployed on the vehicle system and is very suitable for autonomous driving vehicles. By integrating a multimodal framework of large models and deep learning, the present invention can run efficiently on the limited computing resources of unmanned vehicles and achieve real-time environmental perception. This multimodal-based gesture recognition system can not only help autonomous vehicles better understand pedestrians' intentions and simple gesture command recognition, but also provide more reliable navigation decisions for autonomous vehicles in complex traffic environments.

[0079] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0081] Figure 1 Schematic diagram of the ByteTrack target tracking network process;

[0082] Figure 2 Schematic diagram of the structure of the multimodal gesture recognition model. DETAILED DESCRIPTION

[0083] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0084] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0085] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0086] See also Figures 1 and 2 , which is a multimodal pedestrian command gesture recognition method that integrates large models.

[0087] Example

[0088] This embodiment proposes specific implementation steps of a multimodal pedestrian command gesture recognition method that integrates a large model. The method includes the following steps:

[0089] S1. Collect pedestrian datasets from various locations and time periods and divide the datasets;

[0090] S2. Use the pre-trained pedestrian detection model to extract human key points from the dataset, and use GPT-4 to identify the dataset and generate corresponding text prompts;

[0091] S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information;

[0092] S4. establishing a multimodal action recognition model for recognizing gestures, which includes at least a skeleton encoder and a text encoder;

[0093] S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain the recognition results.

[0094] In step S1 of this embodiment, a large number of diverse visual data sources are collected, including videos and images of pedestrian activities. These data are captured by vehicle-mounted cameras in various environments, such as urban streets, suburban areas, crosswalks, and parking lots, at different times of day and night, and in various weather conditions. Furthermore, the dataset includes images of pedestrians with various intentions, segmented from continuous video. These images include images of pedestrians walking toward the road, waiting to cross the road, signaling to stop, signaling to let a vehicle pass, and signaling a U-turn. The dataset is then split into training, validation, and test sets using an 80 / 10 / 10 ratio.

[0095] In step S2 of this embodiment, a pre-trained deep learning model (OpenPose) is used as the pedestrian detection model of the present invention. The OpenPose model can identify key points of the human body in images or videos, including coordinate information of joints (such as elbows, knees, shoulders, etc.). Each key point corresponds to a specific coordinate in the image or video, and these coordinate points can depict the posture of the human body. By connecting these key points with line segments, a skeleton-like structure is formed, which clearly shows the posture and movement of the human body. At the same time, the OpenPose model can process various types of inputs in different application scenarios, such as real-time video streams, static images, etc., to ensure that the dynamic changes of the human body can be tracked and analyzed. Through the identification of these key points, the OpenPose model provides a basis for gesture recognition, posture estimation, motion analysis, etc. The specific process includes:

[0096] S201. Input image: Input a static image or video frame. The input image may contain one or more human bodies.

[0097] S202, Feature Extraction Network: Use deep convolutional neural networks (VGG) to extract high-level features of images. These networks are pre-trained and can effectively extract feature maps from images.

[0098] S203, dual-branch architecture: uses two branches to process image information in parallel:

[0099] S2031, joint heat map: This branch predicts the probability heat map (probability distribution of joint positions) of each joint in the image;

[0100] S2032, Skeletal Association Graph: This branch is used to predict the lines or connections between joints, which helps the model understand the relationship between different joints;

[0101] S204, multi-stage processing: progressively optimizing the predictions through a multi-stage approach, where each stage utilizes the output of the previous stage to further adjust and optimize the current joint detection and connection predictions, including preliminary calculation of heatmaps and correlation maps in stage 1 and continuous optimization of correlation predictions between heatmaps and joints in stage 2 and subsequent stages based on the output of the previous stage;

[0102] S205, non-maximum suppression: After multiple stages, OpenPose will apply non-maximum suppression to retain the best detection results for each joint and avoid redundant detection;

[0103] S206, Post-processing: After obtaining the joint heatmap and skeletal association graph, the system connects the joints into a complete skeleton. The skeletal association graph can effectively identify which joints belong to the same person, avoiding confusion among multiple people's joints.

[0104] S207, output: The final output is a skeleton diagram, which marks the key joints of the human body and their connections (such as shoulders, elbows, hips, knees, etc.);

[0105] Secondly, the process of using GPT-4 to generate text descriptions is to input the dataset into GPT-4 and let it describe the pedestrian's actions in the same scene in the following three sections:

[0106] S208 Paragraph description: Use a paragraph to summarize. For example, the man repeats the same up-and-down waving motion on the side of the road, waving his arms up and down to signal passing drivers to stop. His movements are forceful and clear.

[0107] S209 Synonyms: Collect synonyms for 10 action labels, such as waving to stop a car on the roadside, waving to stop a car, raising hands to signal the driver to stop, waving to stop a car on the roadside, waving to the driver to stop, gesturing to stop a car on the roadside, waving at a car to stop, waving at the driver to stop a car, gesturing to stop a car with gestures, and reaching out to the car to signal the driver to stop.

[0108] S210 Body part descriptions: Collect descriptions of different body parts for each action, for example, the head is tilted slightly forward, the arms are waving vigorously up and down in the air, the palms are spread, the fingers are naturally extended, and the swinging amplitude is moderate. His hips are slightly tilted back to maintain the balance of the body, his legs are firmly on the ground, and the center of gravity of the body is evenly distributed between the two feet.

[0109] In step S3 of this embodiment, the ByteTrack network model is used as the target tracking network of the present invention. The ByteTrack network model is trained with data marked by the openpose model to obtain a model for pedestrian tracking. The extracted skeleton data undergoes certain processing, such as normalization, data enhancement, and joint repair, so that the neural network model can better learn and process this information. In addition, the skeleton data can also be converted into time series data (such as gesture changes in the video) to analyze dynamic gestures. The specific process is as follows: Figure 1 Shown, including:

[0110] S301. Detect input: First, the algorithm detects the input data, namely the skeleton key points in the image frame.

[0111] S302, detection information: Calculate the confidence value based on the detected key point information. If the confidence value is greater than the predefined threshold (track_thresh), proceed to S303; otherwise, jump to S308;

[0112] S303: If the confidence value is high, it means that it has been detected. At this time, the algorithm will try to match the key point with the existing trajectory and determine whether the parameter is_activated is true: if the current trajectory has been activated (is_activated=True), then enter S304; otherwise jump to S306;

[0113] S304, predict the next position of the current trajectory, and compare whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, go to S305; if the match succeeds, jump to S309;

[0114] S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307;

[0115] S306. Match the key points that are not matched with the detection box with the existing inactive (is_activated=false) track again; if the match is successful, convert the existing track to the active state (is_activated=true) and mark the key points as tracked (state=tracked); if the match fails again, remove the tracking box and determine again whether the confidence of the detection box is greater than the threshold high_treshs. If it is greater than the threshold, create a new track, mark the key points as tracked, and mark the track as inactive;

[0116] S307: Check whether the key points that do not match the tracking frame are in the tracked state. If so, proceed to S308; if not, delete the tracks whose continuous loss time exceeds the maximum time (max_time_lost);

[0117] S308 , matching the key points with a confidence score lower than track_thresh with the track of the unmatched tracking box. If the match is successful, proceed to S309 , otherwise proceed to S310 .

[0118] S309: If the match is successful, the state is marked as tracked.

[0119] S310: If the match fails, the state is marked as lost.

[0120] In the ByteTrack network, when a new skeleton keypoint is detected, the algorithm attempts to match it to an existing track. If a match is successful, the track information is updated; otherwise, a new track is created. If no track is matched for a period of time, the target is considered lost and the track is deleted. This process repeats until all skeleton keypoints have been processed.

[0121] In step S4 of this embodiment, a multimodal gesture recognition framework is used, such as Figure 2 As shown, a skeleton encoder E s and a text encoder E t They are used to generate skeleton features and text features respectively.

[0122] The training loss can be expressed as:

[0123]

[0124] Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM. s Cross entropy loss and multi-part contrast loss are used for training, with partial text descriptions as additional guidance. The text guidance content is generated by LLM, and the fully connected layer connects the aligned skeleton features with the text features.

[0125] S41, skeleton encoder:

[0126] The highly efficient and powerful Graph Convolutional Network (GCN) is used as the skeleton encoder backbone network. The skeleton encoder consists of multiple graph convolutional layers and multiple multi-scale temporal convolutional modules. In the backbone network graph layer, the human skeleton can be represented as a graph G = {V, ε}, where V is the set of human joints, |V| = N, and ε is the set of edges. H∈R represents the features of the human joints at the lth layer with feature dimension F. Graph convolution can be expressed as follows:

[0127]

[0128] Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W l is the learnable parameter of layer l, and δ is the activation function.

[0129] S411. The present invention utilizes a multi-scale temporal convolution module for temporal modeling at different temporal speeds. This module consists of four distinct branches, each of which contains 1×1 convolutions to reduce channel dimensionality. There are two temporal convolution branches with variable dilation branches (1 and 2) and a max pooling layer branch, and a fourth branch containing only 1×1 convolutions. The outputs of the four branches are concatenated to generate the final result.

[0130] S412, Skeleton Classification: Skeleton-based action recognition methods map human skeleton data to one-hot encodings of action labels, which are trained using cross-entropy loss:

[0131] L cts =-ylogp θ (x)

[0132] Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

[0133] S42, Text Encoder

[0134] The present invention uses a pre-trained text encoder based on a language model of Transformer architecture as the text encoder E t The input data is given in the form of text and first undergoes standard word segmentation. Then, these segmented features are processed by a series of Transformer modules, and the final output is a feature vector used to represent the text description. For different parts of the human body, the present invention will divide them into several parts, generate description texts for each part, and use them as input to the text encoder, which includes,

[0135] S421. Input text preprocessing

[0136] S4211. Tokenization: First, the text needs to be segmented into smaller units, called tokens. Tokenization can be done at the word, subword, or even character level. For example, the sentence "I love cats" would be tokenized like ["I", "love", "cats"].

[0137] S4212, Tokenization and Mapping to Index: Each token is mapped to a unique integer index, which is done by looking up the vocabulary.

[0138] S422. Word embedding process: The text encoder converts the input token index into a dense vector representation through the embedding layer. These embedded vectors represent the semantic information of each word or subword in a high-dimensional space. The embedding method uses Word2Vec. The vector representation after the embedding is: [[0.1,0.3,...],[0.2,0.5,...],[0.4,0.1,...]], where each word is converted into a high-dimensional vector.

[0139] S423, Positional encoding: Because there is no inherent order perception ability, positional encoding is needed to represent the order information of the input tokens for the Transformer encoder. The process is to add the encoding generated by the sine and cosine functions to the word embedding so that the model can identify the relative position of the token in the sequence;

[0140] S424, contextual information modeling: A multi-head self-attention mechanism is used to enable each token to interact with other tokens in the input sequence without relying on sequential processing. Each word can see all the words in the entire sentence and perform a weighted sum based on their relevance to capture contextual relationships. After passing through multiple Transformer encoder layers, the embedding vector of each input word is gradually converted into a high-dimensional representation that contains the context.

[0141] S425, Global Semantic Representation Generation: After the encoder processes the input sequence, each word will receive a context-enhanced embedding vector. These vectors contain not only the semantic information of the word itself, but also the dependencies between the word and other words. The model uses the last token in the sentence (the [EOS] token) as the global representation vector for the entire sentence.

[0142] S426, output representation

[0143] The final output representation is the semantic representation of the entire sentence (such as in text classification tasks). These representation vectors can be further used for downstream tasks such as classification, generation, matching, etc.

[0144] In step S5 of this embodiment, a multi-part contrastive learning loss is used to train and optimize a multimodal gesture recognition model. Action descriptions are supervised by natural language-based contrastive learning of skeleton data using a skeleton encoder E and a text encoder E. This is a dual-encoder design that encodes the skeleton data and action descriptions separately. The dual encoders jointly optimize by comparing skeleton-text pairs in two directions within a batch:

[0145]

[0146] Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size. Unlike the one-to-one matching between images and text in CLIP, this deployment includes multiple positive matches, and gestures of different categories are considered negative matches. When calculating the loss, KL divergence is chosen as the contrast loss between skeleton and text:

[0147]

[0148] Where D is the entire dataset, y s2t and y t2s Ground truth similarity score, with probability 0 for negative pairs and probability 1 for positive pairs.

[0149] S51. Multi-part contrastive learning:

[0150] The present invention divides the skeleton into four regions and applies contrast loss to the local features and overall features of these regions respectively, using the multi-part contrast loss method. Specifically, the part features are obtained through joint pooling, which is a method of aggregating the joint features within the same group to generate the corresponding part representation. The human body is divided into four parts: head, arms, torso (hips) and legs and feet. The loss function of the multi-part contrast loss can be expressed as follows:

[0151]

[0152] Where K is the total number of groups.

[0153] S52, Pedestrian Gesture Recognition

[0154] When directing traffic, ordinary pedestrians often make some non-standard gestures due to lack of professional training. The following are some common pedestrian gestures:

[0155] To signal a lane change: raise your hand horizontally or wave your palm outwards;

[0156] Signal to stop: raise your hand horizontally and swing it up and down, wave your arm quickly, raise one hand high, and wave your palm outward;

[0157] To signal a U-turn: draw a circle with your arms, an arc with your forearms, and a semicircle with your palms;

[0158] To indicate walking straight: extend your arms straight forward and point them forward, with your palms pushed forward;

[0159] Let the model record the four common pedestrian gestures for recognition;

[0160] Furthermore, we use Transformer as the decoder and the Softmax function as the activation function to output the probabilities of the four command action categories at the current moment. The action with the highest probability is the current command action. The process includes:

[0161] S521. Input Embedding: Input data (feature vectors containing time series) are passed to the Transformer decoder. The Transformer decoder processes these inputs using an attention mechanism. The decoder gradually extracts and integrates the contextual information of the input data through multi-head self-attention layers and feedforward network layers. The attention mechanism allows the model to capture the importance of different positions in the input sequence and process the features at each moment. The decoder outputs a series of hidden vectors (often called hidden states) that contain rich information about the input.

[0162] S522, Linear Transformation: After the output of the Transformer decoder, a linear layer is used to map the decoder's hidden vector to the output space. Because four command action categories (action 1, action 2, action 3, action 4) need to be predicted, the linear layer will convert the hidden state of each time step into four numerical values.

[0163] S523, Softmax activation function: The four values ​​output by the linear layer do not directly represent the probability of the category. These values ​​need to be converted into a probability distribution through the Softmax activation function. The Softmax function converts these four values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1.

[0164] For the prediction at each time step, the Softmax formula is:

[0165]

[0166] Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i;

[0167] S524, Action Prediction: After obtaining the probabilities of the four categories through Softmax output, the category with the highest probability is selected as the command action at the current moment. This process can be expressed as:

[0168] A=argmax(P(y1),P(y2),P(y3),P(y4))

[0169] That is, choose the category with the highest probability among the four categories.

[0170] After evaluating the system using the test set, we check its accuracy in recognizing gestures and generating text descriptions. Based on the evaluation results, we fine-tune the computer vision model and language model to improve performance.

[0171] In practical applications, a portion of the trained gesture detection model is cut out and integrated into self-driving cars to detect pedestrian gestures in real time, identify their intentions, and respond accordingly. After deployment on the vehicle, more data is collected to continuously train and optimize the model.

[0172] The specific implementation process is as follows:

[0173] 1. Dataset Collection

[0174] Collect many different visual data sources: videos and images of pedestrian activities, which are captured by car-mounted cameras in different environments such as urban streets, suburbs, crosswalks, parking lots, and at different time periods such as day and night, as well as in different weather conditions. Among them, the optional dataset NTURGB+D is a dataset widely used for skeleton-based human action recognition, which contains 56,880 skeleton action sequences with 20 subjects. There is also the NW-UCLA dataset, which is recorded by three Kinect V1 sensors from different angles. The skeleton contains 20 joints and 19 bone connections. It includes 1,494 video sequences of 10 action categories.

[0175] 2. Neural Network Target Detection Network:

[0176] (1) Pedestrian dataset preparation: Pedestrian datasets are widely available in open source datasets and can be used directly. Then, a portion of the pedestrian recognition dataset is collected by ourselves, and then annotated using annotation software. The annotated categories are sundry and pedestrian. Finally, the annotated data is divided into a training set: validation set ratio of 5:5 to obtain a complete dataset for training.

[0177] (2) Pedestrian detection model training: Figure 2 The following figure shows the block diagram of the neural network object detection network. This model architecture is used to build the detection network in the PyTorch deep learning framework. The initial learning rate is 0.01. After every 10 epochs, the learning rate is multiplied by 0.1 until the maximum number of training epochs is reached. The batch size range is 64 to 128, and the Adam optimizer is selected.

[0178] (3) Model evaluation: After training is completed, the test set is used to evaluate the performance of the model, and adjustments and optimizations are made based on the evaluation results.

[0179] 3. Target tracking network:

[0180] (1) Pedestrian tracking dataset preparation: Use the trained neural network model to obtain detection results and prepare a continuous pedestrian tracking dataset.

[0181] (2) Pedestrian tracking model training: Use the extracted feature vectors and annotation information to train the ByteTrack network. The model weights can be initialized to weights that have been pre-trained on a large-scale dataset, and then trained on your own dataset. After training, use the test set to evaluate the performance of the model, and adjust and optimize it based on the evaluation results.

[0182] (3) Data association: The ByteTrack algorithm uses a cascade matching algorithm for data association, which calculates the similarity between targets based on their motion and appearance features, and matches the target in the current frame with the existing trajectory.

[0183] (4) Trajectory update: The ByteTrack algorithm updates the trajectory according to the results of data association. For successfully matched targets, their trajectories will be extended. For unmatched targets, they will be regarded as new targets and new trajectories will be created.

[0184] 4. Multimodal action recognition model:

[0185] (1) Data preprocessing

[0186] The data in the dataset is converted to 64 frames. The present invention adopts the currently best CTR-GCN with multi-scale temporal convolution. In the training model stage, the batch size is 110, the epoch is set to 64, the learning rate is set to 0.2, the weight decay is set to 4e-4, the reduction step is set to [90,100], and the warm-up epoch is set to 5.

[0187] (2) Model training:

[0188] The framework of this invention was implemented using PyTorch, with the model seed set to 1. Mixed-precision training was employed to accelerate training and reduce memory usage. A large language model was used as the knowledge engine for gesture recognition. By combining label names with synonyms for body part descriptions for prompts, detailed descriptions of body parts were automatically generated, reducing the complex process of manually annotating data.

[0189] 5. Pedestrian gesture recognition and classification

[0190] Transformer is used as the decoder. The input data is passed into the Transformer decoder. The activation function uses the Softmax function to output the probabilities of the four command action categories at the current moment. The current command action has the highest probability.

[0191] 6. Real-time detection phase of the vehicle-mounted pedestrian command gesture recognition system:

[0192] The model format file with the best results after PyTorch framework training is pt format data, which is converted to onnx format file and then deployed on autonomous driving vehicles for pedestrian command action recognition.

[0193] First, the video data captured by the on-board camera is read. Then the neural network target detection network distinguishes between pedestrians and pedestrians. The detection results are then input into the ByteTrack network to track the pedestrian's command movements. Finally, they are input into the action detection network to identify and classify the action categories.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal pedestrian hand gesture recognition method integrating a large model, characterized by: The method comprises the following steps: S1. Collect pedestrian datasets from various locations and time periods and divide the datasets; S2. Use the pre-trained pedestrian detection model to extract human key points from the dataset and generate corresponding text prompts in the dataset; S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information; S4. establishing a multimodal action recognition model for recognizing gestures, which includes at least a skeleton encoder and a text encoder; S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain recognition results; In step S4, the multimodal gesture recognition model includes at least a skeleton encoder E s and text encoder E t , which are used to generate skeleton features and text features respectively, where Skeleton Encoder E s A graph convolutional network is used as the backbone network, which includes multiple graph convolution layers and multiple multi-scale temporal convolution modules. Text Encoder E t A language model based on the Transformer architecture is used, which first performs standard word segmentation and then processes it through the Transformer module to output a feature vector representing the text description. In step S5, E s Cross entropy loss and multi-part contrastive loss are used for training, and some text descriptions are used as additional guidance. The text guidance content is generated by GPT-4. The fully connected layer connects the aligned skeleton features and text features. The following loss function is used to train the model: Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM; Action description is done with a skeleton encoder E s and a text encoder E t The dual encoder encodes the skeleton data and operation description respectively, where the dual encoder jointly optimizes by comparing the skeleton-text pairs in two directions in the batch: Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size; When calculating the loss, we choose to use KL divergence as the contrast loss between skeleton and text: Where D is the entire dataset, y s2t and y t2s Ground truth similarity score, with probability 0 for negative pairs and 1 for positive pairs; Step S5 includes the following steps: S51. Multi-part contrast learning: The skeleton is divided into several regions, and multi-part contrast loss is applied to the local features and overall features of each region. Part features are obtained by aggregating joint features within the same group through joint pooling. The regions after skeleton division include: head, arms, torso, legs and feet. The loss function of multi-part contrast loss is expressed as: Where K is the total number of groups; S52, Pedestrian Gesture Recognition: N pedestrian gestures are pre-entered into the model, which contain the pedestrian intention information corresponding to the gestures. Then, Transformer is used as the decoder, and the activation function adopts the Softmax function to output the probabilities of N command action categories at the current moment. The current command action has the highest probability.

2. The multimodal pedestrian hand gesture recognition method integrating a large model according to claim 1 is characterized by: In step S1, different visual data sources are collected, which are divided into at least videos and images containing pedestrian activities; The visual data source covers videos and images of pedestrian activities in different scenes, at different times, and under different weather conditions; Furthermore, pedestrian activities cover at least a variety of pedestrian intentions, including: pedestrians walking towards the road, pedestrians waiting to cross the road, pedestrians signaling to stop, pedestrians signaling vehicles to pass, and pedestrians signaling to turn around; All collected visual data sources are divided into training set, validation set and test set according to preset ratios.

3. The multimodal pedestrian command gesture recognition method integrating a large model according to claim 2 is characterized by: In step S2, the pedestrian detection model uses the OpenPose deep learning model, which identifies the coordinate information of the key points of the human body in the video or image and connects the key points with line segments according to the corresponding coordinate information to obtain the skeleton structure of the human body. The process includes: S201, input image: input a static image or video frame, the input image contains one or more human bodies; S202, Feature Extraction Network: Use the pre-trained deep convolutional neural network VGG to extract high-level features of the image; S203, dual-branch architecture: Two branches are used to process the information in the image in parallel, including a joint heat map branch and a skeletal association map branch, wherein: The joint heatmap branch predicts the probability heatmap of each joint in the image, i.e., the probability distribution of joint locations; The skeleton association graph branch is used to predict the lines or connections between joints, helping the model understand the relationship between different joints; S204, multi-stage processing: progressively optimizing the predictions through a multi-stage approach, where each stage utilizes the output of the previous stage to further adjust and optimize the current joint detection and connection predictions; S205, non-maximum suppression: After multiple stages, non-maximum suppression is applied to retain the best detection results of each joint; S206, post-processing stage: after obtaining the joint heat map and the bone association map, the joint points are connected to form a complete skeleton; S207, output: finally output the skeleton diagram, in which the key joints of the human body and their connections are marked; Secondly, the GPT-4 model is used to identify the dataset to generate text prompts of pedestrian actions. The GPT-4 model generates text descriptions in the form of paragraph descriptions, synonym descriptions, and part descriptions.

4. The multimodal pedestrian hand gesture recognition method integrating a large model according to claim 3 is characterized by: In step S3, the target tracking network uses the ByteTrack network model, which converts the extracted skeleton structure data into time series data. The steps are as follows: S301, detecting input data, i.e., skeleton key points in an image frame; S302, calculate the confidence value based on the detected key point information. If the confidence value is greater than the predefined threshold track_thresh, proceed to S303; otherwise, jump to S308; S303, try to match the key point to the existing trajectory, and determine whether the parameter is_activated is true: if the current trajectory has been activated, go to S304; otherwise jump to S306; S304, predict the next position of the current trajectory, and compare whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, go to S305; if the match succeeds, jump to S309; S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307; S306, matching the key points of the unmatched detection frame with the inactivated existing track again; If the match is successful, the existing track is converted to the active state and the key point is marked as tracked; If the matching fails again, the tracking box is removed and the confidence of the detection box is determined again to be greater than the threshold high_treshs. If it is greater than the threshold, a new track is created and the key points are marked as tracked and the track is marked as inactive. S307: Check whether the key points that do not match the tracking frame are in the tracked state. If so, proceed to S308; if not, delete the tracks whose continuous loss time exceeds the maximum time max_time_lost; S308, matching the key points with a confidence score lower than track_thresh with the track of the unmatched tracking box. If the match is successful, proceed to S309, otherwise proceed to S310; S309: Matching is successful, and the status is marked as tracked; S310: Matching fails, and the status is marked as discarded.

5. The multimodal pedestrian hand gesture recognition method integrating a large model according to claim 1 is characterized by: In the skeleton encoder E s In the graph layer, the human skeleton is represented as a graph G = {V, ε}, where V is the set of human joints, |V| = N, and ε is the set of edges; H∈R represents the features of the l-th layer of human joints with feature dimension F, then the graph convolution is expressed as follows: Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W l is the learnable parameter of layer l, and δ is the activation function; The multi-scale temporal convolution module is then used for temporal modeling: the multi-scale temporal convolution module consists of four different branches, each of which contains 1×1 convolution. The first and second branches are temporal convolution branches with variable dilation branches, the third branch is the maximum pooling layer branch, and the fourth branch only contains 1×1 convolution. The outputs of the four branches are connected to generate the final result. The skeleton-based action recognition method maps human skeleton data to one-hot encoding of action labels for classification, and the labels are trained using cross entropy loss: L cts =-ylogp θ (x) Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

6. The multimodal pedestrian command gesture recognition method integrating a large model according to claim 1 is characterized by: Text Encoder E t The data processing process includes: S401, preprocessing the input text, wherein the preprocessing process first performs word segmentation to divide the text into basic units, each basic unit is a token, and each token is mapped to a unique integer index; S402, word embedding process: In the embedding layer, the Word2Vec method is used to convert the input token index into a dense vector representation. The embedded vector represents the semantic information of each word or subword in a high-dimensional space; S403, Positional encoding: Positional encoding is introduced to represent the order information of the input tokens. Positional encoding refers to adding the codes generated by sine and cosine functions to the word embedding, so that the model can identify the relative position of the tokens in the sequence; S404, Contextual Information Modeling: Using a multi-head self-attention mechanism, each token interacts with other tokens in the input sequence, and each token is weighted and summed based on its relevance to other tokens. Through multiple text encoder layers, each word embedding vector in the input is gradually converted into a high-dimensional representation that includes context. S405, global semantic representation generation: use the last token in the sentence, i.e., the [EOS] token, as the global representation vector of the entire sentence; S406. The final output representation is the semantic representation of the entire sentence.

7. The multimodal pedestrian hand gesture recognition method integrating a large model according to claim 1 is characterized by: In step S52, The following processes are included: S521, Input Embedding: The input data containing the time series feature vector is passed to the Transformer decoder. The decoder gradually extracts and integrates the context information of the input data through the multi-head self-attention layer and the feedforward network layer, and the decoder outputs a series of hidden layer vectors. S522, Linear Transformation: Use a linear layer to map the decoder's hidden vector to the output space. The linear layer converts the hidden state of each time step into N numerical values. S523, Softmax activation function: The Softmax activation function converts N values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1; For the prediction at each time step, the Softmax formula is: Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i; S524, Action Prediction: After obtaining the probabilities of N categories through Softmax output, the category with the highest probability is selected as the command action A at the current moment: A=argmax(P(y1),P(y2),…,P(y N )) A represents the final recognized action type, and the pedestrian intention is obtained based on the recognized action type.

Images