Multi-modal pedestrian command gesture recognition method fused with large model

Abstract

The invention relates to a multi-modal pedestrian command gesture recognition method fused with a large model, and belongs to the field of automatic driving. The method comprises the following steps: collecting a pedestrian data set, and carrying out data set division; extracting human body key points from the data set by adopting a pre-trained pedestrian detection model; processing the extracted human body key point information through a target tracking network to obtain dynamic key point track information; establishing a multi-modal action recognition model for recognizing gestures, wherein the multi-modal action recognition model at least comprises a skeleton encoder and a text encoder; and the multi-modal gesture recognition model is trained and optimized through multi-part comparison learning loss, and a recognition result is obtained. The trained model is installed on an automatic driving automobile, pedestrian gestures are distinguished, and the intention of pedestrians is accurately judged. According to the method, a multi-modal framework is adopted, the recognition precision is high, and the recognition speed is high.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving and relates to a multi-modal pedestrian command gesture recognition method integrating a large model. Background Art

[0002] Autonomous vehicles often display overly cautious behavior when dealing with pedestrian interactions. This phenomenon is particularly evident in the current popular autonomous taxi services, such as the autonomous taxi "Robot Run". When these vehicles are driving on the road, they often stop immediately after seeing pedestrians, regardless of whether the pedestrians actually intend to cross the road. This overly courteous behavior not only increases the waiting time for passengers, but may also lead to a decrease in traffic flow and even cause traffic jams. Behind this phenomenon, it reflects that autonomous driving technology still faces many challenges when dealing with complex road environments.

[0003] One of the main reasons why driverless vehicles are overly cautious in front of pedestrians is the inadequacy of current pedestrian gesture recognition technology. Existing pedestrian gesture recognition systems still have a lot of room for improvement in terms of accuracy, response speed, and lightweight models. Many driverless systems rely on pre-set rules and behavior patterns to make decisions, and these rules are often based on the most conservative assumptions to ensure the absolute safety of pedestrians. However, overly conservative algorithm design also brings side effects, that is, in some cases, vehicles may stop frequently when it is not necessary to let pedestrians pass, and even when pedestrians have no clear intention to cross the road, vehicles will still choose to stop and wait. This situation will obviously have an adverse impact on the efficiency of the entire transportation system.

[0004] The current decision-making systems of driverless vehicles mainly rely on a series of rule-based algorithms, which often have difficulty coping with complex and dynamically changing road scenes. For example, on some busy roads or intersections, the movements of pedestrians may be extremely uncertain, and sometimes pedestrians may even suddenly change direction or wander around zebra crossings. It is difficult for traditional algorithms to effectively distinguish whether pedestrians really intend to cross the road. Therefore, driverless vehicles often choose to stop when there is any uncertainty to minimize potential accidents. Although this safety-first principle can reduce the probability of accidents, it also reduces the operating efficiency of driverless vehicles and increases travel time and cost. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a multi-modal pedestrian command gesture recognition method integrating a large model.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-modal pedestrian command gesture recognition method integrating a large model, the method comprising the following steps:

[0008] S1. Collect pedestrian data sets from multiple places and multiple time periods, and divide the data sets;

[0009] S2, use the pre-trained pedestrian detection model to extract human key points from the dataset and generate corresponding text prompts in the dataset;

[0010] S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information;

[0011] S4, establishing a multimodal action recognition model for recognizing gestures, which at least includes a skeleton encoder and a text encoder;

[0012] S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain the recognition result.

[0013] Further, in step S1, different visual data sources are collected, which are at least divided into videos and images containing pedestrian activities; the visual data sources cover videos and images of pedestrian activities in different scenes, different time periods, and different weather conditions; and pedestrian activities at least cover multiple pedestrian intentions, including: pedestrians are walking towards the road, pedestrians are waiting to cross the road, pedestrians signal to stop, pedestrians signal vehicles to pass, and pedestrians signal to turn around; all collected visual data sources are divided into a training set, a validation set, and a test set according to a preset ratio.

[0014] Further, in step S2, the pedestrian detection model adopts the OpenPose deep learning model, which identifies the coordinate information of the key points of the human body in the video or image, and connects the key points with line segments according to the corresponding coordinate information to obtain the skeleton structure of the human body. The process includes:

[0015] S201, input image: input a static image or a video frame, the input image contains one or more human bodies;

[0016] S202, Feature Extraction Network: Use the pre-trained deep convolutional neural network VGG to extract high-level features of the image;

[0017] S203, dual-branch architecture: two branches are used to process the information in the image in parallel, including a joint heat map branch and a bone association map branch, wherein:

[0018] The joint heatmap branch predicts the probability heatmap of each joint in the image, i.e., the probability distribution of joint locations;

[0019] The skeleton association graph branch is used to predict the lines or connections between joints, helping the model understand the relationship between different joints;

[0020] S204, multi-stage processing: gradually optimizing its predictions through a multi-stage approach, where each stage uses the output of the previous stage to further adjust and optimize the current joint detection and connection prediction;

[0021] S205, non-maximum suppression: After multiple stages, non-maximum suppression is applied to retain the best detection results of each joint;

[0022] S206, post-processing stage: after obtaining the joint heat map and the bone association map, the joint points are connected into a complete skeleton;

[0023] S207, output: finally output a skeleton diagram, in which key joints of the human body and their connections are marked;

[0024] Secondly, the GPT-4 model is used to identify the dataset to generate text prompts of pedestrian actions. The GPT-4 model generates text descriptions in the form of paragraph descriptions, synonym descriptions, and part descriptions.

[0025] Further, in step S3, the target tracking network adopts the ByteTrack network model, which converts the extracted skeleton structure data into time series data, and the steps are:

[0026] S301, detecting input data, i.e. skeleton key points in an image frame;

[0027] S302, calculate the confidence value according to the detected key point information, if the confidence value is greater than the predefined threshold track_thresh, then enter S303; otherwise jump to S308;

[0028] S303, try to match the existing trajectory for the key point, and determine whether the parameter is_activated is true: if the current trajectory has been activated, go to S304; otherwise jump to S306;

[0029] S304, predicting the next position of the current trajectory, and comparing whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, proceed to S305; if the match succeeds, jump to S309;

[0030] S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307;

[0031] S306, match the key points of the unmatched detection frame with the inactivated existing track again; if the match is successful, convert the existing track to the activated state, and mark the key points as tracked; if the match fails again, remove the tracking frame, and determine again whether the confidence of the detection frame is greater than the threshold high_treshs, if it is greater than the threshold, create a new track, mark the key points as tracked, and mark the track as inactivated;

[0032] S307, detecting whether the key points that do not match the tracking frame are in the tracked state, if so, proceeding to S308; if not, deleting the tracks whose continuous loss time exceeds the maximum time max_time_lost;

[0033] S308, matching the key points with a confidence level lower than track_thresh with the track of the unmatched tracking box, if the match is successful, proceed to S309, otherwise proceed to S310;

[0034] S309: Matching is successful, and the status is marked as tracked;

[0035] S310: If the match fails, the status is marked as discarded.

[0036] Further, in step S4, the multimodal gesture recognition model includes at least a skeleton encoder E s and text encoder E t , which are used to generate skeleton features and text features respectively, where

[0037] Skeleton Encoder E s A graph convolutional network is used as the backbone network, which includes multiple graph convolutional layers and multiple multi-scale temporal convolutional modules.

[0038] Text EncoderE t A language model based on the Transformer architecture is used, which first performs standard word segmentation and then is processed by the Transformer module to output a feature vector used to represent the text description.

[0039] Furthermore, in the skeleton encoder E s In the graph layer of , the human skeleton is represented as a graph G = {V, ε}, where V is the set of human joints, |V| = N, and ε is the set of edges; H∈R represents the features of the l-th layer of human joints with feature dimension F, then the graph convolution is expressed as follows:

[0040]

[0041] Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W lis the learnable parameter of the lth layer, and δ is the activation function;

[0042] Then use the multi-scale temporal convolution module for temporal modeling: the multi-scale temporal convolution module consists of four different branches, each of which contains 1×1 convolution, where the first and second branches are temporal convolution branches with variable dilation branches, the third branch is the maximum pooling layer branch, and the fourth branch only contains 1×1 convolution; the outputs of the four branches are connected to generate the final result;

[0043] The skeleton-based action recognition method maps human skeleton data to one-hot encoding of action labels for classification, and the labels are trained using cross entropy loss:

[0044] L cts =-ylogp θ (x)

[0045] Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

[0046] Furthermore, the text encoder E t The data processing process includes:

[0047] S401, preprocessing the input text, wherein the preprocessing process first performs word segmentation processing to divide the text into basic units, each basic unit is a token, and each token is mapped to a unique integer index;

[0048] S402, word embedding process: In the embedding layer, the Word2Vec method is used to convert the input token index into a dense vector representation, and the embedded vector represents the semantic information of each word or subword in a high-dimensional space;

[0049] S403, position encoding: position encoding is introduced to represent the order information of the input tokens. Position encoding refers to adding the codes generated by sine and cosine functions to word embeddings so that the model can identify the relative position of the tokens in the sequence;

[0050] S404, context information modeling: using a multi-head self-attention mechanism, each token interacts with other tokens in the input sequence, and each token is weighted and summed according to its correlation with other tokens; through multiple text encoder layers, each word embedding vector of the input is gradually converted into a high-dimensional representation containing context;

[0051] S405, global semantic representation generation: using the last token in the sentence, i.e., the [EOS] token, as the global representation vector of the entire sentence;

[0052] S406. The final output representation is the semantic representation of the entire sentence.

[0053] Further, in step S5, E s The model is trained using cross entropy loss and multi-part contrast loss, and some text descriptions are used as additional guidance. The text guidance content is generated by GPT-4. The fully connected layer connects the aligned skeleton features and text features. The following loss function is used to train the model:

[0054]

[0055] Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM.

[0056] Action description is done with a skeleton encoder E s and a text encoder E t The dual encoders encode the skeleton data and operation descriptions respectively, where the dual encoders jointly optimize by comparing skeleton-text pairs in two directions in the batch:

[0057]

[0058] Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size;

[0059] When calculating the loss, choose to use KL divergence as the contrast loss between skeleton and text:

[0060]

[0061] Where D is the entire data set, y s2t and t2s Ground truth similarity scores, with probability 0 for negative pairs and 1 for positive pairs.

[0062] Further, step S5 includes the following steps:

[0063] S51. Multi-part contrast learning: The skeleton is divided into several regions, and multi-part contrast loss is applied to the local features and overall features of each region respectively. The part features are obtained by aggregating the joint features in the same group through joint pooling. The regions after skeleton division include: head, arms, torso, legs and feet. The loss function of multi-part contrast loss is expressed as:

[0064]

[0065] Where K is the total number of groups;

[0066] S52, pedestrian gesture recognition: N pedestrian gestures are pre-entered into the model, which contain the pedestrian intention information corresponding to the gestures. Then Transformer is used as a decoder, and the activation function adopts the Softmax function to output the probabilities of N command action categories at the current moment. The one with the highest probability is the current command action.

[0067] Further, in step S52, the following process is included:

[0068] S521, input embedding: The input data containing the time series feature vector is passed into the Transformer decoder. The decoder gradually extracts and integrates the context information of the input data through the multi-head self-attention layer and the feedforward network layer. The decoder outputs a series of hidden layer vectors.

[0069] S522, linear transformation: use a linear layer to map the hidden layer vector of the decoder to the output space, and the linear layer converts the hidden state of each time step into N numerical values;

[0070] S523, Softmax activation function: The Softmax activation function converts N values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1;

[0071] For the prediction at each time step, the Softmax formula is:

[0072]

[0073] Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i;

[0074] S524, action prediction: After obtaining the probabilities of N categories through Softmax output, select the category with the largest probability as the command action A at the current moment:

[0075] A = argmax(P(y 1 ),P(y 2 ),…,P(y N ))

[0076] A represents the final recognized action type, and the pedestrian intention is obtained based on the recognized action type.

[0077] The beneficial effects of the present invention are:

[0078] The present invention is applicable to the field of autonomous driving. It uses a vehicle-mounted camera to collect the surrounding environment in real time, and then uses a lightweight multimodal model to distinguish pedestrian gestures, so as to accurately judge the intentions of pedestrians. The method of the present invention adopts a multimodal framework with high recognition accuracy and fast recognition speed. It can be well deployed on the vehicle system and is very suitable for autonomous driving vehicles. By integrating a multimodal framework of large models and deep learning, the present invention can run efficiently on the limited computing resources of unmanned vehicles and achieve real-time environmental perception. This multimodal-based gesture recognition system can not only help autonomous driving vehicles better understand the intentions of pedestrians and simple gesture command recognition, but also provide more reliable navigation decisions for autonomous driving vehicles in complex traffic environments.

[0079] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0081] Figure 1 Schematic diagram of the ByteTrack target tracking network process;

[0082] Figure 2 Schematic diagram of the structure of the multimodal gesture recognition model. DETAILED DESCRIPTION

[0083] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0084] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0085] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0086] See also Figure 1-2 , which is a multimodal pedestrian command gesture recognition method that integrates a large model.

[0087] Example

[0088] This embodiment proposes specific implementation steps of a multi-modal pedestrian command gesture recognition method integrating a large model, and the method includes the following steps:

[0089] S1. Collect pedestrian data sets from multiple places and multiple time periods, and divide the data sets;

[0090] S2. Use the pre-trained pedestrian detection model to extract human key points from the dataset, and use GPT-4 to identify the dataset and generate corresponding text prompts;

[0091] S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information;

[0092] S4, establishing a multimodal action recognition model for recognizing gestures, which at least includes a skeleton encoder and a text encoder;

[0093] S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain the recognition result.

[0094] In step S1 of this embodiment, a large number of different visual data sources are collected, including videos and images of pedestrian activities, which are captured by vehicle-mounted cameras in different environments such as urban streets, suburbs, crosswalks, parking lots, and different time periods such as day and night, and different weather conditions. In addition, the data set contains pictures of pedestrians with various intentions, which are segmented from continuous videos, such as pedestrians walking towards the road, pedestrians waiting to cross the road, pedestrians signaling to stop, pedestrians signaling to let vehicles pass, and pedestrians signaling to turn around, etc. The data set is segmented into a training set, a validation set, and a test set according to a ratio of 80-10-10.

[0095] In step S2 of this embodiment, a pre-trained deep learning model (OpenPose) is used as the pedestrian detection model of the present invention. The OpenPose model can identify key points of the human body in an image or video, including the coordinate information of joints (such as elbows, knees, shoulders, etc.). Each key point corresponds to a specific coordinate in the image or video, and these coordinate points can depict the posture of the human body. By connecting these key points with line segments, a skeleton-like structure is formed, which clearly shows the posture and movement of the human body. At the same time, the OpenPose model can process various types of inputs in different application scenarios, such as real-time video streams, static images, etc., to ensure that the dynamic changes of the human body can be tracked and analyzed. Through the identification of these key points, the OpenPose model provides a basis for gesture recognition, posture estimation, motion analysis, etc. The specific process includes:

[0096] S201, input image: input a static image or a video frame, and the input image may contain one or more human bodies;

[0097] S202, Feature Extraction Network: Use deep convolutional neural network VGG to extract high-level features of images. These networks are pre-trained and can effectively extract feature maps from images;

[0098] S203, dual-branch architecture: two branches are used to process the information in the image in parallel:

[0099] S2031, joint heat map: This branch predicts the probability heat map (probability distribution of joint positions) of each joint in the image;

[0100] S2032, Skeletal Association Graph: This branch is used to predict the lines or connections between joints, which helps the model understand the relationship between different joints;

[0101] S204, multi-stage processing: gradually optimizing the prediction through a multi-stage approach, each stage will use the output of the previous stage to further adjust and optimize the current joint detection and connection prediction, including the initial calculation of heat maps and association maps in stage 1 and the continuous optimization of the association prediction between heat maps and joints in stage 2 and subsequent stages based on the output of the previous stage;

[0102] S205, non-maximum suppression: After multiple stages, OpenPose will apply non-maximum suppression to retain the best detection results for each joint and avoid redundant detection;

[0103] S206, post-processing stage: After obtaining the joint heat map and the bone association map, the system will connect the joint points into a complete skeleton. The bone association map can effectively identify which joint points belong to the same person, avoiding confusion of joint points of multiple people.

[0104] S207, output: The final output is a skeleton diagram, in which the key joints of the human body and their connections (such as shoulders, elbows, hips, knees, etc.) are marked;

[0105] Secondly, the process of using GPT-4 to generate text descriptions is to input the dataset into GPT-4 so that it can be divided into the following three sections to describe the actions of pedestrians in the same scene:

[0106] S208 Paragraph description: Use a paragraph to summarize, for example, this man repeats the same up and down waving motion on the roadside, waving his arms up and down, signaling the passing drivers to stop, his movements are powerful and clear;

[0107] S209 synonyms: collect synonyms for 10 action labels, such as waving to stop on the roadside, waving to stop a car, raising hands to signal the driver to stop, waving to stop a car on the roadside, waving to the driver to stop, gesturing to stop the car on the roadside, waving at the vehicle to stop, waving arms to stop the car, gesturing to stop the car, and reaching out to the car to signal the driver to stop;

[0108] S210 Body part descriptions: Collect descriptions of different body parts for each action, for example, the head is tilted slightly forward, the arms are waving vigorously up and down in the air, the palms are spread, the fingers are naturally extended, and the swinging amplitude is moderate. His hips are slightly tilted back to maintain the balance of the body, his legs are firmly on the ground, and the center of gravity of the body is evenly distributed between the two feet.

[0109] In step S3 of this embodiment, the ByteTrack network model is used as the target tracking network of the present invention. The ByteTrack network model is trained with the data marked by the openpose model to obtain a model for pedestrian tracking. The extracted skeleton data is processed in a certain way, such as normalization, data enhancement, and joint point repair, so that the neural network model can better learn and process this information. In addition, the skeleton data can also be converted into time series data (such as gesture changes in the video) to analyze dynamic gestures. The specific process is as follows: Figure 1 As shown, including:

[0110] S301, detect input: First, the algorithm detects the input data, namely the skeleton key points in the image frame.

[0111] S302, detection information: Calculate the confidence value based on the detected key point information. If the confidence value is greater than the predefined threshold (track_thresh), proceed to S303; otherwise, jump to S308;

[0112] S303: If the confidence value is high, it means that it has been detected. At this time, the algorithm will try to match the existing trajectory for the key point and determine whether the parameter is_activated is true: if the current trajectory has been activated (is_activated=True), then enter S304; otherwise jump to S306;

[0113] S304, predicting the next position of the current trajectory, and comparing whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, proceed to S305; if the match succeeds, jump to S309;

[0114] S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307;

[0115] S306, match the key points that are not matched with the detection box with the existing inactivated (is_activated=false) track again; if the match is successful, convert the existing track to the activated state (is_activated=true), and mark the key points as tracked (state=tracked); if the match fails again, remove the tracking box, and determine again whether the confidence of the detection box is greater than the threshold high_treshs, if it is greater than the threshold, create a new track, mark the key points as tracked, and mark the track as inactivated;

[0116] S307, detecting whether the key points that do not match the tracking frame are in the tracked state, if so, proceeding to S308; if not, deleting the tracks whose continuous loss time exceeds the maximum time (max_time_lost);

[0117] S308 , matching the key points with a confidence level lower than track_thresh with the track of the unmatched tracking box, if the match is successful, proceed to S309 , otherwise proceed to S310 .

[0118] S309: If the match is successful, the state is marked as tracked.

[0119] S310: If the match fails, the state is marked as lost.

[0120] In the ByteTrack network, when a new skeleton keypoint is detected, the algorithm will try to match the existing track. If the match is successful, the track information is updated, otherwise a new track is created. If no track can be matched for a period of time, the target is considered to have disappeared and the track is deleted. This process is repeated until all skeleton keypoints have been processed.

[0121] In step S4 of this embodiment, a multimodal gesture recognition framework is used, such as Figure 2 As shown, a skeleton encoder E s and a text encoder E t They are used to generate skeleton features and text features respectively.

[0122] The training loss can be expressed as:

[0123]

[0124] Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM. s Cross entropy loss and multi-part contrastive loss are used for training, with partial text descriptions as additional guidance. The text guidance content is generated by LLM, and the fully connected layer connects the aligned skeleton features with the text features.

[0125] S41, skeleton encoder:

[0126] The highly efficient and powerful graph convolutional network (GCN) is used as the skeleton encoder backbone network, where the skeleton encoder contains multiple graph convolution layers and multiple multi-scale temporal convolution modules. In the backbone network graph layer, the human skeleton can be represented as a graph G = {V, ε}, where V is the set of human joints, where |V| = N, and ε is the set of edges. H∈R represents the features of the lth layer of human joints with feature dimension F. Graph convolution can be expressed as follows:

[0127]

[0128] Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W l is the learnable parameter of the lth layer, and δ is the activation function.

[0129] S411. At different time speeds, the present invention uses a multi-scale temporal convolution module for temporal modeling. The module consists of four different branches, each of which contains 1×1 convolutions to reduce channel dimensions. There are two temporal convolution branches with variable dilation branches (1 and 2) and a maximum pooling layer branch, and the fourth branch only contains 1×1 convolutions. The outputs of the four branches are connected to generate the final result.

[0130] S412, Skeleton Classification: Skeleton-based action recognition methods map human skeleton data to one-hot encodings of action labels, which are trained using cross-entropy loss:

[0131] L cts =-ylogp θ (x)

[0132] Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

[0133] S42, Text Encoder

[0134] The present invention uses a pre-trained text encoder based on a language model of a Transformer architecture as the text encoder E t The input data is given in the form of text and first undergoes standard word segmentation. Then, the features after word segmentation are processed by a series of Transformer modules, and the final output is a feature vector used to represent the text description. For different parts of the human body, the present invention will divide them into several parts, generate description texts for each part, and use them as the input of the text encoder, which includes,

[0135] S421. Input text preprocessing

[0136] S4211. Tokenization: First, the text needs to be divided into smaller basic units, called tokens. Tokenization can be based on words, subwords, or even characters. For example, the sentence "I love cats" will be divided into tokens like ["I", "love", "cats"].

[0137] S4212, Tokenization and mapping to index: Each token is mapped to a unique integer index, which is done by looking up the vocabulary.

[0138] S422. Word embedding process: The text encoder converts the input token index into a dense vector representation through the embedding layer. These embedded vectors represent the semantic information of each word or subword in a high-dimensional space. The embedding method uses Word2Vec. The vector representation after embedding is: [[0.1,0.3,...],[0.2,0.5,...],[0.4,0.1,...]], where each word is converted into a high-dimensional vector.

[0139] S423, position encoding: Because there is no inherent order perception ability, position encoding needs to be introduced to represent the order information of the input tokens for the Transformer encoder. The process is to add the encoding generated by the sine and cosine functions to the word embedding so that the model can recognize the relative position of the token in the sequence;

[0140] S424, context information modeling: A multi-head self-attention mechanism is used to allow each token to interact with other tokens in the input sequence without relying on sequential processing. Each word can see all the words in the entire sentence and perform a weighted sum based on their relevance to capture contextual relationships. After passing through multiple Transformer encoder layers, each word embedding vector of the input is gradually converted into a high-dimensional representation that contains the context.

[0141] S425, Global semantic representation generation: After the encoder processes the input sequence, each word will get a context-enhanced embedding vector. These vectors contain not only the semantic information of the word itself, but also the dependency relationship between the word and other words. The model uses the last token in the sentence ([EOS] token) as the global representation vector of the entire sentence.

[0142] S426, output representation

[0143] The final output representation is the semantic representation of the entire sentence (such as in text classification tasks). These representation vectors can be further used for downstream tasks such as classification, generation, matching, etc.

[0144] In step S5 of this embodiment, a multi-part contrastive learning loss is used to train and optimize a multimodal gesture recognition model, and the action description is supervised by a skeleton language contrastive learning with a skeleton encoder E and a text encoder E in natural language. This is a dual encoder design that encodes the skeleton data and the action description respectively. The dual encoder jointly optimizes by contrasting the skeleton-text pairs in two directions in the batch:

[0145]

[0146] Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size. Unlike the one-to-one matching of images and texts in CLIP, the deployment here contains multiple positive matches, and gestures of different categories are considered negative matches. When calculating the loss, KL divergence is chosen as the contrast loss between skeleton and text:

[0147]

[0148] Where D is the entire data set, y s2t and t2s Ground truth similarity scores, with probability 0 for negative pairs and 1 for positive pairs.

[0149] S51. Multi-part contrastive learning:

[0150] The present invention divides the skeleton into four regions and applies contrast loss to the local features and overall features of these regions respectively, using the multi-part contrast loss method. Specifically, the part features are obtained by joint pooling, which is a method of aggregating joint features within the same group to generate corresponding part representations. The human body is divided into four parts: head, arms, torso (buttocks) and legs and feet. The loss function of multi-part contrast loss can be expressed as follows:

[0151]

[0152] Where K is the total number of groups.

[0153] S52, Pedestrian gesture recognition

[0154] When ordinary pedestrians direct traffic, they often make some irregular gestures due to lack of professional training. The following are some common pedestrian gestures:

[0155] To signal a lane change: raise your hand, hold it horizontally, and wave it outward with your palm;

[0156] Signal to stop: raise your hand horizontally and swing it up and down, wave your arm quickly, raise one hand high, and wave your palm outward;

[0157] To indicate a U-turn: draw a circle with your arms, an arc with your forearms, and a semicircle with your palms;

[0158] To indicate to walk straight: stretch your arms out and point them forward, with your palms pushed forward;

[0159] Let the model record the four common gestures of pedestrians for recognition of pedestrian gestures;

[0160] Furthermore, Transformer is used as a decoder, and the activation function adopts the Softmax function to output the probabilities of the four command action categories at the current moment. The one with the highest probability is the current command action. The process includes:

[0161] S521, Input Embedding: Input data (feature vectors containing time series) are passed to the decoder part of Transformer. Transformer decoder will process these inputs in combination with attention mechanism; the decoder will gradually extract and integrate the contextual information of the input data through multi-head self-attention layer and feedforward network layer. Through the attention mechanism, the model can capture the importance of different positions in the input sequence and process the features at each moment. The output of the decoder is a series of hidden layer vectors (usually called hidden states), which contain rich information about the input.

[0162] S522, linear transformation: After the output of the Transformer decoder, a linear layer is used to map the decoder's hidden vector to the output space. Because four types of command actions need to be predicted (action 1, action 2, action 3, action 4), the linear layer will convert the hidden state of each time step into four values.

[0163] S523, Softmax activation function: The four values ​​output by the linear layer do not directly represent the probability of the category. These values ​​need to be converted into probability distribution through the Softmax activation function. The Softmax function converts these four values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1.

[0164] For the prediction at each time step, the Softmax formula is:

[0165]

[0166] Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i;

[0167] S524, action prediction: After obtaining the probabilities of the four categories through Softmax output, the category with the largest probability is selected as the command action at the current moment. This process can be expressed as:

[0168] A = argmax(P(y 1 ),P(y 2 ),P(y 3 ),P(y 4 ))

[0169] That is, choose the category with the highest probability among the four categories.

[0170] After evaluating the system using the test set, check its accuracy in recognizing gestures and generating text descriptions, and based on the evaluation results, fine-tune the computer vision model and language model to improve performance.

[0171] In actual applications, a part of the trained gesture detection model is cut and integrated into the self-driving car to detect pedestrian gestures in real time, identify the pedestrian's gesture intention and respond accordingly. After being deployed on the vehicle, more data is collected to continuously train and optimize the model.

[0172] The specific implementation process is as follows:

[0173] 1. Dataset Collection

[0174] Many different visual data sources are collected: videos and images containing pedestrian activities, which are taken by on-board cameras in different environments such as urban streets, suburbs, crosswalks, parking lots, and different time periods such as day and night, as well as different weather conditions. Among them, the optional dataset NTURGB+D, which is a widely used dataset for skeleton-based human action recognition, contains 56,880 skeleton action sequences with 20 subjects. There is also the NW-UCLA dataset, which is recorded from different angles by three Kinect V1 sensors. The skeleton contains 20 joints and 19 bone connections. It includes 1,494 video sequences of 10 action categories.

[0175] 2. Neural network target detection network:

[0176] (1) Pedestrian dataset preparation: Pedestrian datasets are widely available in open source datasets and can be used directly. Then, a part of the pedestrian recognition dataset is collected by ourselves, and then annotated using annotation software. The annotated categories are sundry and pedestrian. Finally, the annotated data is divided into two categories: training set: validation set = 5:5, and finally a complete dataset for training is obtained.

[0177] (2) Pedestrian detection model training: Figure 2 The following is a block diagram of the neural network target detection network. This model architecture is used to build the detection network in the Pytorch deep learning framework. The initial learning rate is 0.01. After every 10 epochs of training, the learning rate is multiplied by 0.1 until the preset maximum number of training rounds is reached. The batch size range is 64 to 128, and the Adam optimizer is selected.

[0178] (3) Model evaluation: After training is completed, use the test set to evaluate the performance of the model and make adjustments and optimizations based on the evaluation results.

[0179] 3. Target tracking network:

[0180] (1) Pedestrian tracking dataset preparation: Use the trained neural network model to obtain detection results and prepare a continuous pedestrian tracking dataset.

[0181] (2) Pedestrian tracking model training: Use the extracted feature vectors and annotation information to train the ByteTrack network. The model weights can be initialized to weights that have been pre-trained on a large-scale dataset, and then trained on your own dataset. After training, use the test set to evaluate the performance of the model and make adjustments and optimizations based on the evaluation results.

[0182] (3) Data association: The ByteTrack algorithm uses a cascade matching algorithm for data association, which calculates the similarity between targets based on their motion and appearance features, and matches the target in the current frame with the existing trajectory.

[0183] (4) Trajectory update: The ByteTrack algorithm updates the trajectory according to the results of data association. For successfully matched targets, their trajectories will be extended. For unmatched targets, they will be regarded as new targets and new trajectories will be created.

[0184] 4. Multimodal action recognition model:

[0185] (1) Data preprocessing

[0186] The data in the dataset is converted into 64 frames. The present invention adopts the CTR-GCN with multi-scale temporal convolution which has the best effect currently. In the training model stage, the batch size is 110, the epoch is set to 64, the learning rate is set to 0.2, the weight decay is set to 4e-4, the reduction step is set to [90, 100], and the warm-up epoch is set to 5 respectively.

[0187] (2) Model training:

[0188] The framework of the present invention is implemented using Pytorch, and the model seed is set to 1. Mixed precision training is used to speed up training and reduce memory usage. A large language model is used as a knowledge engine for gesture recognition. By combining the synonyms of the label name and the body part description for prompts, a detailed description of the body part is automatically generated, reducing the complex process of manually annotating data.

[0189] 5. Pedestrian gesture recognition classification

[0190] Transformer is used as a decoder, and the input data is passed into the Transformer decoder. The activation function uses the Softmax function to output the probabilities of the four command action categories at the current moment. The current command action has the highest probability.

[0191] 6. Real-time detection stage of the vehicle-mounted pedestrian command gesture recognition system:

[0192] The model format file with the best results after training with the Pytorch framework is pt format data, which is converted into onnx format files and then deployed on autonomous driving vehicles for the recognition of pedestrian command actions.

[0193] First, the video data read by the on-board camera is read, and then the neural network target detection network distinguishes between pedestrians and pedestrians. The detection results are then input into the ByteTrack network to track the pedestrian's command actions, and finally input into the action detection network to identify and classify the action category.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-modal pedestrian command gesture recognition method integrating a large model, characterized by: The method comprises the following steps: S1. Collect pedestrian data sets from multiple places and multiple time periods, and divide the data sets; S2, use the pre-trained pedestrian detection model to extract human key points from the dataset and generate corresponding text prompts in the dataset; S3, processing the extracted human body key point information through the target tracking network to obtain dynamic key point trajectory information; S4, establishing a multimodal action recognition model for recognizing gestures, which at least includes a skeleton encoder and a text encoder; S5. Multi-part contrastive learning loss training optimizes the multimodal gesture recognition model to obtain the recognition result.

2. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 1 is characterized in that: In step S1, different visual data sources are collected, which are at least divided into videos and images containing pedestrian activities; The visual data source includes videos and images of pedestrian activities in different scenes, at different times, and under different weather conditions; And pedestrian activities cover at least a variety of pedestrian intentions, including: pedestrians are walking towards the road, pedestrians are waiting to cross the road, pedestrians signal to stop, pedestrians signal vehicles to pass, pedestrians signal to turn around; All collected visual data sources are divided into training sets, validation sets, and test sets according to preset ratios.

3. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 2 is characterized in that: In step S2, the pedestrian detection model uses the OpenPose deep learning model, which identifies the coordinate information of the key points of the human body in the video or image, and connects the key points with line segments according to the corresponding coordinate information to obtain the skeleton structure of the human body. The process includes: S201, input image: input a static image or a video frame, the input image contains one or more human bodies; S202, Feature Extraction Network: Use the pre-trained deep convolutional neural network VGG to extract high-level features of the image; S203, dual-branch architecture: two branches are used to process the information in the image in parallel, including a joint heat map branch and a bone association map branch, wherein: The joint heatmap branch predicts the probability heatmap of each joint in the image, i.e., the probability distribution of joint locations; The skeleton association graph branch is used to predict the lines or connections between joints, helping the model understand the relationship between different joints; S204, multi-stage processing: gradually optimizing its predictions through a multi-stage approach, where each stage uses the output of the previous stage to further adjust and optimize the current joint detection and connection prediction; S205, non-maximum suppression: After multiple stages, non-maximum suppression is applied to retain the best detection results of each joint; S206, post-processing stage: after obtaining the joint heat map and the bone association map, the joint points are connected into a complete skeleton; S207, output: finally output a skeleton diagram, in which key joints of the human body and their connections are marked; Secondly, the GPT-4 model is used to identify the dataset to generate text prompts of pedestrian actions. The GPT-4 model generates text descriptions in the form of paragraph descriptions, synonym descriptions, and part descriptions.

4. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 3 is characterized in that: In step S3, the target tracking network adopts the ByteTrack network model, which converts the extracted skeleton structure data into time series data. The steps are: S301, detecting input data, i.e. skeleton key points in an image frame; S302, calculate the confidence value according to the detected key point information, if the confidence value is greater than the predefined threshold track_thresh, then enter S303; otherwise jump to S308; S303, try to match the existing trajectory for the key point, and determine whether the parameter is_activated is true: if the current trajectory has been activated, go to S304; otherwise jump to S306; S304, predicting the next position of the current trajectory, and comparing whether the position of the predicted trajectory matches the key point object currently detected; if the match fails, proceed to S305; if the match succeeds, jump to S309; S305: If the detection frame is not matched, proceed to S306; if the tracking frame is not matched, proceed to S307; S306, matching the key points of the unmatched detection frame with the inactivated existing tracks again; If the match is successful, the existing track is converted to an active state and the key point is marked as tracked; If the match fails again, remove the tracking box and determine again whether the confidence of the detection box is greater than the threshold high_treshs. If it is greater than the threshold, create a new track, mark the key point as tracked, and mark the track as inactive. S307, detecting whether the key points that do not match the tracking frame are in the tracked state, if so, proceeding to S308; if not, deleting the tracks whose continuous loss time exceeds the maximum time max_time_lost; S308, matching the key points with a confidence level lower than track_thresh with the track of the unmatched tracking box, if the match is successful, proceed to S309, otherwise proceed to S310; S309: Matching is successful, and the status is marked as tracked; S310: If the match fails, the status is marked as discarded.

5. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 4 is characterized in that: In step S4, the multimodal gesture recognition model includes at least a skeleton encoder E s and text encoder E t , which are used to generate skeleton features and text features respectively, where Skeleton Encoder E s A graph convolutional network is used as the backbone network, which includes multiple graph convolutional layers and multiple multi-scale temporal convolutional modules. Text EncoderE t A language model based on the Transformer architecture is used, which first performs standard word segmentation and then is processed by the Transformer module to output a feature vector used to represent the text description.

6. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 5 is characterized in that: In the skeleton encoder E s In the graph layer, the human skeleton is represented as a graph G = {V, ε}, where V is the set of human joints, |V| = N, and ε is the set of edges; H∈R represents the features of the l-th layer of human joints with feature dimension F, then the graph convolution is expressed as follows: Where D∈R N*N is the degree matrix, A is the adjacency matrix representing the joint connection, W l is the learnable parameter of the lth layer, and δ is the activation function; Then use the multi-scale temporal convolution module for temporal modeling: the multi-scale temporal convolution module consists of four different branches, each of which contains 1×1 convolution, where the first and second branches are temporal convolution branches with variable dilation branches, the third branch is the maximum pooling layer branch, and the fourth branch only contains 1×1 convolution; the outputs of the four branches are connected to generate the final result; The skeleton-based action recognition method maps human skeleton data to one-hot encoding of action labels for classification, and the labels are trained using cross entropy loss: L cts =-ylogp θ (x) Where y is the one-hot true operation label, x is the global skeleton feature, and p(x) is the predicted probability distribution.

7. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 6 is characterized in that: Text EncoderE t The data processing process includes: S401, preprocessing the input text, wherein the preprocessing process first performs word segmentation processing to divide the text into basic units, each basic unit is a token, and each token is mapped to a unique integer index; S402, word embedding process: in the embedding layer, the Word2Vec method is used to convert the input token index into a dense vector representation, and the embedded vector represents the semantic information of each word or subword in a high-dimensional space; S403, position encoding: position encoding is introduced to represent the order information of the input tokens. Position encoding refers to adding the codes generated by sine and cosine functions to word embeddings so that the model can identify the relative position of the tokens in the sequence; S404, context information modeling: using a multi-head self-attention mechanism, each token interacts with other tokens in the input sequence, and each token is weighted and summed according to its correlation with other tokens; through multiple text encoder layers, each word embedding vector of the input is gradually converted into a high-dimensional representation containing context; S405, global semantic representation generation: using the last token in the sentence, i.e., the [EOS] token, as the global representation vector of the entire sentence; S406. The final output representation is the semantic representation of the entire sentence.

8. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 7 is characterized in that: In step S5, E s The model is trained using cross entropy loss and multi-part contrast loss, and some text descriptions are used as additional guidance. The text guidance content is generated by GPT-4. The fully connected layer connects the aligned skeleton features and text features. The following loss function is used to train the model: Among them, L cls is the cross entropy classification loss, S∈R B*3*N*T , B is the batch size, the second number in the superscript is the coordinate number, N and T are the joint number and sequence length respectively, λ is a learnable trade-off parameter, and T is the text description generated by LLM; Action description is done with a skeleton encoder E s and a text encoder E t The dual encoders encode the skeleton data and operation descriptions respectively, where the dual encoders jointly optimize by comparing skeleton-text pairs in two directions in the batch: Among them, s and t are the encoding features of the skeleton and text, sim(s i ,t i ) is the cosine similarity, τ is the temperature parameter, and B is the batch size; When calculating the loss, choose to use KL divergence as the contrast loss between skeleton and text: Where D is the entire data set, y s2t and t2s Ground truth similarity scores, with probability 0 for negative pairs and 1 for positive pairs.

9. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 8 is characterized in that: Step S5 includes the following steps: S51. Multi-part contrast learning: The skeleton is divided into several regions, and multi-part contrast loss is applied to the local features and overall features of each region respectively. The part features are obtained by aggregating the joint features in the same group through joint pooling. The regions after skeleton division include: head, arms, torso, legs and feet. The loss function of multi-part contrast loss is expressed as: Where K is the total number of groups; S52, pedestrian gesture recognition: N pedestrian gestures are pre-entered into the model, which contain the pedestrian intention information corresponding to the gestures. Then Transformer is used as a decoder, and the activation function adopts the Softmax function to output the probabilities of N command action categories at the current moment. The one with the highest probability is the current command action.

10. The multi-modal pedestrian command gesture recognition method integrating a large model according to claim 9 is characterized in that: In step S52, The process includes: S521, input embedding: The input data containing the time series feature vector is passed into the Transformer decoder. The decoder gradually extracts and integrates the context information of the input data through the multi-head self-attention layer and the feedforward network layer. The decoder outputs a series of hidden layer vectors. S522, linear transformation: use a linear layer to map the hidden vector of the decoder to the output space, and the linear layer converts the hidden state of each time step into N numerical values; S523, Softmax activation function: The Softmax activation function converts N values ​​into a probability distribution from 0 to 1, and the sum of the probabilities of all categories is 1; For the prediction at each time step, the Softmax formula is: Among them, z i is the raw score of category i, P(y i ) is the predicted probability of category i; S524, action prediction: After obtaining the probabilities of N categories through Softmax output, select the category with the largest probability as the command action A at the current moment: A=argmax(P(y1),P(y2),…,P(y N ))A represents the final recognized action type, and the pedestrian intention is obtained based on the recognized action type.

Images