Traffic police dynamic gesture recognition method and system based on vehicle-mounted federal learning

Abstract

The invention discloses a traffic police dynamic gesture recognition method and system based on vehicle-mounted federal learning, and the method enables a key feature extraction model and a preset global model to be combined as a traffic police gesture recognition model shared by vehicle-mounted terminals of a whole network, and facilitates the improvement of the gesture recognition precision of the traffic police. Different vehicle-mounted terminals are organized to train the models based on the collected traffic police gesture image data through vehicle-mounted federal learning, and then each vehicle-mounted terminal only shares model update data instead of original training data, so that the data privacy of vehicle users is protected, the communication cost for uploading mass data to a central server is saved, and the user experience is improved. Meanwhile, real-time traffic police gesture image data dispersed in different vehicle-mounted devices are fully utilized in vehicle-mounted federation learning, training data from vehicle-mounted images have different observation angles, different environment conditions and the like, the diversity of model training data is enriched, and the training efficiency is improved. And thus, the adaptive capacity of the trained traffic police gesture recognition model in a variable traffic environment is enhanced.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and more specifically, to a method and system for traffic police dynamic gesture recognition based on vehicle-mounted federated learning. Background Art

[0002] With the development of autonomous driving and advanced driver assistance technologies, intelligent connected vehicles are required to automatically and in real time recognize traffic police gestures on the road. Traffic police dynamic gesture recognition has become a key link in improving driving safety and efficiency. However, the existing data centralized learning method requires the collection of massive traffic police gesture images on the road and uniform transmission to the central server for training, which not only creates a burden on data transmission and storage, but also may cause the risk of privacy leakage of passing vehicle users. Traditional methods also fail to fully utilize the traffic police gesture images collected by intelligent connected vehicles under different observation angles and different environmental conditions through on-board cameras, and the accuracy and robustness of the recognition model are low. When using on-board image data for model training, it is necessary to consider the data privacy protection needs of vehicle users. Forcibly collecting on-board image data for centralized learning will face the "data island" problem in practical applications. Summary of the invention

[0003] In order to overcome the defects of low accuracy and low data utilization of traffic police gesture recognition in the prior art, the present invention provides a traffic police dynamic gesture recognition method and system based on vehicle-mounted federated learning.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] The present invention provides a traffic police dynamic gesture recognition method based on vehicle-mounted federated learning, comprising:

[0006] Get traffic police gesture image;

[0007] Inputting the traffic police gesture image into a trained key feature extraction model to obtain key nodes and posture vectors in the traffic police gesture image, and constructing skeleton space features according to the key nodes and posture vectors;

[0008] Constructing an urban road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputting skeleton spatial features into a preset global model, and deploying the key feature extraction model and the global model to all vehicle-mounted terminals;

[0009] Initializing the global model parameters using a central server, and sending the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals;

[0010] The selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, which is then uploaded to the central server by the roadside unit;

[0011] The central server aggregates the collected local model parameters to obtain the final global model parameters;

[0012] The preset global model is trained based on the final model parameters until the preset number of training times is reached to obtain the final global model;

[0013] Traffic police gestures are recognized based on the final global model.

[0014] Preferably, the key nodes include head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, a total of 14 key nodes of the human body; the posture vectors include turning the head, waving arms and turning around, a total of 3 posture vectors.

[0015] Preferably, obtaining a trained key feature extraction model includes:

[0016] Constructing a key feature extraction model, wherein the key feature extraction model includes a feature extraction unit and a branch unit connected in sequence;

[0017] The acquired traffic police gesture image is input into the feature extraction unit for feature extraction to obtain a key feature map;

[0018] Input the key feature graph into the branch unit to obtain the predicted key nodes and predicted posture vector;

[0019] Set the total loss function, calculate the total loss function value according to the predicted key nodes and the real key nodes and the predicted posture vector and the real posture vector, and adjust the parameters of the constructed key feature extraction model;

[0020] When the total loss function value reaches a preset value or reaches a preset number of training times, a trained key feature extraction model is obtained.

[0021] Preferably, the feature extraction unit includes a first convolution layer, a second convolution layer, a first maximum pooling layer, a third convolution layer, a fourth convolution layer, a second maximum pooling layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a third maximum pooling layer, a ninth convolution layer, a tenth convolution layer, an eleventh convolution layer, and a twelfth convolution layer, which are connected in sequence; the output end of the twelfth convolution layer is connected to the input end of the branch unit;

[0022] The branch unit includes a plurality of sequentially connected stage subunits and splicing layers;

[0023] The stage subunit includes an upper branch subunit and a lower branch subunit; the output end of the twelfth convolutional layer is connected to the input ends of the upper branch subunit and the lower branch subunit; the output ends of the upper branch subunit and the lower branch subunit are connected to the input end of the splicing layer; the output end of the twelfth convolutional layer is connected to the input end of each splicing layer.

[0024] Preferably, the total loss function includes an upper branch loss function and a lower branch loss function, specifically:

[0025]

[0026] Where N is the number of branches; is the upper branch loss function; is the loss function of the lower branch;

[0027] Preferably, the formula of the upper branch loss function is:

[0028]

[0029] in, is the real key node; is the predicted key node; W is the binary mask; j is the sequence number of the key node; p is the position of the key node;

[0030] The formula of the lower branch loss function is:

[0031]

[0032] in, is the true posture vector; is the predicted pose vector.

[0033] Preferably, obtaining the preset global model includes:

[0034] Constructing an initial global model, wherein the initial global model includes a first recurrent neural network unit, a first regularization layer, a second recurrent neural network unit, a self-attention layer, a fully connected layer, and a second regularization layer connected in sequence;

[0035] Input the skeleton space features into the constructed initial global model to obtain the predicted traffic police gesture category;

[0036] Set the cross entropy loss function, calculate the cross entropy loss function value according to the predicted traffic police gesture category and the real traffic police gesture category, and adjust the parameters of the constructed initial global model;

[0037] When the cross entropy loss function value reaches the preset value, the preset global model is obtained.

[0038] Preferably, the formula of the cross entropy loss function is:

[0039]

[0040] in, is the predicted probability of the traffic police gesture category; I is the number of skeleton space features; y i is the true category of the i-th skeleton space feature.

[0041] Preferably, the selected vehicle-mounted terminal performs multiple rounds of local model training through an optimization algorithm according to the local traffic police gesture data, and the optimization algorithm is specifically:

[0042]

[0043] Where η is the learning rate; is the momentum term; is the variance term; ε is the gradient parameter;

[0044]

[0045] Among them, β1 is the decay rate of the momentum term; β2 is the decay rate of the variance term; is the momentum term of the previous time step; is the variance term of the previous time step.

[0046] The present invention also provides a traffic police dynamic gesture recognition system based on vehicle-mounted federated learning, which is used to implement the above method, including:

[0047] An image acquisition module, which acquires traffic police gesture images;

[0048] A skeleton construction module, which inputs the traffic police gesture image into a trained key feature extraction model, obtains key nodes and posture vectors in the traffic police gesture image, and constructs skeleton space features according to the key nodes and posture vectors;

[0049] A scenario construction module constructs a city road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputs the skeleton spatial features into a preset global model, and deploys the key feature extraction model and the global model to all vehicle-mounted terminals;

[0050] A parameter initialization module, which uses a central server to initialize the global model parameters, and sends the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals;

[0051] A local training module, wherein the selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data, and is used to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, and then uploaded to the central server by the roadside unit;

[0052] Aggregation module: the central server aggregates the collected local model parameters to obtain the final global model parameters;

[0053] The global training module trains the preset global model based on the final model parameters until the preset number of training times is reached to obtain the final global model;

[0054] The gesture recognition module recognizes the traffic police’s gestures based on the final global model.

[0055] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0056] The present invention proposes a method and system for traffic police dynamic gesture recognition based on on-board federated learning, which combines a key feature extraction model with a preset global model as a traffic police gesture recognition model shared by all on-board terminals in the entire network, thereby helping to improve the accuracy of traffic police gesture recognition. Through on-board federated learning, different on-board terminals are organized to train the above model based on the traffic police gesture image data collected by themselves, and then each on-board terminal only shares the model update data rather than the original training data, thereby protecting the data privacy of vehicle users and saving the communication cost of uploading massive data to a central server. At the same time, on-board federated learning makes full use of real-time traffic police gesture image data scattered in different on-board devices. These training data from on-board images have different observation angles, different environmental conditions, etc., which enriches the diversity of model training data, thereby enhancing the adaptability of the trained traffic police gesture recognition model in a changing traffic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of the traffic police dynamic gesture recognition method based on vehicle-mounted federated learning described in Example 1;

[0058] Figure 2 It is a structural schematic diagram of the key node position described in Example 2;

[0059] Figure 3 It is a structural schematic diagram of the feature extraction model described in Example 2;

[0060] Figure 4 Schematic diagram of the structure of the SA-LSTM network described in Example 2;

[0061] Figure 5 Schematic diagram of the structure of the vehicle-mounted federated learning system described in Example 2;

[0062] Figure 6 A schematic diagram of the loss change and training accuracy of the model described in Example 3 during training;

[0063] Figure 7 This is a structural diagram of the traffic police dynamic gesture recognition system based on vehicle-mounted federated learning described in Example 3. DETAILED DESCRIPTION

[0064] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0065] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0066] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0067] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0068] Example 1

[0069] This embodiment provides a traffic police dynamic gesture recognition method based on vehicle-mounted federated learning, such as Figure 1 As shown, including:

[0070] Get traffic police gesture image;

[0071] Inputting the traffic police gesture image into a trained key feature extraction model to obtain key nodes and posture vectors in the traffic police gesture image, and constructing skeleton space features according to the key nodes and posture vectors;

[0072] Constructing an urban road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputting skeleton spatial features into a preset global model, and deploying the key feature extraction model and the global model to all vehicle-mounted terminals;

[0073] Initializing the global model parameters using a central server, and sending the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals;

[0074] The selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, which is then uploaded to the central server by the roadside unit;

[0075] The central server aggregates the collected local model parameters to obtain the final global model parameters;

[0076] The preset global model is trained based on the final model parameters until the preset number of training times is reached to obtain the final global model;

[0077] Traffic police gestures are recognized based on the final global model.

[0078] This embodiment combines the key feature extraction model and the preset global model as a traffic police gesture recognition model shared by all vehicle terminals in the entire network, which helps to improve the accuracy of traffic police gesture recognition. Through on-board federated learning, different on-board terminals are organized to train the above model based on the traffic police gesture image data collected by themselves. Subsequently, each on-board terminal only shares the model update data instead of the original training data, which protects the data privacy of vehicle users and saves the communication cost of uploading massive data to the central server. At the same time, on-board federated learning makes full use of the real-time traffic police gesture image data scattered in different on-board devices. These training data from on-board images have different observation angles, different environmental conditions, etc., which enriches the diversity of model training data, thereby enhancing the adaptability of the trained traffic police gesture recognition model in a changing traffic environment.

[0079] Example 2

[0080] This embodiment provides a traffic police dynamic gesture recognition method based on vehicle-mounted federated learning, including:

[0081] Get traffic police gesture image;

[0082] Inputting the traffic police gesture image into a trained key feature extraction model to obtain key nodes and posture vectors in the traffic police gesture image, and constructing skeleton space features according to the key nodes and posture vectors;

[0083] Constructing an urban road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputting skeleton spatial features into a preset global model, and deploying the key feature extraction model and the global model to all vehicle-mounted terminals;

[0084] Initializing the global model parameters using a central server, and sending the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals;

[0085] The selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, which is then uploaded to the central server by the roadside unit;

[0086] The central server aggregates the collected local model parameters to obtain the final global model parameters;

[0087] The preset global model is trained based on the final model parameters until the preset number of training times is reached to obtain the final global model;

[0088] Traffic police gestures are recognized based on the final global model.

[0089] like Figure 2 As shown, the key nodes include head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, a total of 14 human key nodes; the posture vectors include turning the head, waving the arms, and turning around, a total of 3 posture vectors.

[0090] Get the trained key feature extraction model, including:

[0091] Constructing a key feature extraction model, wherein the key feature extraction model includes a feature extraction unit and a branch unit connected in sequence;

[0092] The acquired traffic police gesture image is input into the feature extraction unit for feature extraction to obtain a key feature map;

[0093] Input the key feature graph into the branch unit to obtain the predicted key nodes and predicted posture vector;

[0094] Set the total loss function, calculate the total loss function value according to the predicted key nodes and the real key nodes and the predicted posture vector and the real posture vector, and adjust the parameters of the constructed key feature extraction model;

[0095] When the total loss function value reaches a preset value or reaches a preset number of training times, a trained key feature extraction model is obtained.

[0096] like Figure 3 As shown, the feature extraction unit includes a first convolution layer, a second convolution layer, a first maximum pooling layer, a third convolution layer, a fourth convolution layer, a second maximum pooling layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a third maximum pooling layer, a ninth convolution layer, a tenth convolution layer, an eleventh convolution layer, and a twelfth convolution layer, which are sequentially connected; the output end of the twelfth convolution layer is connected to the input end of the branch unit;

[0097] The branch unit includes a plurality of sequentially connected stage subunits and splicing layers;

[0098] The stage subunit includes an upper branch subunit and a lower branch subunit; the output end of the twelfth convolutional layer is connected to the input ends of the upper branch subunit and the lower branch subunit; the output ends of the upper branch subunit and the lower branch subunit are connected to the input end of the splicing layer; the output end of the twelfth convolutional layer is connected to the input end of each splicing layer.

[0099] The total loss function includes an upper branch loss function and a lower branch loss function, specifically:

[0100]

[0101] Where N is the number of branches; is the upper branch loss function; is the loss function of the lower branch;

[0102] The formula of the upper branch loss function is:

[0103]

[0104] in, is the real key node; is the predicted key node; W is the binary mask; j is the sequence number of the key node; p is the position of the key node;

[0105] The formula of the lower branch loss function is:

[0106]

[0107] in, is the true posture vector; is the predicted pose vector.

[0108] Obtaining the preset global model includes:

[0109] Constructing an initial global model, wherein the initial global model includes a first recurrent neural network unit, a first regularization layer, a second recurrent neural network unit, a self-attention layer, a fully connected layer, and a second regularization layer connected in sequence;

[0110] Input the skeleton space features into the constructed initial global model to obtain the predicted traffic police gesture category;

[0111] Set the cross entropy loss function, calculate the cross entropy loss function value according to the predicted traffic police gesture category and the real traffic police gesture category, and adjust the parameters of the constructed initial global model;

[0112] When the cross entropy loss function value reaches the preset value, the preset global model is obtained.

[0113] Preferably, the formula of the cross entropy loss function is:

[0114]

[0115] in, is the predicted probability of the traffic police gesture category; I is the number of skeleton space features; y i is the true category of the i-th skeleton space feature.

[0116] The selected vehicle-mounted terminal performs multiple rounds of local model training using an optimization algorithm based on the local traffic police gesture data. The optimization algorithm is specifically:

[0117]

[0118] Where η is the learning rate; is the momentum term; is the variance term; ε is the gradient parameter;

[0119]

[0120] Among them, β1 is the decay rate of the momentum term; β2 is the decay rate of the variance term; is the momentum term of the previous time step; is the variance term of the previous time step.

[0121] In a specific embodiment, the traffic police gesture modeling based on joint points and skeleton in step 1 includes the following contents: for vehicles, the collected traffic police gesture images come from the vehicle video data set, which is generally the angle of a single camera (i.e., one direction of the traffic police). To this end, the present invention uses 2D form to model the key nodes and skeleton vectors of the traffic police. Considering that the 8 standard gesture signals of the Chinese traffic police include multiple types of gestures such as turning the head, waving arms, waving hands, turning around, etc., the key nodes include head X1, neck X2, left shoulder X3, right shoulder X4, left elbow X5, right elbow X6, left wrist X7, right wrist X8, left hip X9, right hip X10, right hip X11, right hip X12, left hip X13, right hip X14, left hip X15, right hip X16, left hip X17, right hip X18, left hip X19, right hip X20, left hip X21, right hip X22, left hip X3, right hip X4, left hip X5, right hip X6, left hip X7, right hip X8, left hip X9, right hip X10, left hip X11, right hip X12, left hip X13, right hip X14, left hip X15, right hip X16, left hip X17, right hip X18, left hip X19, right hip X19, left hip X10, right hip X11, left hip X12, right hip X13, left hip X14, right hip X15, left hip X16, left hip X17, right hip X18, left hip X19, right hip X19, left hip X10, right hip X19, left hip X11, right hip X11, left hip X1 10 , Left knee X 11 , right knee X 12 , left ankle X 13 and right ankle X 14 There are 14 key nodes of the human body. The modeling of the skeleton vector is based on the pairwise links between the key nodes related to the traffic police posture, which can be composed of the following parts:

[0122] The posture vector part of the head turn:

[0123] The gesture vector part of waving arms and hands:

[0124] The posture vector part of the turn:

[0125] Furthermore, in step 2, the key nodes of the traffic police gesture are extracted using the spatial feature extraction model, and then the skeleton spatial features are calculated. Specifically, the following steps are included:

[0126] S1. Construct a partial affinity domain network model for the key nodes of traffic police gestures. The model takes a color image of size w×h as input, and after a network structure containing multiple stages and two branches, generates the two-dimensional position of the human joint points in the image as output.

[0127] Specifically, the model's implementation process for spatial feature extraction is as follows:

[0128] (1) First, a convolutional neural network (initialized by the first 10 layers of VGG-19 and fine-tuned as the backbone network) is used to perform preliminary feature extraction on the image to generate a set of feature maps F. Then, the feature maps F are input into the first stage of the two branches (the upper branch is used to predict key nodes; the lower branch is used to predict the association between nodes, i.e., the skeleton).

[0129] (2) In the first stage, after passing through each branch network, a confidence graph E of a set of key nodes is output. 1 =α 1 (F) and a set of partial affinity domains L 1 =β 1 (F). Among them, α 1 and β 1 They are the upper branch network and the lower branch network used for reasoning in the first stage respectively; the confidence map of the key node is a heat map used to represent the confidence of a specific key point at each pixel position in the image. j ∈R 2 Represents the true position of the key node j of the human body, then the position p∈R in the confidence map (PCM) of the key node 2 The value of can be defined as:

[0130]

[0131] In this formula, σ controls the propagation of the peak. The formula is essentially a Gaussian distribution centered on the key point, whose value decreases rapidly as the distance from the center of the key point increases. In this way, PCM can generate a clear peak response for each key point, thereby helping the model to accurately locate the position of the key point. The part affinity domain is a vector domain composed of multiple two-dimensional vectors with different confidence levels. It encodes the degree of association between parts. For each skeleton link c, its PAFs vector field can be expressed as:

[0132]

[0133] Where v is the unit vector of the skeleton link and p is the pixel point in the image. The calculation formula of the unit vector is:

[0134]

[0135] (3) In the following N stages (N>1), each stage n will first convert the predictions E of the two branches in the previous stage n-1 , L n-1 Connected with the original image feature F, and then input into the two branch networks α of the current stage n and β n In order to generate more accurate predictions by analyzing the features of the image under different receptive fields, each stage outputs the result E n and L n The expression is as follows:

[0136]

[0137] (4) During the training of the model, two loss functions need to be applied at the end of each stage of the network, one for each branch. The L2 loss is used between the estimated prediction and the true value mapping and field. The loss function uses an intermediate supervision method at each stage to solve the gradient vanishing problem by regularly replenishing the gradient, and the loss function is weighted spatially. Specifically, the loss functions of the upper and lower branches in the nth stage are:

[0138]

[0139] in, is the true value of the confidence map of the key node, is the true value of the part affinity domain, W is the binary mask when the annotation is missing at image position p, that is, W(p) = 0, and W is used to avoid penalizing TF (true example) predictions during training. The overall target loss function is the sum of the loss values ​​of the two branches in all stages, and its definition is:

[0140]

[0141] (5) When the final prediction result is output, it is necessary to obtain the candidate node with the highest confidence through non-maximum suppression (NMS) and use it as the final key node Right now

[0142] Specifically, in the present invention, the partial affinity field (PAFs) network model is mainly used to extract key nodes from the traffic police dynamic gesture data, but the model is not trained using the traffic police dynamic gesture data, but using the open source AIChallenger keypoints dataset for training, which contains 300,000 images and corresponding label files, and the label information includes key node identification and coordinates, skeleton link relationship, etc. When the model is trained, the parameters are set as follows: batch processing number is 30, initial learning rate is 0.001, Adam optimizer, 100 iterations, and the optimal model is saved when the training is completed.

[0143] S2. After the training of the model is completed, the trained model is used to extract the key nodes and the relationship between nodes of the traffic police gesture. However, the relationship between nodes does not use the final output of the lower branch of the network model, but uses the coordinates of the output key nodes to obtain a more accurate skeleton link relationship through formula calculation. Specifically, it is necessary to use the key node coordinates to further obtain the two spatial feature information of the length of the skeleton link (the distance between the key nodes) and the direction of the skeleton link (the unit vector connected by the key nodes). The implementation process is as follows:

[0144] (1) For any skeleton link c, the length of the skeleton link can be calculated using the Euclidean distance formula:

[0145]

[0146] Among them, (u1, v1) and (u2, v2) are the two key nodes X of the skeleton link c respectively. j1 (p) and X j2 (p) is the position coordinate of the

[0147] (2) If the direction of gravity is selected as the reference direction, the skeleton link direction can be defined as the angle between it and the direction of gravity, and the angle can be uniquely determined by calculating the sine and cosine values ​​of the angle. The calculation formula for the sine and cosine values ​​of the angle is as follows:

[0148]

[0149] Where g is a unit vector with the same direction as gravity, R c Represents the direction vector of the skeleton link c.

[0150] The direction of the skeleton link can be determined by vector splicing of the sine and cosine values ​​of the angle, which is:

[0151]

[0152] (3) For any skeleton link c, the length of the skeleton link is The direction of the skeleton link is Then these two spatial feature information can be spliced ​​and integrated into the human body spatial feature information S c , in order to facilitate the further extraction of spatiotemporal features in the future. Its expression is:

[0153]

[0154] Specifically, the trained partial affinity fields (PAFs) network model is used to extract the key nodes of each frame of the processed traffic police dynamic gesture video data or real-time data, and then the skeleton length of each frame of data, the sine value and cosine value of the angle between the skeleton and the gravity direction are calculated based on the output of the key node coordinates. After collecting the spatial feature data of each processed frame in real time, it is input in the form of a time series into the subsequent SA-LSTM model under the on-board federation framework for local model training and model updating, thereby realizing the recognition of traffic police dynamic gestures.

[0155] Step 3: Combine the long short-term memory network (SA-LSTM) based on the self-attention mechanism to extract spatiotemporal features. After extracting the key nodes of the traffic police gestures using the partial affinity field (PAFs) network model in step 2 and calculating the skeleton spatial features, the spatial feature data S of the traffic police dynamic gesture video data in each frame can be obtained. c After subsequent data processing, these sequence data are used as part of the SA-LSTM model input, and the final gesture classification results are obtained after network layer reasoning. The SA-LSTM network consists of a double-layer LSTM, a self-attention layer, and an output layer. The implementation process of spatiotemporal feature extraction is as follows:

[0156] S1. Long short-term memory network (LSTM) is a special type of recurrent neural network, which controls the flow of time series information by introducing three gating mechanisms: forget gate, input gate and output gate. Given an input sequence X = {x t}, where x t is the input at time step t, and the LSTM unit forward propagates and calculates the following at each time step t:

[0157] ① Input gate: It is composed of update gate i t and candidate cell states These two parts, update gate i t Determines the extent to which new information will be stored in the cell state, the candidate cell state Represents the new information itself. The specific calculation formula is as follows:

[0158] it =σ(W i ·[h t-1 ,x t ]+b i )

[0159]

[0160] Among them, i t is the output of the input gate, is the candidate cell state, W i and W C are the weights of the input gate and candidate cell state, respectively, and b i and b C are the corresponding bias terms, tanh represents the hyperbolic tangent function, and the symbol · is usually used to represent the dot product, which is the inner product.

[0161] ②Forget gate: It looks at the current input x t and the hidden state h at the previous time step t-1 , and outputs a value between 0 and 1 through the sigmoid function to indicate the degree of forgetting of each unit in the cell state, thereby determining which information should be discarded from the cell state. The specific calculation formula is as follows:

[0162] f t =σ(W i ·[h t-1 ,x t ]+b f )

[0163] Where σ is the sigmoid activation function, W f and b f are the weight and bias of the forget gate respectively.

[0164] ③Update cell status: cell status C t It is the core of the LSTM unit, which is responsible for storing information in the sequence. At each time step, the cell state is updated through the output information of the forget gate and the update gate, and its implementation expression is as follows:

[0165]

[0166] Among them, C t is the cell state at the current time step, C t-1 is the cell state at the previous time step. The symbol * represents the element-by-element multiplication (Hadamard product). Element-by-element multiplication refers to the multiplication of the elements at corresponding positions of two tensors or matrices of the same shape to obtain a new tensor or matrix with the same shape as the original tensor or matrix.

[0167] ④ Output gate: It uses the sigmoid function to output a value between 0 and 1. t , to indicate the contribution of each unit in the cell state to the hidden state. At the same time, it also updates the hidden state h t , and pass it to the next time step. The relevant calculation formula is as follows:

[0168] h t =σ(W o ·[h t-1 ,x t ]+b o )

[0169] h t =o t *tanh(C t )

[0170] Among them, t is the output of the output gate, W o is the output gate weight, b o is the output gate bias term.

[0171] like Figure 4 As shown in the LSTM network part, the input is the spatial feature data in each frame after data processing and the initial hidden state h of the model t and cell state C t , a two-layer LSTM unit cascade is used (and a dropout regularization layer is used between the two LSTM layers to prevent or reduce overfitting of the model) to first capture the local features of the time series (through LSTM), and finally output the hidden state h t+2 , cell state C t+2 And LSTM output (including the hidden state output of each time step in the sequence), that is N is the time step length of the sequence.

[0172] S2. The self-attention mechanism can dynamically adjust the degree of attention to different time steps according to the content of the input sequence, which helps to capture the long-distance dependencies in the gesture sequence. Compared with LSTM, which can only sequentially learn time series and extract local features, the self-attention mechanism can provide a summary of global information for each time step by generating a context vector, thereby establishing a global dependency. In this SA-LSTM model, after LSTM extracts the temporal features of the gesture, the constructed self-attention layer is used to generate context vectors at the output of LSTM. t+2The relationship between the hidden states of the LSTM is captured in the above figure, which can enhance the focus on key time steps and help the model consider the information of the entire time series when making the final decision, thereby improving the accuracy of gesture recognition. The implementation steps of this self-attention mechanism are as follows:

[0173] ① By inputting X (actually o t+2 ) applies a linear transformation to generate queries (Q), keys (K), and values ​​(V), which allows the model to learn different representations of time series. In the self-attention mechanism, the sources of queries (Q), keys (K), and values ​​(V) are the input matrix X and three biasable weight matrices A q , A k and A v The code is passed through three linear layers L q , L k and L v To achieve this, the formula is as follows:

[0174]

[0175] The weight matrix can learn the complex relationships between input features and map them to new feature representations so that the self-attention mechanism can more effectively capture the correlations between different time steps in the sequence.

[0176] ② Calculate the attention weight W A :First, calculate the dot product of the query (Q) and the key (K) to get the preliminary attention score, then use the scaling factor (usually the inverse of the square root of the key vector dimension) to scale the dot product result to prevent the dot product result from being too large, and then convert the scaled attention score into a probability distribution through the Softmax function, so that the sum of the attention weights of each time step in the time series is 1. The specific calculation formula is as follows:

[0177]

[0178] Among them, d k is the dimension of the key vector.

[0179] ③ Use the obtained attention weight to perform weighted sum on the value (V) to obtain the final attention output Att.

[0180] The formula is as follows:

[0181]

[0182] Where T is the sequence length of the input time series (in frames).

[0183] S3. After the self-attention layer, the above attention output Att is passed through a fully connected layer and a dropout regularization layer, and finally a two-dimensional tensor is output. (i is the sample number of the sequence, there are I sequence samples in total, that is, N = number of time steps * number of batches; f is the category number of the sequence, there are 9 categories in total), which contains the predicted probability of 9 types of gestures (8 standard traffic police gestures and no gesture) at each time step.

[0184] S4. Loss function calculation: First, output Softmax function calculation is performed to achieve normalization. For each sequence sample n and each category f, the calculation formula is as follows:

[0185]

[0186] Among them, Z f is the predicted probability of class f, is the probability of category f in the i-th sample after Softmax transformation. Then the cross entropy loss function is used for calculation, and the expression is as follows:

[0187]

[0188] Among them, y i is the actual category index of the i-th sample. Then the losses of all samples are averaged to get the final loss value.

[0189] Specifically, when the SA-LSTM network model performs local model training in each vehicle terminal under the vehicle federation framework, the model parameters are set as follows: LSTM hidden layer size is 64, dropout regularization layer coefficient is 0.5, and the specific federation parameters are described in the following steps.

[0190] Step 4: Use the federated learning framework based on the FedAvg algorithm to implement distributed learning of each vehicle terminal in the vehicle environment. This step includes the following specific steps:

[0191] S1. Build the framework of the vehicle-mounted federated learning system, such as Figure 5 As shown in the figure, in the urban road network scenario, it is assumed that there are H traffic policemen conducting real-time traffic control at each congested urban intersection, and the entities include a central server, N roadside units (RSUs) and K vehicle terminals. Each vehicle client uses the traffic police gesture image data collected by itself to train the local model and upload the model gradient. The central server is responsible for aggregating the model gradients of all vehicle terminals and sending the updated global model parameters back to each vehicle terminal. The roadside unit serves as a communication relay between the vehicle terminal and the central server. Usually, the vehicle terminal and the roadside unit perform wireless data transmission, and the roadside unit and the central server perform wireless or wired data transmission.

[0192] S2. Initialization of vehicle terminals and global models: Deploy partial affinity domain network models that have been trained with the human key node dataset and the SA-LSTM model to be trained on the K vehicle terminals participating in vehicle federated learning. In the present invention, only the SA-LSTM model is used as a global model, and distributed model training is required in vehicle federated learning. The global model parameter ω0 is initialized by the central server, and the initialized global model parameter ω0 is transmitted to each roadside unit, and then each roadside unit broadcasts the model parameter ω0 to nearby vehicle terminals that have been confirmed to participate in vehicle federated learning, thereby obtaining

[0193] S3. Selection of vehicle terminals in each round of global model training: The number of vehicle terminals set is known to be K, and the proportion of vehicle terminals selected to participate in local model training is ρ. In each round of federated training, the central server calculates the number of selected clients m according to the following formula:

[0194] m=max(Kρ,1)

[0195] Then randomly select m vehicle terminals to form the vehicle terminal set S τ , let m vehicle terminals conduct local model training in parallel. Where τ is the current global model training round.

[0196] S4. Local model training: In each round of global model training, the selected vehicle terminals use the traffic police dynamic gesture data collected locally to train the SA-LSTM model. In order to reduce communication costs and ensure that each vehicle terminal can fully learn the characteristics of its local data, the selected vehicle terminals in each round of global model training will conduct E rounds of local model training and update the local model parameters in each local model training. In each round of local model training, after forward propagation and loss calculation, the Adam optimization algorithm is used to perform gradient descent optimization on the model parameters of each vehicle terminal to minimize the loss function. In this algorithm, the gradient of the current time step t is first calculated in the back propagation, and the calculation formula is as follows:

[0197]

[0198] in, are the local model parameters at the current time step t, L k is the loss function, Represents the gradient operation. Then, the model parameters of each vehicle terminal are optimized by gradient descent. The optimization algorithm can be expressed by the following formula:

[0199]

[0200] Where η is the global learning rate, is the first-order moment estimate of the gradient (i.e., the momentum term), is the second-order moment estimate of the gradient (i.e., the variance term), ε is a very small number used to prevent the denominator from being zero. and The calculation formula is as follows:

[0201]

[0202] Among them, β1 is the decay rate of the first-order moment estimate, β2 is the decay rate of the second-order moment estimate, is the first-order moment estimate of the previous time step, is the second-order moment estimate at the previous time step.

[0203] S5. Local model parameter upload: After local model training and model update are completed, the selected vehicle terminals in each round of global model training upload their updated model parameters to the local model training terminal. Upload to the nearest roadside unit, which will Upload to the central server.

[0204] S6. Global model aggregation and model distribution:

[0205] The central server collects local model parameters uploaded by all vehicle terminals The FedAvg algorithm is used to Perform parameter aggregation to obtain the global model parameter ω of the τ+1th round τ+1 , as shown below:

[0206]

[0207] Among them, θ k is the amount of local training data of vehicle terminal k and The central server updates the global model data based on τ+1 Enter the τ+1th round of global model training. Get the latest global parameter ω τ+1 , and the updated global parameter ω τ+1 It is then downloaded to all vehicle terminals through the roadside unit transmission again.

[0208] S7, iterative optimization: repeat steps S3 to S6 until the model converges or reaches a preset global iteration round.

[0209] Regarding the traffic police dynamic gesture data collected locally by the vehicle terminals in step S4, in actual vehicle scenarios, there are several solutions for online federated learning of the real-time unlabeled data stream collected by the vehicle camera:

[0210] (1) Using a semi-supervised learning algorithm, combined with data stream processing and window technology that can adapt to continuous data input, the global model is optimized by simultaneously using labeled data and unlabeled data for training on each vehicle terminal and fusing the model parameters of each node on the central server, thereby achieving online federated learning.

[0211] (2) Use unsupervised learning algorithms (such as clustering algorithms) to process unlabeled data. Such algorithms do not require labels to learn the intrinsic structure and patterns of data. They learn unbiased representations in decentralized and heterogeneous local data through knowledge distillation and contrastive learning on both sides, and can realize the federated learning framework FedX combined with unsupervised learning.

[0212] (3) Using a self-supervised learning algorithm, by constructing tasks, the vehicle terminal can use the unlabeled video data stream to self-generate pseudo-labels, thereby learning the intrinsic structure and pattern of the data. The constructed self-supervised tasks are then used to train the model on each vehicle terminal, and federal aggregation and global model fine-tuning are completed on the server side.

[0213] In the actual vehicle-mounted scenario where real-time unlabeled data streams collected by vehicle cameras are used for online federated learning, the above method or the method of real-time automatic data labeling on the vehicle side or other methods can be used. Since the present invention does not involve a specific method for implementing online federated learning using real-time unlabeled data streams collected by vehicle cameras, there is no restriction on the implementation method, and any publicly available method in the prior art can be freely adopted.

[0214] Specifically, when performing actual training in the present invention, the federal parameters in the federal framework are set as follows: the number of vehicle-mounted terminals K is 10, the proportion of vehicle-mounted terminals selected to participate in local model training ρ is 0.7, the number of global iterations is 10000, the number of local model training times E in each round of global iterations is 5, the global learning rate is 0.001, and the number of batch processing is 1.

[0215] In general, in the task of traffic police dynamic gesture recognition, under the framework of on-board federated learning, the trained partial affinity domain network model deployed on the on-board terminal is first used, combined with the video data collected by the on-board terminal, to accurately extract the key node information of the traffic police gesture, and calculate the skeleton space features; then, the time series data containing skeleton space features extracted and calculated from the traffic police dynamic gesture data are imported into the SA-LSTM model deployed on the on-board terminal for collaborative training and joint optimization. The optimization process aims to minimize the loss function by adjusting the global model parameters, thereby improving the accuracy of the prediction; finally, the trained SA-LSTM global model is used, combined with the time series data containing skeleton space features extracted from the real-time data of traffic police dynamic gestures by the partial affinity domain network model, to predict the probability of the traffic police dynamic gesture category, and the category with the maximum probability is used as the recognition output of the traffic police dynamic gesture through visualization methods.

[0216] In order to verify and compare the performance of the model of the present invention, the model was tested on the public Chinese traffic police gesture dataset (CTPGD). When the federated framework is not used for centralized training, the accuracy of the model in traffic police dynamic gesture recognition in the evaluation test is 96.11% (in the form of edit distance), which is nearly 4% higher than the traditional CPM+LSTM solution; when the federated framework is used for distributed training, the loss change and training accuracy of the model during training are as follows: Figure 5 As shown in the figure, the global model training converges well. In the evaluation test, the accuracy of the model in identifying the dynamic gestures of traffic police is 95.78% (in the form of edit distance), which achieves the effect of model performance approaching that of centralized training while protecting the data privacy of vehicle users. The present invention can perform dynamic gesture recognition of traffic police on real-time video streams, such as Figure 6 As shown, this is the effect diagram of a frame in real-time recognition.

[0217] Example 3

[0218] This embodiment also provides a traffic police dynamic gesture recognition system based on vehicle-mounted federated learning, such as Figure 7 As shown, the method for implementing embodiment 1 or embodiment 2 includes:

[0219] An image acquisition module, which acquires traffic police gesture images;

[0220] A skeleton construction module, which inputs the traffic police gesture image into a trained key feature extraction model, obtains key nodes and posture vectors in the traffic police gesture image, and constructs skeleton space features according to the key nodes and posture vectors;

[0221] A scenario construction module constructs a city road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputs the skeleton spatial features into a preset global model, and deploys the key feature extraction model and the global model to all vehicle-mounted terminals;

[0222] A parameter initialization module, which uses a central server to initialize the global model parameters, and sends the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals;

[0223] A local training module, wherein the selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data, and is used to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, and then uploaded to the central server by the roadside unit;

[0224] Aggregation module: the central server aggregates the collected local model parameters to obtain the final global model parameters;

[0225] The global training module trains the preset global model based on the final model parameters until the preset number of training times is reached to obtain the final global model;

[0226] The gesture recognition module recognizes the traffic police’s gestures based on the final global model.

[0227] The same or similar reference numerals correspond to the same or similar components;

[0228] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0229] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A traffic police dynamic gesture recognition method based on vehicle-mounted federated learning, characterized in that: include: Get traffic police gesture image; Inputting the traffic police gesture image into a trained key feature extraction model to obtain key nodes and posture vectors in the traffic police gesture image, and constructing skeleton space features according to the key nodes and posture vectors; Constructing an urban road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputting skeleton spatial features into a preset global model, and deploying the key feature extraction model and the global model to all vehicle-mounted terminals; Initializing the global model parameters using a central server, and sending the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals; The selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, which is then uploaded to the central server by the roadside unit; The central server aggregates the collected local model parameters to obtain the final global model parameters; The preset global model is trained based on the final model parameters until the preset number of training times is reached to obtain the final global model; Traffic police gestures are recognized based on the final global model.

2. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 1 is characterized in that: The key nodes include head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, a total of 14 key nodes of the human body; the posture vectors include turning the head, waving arms, and turning around, a total of 3 posture vectors.

3. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 1 is characterized in that: Get the trained key feature extraction model, including: Constructing a key feature extraction model, wherein the key feature extraction model includes a feature extraction unit and a branch unit connected in sequence; The acquired traffic police gesture image is input into the feature extraction unit for feature extraction to obtain a key feature map; Input the key feature graph into the branch unit to obtain the predicted key nodes and predicted posture vector; Set the total loss function, calculate the total loss function value according to the predicted key nodes and the real key nodes and the predicted posture vector and the real posture vector, and adjust the parameters of the constructed key feature extraction model; When the total loss function value reaches a preset value or reaches a preset number of training times, a trained key feature extraction model is obtained.

4. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 3 is characterized in that: The feature extraction unit includes a first convolution layer, a second convolution layer, a first maximum pooling layer, a third convolution layer, a fourth convolution layer, a second maximum pooling layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a third maximum pooling layer, a ninth convolution layer, a tenth convolution layer, an eleventh convolution layer, and a twelfth convolution layer, which are sequentially connected; the output end of the twelfth convolution layer is connected to the input end of the branch unit; The branch unit includes a plurality of sequentially connected stage subunits and splicing layers; The stage subunit includes an upper branch subunit and a lower branch subunit; the output end of the twelfth convolutional layer is connected to the input ends of the upper branch subunit and the lower branch subunit; the output ends of the upper branch subunit and the lower branch subunit are connected to the input end of the splicing layer; the output end of the twelfth convolutional layer is connected to the input end of each splicing layer.

5. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 3 is characterized in that: The total loss function includes an upper branch loss function and a lower branch loss function, specifically: Where N is the number of branches; is the upper branch loss function; is the loss function of the lower branch.

6. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 5 is characterized in that: The formula of the upper branch loss function is: in, is the real key node; is the predicted key node; W is the binary mask; j is the sequence number of the key node; p is the position of the key node; The formula of the lower branch loss function is: in, is the true posture vector; is the predicted pose vector.

7. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 1 is characterized in that: Obtaining the preset global model includes: Constructing an initial global model, wherein the initial global model includes a first recurrent neural network unit, a first regularization layer, a second recurrent neural network unit, a self-attention layer, a fully connected layer, and a second regularization layer connected in sequence; Input the skeleton space features into the constructed initial global model to obtain the predicted traffic police gesture category; Set the cross entropy loss function, calculate the cross entropy loss function value according to the predicted traffic police gesture category and the real traffic police gesture category, and adjust the parameters of the constructed initial global model; When the cross entropy loss function value reaches the preset value, the preset global model is obtained.

8. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 7 is characterized in that: The formula of the cross entropy loss function is: in, is the predicted probability of the traffic police gesture category; I is the number of skeleton space features; y i is the true category of the i-th skeleton space feature.

9. The method for traffic police dynamic gesture recognition based on vehicle-mounted federated learning according to claim 1 is characterized in that: The selected vehicle-mounted terminal performs multiple rounds of local model training using an optimization algorithm based on the local traffic police gesture data. The optimization algorithm is specifically: Where η is the learning rate; is the momentum term; is the variance term; ε is the gradient parameter; Among them, β1 is the decay rate of the momentum term; β2 is the decay rate of the variance term; is the momentum term of the previous time step; is the variance term of the previous time step.

10. A traffic police dynamic gesture recognition system based on vehicle-mounted federated learning, used to implement the method described in claims 1-9, characterized in that: include: An image acquisition module, which acquires traffic police gesture images; A skeleton construction module, which inputs the traffic police gesture image into a trained key feature extraction model, obtains key nodes and posture vectors in the traffic police gesture image, and constructs skeleton space features according to the key nodes and posture vectors; A scenario construction module constructs a city road network scenario, including a central server, several roadside units and several vehicle-mounted terminals; inputs the skeleton spatial features into a preset global model, and deploys the key feature extraction model and the global model to all vehicle-mounted terminals; A parameter initialization module, which uses a central server to initialize the global model parameters, and sends the initialized global model parameters to all roadside units, which are then sent by the roadside units to the selected vehicle terminals; A local training module, wherein the selected vehicle-mounted terminal performs multiple rounds of local model training according to the local traffic police gesture data, and is used to update the local model parameters, and when the preset conditions are met, the updated local model parameters are uploaded to the nearest roadside unit, and then uploaded to the central server by the roadside unit; Aggregation module: the central server aggregates the collected local model parameters to obtain the final global model parameters; The global training module trains the preset global model based on the final model parameters until the preset number of training times is reached to obtain the final global model; The gesture recognition module recognizes the traffic police’s gestures based on the final global model.

Images