A method, system, medium, and device for traffic police gesture recognition based on computer vision and human skeleton.

Abstract

This invention discloses a method, system, medium, and device for recognizing traffic police gestures based on computer vision and human skeleton. The method obtains the recognition results of traffic police gestures in time-series images based on a trained recognition network model. First, spatial and temporal information of the time-series data is extracted through a backbone network. Then, a hierarchical pyramid module combined with human biological structure is used to construct a multi-scale hierarchical structure of the human body, refining feature information hierarchically. Finally, a graph convolutional layer module is used for regression processing and outputs keypoint coordinate information. The keypoints are input into the gesture recognition model to identify the gesture category. This invention combines deep learning and human biological structure, achieving adaptive fusion of three-level features (keypoint-component-region) at the feature level. This improves the model's fine-grained representation capability, effectively solves the long-distance dependency problem in complex poses, and significantly improves recognition accuracy in occluded and unconventional pose scenarios.

Description

Technical Field

[0001] This invention belongs to the technical field of human motion recognition, specifically relating to a traffic police gesture recognition method, system, medium, and device based on computer vision and human skeleton. Background Technology

[0002] With the continuous evolution of deep learning technology, machine vision technology is transcending the limitations of static images and gradually beginning to explore dynamic scenes. To facilitate human life, the demand for intelligent urban transportation and refined security is rapidly increasing, such as semantic understanding of traffic police hand gestures and traffic violations. However, in real-world scenarios, the complexity of parsing increases exponentially, and its implementation is limited by obstacles such as hardware computing power and scene complexity.

[0003] Human motion recognition is a branch of artificial intelligence technology. This technology focuses on parsing the semantic information contained in continuous frame sequence image data, assigning category labels to dynamic behaviors, and is widely used in fields such as intelligent traffic management, seamless human-vehicle collaboration, and robotic services. In recent years, human motion recognition technology has been mainly divided into image sample-based motion recognition technology and human skeleton-based motion recognition technology. The former extracts the temporal and spatial information of samples through temporal network models such as graph convolutional neural networks or deformer neural networks, and then performs motion recognition at the high-dimensional feature level. This method is highly complex, sensitive to occlusion and unconventional movements, and requires a large amount of data to drive it. The latter maps high-dimensional features to a human skeleton topology map and directly performs motion recognition on the topology map. It has low-dimensional and efficient computational characteristics, does not require too much computational cost, and is easy to deploy.

[0004] Currently, existing motion recognition models based on human skeletons deepen the network to model long-distance dependencies between body parts in order to enhance the understanding of human motion coordination. This introduces a lot of irrelevant noise information, focuses too much on dependencies between disconnected keypoints, and ignores the moderating effect of body parts on keypoints, resulting in weakened long-distance dependencies, lack of structural information, and a high error rate. Therefore, there is an urgent need to propose a lightweight and easily deployable motion recognition solution to effectively solve the long-distance dependency problem in complex poses and improve recognition accuracy in occluded and unconventional pose scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, medium, and device for traffic police gesture recognition based on computer vision and human skeleton, in order to solve the above-mentioned problems.

[0006] This invention is mainly achieved through the following technical solutions:

[0007] A traffic police gesture recognition method based on computer vision and human skeleton includes the following steps:

[0008] Step S1: Acquire the detection video and obtain the time sequence image to be detected;

[0009] Step S2: Obtain the traffic police gesture recognition results in the time sequence image to be detected based on the trained recognition network model;

[0010] Step S21: First, extract the human posture skeleton key points using the human skeleton key point detection model;

[0011] Step SA1: First, extract the spatial and temporal information of the time series data through the backbone network;

[0012] Step SA2: Then, using a hierarchical pyramid module combined with human biological structure, a multi-scale hierarchical structure of the human body is constructed, and the feature information is refined hierarchically.

[0013] Step SA3: Finally, the graph convolutional layer module is used for regression processing to output the key point coordinate information;

[0014] Step S22: Input the key points into the gesture recognition model to identify the gesture category.

[0015] To better implement the present invention, further, in step SA1, the backbone network output includes features with an increasing number of semantic elements. F 1. Characteristics F 2 and features F 3; wherein: the feature F 1 is used for region-adaptive grouping, the feature F 2 is used for adaptive grouping of components, the feature F 3 is used as key point feature information.

[0016] To better implement this invention, further, in step SA2, the key points, components, and regions are classified according to three aspects, and an adaptive grouping module is used to realize multi-scale dynamic recombination of key points, components, and regions; then, aggregation is performed by group through an average pooling layer; finally, a self-attention mechanism is used to allow strong connections between key points and components in the same region; the processing of the adaptive grouping module includes the following steps:

[0017] (1) Flatten the feature vectors corresponding to the regions and components respectively to form feature matrices;

[0018] (2) Using the number of groups as the number of clusters, dynamic clustering is performed using the differentiable k-means algorithm to adaptively divide the feature information into K semantic groups, with 4 groups for regions and 8 groups for components.

[0019] To better realize the present invention, further, in step SA2, the grouping of each level conforms to the human biological structure through feature consistency constraints; the key point features within the same component are similar to the features of the component, and the component features within the same region are similar to the features of the region.

[0020] To better realize the present invention, the loss function of the recognition network model further includes feature consistency constraint, skeletal keypoint detection loss function and category classification recognition loss function; the feature consistency constraint adopts the mean squared error loss function, the skeletal keypoint detection loss function adopts the smoothing L1 loss function, and the category classification recognition loss function adopts the cross-entropy loss function.

[0021] To better realize the present invention, in step S22, the gesture recognition model analyzes the key points of the skeleton from coarse to fine based on feature information, and combines the joint motion acceleration variables to focus on the intensity, suddenness or fine dynamic changes of the action.

[0022] This invention is mainly achieved through the following technical solutions:

[0023] A traffic police gesture recognition system based on computer vision and human skeleton, based on the aforementioned traffic police gesture recognition method based on computer vision and human skeleton, includes a video acquisition and processing module, a model training module, and a gesture detection module; the video acquisition module is used to acquire and process detection videos to obtain time-series image data; the model training module is used to train a recognition network model using a training set; the gesture detection module is used to output traffic police gesture recognition results based on the trained recognition network model;

[0024] The recognition network model includes a human skeleton key point detection model and a gesture recognition model. The human skeleton key point detection model includes a backbone network, a hierarchical pyramid module, and a graph convolutional layer module arranged from front to back.

[0025] To better implement this invention, the hierarchical pyramid module further includes a first to a third branch arranged in parallel, and a feature splicing layer and a self-attention layer connected to each other. The first to third branches are respectively arranged from front to back as a batching layer, an adaptive grouping module, and a feature consistency constraint. The first and second branches are respectively connected to the feature splicing layer through an average pooling layer, and the third branch is respectively connected to the feature splicing layer and the self-attention layer; the features... F 1. Characteristics F 2 and features F 3. Enter the first branch to the third branch respectively.

[0026] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a traffic police gesture recognition method based on computer vision and human skeleton as described above.

[0027] An electronic device includes a memory and a processor; the memory stores a computer program; the processor is configured to execute the computer program in the memory to implement a traffic police gesture recognition method based on computer vision and human skeleton, as described above.

[0028] The beneficial effects of this invention are as follows:

[0029] (1) This invention first uses a human skeleton key point detection model to extract key points of human posture skeleton. According to the requirements of traffic police gestures, a total of 47 key points are extracted. Then, the key points are input into the gesture recognition model to identify the gesture category. This invention combines deep learning and human biological structure to achieve adaptive fusion of three-level features at the feature level: key points, components, and regions. This improves the fine-grained representation capability of the model and effectively solves the long-distance dependency problem in complex postures. The recognition accuracy is greatly improved in occlusion and unconventional posture scenarios. This invention achieves accurate recognition and real-time feedback of traffic police gestures, which can ensure that the vehicle can correctly respond to the traffic police's commands when autonomous driving (assisted driving) is activated. At the same time, this invention can also be used in the teaching of traffic police and drivers.

[0030] (2) The present invention uses a human skeleton key point detection model to first extract spatial and temporal information of time series data through the backbone network, and then the hierarchical pyramid module combines human biological structure to construct human hierarchical structure at multiple scales, refine feature information hierarchically, and then output human skeleton key points to the gesture recognition model. Finally, the gesture recognition model identifies the gesture category. The hierarchical pyramid module classifies the human body according to key points, components and regions, and uses a self-attention mechanism to allow strong links between key points and components in the same region, which solves the problems of long-distance dependence and structural ambiguity in complex pose estimation. Attached Figure Description

[0031] Figure 1 This is a flowchart of the traffic police gesture recognition method based on computer vision and human skeleton according to the present invention;

[0032] Figure 2 A schematic diagram of the structure of a human skeleton key point detection model;

[0033] Figure 3 This is a structural diagram of the hierarchical feature pyramid module;

[0034] Figure 4 This is a schematic diagram of the gesture recognition model.

[0035] Figure 5 A schematic diagram of human skeletal key point detection for gesture examples;

[0036] Figure 6 This is a trend graph showing the iterative loss value and accuracy on the validation set in this invention. Detailed Implementation

[0037] Example 1:

[0038] A traffic police gesture recognition method based on computer vision and human skeleton includes the following steps:

[0039] 1. Data acquisition and processing;

[0040] Video images of traffic police hand gestures are collected in various scenarios and environments. Then, key points of the human body and gesture categories are manually labeled. The time-series image data is divided into training and testing sets according to a certain format for training and testing of model algorithms.

[0041] Preferably, 15 original video files can be read. During the shooting process, each action is broken down into 15 still actions, and the still actions are shot into original video files. During processing, one frame is extracted from each video and the frame sequence is combined according to the rule of "15 frames forward + 15 frames backward" to form a continuous video image of 30 frames. In this way, we only need to shoot simple static videos to obtain a large amount of dynamic datasets for model training.

[0042] Generate up to 200 new video files (based on the minimum frame rate of the 15 original videos), stored in the output_videos folder. Each new video contains 30 frames, with the first 15 frames in forward order of the original videos and the last 15 frames in reverse order. The video format is MP4, with the same resolution as the original videos, and the frame rate remains at 30 frames per second.

[0043] Iterate through each original video, reading frame by frame, and store the valid frames in the original_frames list. For the generated t-th video, the first 15 frames are taken from the t-th frame of the i-th original video (i=1~15), and the last 15 frames are taken from the total number of frames - 1 - t of the i-th video (i=15~1). Merge the forward and reverse frames to generate a 30-frame sequence.

[0044] The original_frames is a two-dimensional list used to store all frames of 15 original videos. Each inner list corresponds to all frames of one video and is used for subsequent compositing.

[0045] The total_frames_list is a one-dimensional list that records the total number of frames in each original video. It is used to calculate the minimum common frame count and determine the number of composite videos based on the minimum frame count.

[0046] `combined_frames` temporarily stores a 30-frame sequence (15 forward frames + 15 reverse frames) of a single output video, which is then used to synthesize these frames into a coherent video.

[0047] The height and width parameters store the height and width of the video frame and are used to set the resolution of the output video. They are obtained from the first frame of the original video.

[0048] fps = 30 is used to set the frame rate of the output video.

[0049] The generated videos are stored in the output_videos folder. Each video is named starting from 1 according to the generation order and stored in mp4 format.

[0050] Process steps: Initialize empty lists `original_frames` and `total_frames_list`. Iterate through the original video files 1-15, reading each frame and storing it in the `frames` list. If `frames` is not empty, add it to `original_frames` and `total_frames_list`. Calculate the minimum number of frames, `min_total_frames`, to determine the number of output videos (the smaller of 200 and the minimum number of frames). Obtain the video resolution from the original videos and fix the generated video frame rate (30 FPS). Create an output folder `output_videos`. Iterate through each output video, generating a forward frame sequence, then a reverse frame sequence, and finally merge the frame sequences and store them in MP4 format.

[0051] 2. Build and train the recognition network model;

[0052] Build the network structure according to the application scenario and target action characteristics, such as Figure 2 As shown, the entire model is divided into a human skeleton key point detection model and a gesture recognition model. The human skeleton key point detection model first extracts the spatial and temporal information of the time series data through the backbone network. Then, the hierarchical pyramid module combines the human biological structure to construct the human hierarchical structure at multiple scales and refine the feature information hierarchically. Next, it outputs the human skeleton key points to the gesture recognition model. Finally, the gesture recognition model identifies the gesture category.

[0053] like Figure 1As shown, the recognition network model consists of a human skeleton keypoint detection model and a gesture recognition model. Training data is used to train both the human skeleton keypoint detection model and the gesture recognition model, and the optimal model is selected for testing. After inputting test samples, the human skeleton keypoint detection model is first used to extract the human posture skeleton keypoints, such as... Figure 5 As shown, (a) is a schematic diagram of human skeletal key point detection for gesture example one, and (b) is a schematic diagram of human skeletal key point detection for gesture example two. Based on the traffic police gesture requirements, a total of 47 key points were extracted. These key points were then input into the gesture recognition model to identify the gesture category. The gesture recognition model uses transfer learning of a general recognition model, which can reduce the model's dependence on the amount of data.

[0054] like Figure 2 As shown, the human skeleton keypoint detection model mainly consists of a backbone network, a hierarchical feature pyramid, and graph convolutional layer modules. Specifically, the backbone network extracts spatial and temporal information from time-series data and outputs features. F 1. Characteristics F 2 and features F 3. Representing feature information with different semantic meanings. Among them, features... F 1. Contains minimal semantics, used for region-adaptive grouping; Features F 2. Includes semantics, secondly, for adaptive grouping of components; features F 3 contains the most semantics and is mainly used for key point feature information. Then, the hierarchical pyramid module combines human biological structure to construct a multi-scale human hierarchical structure, refine the feature information hierarchically, and finally regress the key point coordinate information through the graph convolutional layer module.

[0055] like Figure 3 As shown, the hierarchical feature pyramid mainly consists of a layer-by-layer batching, an adaptive grouping module, feature consistency constraints, an average pooling layer (AP), a feature concatenation layer, and a self-attention mechanism. Specifically, it is hierarchically classified into keypoints, parts, and regions, and the self-attention mechanism allows strong connections between keypoints and parts within the same region, solving the problems of long-distance dependencies and structural ambiguity in complex pose estimation. The adaptive grouping module mainly uses the differentiable k-means algorithm for grouping calculation. The part level is mainly divided into 8 groups, and the region level is mainly divided into 4 groups. The number of groups is used as the cluster number for the k-means algorithm. Through feature-driven online clustering, multi-scale dynamic reorganization of keypoints, parts, and regions is achieved. The processing flow of the differentiable k-means algorithm is as follows:

[0056] ① Flatten the feature vectors corresponding to the region / component to form a feature matrix;

[0057] ② Using the number of groups as the number of clusters, dynamic clustering is performed using the kmeans algorithm to adaptively divide the feature information into K semantic groups, with 4 groups for regions and 8 groups for components.

[0058] like Figure 3 As shown, the adaptive grouping module is followed by an average pooling layer, which aggregates data by group, preserving multi-level information from local details to global semantics, thus enhancing the modeling capability of human substructures. Furthermore, to ensure that the grouping of each level in the pyramid structure conforms to human biological structure, a feature consistency constraint is added. This means that the keypoint features within the same component should be similar to the features of that component, and the component features within the same region should be similar to the features of that region, ensuring the coherence of feature representations. The formula is as follows:

[0059] ;

[0060] ;

[0061] Where: L k2p L represents the key-component feature consistency loss. p2r Let represent the component-region feature consistency loss, i represent the sample index, N represent the batch size, k represent the key point feature vector, p represent the component feature vector, and r represent the region feature vector.

[0062] like Figure 4 As shown, the gesture recognition model adopts a cascaded structure, analyzing key skeletal points from coarse to fine in terms of feature information, and combining it with joint motion acceleration variables to focus on the intensity, suddenness, or fine dynamic changes of the movement, thereby enhancing the model's anti-interference ability. The formula for calculating joint motion acceleration is as follows:

[0063] ;

[0064] Where: A represents the joint motion acceleration, P represents the two-dimensional joint position coordinate tensor, i represents the sample index, t represents the effective time frame, j represents the key point index; Δt is the time interval;

[0065] d represents the coordinate dimension; when d=0, it represents the x-coordinate, and when d=1, it represents the y-coordinate.

[0066] The loss functions are divided into feature consistency constraint, skeletal keypoint detection loss function, and category classification recognition loss function. The feature consistency constraint adopts the mean squared error loss function, the skeletal keypoint detection loss function adopts the smooth L1 loss function, and the category classification recognition loss function adopts the cross-entropy loss function.

[0067] Before training begins, model weights and parameters are randomly initialized, termination conditions such as the maximum number of iterations and hyperparameters are preset, the learning rate decay strategy is adjusted, multiple rounds of iterative optimization are performed, the convergence of the loss value is monitored, and the optimal model is selected through evaluation in real-world scenarios. Figure 6 As shown, (a) is a graph showing the decreasing trend of the loss value during the iteration process of this invention; (b) is a graph showing the accuracy trend on the validation set. The recognition accuracy of this invention is greater than 95%.

[0068] 3. Based on the trained recognition network model, the recognition results of traffic police gestures in the time series images to be detected are obtained.

[0069] In summary, the action recognition method constructed in this invention possesses higher fine-grained representation capabilities and significantly improves recognition accuracy in occluded and unconventional pose scenarios. Experiments show that the proposed method incorporates human biological structures, achieving adaptive fusion of keypoint-component-region three-level features at the feature level compared to existing detection methods. This enhances the model's fine-grained representation capabilities and effectively addresses the long-distance dependency problem arising in complex poses.

[0070] Example 2:

[0071] A traffic police gesture recognition system based on computer vision and human skeleton includes a video acquisition and processing module, a model training module, and a gesture detection module. The video acquisition module is used to acquire and process detection videos to obtain time-series image data. The model training module is used to train a recognition network model using a training set. The gesture detection module is used to output traffic police gesture recognition results based on the trained recognition network model.

[0072] like Figure 2 As shown, the recognition network model includes a human skeleton keypoint detection model and a gesture recognition model. The human skeleton keypoint detection model includes a backbone network, hierarchical pyramid modules, and graph convolutional layer modules arranged sequentially from front to back. The backbone network includes several graph convolutional layer modules and three deformer modules set in the branches of the graph convolutional modules, used to output features F1 to F3. Figure 3 As shown, the hierarchical pyramid module includes a first to a third branch arranged in parallel, and a feature splicing layer and a self-attention layer connected to each other. The first to third branches are respectively configured with a batch normalization layer, an adaptive grouping module, and a feature consistency constraint from front to back. The first and second branches are connected to the feature splicing layer through average pooling layers, and the third branch is connected to both the feature splicing layer and the self-attention layer. F 1. Characteristics F 2 and features F 3. Enter the first branch to the third branch respectively.

[0073] like Figure 4 As shown, the gesture recognition model includes several convolutional residual blocks, and the middle branches and tail ends of these convolutional residual blocks are respectively provided with a first fully connected layer to a third fully connected layer. The gesture recognition model is used to perform coarse-to-fine analysis of skeletal key points based on feature information, and combines joint motion acceleration variables to focus on the intensity, suddenness, or subtle dynamic changes of the movement, thereby enhancing the model's anti-interference ability.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A traffic police gesture recognition method based on computer vision and human skeleton, characterized in that, Includes the following steps: Step S1: Acquire the detection video and obtain the time sequence image to be detected; Step S2: Obtain the traffic police gesture recognition results in the time sequence image to be detected based on the trained recognition network model; Step S21: First, extract the human posture skeleton key points using the human skeleton key point detection model; Step SA1: First, extract the spatial and temporal information of the time series data through the backbone network; In step SA1, the backbone network output contains features with an increasing number of semantic elements. F 1. Characteristics F 2 and features F 3; Wherein: the feature F 1 is used for region-adaptive grouping, the feature F 2 is used for adaptive grouping of components, the feature F 3 is used as key point feature information; Step SA2: Then, using a hierarchical pyramid module combined with human biological structure, a multi-scale hierarchical structure of the human body is constructed, and the feature information is refined hierarchically. Step SA3: Finally, the graph convolutional layer module is used for regression processing to output the key point coordinate information; Step S22: Input the key points into the gesture recognition model to identify the gesture category; In step SA2, the classification is performed according to three aspects: key points, components, and regions. An adaptive grouping module is used to achieve multi-scale dynamic reorganization of key points, components, and regions. Then, aggregation is performed by group through an average pooling layer. Finally, a self-attention mechanism is used to allow strong connections between key points and components in the same region. The processing of the adaptive grouping module includes the following steps: (1) Flatten the feature vectors corresponding to the regions and components respectively to form feature matrices; (2) Using the number of groups as the number of clusters, dynamic clustering is performed using the differentiable k-means algorithm to adaptively divide the feature information into K semantic groups, with 4 groups for regions and 8 groups for components; In step SA2, feature consistency constraints are used to ensure that the grouping of each level conforms to the human biological structure; the key point features within the same component are similar to the features of that component, and the component features within the same region are similar to the features of that region. The hierarchical pyramid module includes a first to a third branch arranged in parallel, and a feature splicing layer and a self-attention layer connected to each other. The first to third branches are respectively configured with a batch normalization layer, an adaptive grouping module, and a feature consistency constraint from front to back. The first and second branches are connected to the feature splicing layer through an average pooling layer, and the third branch is connected to both the feature splicing layer and the self-attention layer. F 1. Characteristics F 2 and features F 3. Enter the first branch to the third branch respectively.

2. The traffic police gesture recognition method based on computer vision and human skeleton as described in claim 1, characterized in that, The loss function of the recognition network model includes feature consistency constraint, skeletal keypoint detection loss function, and category classification recognition loss function; the feature consistency constraint adopts the mean squared error loss function, the skeletal keypoint detection loss function adopts the smoothed L1 loss function, and the category classification recognition loss function adopts the cross-entropy loss function.

3. The traffic police gesture recognition method based on computer vision and human skeleton as described in claim 1, characterized in that, In step S22, the gesture recognition model analyzes the key points of the skeleton from coarse to fine based on feature information, and combines the joint motion acceleration variables to focus on the intensity, suddenness or fine dynamic changes of the action.

4. A traffic police gesture recognition system based on computer vision and human skeleton, based on the traffic police gesture recognition method based on computer vision and human skeleton as described in any one of claims 1-3, characterized in that, It includes a video acquisition and processing module, a model training module, and a gesture detection module; the video acquisition and processing module is used to acquire and process the detected video to obtain time-series image data; the model training module is used to train a recognition network model using a training set; the gesture detection module is used to output traffic police gesture recognition results based on the trained recognition network model; The recognition network model includes a human skeleton key point detection model and a gesture recognition model. The human skeleton key point detection model includes a backbone network, a hierarchical pyramid module, and a graph convolutional layer module arranged from front to back.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the traffic police gesture recognition method based on computer vision and human skeleton as described in any one of claims 1-3.

6. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the traffic police gesture recognition method based on computer vision and human skeleton as described in any one of claims 1-3.

Images