Dual-channel video collaborative perception behavior gesture recognition method

Abstract

This invention provides a dual-path video collaborative perception method for behavior and posture recognition, comprising a dual-path video collaborative perception system including camera modules on left and right smart light poles, an edge computing module, and an interaction module. The camera modules on the left and right smart light poles are each deployed with a single camera on one side of the pole for collecting pedestrian video data. The edge computing module is a lightweight pedestrian action and posture recognition neural network deployed at the end of the smart light pole for processing pedestrian video data, i.e., inferring the real-time movement category of pedestrians. The interaction module is used to share the pedestrian posture information acquired by the two neural networks, enabling game-theoretic optimization of the results. This recognition algorithm can be used for mining street pedestrian video surveillance information and real-time emergency response processing. It can process pedestrian video surveillance information in real-time, efficiently, and accurately with low computing resources, saving manpower and material resources, and ensuring the safety of pedestrians and public and private property.

Description

Technical Field

[0001] This invention belongs to the field of computer vision algorithm technology, specifically relating to a behavior pose recognition method based on dual-path video collaborative perception. Background Technology

[0002] Currently, surveillance cameras are practically ubiquitous on every street in people's daily lives, constantly recording pedestrians and vehicles. However, aside from traffic violation penalties, due to the sheer volume of recorded information and the limitations of manpower, much of this data is only retrieved after an unexpected situation occurs, potentially delaying optimal intervention. Examples include pedestrians suffering sudden heart attacks, elderly people falling and unable to get up, or theft and vandalism. But with the continuous advancement of computer vision algorithms and the decreasing cost of hardware with equivalent computing power, it has become possible for smart terminals to process this information and respond in real time.

[0003] However, most existing motion and pose recognition algorithms are based on single-channel video, which cannot effectively address the problem of pedestrian joint occlusion with relatively low computational power.

[0004] In summary, existing technologies have some shortcomings and deficiencies in further mining and processing pedestrian behavior and posture information in surveillance, so it is necessary to develop new methods to handle current application scenarios. Summary of the Invention

[0005] This invention addresses the aforementioned problems and aims to provide a method for behavior and pose recognition in videos based on dual-path video collaborative perception, thereby achieving this with lower computational power. The invention employs the following technical solution:

[0006] This invention provides a behavior pose recognition method based on dual-path video collaborative perception, characterized by the following steps:

[0007] Step S1: Construct a dual-path video collaborative perception system, including two perception units set on both sides of the road. Each perception unit includes: a camera for capturing pedestrian posture images on the road; an edge computing module containing a neural network model for recognizing pedestrian behavior postures based on the input pedestrian posture images; and an interaction module for interacting with at least the other perception unit, wherein the two cameras have overlapping field of view areas.

[0008] Step S2: The road is captured by the two cameras, and pedestrians on the road are identified based on the captured video information. It is then determined whether the pedestrian is in the overlapping field of view.

[0009] If step S3 is incorrect after step S2, the two sensing units share the video information, and the two neural network models use the second rolling temporal game optimization algorithm to identify pedestrian behavior and posture in the video information. The second rolling temporal game optimization algorithm is as follows: when only one sensing unit captures a pedestrian, the recognition result of the neural network model of that sensing unit is used; as time goes by, the spatial position of the pedestrian changes, and the sensing unit that can capture the pedestrian changes accordingly; when there is still only one sensing unit that can capture the pedestrian, the recognition result of the neural network model of that sensing unit is used.

[0010] In step S4, if step S2 is correct, the two neural network models will share the process feature output and weight parameters. Both neural network models will predict and score the probability that the pedestrian's action belongs to each action category, and the action category with the highest score in the two predictions will be used as the recognition result.

[0011] The dual-path video collaborative perception behavior posture recognition method provided by the present invention may also have the following technical features, wherein the neural network model includes: a lightweight posture recognition network based on MobileNet, which takes the video captured by the camera on one side after preprocessing as input and outputs a heatmap of pedestrian skeleton point positions for multiple consecutive frames; and an action recognition network based on 3D-CNN, wherein the interaction module fills in the missing joint point positions in the pedestrian skeleton point position heatmap, and uses the filling result as the input of the action recognition network, and outputs the action category of each pedestrian.

[0012] The dual-path video collaborative perception behavior posture recognition method provided by the present invention may also have the following technical features: the lightweight posture recognition network serves as the first stage, the action recognition network serves as the second stage, and the interaction module records the position, relative position, and position change features of each skeletal point of the pedestrian output by the first stage of the two neural network models respectively. When the pedestrian is in the overlapping visual field area, if the skeletal point of the pedestrian in the visual field of one of the perception units is occluded, the position of the occluded skeletal point is inferred and filled in using the pedestrian motion feature information obtained by the other perception unit.

[0013] The dual-path video collaborative perception behavior and posture recognition method provided by the present invention may also have the following technical features, wherein the perception unit is installed on a smart light pole, and step S2 includes the following sub-steps:

[0014] Step S2-1: Identify pedestrians in the video information using a predetermined target recognition algorithm;

[0015] Step S2-2: Use the geographical location information of the smart light pole itself and the video information captured by the camera at the smart light pole to perform spatiotemporal positioning;

[0016] Step S2-3: Construct the pedestrian's ID code;

[0017] Step S2-4: Determine whether the pedestrian is in the overlapping field of vision area.

[0018] The dual-path video collaborative perception behavior pose recognition method provided by this invention may also have the following technical features, wherein the training process of the neural network model includes the following steps:

[0019] Step S1-1: Construct the dual-path video collaborative perception system by using the cameras on both sides of the road to collect synchronized time-series video data of various pedestrian behaviors and postures from two different perspectives.

[0020] Step S1-2: Extract features from the video data collected in step S1-1, extract joint pose information, and combine different action features to perform data labeling and classification to construct a dual-channel paired pedestrian behavior pose dataset.

[0021] Steps S1-3: Based on the dataset, the two neural network models are trained separately using their respective captured video data and annotations, and their model parameters are adjusted.

[0022] Steps S1-4: Based on the dataset, the two neural network models form a game network through the interaction module. The model parameters of the two neural networks are optimized by the first rolling time-domain game optimization algorithm. The first rolling time-domain game optimization algorithm is as follows: the two neural network models compare the behavior posture recognition results and recognition accuracy of the same pedestrian, and optimize the parameters of the other neural network model with the neural network model with higher recognition accuracy as the target, until the recognition accuracy of the two neural network models are consistent. Then, they are trained separately and the above process is repeated.

[0023] Steps S1-5: The two neural network models form a feature-sharing network through the interaction module, sharing training parameters and feature outputs for contrast cases, inaccurate cases, or low-accuracy dead loop cases.

[0024] The dual-path video collaborative perception behavior pose recognition method provided by this invention may also have the following technical features, wherein the training process of the neural network model further includes the following steps:

[0025] Steps S1-6: Based on the dataset, obtain the recognition accuracy of the two neural network models, and determine whether the recognition accuracy reaches the predetermined accuracy.

[0026] If the determination in steps S1-7 and S1-6 is yes, the two neural network models are deployed to the two sensing units respectively for application;

[0027] If steps S1-8 and S1-6 are deemed incorrect, the keyframes of the cases in steps S1-5 are stored and a long-tail dataset is constructed for re-optimizing the neural network model.

[0028] The dual-path video collaborative perception behavior pose recognition method provided by this invention may also have the following technical features, and further includes the following steps:

[0029] Step S5: Determine whether the behavior pose recognition result of the neural network model is ideal;

[0030] If step S5 determines no, the video information is sent to the backend to construct a long-tail dataset;

[0031] Step S7: The neural network model is further trained and optimized using the long-tail dataset, and the trained neural network model is transmitted remotely to the sensing unit for upgrading.

[0032] Invention Function and Effect

[0033] The dual-path video collaborative perception behavior posture recognition method of the present invention has the following advantages compared with the existing pedestrian behavior posture recognition methods:

[0034] 1. By setting up sensing units on both sides of the road, each sensing unit contains a neural network model capable of recognizing pedestrian behavior and posture, edge computing power can be fully utilized to perform pedestrian behavior and posture recognition tasks, reducing the computing load on the cloud and making multi-terminal real-time pedestrian behavior and posture recognition possible.

[0035] 2. Pedestrian action and posture recognition is achieved by using dual-channel video data captured by sensing units on both sides of the road, which effectively overcomes the problem of low recognition accuracy caused by partial occlusion of pedestrian bodies when using single-channel video recognition. In addition, the neural network models on both sides form a game network to optimize parameters and recognition results, which effectively improves the accuracy and completeness of pedestrian information recognition. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the behavior pose recognition method of dual-path video collaborative perception in an embodiment of the present invention;

[0037] Figure 2 This is a flowchart of the behavior pose recognition method of dual-path video collaborative perception in an embodiment of the present invention;

[0038] Figure 3 This is a flowchart of the training process for the pedestrian behavior posture recognition neural network model in an embodiment of the present invention;

[0039] Figure 4 This is a detailed flowchart of the behavior pose recognition method of dual-path video collaborative perception in an embodiment of the present invention. Detailed Implementation

[0040] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes the behavior posture recognition method of dual-path video collaborative perception of this invention in detail with reference to embodiments and accompanying drawings.

[0041] <Example>

[0042] Figure 1 , Figure 2 These are the schematic diagram and flowchart of the behavior pose recognition method for dual-path video collaborative perception in this embodiment.

[0043] like Figure 1-2 As shown, the behavior pose recognition method of dual-path video collaborative perception includes the following steps:

[0044] Step S1: Construct a dual-path video collaborative perception system, including two perception units on both sides of the road, which include cameras and neural network models;

[0045] Step S2: Obtain dual-path video of the road, identify pedestrians, and determine whether the pedestrians are in the overlapping field of view area;

[0046] If step S3 is not determined in step S2, the two sensing units will share video information, and the neural network model will use a rolling temporal game optimization algorithm to identify pedestrian behavior and posture in the video information.

[0047] If step S4 is true and step S2 is true, the neural network model will share the process feature output and weight parameters to perform pedestrian behavior posture recognition.

[0048] Step S5: Determine whether the behavior pose recognition result is satisfactory;

[0049] If step S6 and step S5 are not correct, construct the long-tail dataset;

[0050] Step S7: Use the long-tail dataset to further optimize and update the neural network model.

[0051] The steps described above will be explained in detail below.

[0052] Step S1: Construct a dual-path video collaborative perception system. The system includes two perception units, each of which includes a camera, an edge computing module, and an interaction module.

[0053] In this embodiment, a dual-path video collaborative perception system is constructed using a pair of smart light poles on both sides of the road. Each smart light pole is equipped with one camera, one edge computing module, and one interaction module, all connected to the same control module. The smart light pole also has conventional features such as a positioning module.

[0054] Two cameras are used to capture images of pedestrian postures on the road from two different perspectives. An edge computing module is used to deploy a pedestrian posture recognition neural network model, which identifies pedestrian behavior and postures based on the images captured by the corresponding cameras. An interaction module is used to interact with another sensing unit.

[0055] The pedestrian pose recognition neural network model consists of two parts: a lightweight pose recognition network based on MobileNet and an action recognition network based on 3D-CNN. The first stage, the lightweight pose recognition network based on MobileNet, takes video footage captured by a camera on one side (appropriately cropped and compressed) as input and outputs a multi-frame heatmap of pedestrian skeleton positions for each pedestrian. An interactive module fills in missing joint positions in the multi-frame heatmap output by the lightweight pose recognition network. The filled-in result serves as the input to the second stage, the 3D-CNN-based action recognition network, whose output is the action category for each pedestrian.

[0056] The appropriate cropping and compression process involves: first, randomly sampling keyframes within each time unit of several consecutive video clips; then, cropping the consecutive keyframes to a suitable aspect ratio and reducing them to a suitable resolution for processing by the neural network model. The pedestrian skeleton point location heatmap reflects the probability of human joints appearing in various regions, and is a visualization of the Gaussian distribution of these probabilities.

[0057] The interaction module also records the positions, relative positions, and position change features of each skeletal point of the pedestrian, output by the first stage of the two neural network models. When the pedestrian is in the overlapping field of view of the two cameras, if the pedestrian skeletal points in one side of the field of view are occluded, the positions of the unidentified skeletal points are inferred and filled in using the pedestrian motion feature information obtained from the unoccluded side, thereby improving the accuracy of the action recognition results optimized by the game between the two neural networks. In addition, if the pedestrian skeletal point position heatmap output by the first stage only shows skeletal point occlusion or missing points in a few discontinuous or a small number of continuous video frames, the relevant frames are directly removed and not used as input for the next stage, but the position and motion feature information of the unoccluded skeletal points is still recorded and retained in the interaction module.

[0058] Figure 3 This is a schematic diagram of the training process of the pedestrian behavior posture recognition neural network model in this embodiment.

[0059] like Figure 3 As shown, the training process of the pedestrian behavior pose recognition neural network specifically includes the following steps:

[0060] Step S1-1: Similarly, construct the dual-path video collaborative perception system described above, and use the cameras on both sides to collect synchronous time-series video data of various pedestrian behaviors and postures from two different perspectives.

[0061] The types of dual-channel video data collected include, but are not limited to, pedestrian spatiotemporal localization, pedestrian joint motion information extraction, behavioral action classification (including but not limited to squatting, standing, walking, running, lying down, punching, kicking), and various combinations of videos taken from lampposts at different locations.

[0062] Steps S1-2 involve extracting features from video data of pedestrians from different perspectives, extracting joint posture information, and combining different action features to perform data labeling and classification, thereby constructing a dual-channel paired pedestrian behavior posture dataset for learning and training of neural network models on both sides.

[0063] Steps S1-3: Based on the above dataset, the two neural network models are trained separately using their respective video data and annotations, their model parameters are adjusted, joint motion information is extracted, and behavior and posture classification is performed.

[0064] Steps S1-4: Based on the above dataset, the two neural network models form a game network through an interaction module. The model parameters of the two neural networks are optimized online using the first rolling time-domain game optimization algorithm to further improve the accuracy of key motion information extraction.

[0065] The first rolling time-domain game optimization algorithm in the model training phase refers to comparing the results and accuracy of the neural network models of the two smart light poles in recognizing the same pedestrian behavior, and making mutual corrections. That is, taking the neural network model with higher accuracy as the target, the parameters of the neural network model with lower accuracy are optimized until the recognition accuracy of the two models is consistent. Then, their respective training optimizations are carried out, and the above process is repeated.

[0066] In steps S1-5, the two neural network models form a feature-sharing network through an interaction module. For contrast cases, inaccurate cases, or low-accuracy dead loop cases, they share training parameters and feature outputs, and improve training accuracy through a shared, complementary, and synergistic enhancement mode.

[0067] Steps S1-6: Based on the above dataset, obtain the recognition accuracy of the two neural network models and determine whether the recognition accuracy is ideal (i.e., whether it reaches the predetermined accuracy).

[0068] If steps S1-7 and S1-6 are deemed correct, the trained neural network model will be deployed to the two smart light poles for application.

[0069] If steps S1-8 and S1-6 are deemed incorrect, the keyframes of the cases in step S1-5 will be automatically stored and sent to the back-end administrator. The administrator will then intervene and manually process the data, constructing a long-tail dataset to re-optimize or replace the model.

[0070] In this process, managers manually process relevant video information, set correct labeled data for it, construct a long-tail dataset, and further optimize and update the model based on the long-tail dataset. In addition to resampling, reweighting, and transfer learning, optimization methods can also be used to create supplementary training data based on the characteristics of specific long-tail data and the relative stability of the characteristics of the smart light pole itself, so as to enhance the recognition accuracy of specific situations.

[0071] Step S2: The road is captured by cameras on both sides. Based on the captured video information (video clips), pedestrians on the road are identified, and it is determined whether the pedestrian is in the overlapping field of view area.

[0072] Figure 4 This is a detailed flowchart of the behavior pose recognition method of dual-path video collaborative perception in this embodiment.

[0073] like Figure 4 As shown, step S2 includes the following sub-steps:

[0074] Step S2-1: Use a predetermined target recognition algorithm to identify pedestrians in the video information, that is, pedestrians appear in the area covered by the camera of the smart light pole.

[0075] Human target recognition algorithms are existing technologies, so I won't go into detail.

[0076] Step S2-2: Use the smart light pole's own geographical location information and the video information captured by the light pole's camera to perform spatiotemporal positioning.

[0077] Step S2-3: Construct pedestrian ID codes.

[0078] Step S2-4: Determine whether the pedestrian is in the overlapping field of view of the two cameras.

[0079] That is, to determine whether the pedestrian can be identified from the videos of both cameras.

[0080] If step S3 and step S2 are both incorrect, meaning the pedestrian is in a non-overlapping field of view, the two sensing units will share the video information captured and use the second rolling temporal game optimization method to detect the pedestrian's behavior and posture in the video information.

[0081] like Figure 4 As shown, step S3 includes the following sub-steps:

[0082] In step S3-1, the two sensing units share the captured video information through the interaction module to perform individual pedestrian behavior and posture recognition in a video sharing manner.

[0083] In step S3-2, the two neural networks use the second rolling temporal game optimization algorithm to detect pedestrian behavior and pose in the video information, thereby achieving complementary blind spots in the pedestrian ID index.

[0084] The second rolling time-domain game optimization algorithm in the application phase refers to the following: when only one side can capture pedestrians, the prediction result of the neural network model on that side is used; as time goes by, the spatial position of pedestrians will change, and the smart light poles that can capture pedestrians will also change. When the number of smart light poles that can capture pedestrians is still one, the recognition result of the model on that smart light pole side is used.

[0085] Step S4: When the determination in step S2 is yes, that is, the pedestrian is in the overlapping visual field area, the two neural networks of the two perception units share the process feature output and weight parameters through the interaction module, thereby performing pedestrian behavior and posture detection in video information and improving the accuracy and completeness of information.

[0086] Specifically, the two neural networks output process features through an interaction module and share the weight parameters of the corresponding network parts for identifying contrasting cases, inaccurate cases, or low-accuracy dead loop cases, thereby improving information accuracy and completeness. When pedestrians can be captured on both sides, the neural network models on both sides predict and score the probability of the pedestrian's current action belonging to each action category. At this time, the action category with the highest score among the two predictions is used as the recognition result.

[0087] Furthermore, it should be noted that the pedestrian behavior and posture recognition in the above steps continues from the moment the pedestrian appears in the area covered by the camera until the pedestrian leaves the area, and during this process, the parameters of the neural network are continuously optimized online through rolling time-domain game theory.

[0088] Step S5: Determine whether the pedestrian behavior posture recognition result is ideal. If the result is ideal, proceed to the end state.

[0089] In this embodiment, a score threshold is preset. When the model's prediction score for the probability of the action category of the recognition result is lower than the threshold, it is judged as unsatisfactory, and otherwise it is judged as ideal. When the judgment is correct, the system enters the end state.

[0090] Step S6: If the determination in step S5 is negative, i.e., the recognition result is not ideal, the video information is automatically sent to the back-end administrator. The administrator intervenes to process it manually and stores it in the long-tail dataset.

[0091] In this process, after the administrators set the correct annotation data for the relevant video clips, they then stored them in the long-tail dataset.

[0092] Step S7: Use the long-tail dataset to further train and optimize the neural network, and then transmit the trained model to the sensing unit remotely for upgrade.

[0093] In this embodiment, the parts not described in detail are well-known technologies in the art.

[0094] Functions and effects of the embodiments

[0095] The behavior pose recognition method based on dual-path video collaborative perception provided in this embodiment has the following advantages compared with existing behavior pose recognition methods:

[0096] 1. The method of the embodiment is deployed at the smart light pole. With its lightweight structure, it can make full use of edge computing power to perform pedestrian behavior and posture recognition tasks, reduce the computing power load on the cloud, and make multi-terminal real-time pedestrian behavior and posture recognition possible.

[0097] 2. By using dual-channel video data captured by cameras on smart light poles on both sides of the road for pedestrian action and posture recognition, the problem of low recognition accuracy caused by partial occlusion of pedestrian bodies during single-channel video recognition is effectively overcome. In addition, the neural network models on both sides form a game network to optimize parameters and recognition results, effectively improving the accuracy and completeness of pedestrian information recognition.

[0098] 3. The method in the embodiment clarifies the process of model training and improvement as well as the construction and utilization of long-tail data, enabling the method to become increasingly accurate after actual deployment and application, and has the characteristics of advanced technology and sustainable development.

[0099] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

[0100] In the above embodiment, the two sensing units are installed on the smart light pole. In an alternative, the two sensing units can also be installed on both sides of the road in other forms, such as by mounting them on both sides of the road or on the exterior walls of buildings on both sides of the road with brackets, which can also achieve the technical effect of the present invention.

Claims

1. A method for recognizing behavior gesture by dual-path video co-perception, characterized in that, The method comprises the following steps: Step S1, constructing a dual-channel video collaborative perception system, comprising two perception units arranged on both sides of a road, each of the perception units comprising: a camera for capturing a pedestrian posture image on the road; an edge computing module comprising a neural network model for identifying a pedestrian behavior posture based on an input of the pedestrian posture image; and an interaction module for at least interacting with another of the perception units, wherein the two cameras have an overlapping field of view region; Step S2, capturing the road by the two cameras, identifying a pedestrian on the road based on the captured video information, and determining whether the pedestrian is in the overlapping field of view region; Step S3, when the determination in Step S2 is negative, the two perception units share the video information, and the two neural network models identify a pedestrian behavior posture in the video information by a second rolling time domain game optimization algorithm, wherein the second rolling time domain game optimization algorithm is as follows: when only one of the perception units captures the pedestrian, the identification result of the neural network model of the perception unit is adopted; as time elapses, the spatial position of the pedestrian changes, the perception unit capable of capturing the pedestrian changes accordingly, and when the perception unit capable of capturing the pedestrian is still one, the identification result of the neural network model of the perception unit is adopted; Step S4, when the determination in Step S2 is positive, the two neural network models share process features and weight parameters, the two neural network models both predict and score the possibility of an action of the pedestrian belonging to each action category, and the action category with the highest score in the two groups of predictions is adopted as the identification result.

2. The dual-path video co-perception based action pose recognition method of claim 1, characterized in that: wherein the neural network model comprises: a lightweight posture recognition network based on MobileNet, taking a video captured by the camera on one side after preprocessing as input, and outputting a pedestrian skeleton point position heat map of continuous multiple frames of each pedestrian; and an action recognition network based on 3D-CNN, the interaction module fills in missing joint point positions in the pedestrian skeleton point position heat map, and takes the filling result as input of the action recognition network, and outputs an action category of each pedestrian.

3. The dual-channel video collaborative perception behavior posture identification method according to claim 2, characterized in that: wherein the lightweight posture recognition network is a first stage, and the action recognition network is a second stage, the interaction module records the position, relative position and position change feature of each skeleton point of the pedestrian output by the first stage of the two neural network models, when the pedestrian is in the overlapping field of view region, if the skeleton point of the pedestrian in the field of view of one of the perception units is blocked, the skeleton point position is inferred and filled by using the motion feature information of the pedestrian obtained by the other perception unit.

4. The dual-path video co-perception based action pose recognition method of claim 1, characterized in that: wherein the perception unit is arranged on a smart lamp pole, Step S2 comprises the following substeps: Step S2-1, identifying the pedestrian in the video information by using a predetermined target recognition algorithm; Step S2-2, spatio-temporal positioning is performed by using the geographical position information of the smart lamp pole itself and the video information captured by the camera at the smart lamp pole; Step S2-3, the ID code of the pedestrian is constructed; Step S2-4, it is judged whether the pedestrian is in the overlapping field of view area.

5. The behavior gesture recognition method of the dual-channel video collaborative perception according to claim 1, characterized in that: wherein The training process of the neural network model comprises the following steps: Step S1-1, the dual-channel video collaborative perception system is constructed, and video data of synchronous time series of multiple behavior gestures of pedestrians under two different visual angles is collected by using the cameras on both sides of the road; Step S1-2, feature extraction is performed on the video data collected in step S1-1, joint posture information is extracted, data labeling classification is performed in combination with different motion features, and a data set of dual-channel paired pedestrian behavior gestures is constructed; Step S1-3, based on the data set, the two neural network models are individually trained by using the video data and its labeling of each camera, and the model parameters are adjusted; Step S1-4, based on the data set, the two neural network models constitute a game network through the interaction module, and the model parameters of the two neural networks are optimized through a first rolling time domain game optimization algorithm, Wherein, the first rolling time domain game optimization algorithm is: The behavior gesture recognition results and the recognition accuracy of the two neural network models for the same pedestrian are compared, the neural network model with higher recognition accuracy is taken as the target, the parameters of the other neural network model are optimized, until the recognition accuracies of the two neural network models are consistent, and then each is individually trained, and the above process is repeated; Step S1-5, the two neural network models constitute a feature sharing network through the interaction module, and share the training parameters and feature outputs for contrast cases, misalignment cases or low accuracy dead loop cases.

6. The dual-path video co-perception based gesture recognition method of claim 5, Characterized in that: Wherein, the training process of the neural network model further comprises the following steps: Step S1-6, based on the data set, the recognition accuracies of the two neural network models are obtained, and it is judged whether the recognition accuracy reaches a predetermined accuracy; Step S1-7, if the judgment of step S1-6 is yes, the two neural network models are respectively deployed to the two perception units for application; Step S1-8, if the judgment of step S1-6 is no, the key frames of the cases in step S1-5 are stored and a long tail data set is constructed, which is used to re-optimize the neural network model.

7. The dual-path video co-aptitude behavioral gesture recognition method of claim 1, wherein, Further comprising the following steps: Step S5, it is judged whether the behavior gesture recognition result of the neural network model is ideal; Step S6, if the judgment of step S5 is no, the video information is sent to the background to construct a long tail data set; Step S7, the neural network model is further trained and optimized by using the long tail data set, and the trained neural network model is transmitted to the perception unit in a remote manner for upgrading.

Images