Industrial packing action recognition method based on multi-view adaptive skeleton network

Abstract

The present invention discloses an industrial packing action recognition method of a multi-view adaptive skeleton network. The present invention first uses a stacked differential image as input, instead of the original RGB image, to better extract motion features, thereby reducing the interference of complex backgrounds. After the skeleton data is obtained, it is passed into the adaptive view conversion network to automatically find the best observation angle, which is used to improve the recognition accuracy of workers' complex actions. In addition, a dual-view result is introduced to solve the problem of human body occlusion. The cameras under the two views respectively obtain complementary motion information, which is subsequently passed into a three-layer stacked long short-term memory network for action recognition. Finally, the classification scores under different streams are fused to obtain the recognition result. In addition, in order to identify different subtle movements, a time domain attention module is introduced, and the intercepted local image is used to pass it into the subsequent ResNeXt network, which further improves the recognition accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of action recognition, and in particular relates to an industrial packing action recognition method based on a multi-view adaptive skeleton network. Background Art

[0002] Motion recognition is becoming increasingly important in industrial manufacturing. In complex production environments, by identifying workers' movements and postures, production efficiency and quality can be improved. With the continuous improvement of computer vision technology, motion recognition algorithms have been widely used in video surveillance, human-computer interaction, assisted driving, and medical image analysis.

[0003] Behavior recognition technology is mainly divided into manual feature recognition methods and deep learning methods. Manually constructed feature methods usually model the skeleton and then use mathematical languages ​​such as Grassmann manifolds and Lie groups to represent the features. However, manually extracted features have limitations and often require complex parameter adjustments and optimizations. Manually crafted features require a lot of experience and testing, so they are difficult to implement.

[0004] Deep learning methods are the mainstream approach for skeletal motion recognition and have achieved excellent results. Currently, the most popular deep learning methods are CNN-based methods, RNN-based methods, and GCN-based methods. CNN-based methods are more commonly used to process two-dimensional image tasks, and many researchers have also used deep learning methods to extract motion features from RGB images, optical flow, differential images, and human skeleton data.

[0005] Traditional behavior recognition research often does not consider the problem of data collection failure. Directly applying the above model in a complex factory environment will not achieve good results for the following reasons: 1. There are objects or human bodies blocking the workshop; 2. Frequent changes in light and chaotic backgrounds affect data collection and recognition; 3. Some similar actions cannot be accurately recognized. These problems bring great difficulties to research work. Summary of the invention

[0006] In view of the above problems existing in industrial scenarios, the present invention proposes an industrial packing action recognition method based on a multi-view adaptive skeleton network.

[0007] The present invention first uses stacked differential images as input, instead of the original RGB images, to better extract motion features, thereby reducing the interference of complex backgrounds. After obtaining the skeleton data, it is passed into the adaptive view conversion network to automatically find the best observation angle to improve the recognition accuracy of workers' complex movements. In addition, dual-view results are introduced to solve the problem of human body occlusion. The cameras under the two views respectively obtain complementary motion information, which is subsequently passed into the long short-term memory network for action recognition. Finally, the classification scores under different streams are fused to obtain the recognition results. In addition, in order to identify different subtle movements, a time domain attention module is introduced, and the intercepted local image is used to pass it into the subsequent ResNeXt network, which further improves the recognition accuracy.

[0008] In a first aspect, the present invention provides an industrial packing action recognition method based on a multi-view adaptive skeleton network, comprising the following steps:

[0009] Step S1: Obtain complementary RGB videos under two different viewing angles;

[0010] Step S2: Processing the complementary RGB videos at two different viewing angles to obtain differential image sequences at two different viewing angles;

[0011] Step S3: using a 3D posture estimation algorithm to extract a human skeleton image sequence from the difference image sequences under two different viewing angles;

[0012] Step S4: using an adaptive view conversion network to obtain human skeleton images at two optimal viewing angles;

[0013] Step S5: The human skeleton images at two best observation angles are respectively input into the behavior recognition module for behavior recognition;

[0014] Step S6: The temporal attention module obtains the output results of the two recognition unit LSTM networks in the behavior recognition module and generates a temporal attention weight A T ;

[0015] Step S7: Determine the position of the hand according to the human skeleton image sequence obtained in step S3, and then crop the complementary RGB videos under two different perspectives obtained in step S1 to obtain a partial image of the hand; finally, input the partial image of the hand into the ResNeXt network to extract the hand feature matrix g;

[0016] Step S8: The hand feature matrix g output by the ResNeXt network and the temporal attention weight A generated by the temporal attention module T Multiply linearly in the time dimension, then add the multiplication result to the original feature matrix g bit by bit, and finally get the time domain modified feature matrix g T ;

[0017] Step S9: The time domain corrected feature matrix g T Obtain hand action recognition results through the fully connected layer and softmax layer;

[0018] Step S10: weighted fusion of the recognition result output by the behavior recognition module in step S5 and the hand action recognition result obtained in step S9 to obtain the industrial packing action recognition result.

[0019] In a second aspect, the present invention provides an industrial packaging action recognition system, comprising:

[0020] A data acquisition module, used to acquire complementary RGB videos at two different viewing angles;

[0021] A differential image extraction module is used to process complementary RGB videos at two different viewing angles to obtain differential image sequences at two different viewing angles;

[0022] A human skeleton image extraction module is used to extract a human skeleton image sequence from a differential image sequence under two different viewing angles using a 3D posture estimation algorithm;

[0023] An adaptive view conversion network is used to perform view conversion on the human skeleton image sequence output by the human skeleton image extraction module to obtain human skeleton images of two optimal observation perspectives;

[0024] A behavior recognition module is used to perform behavior recognition on human skeleton images at two optimal observation angles;

[0025] The temporal attention module is used to obtain the output results of the two recognition unit LSTM networks in the behavior recognition module and generate the temporal attention weight A T ;

[0026] The hand feature extraction module determines the position of the hand according to the human skeleton image sequence output by the human skeleton image extraction module, and then crops the complementary RGB videos under two different perspectives obtained by the data acquisition module to obtain a local hand image; finally, the local hand image is input into the ResNeXt network to extract the hand feature matrix g;

[0027] The time domain correction module combines the hand feature matrix g output by the hand feature extraction module with the time domain attention weight A generated by the time domain attention module T Multiply linearly in the time dimension, then add the multiplication result to the original feature matrix g bit by bit, and finally get the time domain modified feature matrix g T ;

[0028] The hand action recognition module uses the time domain correction feature matrix g output by the time domain correction module TObtain hand action recognition results through the fully connected layer and softmax layer;

[0029] The industrial packing action recognition module performs weighted fusion on the recognition result output by the behavior recognition module and the hand action recognition result obtained by the hand action recognition module to obtain the industrial packing action recognition result.

[0030] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the described method.

[0031] In a fourth aspect, the present invention provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the described method.

[0032] Beneficial effects of the present invention:

[0033] The present invention proposes an industrial packing action recognition system of a multi-view adaptive skeleton network, which uses differential images as the input of the model, and uses multiple views in complementary directions to solve the problem of human body occlusion. The differential human skeleton under two complementary views is passed into the adaptive view conversion network. In the adaptive view conversion network, the skeleton data is rotated to the optimal virtual observation angle, and the converted skeleton data is passed into the long short-term memory network (LSTM), and then the classification scores under different streams are fused to obtain the recognition result. In addition, in order to solve the problem of subtle action recognition, the present invention adopts a local positioning image convolution network combined with an attention mechanism, which is passed into the ResNeXt network for recognition. Finally, the results of skeleton and local image recognition are fused to predict the worker's behavior and action, effectively improving the accuracy of behavior recognition in industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the industrial packing action recognition method of the present invention;

[0035] Figure 2 Schematic diagram of the attention unit in the temporal attention network of the present invention;

[0036] Figure 3 To observe the changes of viewpoints for virtual observation;

[0037] Figure 4 This is an action sequence for an industrial box packing scenario; (a) is nailing inside the box; (b) is closing the box lid; (c) is caulking the box lid; (d) is nailing outside the box; (e) is registering information; and (f) is other actions.

[0038] Figure 5 It is a multi-view skeleton observation; (a) is the original image (RGB); (b) is the observation view 1; (c) is the observation view 2; (d) is the observation view 3;

[0039] Figure 6 is the multi-view skeleton occlusion; (a) is the view Figure 1 (RGB); (b) is visual Figure 2 (RGB); (c) is the visual Figure 1 (skeleton); (d) is visual Figure 2 (skeleton). DETAILED DESCRIPTION

[0040] The action recognition method provided by the present invention will be described in detail below.

[0041] A multi-view adaptive skeleton network method for industrial box packing action recognition, such as Figure 1 The steps shown include:

[0042] Step S1: Obtain complementary RGB videos under two different viewing angles;

[0043] Complementary RGB videos are obtained by two cameras in relative positions, such as one camera located in front of the worker and the other located behind the worker. The complementary RGB videos collected by the two cameras can effectively solve the problem of the worker's body being blocked by himself or the cargo.

[0044] Step S2: Processing the complementary RGB videos at two different viewing angles to obtain differential image sequences at two different viewing angles;

[0045] Most action recognition methods use RGB images or human skeleton data as network input to extract features, the most direct representatives of which are CNN and GCN networks. However, in actual industrial scenarios, due to problems such as background clutter, frequent light changes, and human body occlusion, the above two methods are difficult to effectively extract action features, which in turn causes relatively large interference to subsequent action recognition. Based on this, the present invention introduces a residual frames (RF) module to reduce this interference and improve network performance.

[0046] For traditional 3D convolutional networks, stacked frames are used as input, and the input shape is T×H×W×C, which represents an RGB image with a T frame height of H, a width of W, and a channel C of 3. The convolution kernel of each three-dimensional convolutional layer is also three-dimensional, and three-dimensional data operations are performed simultaneously for each layer. However, this is when the motion features and spatial features can be perfectly extracted. The present invention obtains a differential image by subtracting two adjacent frames, retaining the differences between the two frames. Due to the nature of the differential image, in a single differential image, the motion information is stored in the spatial axis. Improving the 2D convolutional network with the differential image has been proven to be quite effective. However, when faced with more complex actions, the duration is generally longer and cannot be represented by a single frame, so continuous multi-frame differential images are required. In the stacked differential image, the action information exists not only on the spatial axis, but also on the time axis, and the connection between two adjacent frames is also part of the running information.

[0047] Among the traditional moving target detection algorithms, the difference method and the optical flow method are the most commonly used methods. Although the optical flow method has good effects, it is complex in calculation and has high hardware requirements, so it is rarely used in real-time industrial scenarios. In comparison, the difference method has a very low computational cost, which can even be ignored, making it one of the methods more suitable for practical applications. At the same time, the difference method is further divided into background difference and inter-frame difference. The former uses an algorithm to separate the foreground and background. The disadvantage is that it cannot be optimally separated in scenarios where the light in the factory changes frequently. The latter uses the difference between two adjacent frames in the video to detect moving objects, but large holes will be generated inside the object. Note that the use of the inter-frame difference method can well extract the contour of the moving object, but it will cause holes inside the object and lose some motion information. However, in practical applications, the skeleton extraction algorithm can be combined to obtain the skeleton data of the moving object from the differential image and pass it into the subsequent main network. Specifically:

[0048] Process the consecutive RGB images at each viewing angle into differential images;

[0049] RF i～j =|F i～j -F i+1～j+1 | (1)

[0050] where F i represents the i-th frame image, F i～j Represents the image of the i-j consecutive frames, RF i～j represents the stacked difference image of the i-th to j-th consecutive frames;

[0051] Step S3: Use a 3D pose estimation algorithm to extract a human skeleton image sequence from a differential image sequence under two different perspectives; the 3D pose estimation algorithm is a pose estimation algorithm based on RGB images. Continuous differential images are segmented into dense probabilistic body component labels, and components are defined as parts that are similar in the skeletal joint space. The inferred components are then reprojected into the world space coordinate system, and the spatial distribution pattern of each component is localized to form a prediction.

[0052] Step S4: using an adaptive view conversion network to obtain human skeleton images at two optimal viewing angles;

[0053] The 3D skeleton data obtained from step S3 is often a few actions in an action sequence. However, it is difficult to observe and identify all actions from a fixed perspective. For some actions, it is easy to identify from one angle, but difficult to identify from other angles. Therefore, it is important to find a suitable observation angle. In actual production operations, there are many viewpoints that can observe and identify different types of behaviors. In order to find the most suitable view to improve recognition performance, the present invention proposes an adaptive view conversion network. The proposed adaptive view conversion network can help the skeleton to automatically find the best observation viewpoint and convert the skeleton data to the new viewpoint.

[0054] like Figure 3 The captured raw skeleton data is expressed in the camera coordinate system (global coordinate system), and its coordinate origin is at the center of the camera sensor. In order to fix the initial observation orientation of each action, the global coordinate system is converted to a new coordinate system O with the center of the skeleton hip joint as the origin. The original skeleton is rotated counterclockwise around the X-axis and Z-axis of the global coordinate system to automatically obtain the best observation angle. There are several considerations for using this rotation method. (1) For a behavior action, it can be observed from different perspectives, but in some perspectives it is difficult to identify the accurate action due to human body overlap, occlusion and other reasons. Therefore, it is reasonable to observe from different perspectives. The dual view is used to solve the problem of obstacle occlusion, and the perspective conversion module is used to reduce the interference of human body overlap, provide the best observation angle for each action to the greatest extent, and improve the recognition accuracy. (2) In actual industrial scenarios, most action recognition errors are caused by observations from different horizontal (Z-axis) and vertical (X-axis) perspectives. By rotating the original skeleton data captured by the two cameras counterclockwise on the Z-axis and counterclockwise on the X-axis, all perspectives in the global coordinate system can be covered. Combining multiple views can obtain the best observation results. Specifically:

[0055] Given a human skeleton image sequence S in the global coordinate system O, the coordinates of the jth skeleton joint in the image at the tth frame are denoted by v t,j=[x t,j ,y t,j ,z t,j ], where t∈(1,...,T),j∈(1,...,J), T represents the total number of frames in the human skeleton image sequence, and J represents the total number of skeleton joints in each frame. The set of skeleton sequences represented in the tth frame is V t = {v t,1 ,...,v t,J}.

[0056] In a global coordinate system O of a certain perspective, assume that the virtual observation viewpoint of the tth frame is represented by a translation vector d t ∈R 3 , and two counterclockwise rotations around the X and Z axes α t ,β t The rotation angle of radians is expressed. All joint coordinates in the same frame share a rotation angle, and the skeleton joint coordinates under the new viewing angle are expressed as:

[0057] v' t,j =[x' t,j ,y' t,j ,z' t,j ] T =R t (v t,j -d t ) (2)

[0058] Where R t Expressed as

[0059]

[0060] in The rotation matrices representing rotations around the X-axis and Z-axis are expressed using the following formulas:

[0061]

[0062]

[0063] In a complete action sequence, each action will have its most appropriate observation angle, and the adaptive view conversion network is used to find the optimal perspective transformation parameter α t ,β t ,d t Finally, the skeleton sequence coordinates V' in the new coordinate system are calculated by formula (2) t ={v' t,1 ,...,v' t,J}.

[0064] The adaptive view conversion network includes two parallel conversion units, each of which includes two LSTM network branches connected in series and a fully connected layer; one LSTM network branch learns the rotation matrix of the node, and the other LSTM network branch learns the translation vector.

[0065] Step S5: The human skeleton images at two best observation angles are respectively input into the behavior recognition module for behavior recognition;

[0066] The behavior recognition module includes two parallel recognition units and a fusion module; each recognition unit includes a layer of LSTM network, a global average pooling module (GAP), and a fully connected module;

[0067] The fusion module is used to fuse the recognition results output by the two recognition units; specifically:

[0068] score(k|S)=w1×p(k|S1)+w2×p(k|S2) (6)

[0069] Where w1 and w2 are the weights of each flow.

[0070] Where p(k|S1) and p(k|S2) represent the recognition posterior probabilities output by the two recognition units; k represents the kth type of behavior, S1 and S2 represent the human skeleton images at the two best observation angles;

[0071] In an actual industrial production environment, due to the problem of obstacles blocking the view, the motion data obtained from a single perspective may be lost, resulting in a decrease in the accuracy of subsequent behavior recognition. In addition, the factory environment is extremely complex, and it is impossible to find an optimal perspective to observe all actions as much as possible. In order to obtain the most complete motion data during the model training and testing phases, the present invention uses a dual-view structure, using two cameras with different perspectives (S1, S2) to obtain video data of the packing process as the input of the model. The dual-view and adaptive view conversion modules are used to perform action recognition on the factory's packing workers to obtain the most complete, effective and accurate data, thereby improving the accuracy of behavior recognition.

[0072] Compared with a single view, the advantage of multi-view is that it can contain additional consistency and complementary information, from which some features that are more helpful to the results can be learned. Among them, the consistency information in different views means that some features are the same and shared, while the complementary information is unique to each perspective and can serve as supplementary information for the action. Taking full advantage of the consistency and complementarity of multiple views can optimize the extraction of the required effective features.

[0073] Step S6: The temporal attention module obtains the output results of the two recognition unit LSTM networks in the behavior recognition module and generates a temporal attention weight A T ;

[0074] The temporal attention network includes two parallel attention units, each of which includes a three-layer recursive neural network, two fully connected layers, and a softmax layer; the tanh activation function is used between the two fully connected layers to correct the eigenvalues. The calculation of each attention unit can be expressed as follows:

[0075]

[0076] A T =softmax(s T ) (9)

[0077] in represents the weights and bias values ​​of the two fully connected layers, s T Represents the attention vector value in the time dimension, Represents the feature matrix output from the recognition unit LSTM network.

[0078] Skeleton sequence is a way of expressing human body movement in space and time dimensions. In a complete skeleton sequence, skeleton data can represent the human body structure located in space, and also capture the key frames of high-frequency limb movement in time domain. This also requires the design of a spatiotemporal attention mechanism for analyzing video action recognition. Structuring the attention mechanism of time and space dimensions can better learn the features of key parts in different dimensions. The present invention also adopts this method to decouple the spatiotemporal attention and apply it to specific modules. According to the positioning of the skeleton joint points, the perspective is focused on the hand joints in the spatial domain, and then the time domain features of the skeleton sequence are combined into the attention mechanism.

[0079] Step S7: Determine the position of the hand according to the human skeleton image sequence obtained in step S3, and then crop the complementary RGB videos under two different perspectives obtained in step S1 to obtain a partial image of the hand; finally, input the partial image of the hand into the ResNeXt network to extract the hand feature matrix g;

[0080] The ResNeXt is a convolutional neural network. ResNeXt consists of a group of residual blocks with the same topological structure, and its aggregation transformation can be expressed as:

[0081]

[0082] where φ i(x) represents any function with the same topological structure, and C is a cardinality, which represents the number of branches with the same structure in the model. Parameter C is also used to describe the complexity of the model. The value of parameter C can be set to any number, preferably 32. That is, the ResNeXt-50 (32x4D) network architecture is used, and 4D represents that each path in the first set of convolutional blocks accepts features of 4 channels.

[0083] For human motion recognition, different parts of the body have different importance. For example, in the action of wearing a helmet, the weight of the hands and head is higher. In the factory scenario, the part of the body that workers rely on most is their hands, and most tasks are completed with hands as the main focus. Therefore, more attention needs to be paid to some movements of the workers' hands. Specifically, there are the following two problems. First, it is difficult to handle some similar movements during exercise. Second, some non-compliant "fake movements", that is, even if the worker makes the same body movement, but does not hold the corresponding tools in his hands, the movement is considered invalid.

[0084] In order to solve these problems, the present invention adopts the architecture of attention module combined with convolutional neural network. According to the positioning of the hand joint coordinates in the skeleton, the local image in the original image is cropped to form a local image action sequence, and the action sequence is passed into the convolutional neural network for recognition, which is used to determine some subtle action differences and "false action" problems. Due to the problems of blur, object occlusion and background interference caused by motion, it is impossible to accurately identify the specific action sequence by only tracking and identifying the hand movements of the workers. Therefore, the main network of action recognition and the sub-network of local image recognition are combined to greatly improve the accuracy of action recognition. Experimental results have proved this. According to experience, setting the fusion weight ratio of the main network and the sub-network to 3 to 1 has the best results.

[0085] The present invention adopts ResNeXt to recognize hand movements.

[0086] Step S8: The hand feature matrix g output by the ResNeXt network and the temporal attention weight A generated by the temporal attention module T Multiply linearly in the time dimension, then add the multiplication result to the original feature matrix g bit by bit, and finally get the time domain modified feature matrix g T .

[0087] g T It can be calculated by the following formula:

[0088]

[0089] Among them, reshape(x) represents the operation of converting the dimension of the feature vector x to be consistent with the dimension of the feature map g.

[0090] Step S9: The time domain corrected feature matrix g T The hand action recognition results are obtained through the fully connected layer and the softmax layer.

[0091] The corrected feature map needs to be converted into the category of worker behavior for output. First, the feature map is converted into a one-dimensional vector using average pooling and input into the fully connected layer. The fully connected layer converts several inputs into outputs of the number of categories that need to be identified. The softmax layer then converts the value of the category vector into a value between 0 and 1, that is, the probability of each corresponding category.

[0092] Step S10: Perform weighted fusion on the recognition result output by the behavior recognition module in step S5 and the hand action recognition result obtained in step S9 to obtain the industrial packing action recognition result. Figure 4 .

[0093] The recognition results of steps S5 and S9, i.e. the worker's human skeleton and hand details, will jointly determine the worker's current behavior category. The dual-stream recognition architecture not only takes advantage of the accuracy of the skeleton data, but also makes up for the differences in small movements. The final recognition results are weighted and fused, and the recognition results of the boxing action are finally output.

[0094] Pytorch was used as the deep learning platform and trained on two NVIDIA RTX 3080Ti GPUs. The video segments obtained depth maps through the Kinect camera. The data was horizontally inverted before entering the network, and time jitter was applied during the training phase. Three LSTM layers were stacked together with a packet loss rate of 0.5. In the CNN model (ResNeXt network), the momentum was set to 0.9, the weight decay was set to 0.0005, and the batch size was set to 32. The stochastic gradient descent algorithm (SGD) and the root mean square propagation algorithm (RMSPROP) were used to train the CNN network and the LSTM network, respectively. The learning rate of both models was initially set to 0.01, and then the learning rate was reduced according to a fixed schedule. In this model, a total of 350 rounds of training were performed, and the learning rate was reduced every 50 rounds. The step size of the LSTM model was set to 20.

[0095] NTU RGB+D experimental results analysis

[0096] The data of the NTU-60 dataset is collected by the Kinect depth camera. It is currently the largest skeleton-based dataset, with 56,880 skeleton sequences, including 60 different behavior types, and each skeleton contains 25 joints. The dataset was collected from 40 volunteers through 3 Kinect cameras in different orientations (-45, 0, 45), and RGB image data, depth data and skeleton data were provided for each frame in the video segment. In order to evaluate, this implementation adopts two evaluation methods: cross-subject evaluation (CS) and cross-view evaluation (CV). In the CS evaluation, the action sequences of the 40 volunteers are divided into training sets and test sets. In the CV evaluation, the samples obtained at 0 and -45 angles are used for training, and the rest are used for testing. NTU-60 is a single-view action dataset, and the present invention uses a single-view model for evaluation. Table 1 shows the evaluation results of various methods on the NTU RGB+60 dataset. It can be seen that the model of this embodiment achieved 85.52% and 93.64% performance in the CS protocol and CV protocol, respectively, which is better than other popular network models. Generally speaking, the method based on GCN (Yang et al., 2020; Tang et al., 2018; Liu et al., 2021) is better than the method based on RNN (Liu et al., 2016; Song et al., 2017; Zhang et al., 2017), and the method considering multi-view and adaptive view conversion modules (Liu et al., 2021; Zhang et al., 2019) will perform better than the traditional method. However, the method of this embodiment takes into account the local positioning image, so that the final result is 1.66% and 1.45% better than the best AMV-GCN model in CS and CV. The single view model of this embodiment combined with the CNN network can also achieve good performance.

[0097] Table 1 Comparison with mainstream action recognition methods in NTU RGB-D

[0098]

[0099] Note: Black fonts indicate the best results.

[0100] Analysis of MBAD experimental results

[0101] On the MBAD dataset, the mainstream behavior recognition methods and the method of the present invention are compared. Table 2 shows the experimental results, and it can be observed that the method of this embodiment is superior to other methods in terms of accuracy. CNN-based methods usually perform better than RNN-based methods, and GCN-based methods have intermediate performance. At the same time, in order to explore the spatiotemporal information of the skeleton, the CNN and RNN network structures are mixed (Zhang et al., 2019), which can improve the accuracy and recall rate, but the method of the present invention is still 3.96% higher in accuracy and 3.81% in recall. We infer that this is because the model of the present invention takes into account differential networks and multiple views. In order to analyze the necessity of each module in the module of the present invention, different experiments are designed for evaluation.

[0102] Table 2 Comparison of MBAD with mainstream behavior recognition methods

[0103]

[0104] Note: Black fonts indicate the best results.

[0105] Table 3 shows the difference between the main network based on RGB images and differential images as input. Using only differential images as network input may cause some body joints to be lost, resulting in lower accuracy. Using RGB images as input can obtain complete observation data, but it is difficult to extract skeleton data later due to the complex factory environment (occlusion, light changes). Using differential images as input combined with the 3D pose estimation algorithm RMPE can capture complete skeleton data and achieve the highest accuracy on the MBAD dataset.

[0106] Table 3 Evaluation of the main network based on RGB and difference images

[0107]

[0108] Note: Black fonts represent the optimal results, and RF is the difference image.

[0109] Table 4 shows the impact of adaptive view conversion network and multi-view on the experimental results. The two views S1 and S2 use the adaptive view conversion network, and we assume that S01 and S02 do not use the adaptive view conversion network and are directly recognized. The recognition efficiency is evaluated by comparing the results of the two pairs of streams. The experimental results show that the recognition performance under single view angles S1 and S2 is better than that of S01 and S02. Although the frame rate fps will decrease, it is still within an acceptable range. In addition, the fusion method of multiple views will also affect the recognition performance. It can be seen from the data in the table that simply fusing two different single-stream output results can improve the performance, and the weighted average method has the best effect, which is 3.83% and 3.03% higher than the single-stream S1 and S2 respectively. The results show that the multi-stream fusion module is beneficial to action recognition, and the weighted fusion method is better than the average fusion. Figure 5 The results of observing the same moving skeleton under different views are shown. After the skeleton data obtained under a certain view is transformed by the adaptive conversion module, the skeleton sequence can be observed from the best observation angle. The experiment found that the module's learning is more inclined to observe the action from the front, because the front observation can disperse the skeleton as much as possible. In contrast, the side observation has the highest degree of mutual occlusion of the skeleton and the worst observation effect. Figure 6 The effect of the multi-view module in solving the occlusion problem is demonstrated. It can be seen that some actions will have object occlusion and human self-occlusion problems under a certain shooting angle. The occlusion problem can be solved by two complementary views, that is, the occluded action can be well recognized in one of the views. In summary, we can draw the following conclusions: (1) In a factory environment, the importance of data obtained from different perspectives is obviously different. We give greater weight to more important perspectives according to the experimental environment and select the best observation perspective, which can greatly improve the recognition performance. (2) Multi-view can better solve the occlusion problem in the factory environment and increase the fault tolerance of the experimental results.

[0110] Table 4 Evaluation of the performance of the adaptive view conversion network and multi-view fusion module

[0111]

[0112] Note: The black font indicates the optimal result, Wei is the weighted average, and Ave is the arithmetic average.

[0113] The present invention proposes a packing behavior detection method for actual production scenarios. The method uses a dual-stream network model. The main network is an adaptive multi-view RNN network. Two depth cameras under complementary perspectives are used to collect data from the same workstation, and the incoming RGB image is converted into a differential image for extracting skeleton information. The skeleton data is subsequently passed into the adaptive view conversion network to obtain the best skeleton observation point. The input behavior recognition module uses weighted fusion of two view features to obtain the recognition result. The main network solves the influence of occlusion and background confusion. In order to make up for the problem of insufficient recognition accuracy of "false actions" and similar actions, the sub-network ResNeXt adds skeleton-located hand image recognition, and the intercepted local positioning image is passed into the ResNeXt network for recognition. Finally, the recognition results of the main network and the sub-network are fused. The packing behavior recognition method proposed in the present invention has an accuracy of 92.31% in the data set MBAD, which is better than several other mainstream behavior recognition methods. In a complex factory environment where the human body is frequently blocked, the method of the present invention can accurately identify the packing actions of packers, while ensuring the fluency of the network and meeting actual production needs.

Claims

1. A multi-view adaptive skeleton network industrial packing action recognition method, characterized by The steps include: Step S1: Obtain complementary RGB videos under two different viewing angles; Step S2: Processing the complementary RGB videos at two different viewing angles to obtain differential image sequences at two different viewing angles; Step S3: using a 3D posture estimation algorithm to extract a human skeleton image sequence from the difference image sequences under two different viewing angles; Step S4: using an adaptive view conversion network to obtain human skeleton images at two optimal viewing angles; specifically: Given a human skeleton image sequence S in the global coordinate system O, the coordinates of the jth skeleton joint in the image at the tth frame are denoted by v t,j =[x t,j ,y t,j ,z t,j ], where t∈(1,...,T),j∈(1,...,J), T represents the total number of frames in the human skeleton image sequence, and J represents the total number of skeleton joints in each frame; the set of skeleton sequences represented in the tth frame is V t = {v t,1 ,...,v t,J }; In a global coordinate system O of a certain perspective, assume that the virtual observation viewpoint of the tth frame is represented by a translation vector d t ∈R 3 , and two counterclockwise rotations around the X and Z axes α t ,β t The rotation angle of radians is expressed; all joint coordinates in the same frame share a rotation angle, and the skeleton joint coordinates under the new observation angle are expressed as: Where R t Expressed as in The rotation matrices representing rotations around the X-axis and Z-axis are expressed using the following formulas: In a complete action sequence, each action will have its most appropriate observation angle, and the adaptive view conversion network is used to find the optimal perspective transformation parameter α t ,β t ,d t Finally, the skeleton sequence coordinates V′ in the new coordinate system are calculated by formula (2) t = {v′ t,1 ,...,v′ t,J }; Step S5: The human skeleton images at two best observation angles are respectively input into the behavior recognition module for behavior recognition; Step S6: The time domain attention network obtains the output results of the two recognition unit LSTM networks in the behavior recognition module and generates the time domain attention weight A T ; Step S7: Determine the position of the hand according to the human skeleton image sequence obtained in step S3, and then crop the complementary RGB videos under two different perspectives obtained in step S1 to obtain a partial image of the hand; finally, input the partial image of the hand into the ResNeXt network to extract the hand feature matrix g; Step S8: The hand feature matrix g output by the ResNeXt network and the temporal attention weight A generated by the temporal attention module T Multiply linearly in the time dimension, then add the multiplication result to the original feature matrix g bit by bit, and finally get the time domain modified feature matrix g T ; Step S9: The time domain corrected feature matrix g T Obtain hand action recognition results through the fully connected layer and softmax layer; Step S10: weighted fusion of the recognition result output by the behavior recognition module in step S5 and the hand action recognition result obtained in step S9 to obtain the industrial packing action recognition result.

2. The method according to claim 1, characterized in that Step S2 specifically includes: Process the consecutive RGB images at each viewing angle into differential images; RF i～j =|F i～j -F i+1～j+1 | (1) Among them, F i represents the i-th frame image, F i～j Represents the image of the i-j consecutive frames, RF i～j Represents the stacked difference image of i-j consecutive frames.

3. The method according to claim 1, characterized in that In step S4, the adaptive view conversion network includes two parallel conversion units, each of which includes two LSTM network branches connected in series and a fully connected layer; one LSTM network branch learns the rotation matrix of the node, and the other LSTM network branch learns the translation vector.

4. The method according to claim 1 or 3, characterized in that The behavior recognition module in step S5 includes two parallel recognition units and a fusion module; each recognition unit includes a layer of LSTM network, a global average pooling module, and a fully connected module; The fusion module is used to fuse the recognition results output by the two recognition units; specifically: score(k|S)=w1×p(k|S1)+w2×p(k|S2) (6) Where w1 and w2 are the weights of each flow; Where p(k|S1) and p(k|S2) represent the recognition posterior probabilities output by the two recognition units; k represents the kth type of behavior, and S1 and S2 represent the human skeleton images at the two best observation angles.

5. The method according to claim 1, characterized in that The time-domain attention network in step S6 includes two parallel attention units, each of which includes a three-layer recursive neural network, two fully connected layers and a Softmax layer; a tanh activation function is used between the two fully connected layers to correct the eigenvalues; The calculation of each attention unit can be expressed as follows: A T =softmax(s T ) (9) in represents the weights and bias values ​​of the two fully connected layers, s T Represents the attention vector value in the time dimension, Represents the feature matrix output from the recognition unit LSTM network.

6. The method according to claim 1, characterized in that In step S7, the ResNeXt is used as a convolutional neural network. ResNeXt is composed of a group of residual blocks with the same topological structure, and its aggregation transformation can be expressed as: where φ i (x) represents any function with the same topological structure, and C is a cardinality, which represents the number of branches with the same structure in the model.

7. The method according to claim 1, characterized in that The characteristic matrix g corrected in the time domain in step S8 T Calculated by the following formula: Among them, reshape(x) represents the operation of converting the dimension of the feature vector x to be consistent with the dimension of the feature map g.

8. An industrial packing action recognition system implementing the method described in any one of claims 1 to 7, characterized in that include: A data acquisition module, used to acquire complementary RGB videos at two different viewing angles; A differential image extraction module is used to process complementary RGB videos at two different viewing angles to obtain differential image sequences at two different viewing angles; A human skeleton image extraction module is used to extract a human skeleton image sequence from a differential image sequence under two different viewing angles using a 3D posture estimation algorithm; An adaptive view conversion network is used to perform view conversion on the human skeleton image sequence output by the human skeleton image extraction module to obtain human skeleton images of two optimal observation perspectives; A behavior recognition module is used to perform behavior recognition on human skeleton images at two optimal observation angles; The temporal attention module is used to obtain the output results of the two recognition unit LSTM networks in the behavior recognition module and generate the temporal attention weight A T ; The hand feature extraction module determines the position of the hand according to the human skeleton image sequence output by the human skeleton image extraction module, and then crops the complementary RGB videos under two different perspectives obtained by the data acquisition module to obtain a local hand image; finally, the local hand image is input into the ResNeXt network to extract the hand feature matrix g; The time domain correction module combines the hand feature matrix g output by the hand feature extraction module with the time domain attention weight A generated by the time domain attention module T Multiply linearly in the time dimension, then add the multiplication result to the original feature matrix g bit by bit, and finally get the time domain modified feature matrix g T ; The hand action recognition module uses the time domain correction feature matrix g output by the time domain correction module T Obtain hand action recognition results through the fully connected layer and softmax layer; The industrial packing action recognition module performs weighted fusion on the recognition result output by the behavior recognition module and the hand action recognition result obtained by the hand action recognition module to obtain the industrial packing action recognition result.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the method according to any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method described in any one of claims 1 to 7.

Images