Behavior recognition method, device, storage medium and electronic device

Abstract

The present disclosure relates to the field of computer technology, and specifically to a behavior recognition method, a behavior recognition device, a storage medium, and an electronic device. The behavior recognition method includes: obtaining multiple skeletal data features of a target object; using a branch displacement convolution subnetwork to perform branch processing on the multiple skeletal data features to obtain multiple feature maps; using a fusion displacement convolution subnetwork to perform mainstream fusion on the multiple feature maps to obtain a feature vector; and using a fully connected layer to perform recognition based on the feature vector to obtain the behavior information of the target object. The behavior recognition method provided by the present disclosure can solve the problem of large amount of calculation in the prior art when performing behavior recognition based on bones.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a behavior recognition method, a behavior recognition device, a storage medium, and an electronic device. Background Art

[0002] In elderly care, health care, and financial services, high-precision automated behavior recognition and processing scenarios are required. Usually, behavior recognition tasks need to be performed based on skeletons. The recognition results can be used for behavior analysis or prediction, which has high use value.

[0003] At present, the mainstream behavior recognition algorithms are all based on deep learning and neural network methods. According to the mode and method used, they are mainly divided into: (1) methods based on 2D convolutional neural network + recurrent neural network, the more representative methods are: LRCN (Long-term Recurrent Convolutional Networks), etc.; (2) methods based on 2D convolutional neural network + optical flow, the more representative methods are: TSN (Temporal Segment Networks), etc.; (3) methods based on 3D convolutional neural network, the more representative methods are: C3D (Convolutional 3D, three-dimensional convolutional neural network), I3D (Inflated 3D, two-stream three-dimensional convolutional neural network), etc.; (4) methods based on skeleton points + neural network, the more representative methods are: ST-GCN (Spatial Temporal Graph Convolutional Networks, spatiotemporal convolutional network), etc.

[0004] In behavior recognition with people and other target objects as the main body and center, the skeleton point + neural network method has stronger robustness and has relatively good results in practice. However, this modality is usually very complex and over-parameterized, and the network usually contains a multi-stream structure with a large number of model parameters, which will lead to a complex training process and high computational cost. Therefore, it is necessary to propose a faster, stronger, and more efficient skeleton-based behavior recognition model to complete this task.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a behavior recognition method, a behavior recognition device, a storage medium and an electronic device, aiming to solve the problem of large amount of calculation when performing behavior recognition based on skeletons in the prior art.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of an embodiment of the present disclosure, a behavior recognition method is provided, including: obtaining multiple skeleton data features of a target object; performing branch processing on the multiple skeleton data features using a branch displacement convolution subnetwork to obtain multiple feature maps; performing mainstream fusion of the multiple feature maps using a fusion displacement convolution subnetwork to obtain a feature vector; and using a fully connected layer to perform recognition based on the feature vector to obtain behavior information of the target object.

[0009] According to some embodiments of the present disclosure, based on the aforementioned scheme, the method of obtaining multiple bone data features of the target object includes: obtaining image data collected by a camera within a preset time according to a preset frame rate set; detecting the bone key points of the target object in each frame of the image data to obtain the bone key point sequence; performing feature processing on the bone key point sequence to obtain the multiple bone data features; wherein the multiple bone data features include at least two of the bone key point sequence features of the first coordinate system, the bone key point features of the second coordinate system, the speed features and the skeleton features.

[0010] According to some embodiments of the present disclosure, based on the aforementioned scheme, the use of a branched displacement convolutional subnetwork to branch the multiple skeleton data features to obtain multiple feature maps includes: inputting each skeleton data feature in parallel into the pre-trained branched displacement convolutional subnetwork corresponding to the skeleton data features to obtain feature maps corresponding to each skeleton data feature; wherein the branched displacement convolutional subnetwork is constructed in series using a batch normalization layer, a displacement map convolution initial block, and at least one displacement map convolution block.

[0011] According to some embodiments of the present disclosure, based on the aforementioned scheme, the multiple feature maps are mainstream fused through a fused displacement convolution sub-network to obtain a feature vector, including: uniformly inputting the multiple feature maps into the pre-trained fused displacement convolution sub-network to obtain the feature vector; wherein the fused displacement convolution sub-network is constructed by serially connecting at least one displacement map convolution block in sequence.

[0012] According to some embodiments of the present disclosure, based on the aforementioned scheme, the displacement map convolution block includes a spatial displacement operation module, a spatial point-by-point convolution module, a temporal displacement operation module and a temporal point-by-point convolution module.

[0013] According to some embodiments of the present disclosure, based on the aforementioned scheme, the method also includes performing model training on the branch displacement convolution subnetwork, the fused displacement convolution subnetwork and the fully connected layer, and the model training includes: obtaining a skeleton data training set and behavior labels of the skeleton data training set; constructing an original input graph according to the skeleton data training set, and performing edge deletion on the original input graph to obtain a target input graph; based on the target input graph, using the branch displacement convolution subnetwork, the fused displacement convolution subnetwork and the fully connected layer to perform recognition to obtain recognition behavior information; comparing the recognition behavior information with the behavior label to modify the model parameters of the branch displacement convolution subnetwork, the fused displacement convolution subnetwork and the fully connected layer.

[0014] According to some embodiments of the present disclosure, based on the aforementioned scheme, the edge deletion of the original input graph to obtain the target input graph includes: constructing an original adjacency matrix according to the displacement relationship in the original input graph; calculating the target adjacency matrix corresponding to the target input graph based on the original adjacency matrix and a preset random edge retention rate; or constructing an attention template adjacency matrix according to the displacement relationship in the original input graph; calculating the edge retention probability of each edge of the original input graph based on the attention template adjacency matrix and a preset adaptive edge deletion parameter; and obtaining the target adjacency matrix corresponding to the target input graph according to the edge retention probability.

[0015] According to a second aspect of an embodiment of the present disclosure, a behavior recognition device is provided, including: an acquisition module, used to acquire a plurality of skeletal data features of a target object within a preset time; a branch module, used to use a branch displacement convolution subnetwork to perform branch processing on the plurality of skeletal data features to obtain a plurality of feature maps; a fusion module, used to perform mainstream fusion of the plurality of feature maps through a fusion displacement convolution subnetwork to obtain a feature vector; and an identification module, used to use a fully connected layer to perform identification based on the feature vector to obtain behavior information of the target object.

[0016] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the behavior recognition method in the above embodiment is implemented.

[0017] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the behavior recognition method as in the above-mentioned embodiment.

[0018] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0019] In the technical solutions provided by some embodiments of the present disclosure, multiple skeleton data features of the target object within a preset time are first processed by a branch displacement convolution subnetwork, and then multiple feature maps are fused by a fusion displacement convolution subnetwork, and finally the behavior information is identified according to the feature vector by a fully connected layer. On the one hand, the multi-stream structure using multiple skeleton data features for behavior recognition can provide richer and more discriminative features and improve the accuracy of the model; on the other hand, the branch displacement convolution subnetwork and the fusion displacement convolution subnetwork are designed using the idea of ​​early fusion, multiple input branches are designed, and a mainstream is applied after multiple feature branches are connected in series, so that the fusion displacement convolution subnetwork can participate in the skeleton data fusion prematurely, which not only retains the rich input characteristics, but also significantly suppresses the complexity and redundancy of the model, making the training process easier to converge.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A flowchart schematically illustrates a behavior recognition method in an exemplary embodiment of the present disclosure;

[0023] FIG. 2( a ) schematically shows a schematic diagram of skeleton key points with N=18 in an exemplary embodiment of the present disclosure;

[0024] FIG2( b ) schematically shows a schematic diagram of skeleton key points with N=25 in an exemplary embodiment of the present disclosure;

[0025] FIG3( a ) schematically shows a schematic diagram of a late fusion architecture in an exemplary embodiment of the present disclosure;

[0026] FIG3( b ) schematically shows a schematic diagram of an early fusion architecture in an exemplary embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram schematically illustrates an early fusion multi-input branch architecture in an exemplary embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram schematically illustrates a displacement graph convolution block structure in an exemplary embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram schematically illustrates a flow chart of a model training method in an exemplary embodiment of the present disclosure;

[0030] FIG. 7( a ) schematically shows a general non-local displacement map in an exemplary embodiment of the present disclosure;

[0031] FIG7( b ) schematically shows a schematic diagram of a common non-local displacement process in an exemplary embodiment of the present disclosure;

[0032] FIG7( c ) schematically shows a common non-local displacement feature map in an exemplary embodiment of the present disclosure;

[0033] FIG8( a ) schematically shows a non-local displacement graph with edge deletion in an exemplary embodiment of the present disclosure;

[0034] FIG8( b ) schematically shows a schematic diagram of a non-local displacement process with edge deletion in an exemplary embodiment of the present disclosure;

[0035] FIG8( c ) schematically shows a non-local displacement feature map with edge deletion in an exemplary embodiment of the present disclosure;

[0036] FIG9( a) schematically shows a schematic diagram of skeleton key points at 0 s in an exemplary embodiment of the present disclosure;

[0037] FIG9( b) schematically shows a schematic diagram of skeleton key points at 5 seconds in an exemplary embodiment of the present disclosure;

[0038] FIG9( c ) schematically shows a schematic diagram of skeleton key points at 10 seconds in an exemplary embodiment of the present disclosure;

[0039] Fig.10 A schematic diagram schematically shows the composition of a behavior recognition device in an exemplary embodiment of the present disclosure;

[0040] Fig.11 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0041] Fig.12 The structure diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0043] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0045] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0046] In the behavior recognition with people and other target objects as the main body and center, the skeleton point + neural network method has stronger robustness. Among all the methods based on skeleton point + neural network, the skeleton point + spatiotemporal graph convolutional network method can better utilize the non-Euclidean topological structure characteristics of the skeleton and learn distinctive and rich features from the skeleton sequence, such as the SOTA (state-of-the-art) skeleton point + spatiotemporal graph convolutional network model, compared with the skeleton point + convolutional neural network and skeleton point + recurrent neural network methods.

[0047] For the SOTA skeleton point + spatiotemporal graph convolutional network model, on the one hand, the network structure is usually very complex and over-parameterized; on the other hand, the network usually contains a multi-stream structure with a large number of model parameters, which leads to a complex training process and high computational cost.

[0048] Therefore, in order to reduce the redundant and huge amount of calculation caused by GCN convolution operations and multi-stream late fusion, the technical solution of the present application is proposed, and an early fusion multi-input branch shift graph convolutional network structure (MultipleInput Branches Shift Graph Convolutional Network with DropEdge, MIBSD-GCN) is proposed.

[0049] The implementation details of the technical solution of the embodiment of the present disclosure are elaborated in detail below.

[0050] Figure 1 The following is a schematic diagram showing a flow chart of a behavior recognition method in an exemplary embodiment of the present disclosure. Figure 1 As shown, the behavior recognition method includes steps S1 to S4:

[0051] Step S1, obtaining multiple bone data features of the target object within a preset time;

[0052] Step S2, using a branch shift convolutional subnetwork to perform branch processing on the multiple skeleton data features to obtain multiple feature maps;

[0053] Step S3, performing mainstream fusion of the multiple feature maps by fusing the shifted convolutional sub-network to obtain a feature vector;

[0054] Step S4: using a fully connected layer to identify the feature vector to obtain the behavior information of the target object.

[0055] In the technical solutions provided by some embodiments of the present disclosure, multiple skeleton data features of the target object within a preset time are first processed by a branch displacement convolution subnetwork, and then multiple feature maps are fused by a fusion displacement convolution subnetwork, and finally the behavior information is identified according to the feature vector by a fully connected layer. On the one hand, the multi-stream structure using multiple skeleton data features for behavior recognition can provide richer and more discriminative features and improve the accuracy of the model; on the other hand, the branch displacement convolution subnetwork and the fusion displacement convolution subnetwork are designed using the idea of ​​early fusion, multiple input branches are designed, and a mainstream is applied after multiple feature branches are connected in series, so that the fusion displacement convolution subnetwork can participate in the skeleton data fusion prematurely, which not only retains the rich input characteristics, but also significantly suppresses the complexity and redundancy of the model, making the training process easier to converge.

[0056] The following will describe in more detail the various steps of the behavior recognition method in this example implementation with reference to the accompanying drawings and embodiments.

[0057] In step S1, multiple bone data features of a target object within a preset time are obtained.

[0058] In one embodiment of the present disclosure, the present application is based on skeleton points for behavior recognition, and the starting point is the skeleton, without involving extraction of skeleton points and posture estimation. Therefore, it is necessary to obtain the skeleton data features of the target object, and a dynamic sequence data with a time length is used as the analysis basis. Among them, the preset time can be specifically set according to the needs, such as 5 seconds, 3 seconds, etc.

[0059] In order to perform bone fusion analysis, multiple bone data features are also required, wherein the multiple bone data features include at least two of the bone key point features of the first coordinate system, the bone key point features of the second coordinate system, the speed features and the skeleton features.

[0060] It should be noted that at different moments within the preset time, the skeletal key points of the target object have corresponding positions. Therefore, the skeletal key point features of the first coordinate system, the skeletal key point features of the second coordinate system, the speed features and the skeleton features are all sequence data features at different moments.

[0061] The skeleton key point features of the first coordinate system may be skeleton key point features of the Cartesian coordinate system (Cartesian Coordinates), including the absolute coordinates and relative coordinate information of each skeleton point in the Cartesian coordinate system at different times.

[0062] The skeleton key point features of the second coordinate system may be skeleton key point features of the spherical coordinate system (SphericalCoordinates), including the absolute coordinates and relative coordinate information of each skeleton point in the spherical coordinate system at different times.

[0063] Velocity refers to the first-order difference and second-order difference information of the changes in bone movement in adjacent moments.

[0064] Skeleton features (Bone) refer to the length and angle information of each bone.

[0065] Further, in one embodiment of the present disclosure, in step S1, a skeleton key point sequence may be first obtained, and then the skeleton key point sequence may be converted into a plurality of skeleton data features. The step of obtaining a plurality of skeleton data features of the target object includes:

[0066] Step S11, obtaining image data collected by the camera within a preset time according to a preset frame rate set;

[0067] Step S12, detecting the skeleton key points of the target object in each frame of the image data to obtain the skeleton key point sequence;

[0068] Step S13, performing feature processing on the skeleton key point sequence to obtain the multiple skeleton data features.

[0069] Step S11 to step S12 is the process of extracting the skeleton key point sequence using the camera. When collecting using the camera, image data within a preset time can be collected, or all image data can be obtained and image data for a period of time can be intercepted therefrom.

[0070] The acquired skeleton key point sequence may be 2D coordinate data of the skeleton points, or 3D coordinate data, depending on the environment space where the target object is located.

[0071] In the extraction process, there are subtle differences depending on the function of the camera. For example, you can use an ordinary optical camera to collect image data, and then use classic human posture estimation algorithms to detect skeleton key points, such as OpenPose, deep learning network, etc. You can also use a depth camera with a special sensor, such as the commonly used Microsoft Kinect camera, which can directly obtain image data with depth information and then detect skeleton key points.

[0072] Specifically, taking the detection of human key points as an example, Figures 2(a) and 2(b) respectively schematically illustrate schematic diagrams of skeletal key points in an exemplary embodiment of the present disclosure. Wherein, Figure 2(a) shows a schematic diagram of skeletal key points of N=18, that is, a human body is composed of 18 skeletal key points, and Figure 2(b) shows a schematic diagram of skeletal key points of N=25, that is, a human body is composed of 25 skeletal key points, and the detected skeletal key point sequence includes relevant information of these skeletal key points.

[0073] In actual operation, a depth camera can be used to obtain real-time image data at a specified frame rate FPS. T frames are divided into segments, and no more than p skeleton sequences of people are extracted in each segment. Each human body contains N skeleton key points. ti represents the i-th skeleton key point in the t-th frame in the segment, then a human skeleton sequence in the video segment can be represented as V = {v ti |t=1,…,T,i=1,…,N}. According to different definitions of skeleton key points, N=18 or N=25 are commonly used, as shown in FIG2(a) and FIG2(b), and one method can be selected to implement it.

[0074] In step S13, in order to supplement and enhance different feature expressions, the skeleton key point sequence can be feature processed to obtain multiple skeleton data features. For example, the acquired skeleton key point sequence can be converted into 4 branches and 8 data formats, including skeleton joint point features (absolute coordinates and relative coordinates) of Cartesian coordinate system, skeleton joint point features (absolute coordinates and relative coordinates) of spherical coordinate system, speed features (first-order difference and second-order difference information of motion changes in adjacent moments) and skeleton features (bone length and angle).

[0075] Among them, converting the skeleton key point sequence into the skeleton key point features, speed features, and skeleton features of the Cartesian coordinate system is relatively common in the field of skeleton point-based behavior recognition. The conversion process can be completed using existing technology and will not be elaborated here.

[0076] It should be noted that the data representation of the spherical coordinate branch is a feature of the present disclosure that distinguishes it from other solutions. The specific conversion method can be performed according to the skeleton key point features of the Cartesian coordinate system, which is specifically introduced as follows.

[0077] The spherical coordinate skeleton point features can more clearly display the changing process of specific skeleton movements from another coordinate system, such as waving, turning around, falling, etc. in human skeleton recognition, thereby better improving the accuracy of action recognition.

[0078] Assume that the key points of human skeleton v ti The three-dimensional coordinates are (x, y, z). The conversion relationship between the Cartesian coordinate system and the corresponding spherical coordinate system is shown in formulas (1)-(3):

[0079]

[0080] Among them, r is from the origin o to the key point v ti The distance between the origin o and the key point v ti The angle between the line connecting and the z-axis is expressed as the polar angle. It is the connection between the origin o and the key point v ti The angle between the projection line of the line connecting the two on the xy plane and the x-axis is expressed as the azimuth.

[0081] According to the above method, it is possible to convert between the Cartesian coordinate system and the spherical coordinate system. In the calculation, it should be ensured that θ is within the range (-π / 2, π / 2), and In the range (-π, π).

[0082] In one embodiment of the present disclosure, an ablation experiment may be used to illustrate the technical effect of introducing spherical coordinate skeleton point features.

[0083] The ablation experiment uses the NTU RGB+D 60X-View dataset to design a comparative experiment. The dataset contains 60 types of actions, with a total of 56,880 samples, of which 40 are daily behavior actions, 9 are health-related actions, and 11 are two-person interactions. These actions are performed by 40 people aged 10 to 35. The dataset is collected by the Microsoft Kinect v2 sensor and uses three cameras at different angles. The collected data includes depth information, 3D skeleton information, RGB frames, and infrared sequences.

[0084] In the experimental design, the network composed of different unit blocks is used to consider the model accuracy of "3-stream" and "4-stream". Among them, "3-stream" uses three skeletal data features of Cartesian coordinate system skeletal key point features, speed features and skeleton features for behavior recognition; "4-stream" uses four skeletal data features of Cartesian coordinate system skeletal key point features, spherical coordinate system skeletal key point features, speed features and skeleton features for behavior recognition.

[0085] ResGCN is the basic displacement graph convolution network model constructed using the basic unit block ResGCN, which is an implementation method of the displacement graph convolution network model in the prior art; Shift-GCN is the displacement graph convolution network model proposed in the present disclosure that integrates the early fusion architecture (MIB) and the Shift-GCN block, which is a new network model proposed in the present disclosure. Here we only compare the technical effects of introducing the spherical coordinate skeleton point features, so the model content will be introduced in detail later.

[0086] Table 1 Accuracy comparison of different multi-input branches using ResGCN

[0087]

[0088]

[0089] Table 1 shows the comparison of model accuracy before and after the introduction of spherical coordinate skeleton point features when using ResGCN as the unit block of the network for behavior recognition. As shown in Table 1, the calculation accuracy of the model is 96.0% when the 3-stream input branch is used, and the model accuracy is 96.2% when the 4-stream input branch is used, that is, after the introduction of spherical coordinate skeleton point features, the accuracy is improved by 0.2% compared to 96.0%.

[0090] Table 2 Accuracy comparison of different multi-input branches using Shift-GCN

[0091]

[0092] Table 2 shows the comparison of model accuracy before and after the introduction of spherical coordinate skeleton point features when using Shift-GCN as the unit block of the network for behavior recognition. As shown in Table 2, the calculation accuracy of the model is 95.8% when the 3-stream input branch is used, and the accuracy is 96.0% when the 4-stream input branch is used. After the introduction of spherical coordinate skeleton point features, the accuracy is also improved.

[0093] Therefore, according to the results in Table 1 and Table 2, after adding the spherical coordinate features, the accuracy of the neural network model built with ResGCN or Shift-GCN as the unit block has been improved by 0.2%, which verifies the effectiveness of improving the model accuracy after adding the spherical coordinate features.

[0094] Based on the above method, a multi-stream structure is used as the basis for action recognition, which provides richer and more discriminative features, thereby improving the accuracy of the model. In particular, the spherical coordinate skeleton point feature is introduced as one of the input branches to convert the Cartesian coordinates (x, y, z) of the skeleton into spherical coordinates. The polar coordinate representation of the skeleton is obtained, so that the information and features of spatial position and motion changes are enhanced and supplemented, which can effectively improve the accuracy of action recognition.

[0095] Although the fractional layer fusion of the multi-stream structure brings excellent model accuracy, the model size and computational cost are greatly increased, and many parameters in the multi-stream model are redundant. Therefore, the problem of reducing the amount of computation needs to be further addressed in the subsequent skeletal fusion analysis.

[0096] At present, most multi-stream structures use late fusion for action recognition, but generally speaking, the late fusion has a large amount of computation and requires repeated training. Therefore, the present invention designs an early fusion multi-input branch architecture model to achieve this. In the early stage of the model, multiple input branches are fused, and then a main stream is used to extract the overall recognition features.

[0097] Figures 3(a) and 3(b) schematically illustrate schematic diagrams of two fusion architectures in an exemplary embodiment of the present disclosure, wherein Figure 3(a) illustrates a schematic diagram of a late fusion architecture, and Figure 3(b) illustrates a schematic diagram of an early fusion architecture. Among them, Data 1 and Data 2 are multi-stream data input into the neural network, and the cell Block is a component unit calculation block of the network, and finally the data is output through the classifier. Of course, Figure 3 is only an example structure. In actual applications, the data can have a structure with more streams, and the composition of the cells in the network model is also different.

[0098] In Figure 3(a), data 1 and data 2 are calculated in three cells respectively and finally fused into the classifier; in Figure 3(b), data 1 and data 2 are fused after being calculated in one cell, and subsequent cells 2 and 3 only need to calculate the fused data, omitting the calculation process of cells 2 and 3.

[0099] As shown in Figure 3(b), compared with the late fusion architecture, the early fusion architecture can fuse multiple input branches in the early stage of the model and apply a mainstream after connecting multiple feature branches in series. This not only retains rich input characteristics, but also significantly suppresses the complexity and redundancy of the model, making the training process easier to converge and reducing the amount of computation.

[0100] Steps S2 to S4 are to use a pre-trained early fusion multi-input branch architecture model to perform action recognition using multiple skeleton data features. Figure 4 A schematic diagram schematically illustrates a multi-input branch architecture of early fusion in an exemplary embodiment of the present disclosure, with reference to Figure 4 As shown in the figure, the early fusion multi-input branch architecture model includes three parts: branch shift convolution sub-network, fusion shift convolution sub-network and fully connected layer.

[0101] In step S2, a branch displacement convolutional subnetwork is used to perform branch processing on the multiple skeleton data features to obtain multiple feature maps.

[0102] In one embodiment of the present disclosure, the branching and shifting convolutional subnetwork is used to perform branching processing on the multiple skeleton data features to obtain multiple feature maps, including: inputting each skeleton data feature in parallel into the pre-trained branching and shifting convolutional subnetwork corresponding to the skeleton data feature to obtain feature maps corresponding to each skeleton data feature.

[0103] Specifically, there are multiple branch input ports in the branch displacement convolution subnetwork, and different bone data features have corresponding input ports. The branch displacement convolution subnetwork can be constructed by serially connecting a batch normalization layer, a displacement map convolution initial block, and at least one displacement map convolution block.

[0104] refer to Figure 4 The branch shift convolution subnetwork shown includes four branch inputs, namely, Cartesian Coordinates, Spherical Coordinates, Velocity, and Bone, and is composed of an ordered stacking of batch normalization layers, an initial block implemented by shift graph convolution, and two shift graph convolution blocks. Therefore, in step S2, the different skeletal data features obtained in step S1 are input to the corresponding entrances in parallel, and the feature maps are fused using a series operation. The feature maps corresponding to each skeletal data feature can be obtained through the pre-trained branch shift convolution subnetwork.

[0105] It should be noted that Figure 4 The network structure shown is only an exemplary description. In actual applications, different bone data features can be designed for fusion analysis according to the needs of fusion. The content and the number of branches can be changed, but must not be less than two.

[0106] In the branch shift convolution subnetwork, the batch normalization layer (Batchnorm) is used to normalize each batch of input data.

[0107] The core idea of ​​the Shift-GCN block is to combine the shift operation of information exchange between nodes on the feature map with 1*1 point convolution to achieve an effect similar to that of GCN convolution while reducing the amount of computation of the GCN block. In terms of spatial modeling, a fully connected spatial shift operation and point-by-point convolution operation are performed on the nodes of each frame t in the skeleton sequence; in terms of temporal feature extraction, a temporal shift operation with adaptive parameters and point-by-point convolution are used to learn contextual features.

[0108] The difference between the initial shift-GCN block and the shifted graph convolution block is that the residual connection structure is not applicable.

[0109] In one embodiment of the present disclosure, the displacement map convolution block includes a spatial displacement operation module, a spatial point-by-point convolution module, a temporal displacement operation module and a temporal point-by-point convolution module.

[0110] Figure 5 A schematic diagram schematically shows a displacement graph convolution block structure in an exemplary embodiment of the present disclosure, referring to Figure 5 As shown in the figure, a shift graph convolution block (Shift-GCN) is composed of a spatial shift operation module (S-Shift), a spatial pointwise convolution module (S-Pointwise), a spatial shift operation module (S-Shift), a temporal shift operation module (T-Shift), a temporal pointwise convolution module (T-Pointwise), and a temporal shift operation module (T-Shift) in series. Among them, S-Shift, S-Pointwise, and S-Shift perform shift calculations in space, and T-Shift, T-Pointwise, and T-Shift perform shift calculations in time.

[0111] For example, if the input and output channels are both 256, the channel reduction rate r is 4, and the time window size L is 8, then the basic block contains a total of 256×256×8=524288 parameters, while Shift-GCN only contains 256×64+64×64×8+64×256=65536 parameters, which is almost 8 times the basic block compared to 65536.

[0112] Therefore, the combination of temporal and spatial displacement operations and lightweight point-by-point volumes enables the use of displacement graph convolution blocks with non-local spatial displacement operations and adaptive temporal displacement operations as the unit blocks of the network structure, which can greatly reduce the amount of computation.

[0113] In step S3, the plurality of feature maps are mainstream fused by fusion shift convolution sub-network to obtain a feature vector.

[0114] In one embodiment of the present disclosure, the mainstream fusion of the multiple feature maps through the fusion shift convolution sub-network to obtain the feature vector includes: uniformly inputting the multiple feature maps into the pre-trained fusion shift convolution sub-network to obtain the feature vector.

[0115] Specifically, the fused shift convolution subnetwork is a pre-trained neural network with only one input, so multiple feature maps need to be uniformly input into the fused shift convolution subnetwork to output a feature vector. The fused shift convolution subnetwork includes at least one shift map convolution block, which has the same structure as the shift map convolution block in the branch shift convolution subnetwork, so we will not go into details here.

[0116] refer to Figure 4 The fused shift convolution subnetwork shown includes a total of six shift graph convolution blocks (Shift-GCN). Therefore, in step S3, the feature maps corresponding to the four bone data features, namely, the skeleton key point features (Cartesian Coordinates), the skeleton key point features (Spherical Coordinates) of the spherical coordinate system, the velocity features (Velocity) and the skeleton features (Bone), are respectively input into the mainstream constructed by the six shift graph convolution blocks (Shift-GCN). Finally, the output feature maps of the mainstream are globally averaged into feature vectors representing the input multiple bone data features.

[0117] At the same time, the Part-wise Attention (PartAtt) mechanism is introduced to emphasize the importance of different bones in the entire action sequence in the mainstream. The attention mechanism is a data processing method in machine learning, which is widely used in various types of machine learning tasks such as natural language processing, image recognition and speech recognition. I will not go into details here.

[0118] In step S4, the behavior information of the target object is obtained by identifying the feature vector using a fully connected layer.

[0119] In one embodiment of the present disclosure, the fully connected layer (FC) plays the role of a "classifier" in the entire convolutional neural network, and can map the feature vector to the sample label space to determine the final action class, that is, the behavior information.

[0120] Next, an ablation experiment is also used to illustrate the technical effect of the early fusion multi-input branch architecture model provided by the present disclosure. The ablation experiment uniformly uses NTU RDB+D 60X-view to design a comparative experiment.

[0121] Among them, "3-stream" uses three features, namely, the skeleton key point features of the Cartesian coordinate system, the velocity features, and the skeleton features, for behavior recognition; "4-stream" uses four features, namely, the skeleton key point features of the Cartesian coordinate system, the skeleton key point features of the spherical coordinate system, the velocity features, and the skeleton features, for behavior recognition.

[0122] ResGCN is the basic shifted graph convolution network model built using the basic unit block ResGCN; Shift-GCN is the shifted graph convolution network model proposed in this disclosure that integrates the early fusion architecture (MIB) and the Shift-GCN block.

[0123] Table 3 Comparison of computational complexity and accuracy of different models under three input features

[0124]

[0125] Table 3 shows the computational complexity and accuracy comparison of the ResGCN model and the Shift-GCN model using three input features. As shown in Table 3, although the accuracy of the Shift-GCN model dropped from 96.0% to 95.8%, the computational complexity dropped sharply from 18.52G to 2.89G.

[0126] Among them, FLOPs refers to Floating-point Operations Per Second, which is 1 billion floating-point operations per second and is often used as a GPU performance parameter.

[0127] Table 4 Comparison of computational complexity and accuracy of different models under four input features

[0128]

[0129] Table 4 shows the computational complexity and accuracy comparison of the ResGCN model and the Shift-GCN model using four input features. As shown in Table 4, the accuracy of the Shift-GCN model dropped from 96.2% to 96.0%, but the computational complexity dropped sharply from 20.32G to 2.96G.

[0130] Therefore, based on the calculation results shown in Tables 3 and 4, after integrating the early fusion architecture (MIB) and the Shift-GCN block, although the accuracy drops by 0.2 points, the amount of calculation is reduced by more than 6 times.

[0131] Based on the above method, in terms of overall architecture, in order to comprehensively consider accuracy and complexity, the present disclosure constructs an early fusion multi-input branch architecture model (MIBSD-GCN), which fuses multiple input branches in the early stage of the model and applies a main stream after multiple feature branches are connected in series. This architecture not only retains rich input features, but also significantly suppresses the complexity and redundancy of the model, making it easier for the training process to converge.

[0132] In terms of unit module selection, in order to balance accuracy and complexity and optimize the entire model, a displacement graph convolution block with non-local spatial displacement operations and adaptive temporal displacement operations is selected as the unit block of the network structure to replace the heavy regularized graph convolution. Due to the clever combination of displacement operations and lightweight point-by-point convolutions, the computational complexity of the displacement graph convolution is reduced by more than several times compared to conventional graph convolution networks.

[0133] Therefore, improvements are made to both the overall architecture and the unit modules, and the high-performance early fusion architecture (MIB) and Shift-GCN unit blocks are integrated to take advantage of their respective advantages. That is, a lightweight shift graph convolutional network is embedded in the multi-branch structure of early fusion. This greatly reduces redundant calculations from both the network structure and unit module perspectives, and strikes a good balance between accuracy and complexity, resulting in a model that excels in both accuracy and computational complexity.

[0134] In one embodiment of the present disclosure, the method further includes performing model training on the early fused multi-input branch architecture model to obtain a branch shift convolution subnetwork, a fused shift convolution subnetwork, and a fully connected layer.

[0135] Figure 6 A schematic diagram of a model training method in an exemplary embodiment of the present disclosure is schematically shown, with reference to Figure 6 As shown, the model training method includes:

[0136] Step S61, obtaining a skeleton data training set and a behavior label of the skeleton data training set;

[0137] Step S62, constructing an original input graph according to the skeleton data training set, and performing edge deletion on the original input graph to obtain a target input graph;

[0138] Step S63, based on the target input graph, using the branch shift convolutional subnetwork, the fused shift convolutional subnetwork and the fully connected layer to perform recognition to obtain recognition behavior information;

[0139] Step S64, comparing the identified behavior information with the behavior label to modify the model parameters of the branch shift convolution subnetwork, the fused shift convolution subnetwork and the fully connected layer.

[0140] Specifically, its training process is roughly the same as that of ordinary neural network training. Both use real behavior labels to train the data training set, and train the various parameters in the model by comparing the recognized behavior information with the behavior labels, and finally obtain a trained model.

[0141] Different from the prior art, in step S62, the input graph is not directly used for behavior recognition, but an edge deletion (DropEdge) mechanism is introduced in the training process.

[0142] In non-local displacement graph convolution, all nodes in the input graph are interconnected, and the connection strength between different nodes is the same. As the number of connections between nodes increases, the connection relationship becomes more complex, which causes the overfitting and oversmoothing of the calculation results to become more serious. Therefore, through the edge deletion mechanism, different randomly deformed copies of the original graph are actually generated, which increases the randomness and diversity of the input data, thereby better preventing overfitting. In addition, edge deletion is regarded as a message passing reducer in graph convolution. Deleting certain edges makes the node connections more sparse, thereby avoiding oversmoothing to some extent,

[0143] Therefore, the present disclosure proposes an edge deletion operation in a displacement graph convolution.

[0144] When operating a non-local displacement graph, given a spatial skeleton feature graph, the moving distance of the i-th channel is i modulo the number of skeleton key points N, and the moving-out channel is used to fill the corresponding moving-in channel. Figures 7(a), 7(b) and 7(c) respectively schematically illustrate schematic diagrams of ordinary non-local displacement graph operations in exemplary embodiments of the present disclosure. Figure 7(a) shows a displacement graph of node 1 after non-local displacement, Figure 7(b) shows a schematic diagram of the non-local displacement process of node 1, and Figure 7(c) shows a displacement feature graph after non-local displacement.

[0145] Considering that the non-local displacement graph operation is a directed operation, the entire displacement graph is considered as a directed graph and all edges are considered as directed edges. When planning to delete an edge from one node to another, we only need to stop the displacement operation from the outgoing node to the incoming node and keep the incoming channel at its original value on the feature map instead of being filled with the value of the outgoing channel.

[0146] Figures 8(a), 8(b) and 8(c) respectively schematically illustrate the schematic diagrams of the non-local displacement graph operation with edge deletion in the exemplary embodiment of the present disclosure. Referring to Figure 8(a), the edges from node 3 to node 1 and node 6 to node 1 are deleted, so in the shift-in channels corresponding to node 3 to node 1 and node 6 to node 1, the feature graph maintains the original value and no displacement operation is required. Figure 8(a) shows the displacement graph of the non-local displacement with edge deletion, and Figure 8(b) shows the schematic diagram of the non-local displacement process with edge deletion, in which the deleted edges are marked, that is, the edges corresponding to the "×" symbol, and Figure 8(c) shows the displacement feature graph after the non-local displacement with edge deletion.

[0147] Just like the common non-local shifted graph convolution, the shifted graph operation is connected in series with the point convolution operation to form a shifted graph convolution module with edge deletion mechanism. It is worth noting that the edge deletion mechanism is only effective during training, and the edge deletion strategy is not used during validation and testing.

[0148] In one embodiment of the present disclosure, when edge deletion is performed in step S62, it is also necessary to determine the edge to be deleted. There are two ways to do this: one is a random edge deletion strategy, and the other is an adaptive attention-guided edge deletion strategy.

[0149] Specifically, for the first random edge deletion strategy, the step of deleting edges from the original input graph to obtain a target input graph includes:

[0150] Step S6211, constructing an original adjacency matrix according to the displacement relationship in the original input graph;

[0151] Step S6212, calculating a target adjacency matrix corresponding to the target input graph based on the original adjacency matrix and a preset random edge retention rate.

[0152] Further, in step S6211, consistent with any ordinary graph convolution, the adjacency matrix A is first used shit To represent the displacement relationship in the directed graph. shift is a matrix whose elements A shift ={a ij |i=1,...,N,j=1,...N} represents the displacement relationship between corresponding nodes. i Indicates the node to be moved in and v j Indicates the removal of the node. If there is a i to v j The edges of j →v i , therefore, the elements in the matrix can be expressed as follows:

[0153]

[0154] A in the nonlocal displacement map shift All elements of are 1.

[0155] In step S6212, when the random edge deletion strategy is in effect, the non-local displacement graph randomly discards a certain proportion of edges at each stage of training the model. Formally, the non-zero elements of N×N×(1-p) in the adjacency matrix are randomly forced to zero, where N×N is the total number of edges, p is the random edge retention rate, that is, the probability that an edge is retained, which is a preset value, and 1-p is the random edge deletion rate.

[0156] From this we can calculate A drop , A drop is a sparse matrix expanded by a random-sized subset of the original edges, that is, the deleted edges. shift-drop Represents the generated adjacency matrix, then its calculation relationship can be expressed as follows:

[0157] A shift-drop =A shift -A drop (5)

[0158] A shift-drop Controls whether to perform a shift operation between corresponding nodes of the non-local shift graph convolution. If A shift-drop If the element in is 1, execute v j →v i Normal displacement operation, otherwise, no displacement operation is performed, and the corresponding v on the feature map j →v i The displacement channels retain their original values.

[0159] Specifically, compared to using the same A in the same training batch shift-drop , we prefer to obtain it by independent calculation layer by layer, denoted as A l shift-drop This layer-by-layer edge deletion mode brings more randomness and deformation to the original data and brings higher performance than the batch edge deletion mode.

[0160] For the second adaptive attention-guided edge deletion strategy, the step of performing edge deletion on the original input graph to obtain a target input graph includes:

[0161] Step S6221, constructing an attention template adjacency matrix according to the displacement relationship in the original input graph;

[0162] Step S6222, calculating the edge retention probability of each edge of the original input graph based on the attention template adjacency matrix and the preset adaptive edge deletion parameter;

[0163] Step S6223, obtaining a target adjacency matrix corresponding to the target input graph according to the edge retention rate.

[0164] Specifically, in non-local displacement graph convolution, the connection strength between different nodes is the same, and the probability of all edges being discarded is also the same. However, in actual action recognition, the importance of each edge is different. Therefore, when deleting edges, it is hoped that edges that contribute more to action recognition will be retained with a higher probability, while edges that contribute less will be discarded with a higher probability.

[0165] Based on this idea, an adaptive attention-guided edge deletion mechanism is proposed to make important edges have a higher sampling probability. The probability calculation method is as follows:

[0166] P l =d·tanh(k·M l A )+(1-d) (6)

[0167] Among them, P l is the edge preserving matrix, M l A is the attention template adjacency matrix, d is the preset adaptive edge deletion parameter (Attention-guided drop parameter), which is used to control the probability interval of edge deletion, and k is the scale parameter.

[0168] P l and M l A It's with A shift Matrices of the same size, P l The value in represents the probability that the edge is retained, M l A is the attention template adjacency matrix, which is a learnable attention template. In order to obtain the distribution of the attention area, it is usually implicitly assumed that the absolute value of the activation represents the importance of a unit, so we follow this assumption and generate M by averaging the absolute values ​​of the channels corresponding to the edge. l A .

[0169] Based on the constraints of the above formula (6), the edge is deleted according to the calculated edge retention probability, and the probability of retaining the edge is controlled within the interval [1-2*d,1], and the probability distribution is P l The Bernoulli distribution samples are used to obtain the result after the final edge deletion.

[0170] Next, an ablation experiment is also used to illustrate the technical effect of introducing the edge deletion mechanism in the model of the present disclosure. The ablation experiment still uses NTU RDB+D 60X-view to design a comparative experiment.

[0171] Among them, the batch-wise deletion strategy is to randomly discard some edges according to probability in each training batch, and the layer-wise deletion strategy is to randomly discard some edges according to probability in each layer.

[0172] It should be noted that under the random edge deletion strategy, both methods are possible, while under the adaptive attention-guided edge deletion strategy, only deletion can be performed on a per-layer basis. “-” means that the edge deletion strategy is not used.

[0173] Table 5 Comparison of the accuracy of random edge deletion strategies with different random edge deletion rates

[0174]

[0175] As shown in Table 5, with the random edge deletion rate of 0 as the benchmark, when the random edge deletion rate increases, the model accuracy of both Batch-wise and Layer-wise strategies increases. And for the same random edge deletion rate, the accuracy of Layer-wise is better than that of Batch-wise.

[0176] Table 6 Comparison of the accuracy of random edge deletion strategies with different random edge deletion rates

[0177]

[0178] In Table 6, taking the adaptive edge deletion parameter as 0 as the baseline, when the adaptive edge deletion parameter increases, the model accuracy also increases.

[0179] Therefore, referring to Tables 5 and 6, it can be clearly seen that in the random DropEdge strategy, Layer-wise is better than Batch-wise, and Attention-guided DropEdge is better than the random DropEdge strategy, and they are all better than the model that does not use the DropEdge strategy.

[0180] Based on the above method, considering that in this scheme, non-local Shift-GCN unit blocks are embedded in the early fusion architecture MIB, the connection relationship becomes complicated, which leads to a higher risk of overfitting and over-smoothing. Therefore, the edge deletion mechanism is introduced in the process of model training, which enhances the data to a certain extent, makes GCN more sparse, and information transmission more concentrated, thereby reducing the risk of overfitting and over-smoothing.

[0181] In particular, the adaptive attention-guided edge removal strategy can also utilize the learnable attention template to retain the edges that contribute more to action recognition with a higher probability, and discard the edges that contribute less with a higher probability, ensuring the accuracy of the model.

[0182] Next, the model recognition effects of the technical solution provided by the present disclosure are compared with those of other technical solutions known in the prior art, and the results are entered into Table 7 to illustrate the advantages of the solution.

[0183] Table 7 Accuracy comparison of different network models

[0184]

[0185] As shown in Table 7, compared with the methods in the prior art, the MIBSD-GCN provided in this application has a very obvious computational advantage (2.96G as shown in Table 7) while maintaining a higher accuracy (96.6% as shown in Table 7).

[0186] It should be noted that the experimental results in this application are only exemplary and cannot limit the present disclosure.

[0187] In one embodiment of the present disclosure, behavior recognition is performed using image data collected from an office building as an example. A depth camera is used to collect data, and data sampling is performed at a specified frame rate of FPS=30, with 150 frames (i.e., 5 seconds) as one segment. The coordinates of the 3D skeleton points and joint points of each frame are extracted in each video segment.

[0188] Figures 9(a), 9(b) and 9(c) respectively show schematic diagrams of skeletal key points at three consecutive moments, Figure 9(a) is a schematic diagram of skeletal key points at 0s, Figure 9(b) is a schematic diagram of skeletal key points at 5s, and Figure 9(c) is a schematic diagram of skeletal key points at 10s.

[0189] After obtaining the skeletal key point sequence, it is converted into four skeletal data features: skeletal key point features of the Cartesian coordinate system, skeletal key point features of the spherical coordinate system, velocity features and skeleton features. The early fusion multi-input branch architecture model is used for recognition and analysis to obtain the final action information, whose action category is "falling".

[0190] The behavior recognition method provided by this disclosure mainly solves the problem that the existing technical solutions are highly complex and have insufficient accuracy, enhances the practicality of the model, and improves the system performance of automated behavior recognition and the proportion of automated processing tasks. The main contributions are summarized as follows:

[0191] (1) We designed an early fusion multi-input branch architecture model that takes input from four independent spatiotemporal feature sequences, including skeleton joints in Cartesian coordinates, skeleton joints in spherical coordinates, velocity, and skeleton features. In particular, we added novel skeleton joints in spherical coordinates to the input branch and demonstrated the effectiveness of this method.

[0192] (2) The displacement map convolution block consisting of displacement map operation and lightweight point-by-point convolution is embedded into the early fusion multi-input branch network to reduce the computational complexity of the unit graph convolution module.

[0193] (3) We propose edge deletion rules in the displacement graph convolution and an adaptive attention-guided edge deletion strategy to prevent overfitting and oversmoothing.

[0194] This method can be attached to the intelligent video analysis platform service system and applied to many business scenarios, including but not limited to elderly care community monitoring, intelligent building monitoring, intelligent visual interaction and other scenarios that require high-precision automated behavior recognition and processing. For example, in the elderly care / big health monitoring system, it can timely and accurately predict risk behaviors in the elderly care community, such as: falls, elderly abuse, etc., so that the elderly can get medical care in time and reduce the probability of danger and accidents for the elderly; in the intelligent human-computer interaction system, it can identify human interaction behaviors and execute specific instructions contactlessly, making human-computer interaction more intelligent; in the insurance double-recording quality inspection system, it can identify specific quality inspection regulations, realize automatic review, improve business efficiency, and reduce labor costs.

[0195] Fig.10 The following is a schematic diagram showing the composition of a behavior recognition device in an exemplary embodiment of the present disclosure. Fig.10 As shown, the behavior recognition device 1000 may include an acquisition module 1001, a branch module 1002, a fusion module 1003 and a recognition module 1004. Among them:

[0196] An acquisition module 1001 is used to acquire multiple bone data features of a target object within a preset time;

[0197] A branch module 1002 is used to perform branch processing on the multiple skeleton data features using a branch displacement convolution subnetwork to obtain multiple feature maps;

[0198] A fusion module 1003 is used to perform mainstream fusion of the multiple feature maps by fusing the shifted convolution sub-network to obtain a feature vector;

[0199] The recognition module 1004 is used to obtain the behavior information of the target object by using the fully connected layer to perform recognition according to the feature vector.

[0200] According to an exemplary embodiment of the present disclosure, the acquisition module 1001 is used to acquire image data within a preset time collected by a camera according to a preset frame rate set; detect the skeleton key points of the target object in each frame of the image data to obtain the skeleton key point sequence; perform feature processing on the skeleton key point sequence to obtain the multiple skeleton data features; wherein the multiple skeleton data features include at least two of the skeleton key point sequence features of the first coordinate system, the skeleton key point features of the second coordinate system, the speed features and the skeleton features.

[0201] According to an exemplary embodiment of the present disclosure, the branch module 1002 is used to input various skeleton data features in parallel into the pre-trained branch displacement convolution sub-network corresponding to the skeleton data features, so as to obtain feature maps corresponding to each skeleton data feature; wherein the branch displacement convolution sub-network is constructed in series using a batch normalization layer, a displacement map convolution initial block and at least one displacement map convolution block.

[0202] According to an exemplary embodiment of the present disclosure, the fusion module 1003 is used to uniformly input the multiple feature maps into the pre-trained fusion displacement convolution sub-network to obtain the feature vector; wherein the fusion displacement convolution sub-network is constructed by serially connecting at least one displacement map convolution block in sequence.

[0203] According to an exemplary embodiment of the present disclosure, the displacement map convolution block includes a spatial displacement operation module, a spatial point-by-point convolution module, a temporal displacement operation module and a temporal point-by-point convolution module.

[0204] According to an exemplary embodiment of the present disclosure, the behavior recognition device 1000 may also include a training module, which is used to obtain a skeleton data training set and a behavior label of the skeleton data training set; construct an original input graph according to the skeleton data training set, and delete edges of the original input graph to obtain a target input graph; based on the target input graph, use the branch displacement convolution subnetwork, the fused displacement convolution subnetwork and the fully connected layer to perform recognition to obtain recognized behavior information; compare the recognized behavior information with the behavior label to modify the model parameters of the branch displacement convolution subnetwork, the fused displacement convolution subnetwork and the fully connected layer.

[0205] According to an exemplary embodiment of the present disclosure, the training module also includes an edge deletion unit, which is used to construct an original adjacency matrix according to the displacement relationship in the original input graph; calculate the target adjacency matrix corresponding to the target input graph based on the original adjacency matrix and a preset random edge retention rate; or construct an attention template adjacency matrix according to the displacement relationship in the original input graph; calculate the edge retention probability of each edge of the original input graph based on the attention template adjacency matrix and a preset adaptive edge deletion parameter; and obtain the target adjacency matrix corresponding to the target input graph according to the edge retention probability.

[0206] The specific details of each module in the above-mentioned behavior recognition device 1000 have been described in detail in the corresponding behavior recognition method, so they will not be repeated here.

[0207] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0208] In an exemplary embodiment of the present disclosure, a storage medium capable of implementing the above method is also provided. Fig.11 A schematic diagram schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure, such as Fig.11 As shown, a program product 1100 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a mobile phone. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0209] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. Fig.12 The structure diagram of a computer system of an electronic device in an exemplary embodiment of the present disclosure is schematically shown.

[0210] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0211] like Fig.12As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0212] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0213] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present disclosure are executed.

[0214] It should be noted that the computer-readable medium shown in the embodiment of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0215] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0216] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0217] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0218] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0219] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0220] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0221] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A behavior recognition method, characterized in that: include: Acquire multiple bone data features of the target object; wherein the multiple bone data features include velocity features, skeleton features, bone key point sequence features in a Cartesian coordinate system, and bone key point features in a spherical coordinate system; Using a branch displacement convolution sub-network to perform branch processing on the multiple skeleton data features to obtain multiple feature maps; using a branch displacement convolution sub-network to perform branch processing on the multiple skeleton data features to obtain multiple feature maps includes: inputting each skeleton data feature in parallel to the pre-trained branch displacement convolution sub-network corresponding to the skeleton data feature to obtain feature maps corresponding to each skeleton data feature; wherein the branch displacement convolution sub-network is constructed by sequentially connecting a batch normalization layer, a displacement map convolution initial block and at least one displacement map convolution block in series; The plurality of feature maps are mainstream fused by a fused displacement convolution subnetwork to obtain a feature vector; wherein the fused displacement convolution subnetwork is constructed by sequentially connecting at least one displacement map convolution block; the displacement map convolution block includes a spatial displacement operation module, a spatial point-by-point convolution module, a temporal displacement operation module and a temporal point-by-point convolution module; Using a fully connected layer to identify the feature vector to obtain the behavior information of the target object; The method also includes model training for the branch shift convolution subnetwork, the fused shift convolution subnetwork and the fully connected layer; the model training includes edge deletion of the original input graph to obtain a target input graph; the edge deletion includes edge deletion using an adaptive attention-guided edge deletion strategy, the adaptive attention-guided edge deletion strategy includes constructing an attention template adjacency matrix based on the displacement relationship in the original input graph; calculating the edge retention probability of each edge of the original input graph based on the attention template adjacency matrix and preset adaptive edge deletion parameters; and obtaining a target adjacency matrix corresponding to the target input graph based on the edge retention probability.

2. The behavior recognition method according to claim 1, characterized in that: The step of obtaining multiple bone data features of the target object includes: Obtain image data collected by the camera within a preset time according to a preset frame rate set; Detecting the skeleton key points of the target object in each frame of the image data to obtain the skeleton key point sequence; The skeleton key point sequence is subjected to feature processing to obtain the plurality of skeleton data features.

3. The behavior recognition method according to claim 1, characterized in that: The step of performing mainstream fusion of the plurality of feature maps by fusing the shifted convolutional subnetwork to obtain a feature vector includes: The multiple feature maps are uniformly input into the pre-trained fused shift convolution sub-network to obtain the feature vector.

4. The behavior recognition method according to claim 1, characterized in that: The model training includes: Obtaining a skeleton data training set and a behavior label of the skeleton data training set; Constructing an original input graph according to the skeleton data training set, and performing edge deletion on the original input graph to obtain a target input graph; Based on the target input graph, the branch displacement convolution subnetwork, the fusion displacement convolution subnetwork and the fully connected layer are used to perform recognition to obtain recognition behavior information; The identified behavior information is compared with the behavior label to modify model parameters of the branch shift convolution subnetwork, the fused shift convolution subnetwork, and the fully connected layer.

5. A behavior recognition device, characterized in that: include: An acquisition module, used to acquire multiple bone data features of a target object within a preset time; wherein the multiple bone data features include speed features, skeleton features, bone key point sequence features of a Cartesian coordinate system, and bone key point features of a spherical coordinate system; A branch module, used for performing branch processing on the plurality of skeleton data features using a branch displacement convolution subnetwork to obtain a plurality of feature maps; the branch processing on the plurality of skeleton data features using a branch displacement convolution subnetwork to obtain a plurality of feature maps comprises: inputting each skeleton data feature in parallel into the pre-trained branch displacement convolution subnetwork corresponding to the skeleton data feature, to obtain feature maps corresponding to each skeleton data feature; wherein the branch displacement convolution subnetwork is constructed by sequentially connecting a batch normalization layer, a displacement map convolution initial block and at least one displacement map convolution block in series; A fusion module, used for mainstream fusion of the multiple feature maps to obtain a feature vector through a fusion displacement convolution subnetwork; wherein the fusion displacement convolution subnetwork is constructed by serially connecting at least one displacement map convolution block; the displacement map convolution block includes a spatial displacement operation module, a spatial point-by-point convolution module, a temporal displacement operation module and a temporal point-by-point convolution module; A recognition module, used to obtain the behavior information of the target object by using a fully connected layer to identify the feature vector; A training module is used to perform model training on the branch shifted convolutional subnetwork, the fused shifted convolutional subnetwork and the fully connected layer; the model training includes performing edge deletion on the original input graph to obtain a target input graph; the edge deletion includes performing edge deletion using an adaptive attention-guided edge deletion strategy, and the adaptive attention-guided edge deletion strategy includes constructing an attention template adjacency matrix based on the displacement relationship in the original input graph; calculating the edge retention probability of each edge of the original input graph based on the attention template adjacency matrix and preset adaptive edge deletion parameters; and obtaining a target adjacency matrix corresponding to the target input graph based on the edge retention probability.

6. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the behavior recognition method according to any one of claims 1 to 4.

7. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the behavior recognition method as described in any one of claims 1 to 4.

Images