Behavior recognition method and device, electronic equipment and computer readable storage medium

Abstract

This application provides a behavior recognition method, apparatus, electronic device, and computer-readable storage medium. The method combines graph convolutional layers and the encoder in the Transformer algorithm to perform behavior recognition on skeletal data, thereby enabling behavior recognition of skeletal data using a small model based on graph convolutional layers and a Transformer encoder. It also achieves comprehensive analysis of skeletal data using graph convolutional layers. Although the Transformer encoder used in this method has only two layers, it can effectively identify the temporal correlation of skeletal data, and the small two-layer structure allows for faster recognition and inference speed, resulting in a small recognition model with high accuracy. Furthermore, this method optimizes the graph convolutional layer network using global points, allowing the features extracted by the graph convolutional layer network to represent the global features of the skeleton, thereby further improving the accuracy of behavior recognition.

Description

Technical Field

[0001] This application relates to the field of intelligent behavioral recognition, and more specifically, to a behavioral recognition method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Skeleton-based human motion recognition methods analyze human behavior by examining temporal and spatial changes in the human skeleton. Because skeletons are unaffected by clothing changes, lighting conditions, or complex backgrounds, this recognition method is highly reliable and has a wide range of applications. Typically, during the recognition process, the human skeleton is segmented into different body parts, and the recognition system identifies the position of each body part to detect posture. This can improve security in law enforcement facilities, schools, airports, banks, commercial spaces, or office buildings. Furthermore, it can detect suspicious or abnormal behavior, making it of significant importance and application value in fields such as security and behavioral monitoring.

[0003] Currently, intelligent behavior analysis of skeletal data is a relatively mature intelligent behavior analysis technology. The method for skeletal behavior recognition is generally to use deep neural networks for automatic recognition. However, the cutting-edge network models used by the analysis systems or methods currently applied to complete intelligent behavior recognition based on skeletal data are extremely large, making it difficult to deploy on low-computing-power devices. Furthermore, the recognition process is complex and slow. Summary of the Invention

[0004] The purpose of this application is to provide a behavior recognition method, device, electronic device, and computer-readable storage medium to solve the problems of large models, difficult deployment, and complex and slow recognition process in current intelligent behavior recognition methods based on skeletal data.

[0005] In a first aspect, the present invention provides a behavior recognition method, comprising: constructing and obtaining skeletal input information based on skeletal information to be recognized and target global point information; wherein, the target global point refers to a point in the skeletal information to be recognized where the information obtained by each skeletal point is consistent in proportion; extracting skeletal features from the skeletal input information through a target graph convolutional layer to obtain first feature information; wherein, the first feature information represents the information after the skeletal information to be recognized and the target global point information have been integrated; and recognizing the behavior corresponding to the skeletal information to be recognized based on a target Transformer encoder and the first feature information.

[0006] The behavior recognition method described above combines graph convolutional layers and the encoder in the Transformer algorithm to perform behavior recognition on skeletal data. This allows for behavior recognition of skeletal data using a small model based on graph convolutional layers and a Transformer encoder. The graph convolutional layers provide a thorough analysis of the skeletal data, and the Transformer encoder, although only two layers, effectively identifies the temporal correlations of the skeletal data. Furthermore, the small two-layer structure enables faster recognition and inference speeds, resulting in a small recognition model with high accuracy. Additionally, the graph convolutional layer network is optimized using global points, allowing the features extracted by the network to represent global features of the skeleton, further improving the accuracy of this approach.

[0007] In an optional implementation of the first aspect, the behavior corresponding to the skeleton information to be identified is identified based on the target Transformer encoder and the first feature information, including: extracting the target global point feature information from the first feature information, and generating encoded input data based on the target global point feature information and the target time vector; extracting time features from the encoded input data through the target Transformer encoder to obtain second feature information, the second feature information representing the information after the target global point feature information and the target time vector are fully integrated; and identifying the behavior corresponding to the skeleton information to be identified based on the second feature information.

[0008] In the above implementation, this solution introduces a target time vector, so that the time dimension features in the second feature information can be represented by the target time vector after being combined with it, thereby simplifying the algorithm and improving accuracy.

[0009] In an optional implementation of the first aspect, the behavior corresponding to the skeleton information to be identified is identified based on the second feature information, including: extracting target time vector feature information from the second feature information; inputting the target time vector feature information into a multilayer perceptron to obtain the probability corresponding to each behavior type output by the multilayer perceptron; and determining the behavior corresponding to the skeleton information to be identified based on the probability corresponding to each behavior type.

[0010] In the above implementation, this solution uses a multilayer perceptron to identify behavior types based on the second feature information, thereby ensuring the accuracy of behavior identification.

[0011] In an optional implementation of the first aspect, generating coded input data based on target global point feature information and target time vector includes: concatenating target global point feature information and target time vector to obtain first concatenated data; and adding position encoding to the first concatenated data to obtain coded input data.

[0012] In the above implementation, this scheme adds positional encoding on the basis of introducing the target time vector, so that the standalone Transformer encoder can also recognize the temporal sequence in the skeletal information, thereby making the recognized features temporally correlated and improving the recognition accuracy of the Transformer encoder.

[0013] In an optional implementation of the first aspect, feature extraction of the skeletal input information through a target graph convolutional layer to obtain first feature information includes: performing feature upscaling on the skeletal input information through a target graph convolutional layer; and performing information fusion on the feature-upgraded skeletal input information to obtain the first feature information.

[0014] In an optional implementation of the first aspect, information fusion is performed on the skeletal input information after feature dimensionality enhancement to obtain first feature information, including: generating the first feature information based on the skeletal input information after feature dimensionality enhancement and the target graph adjacency matrix, wherein the target graph adjacency matrix is ​​obtained by pre-training a preset graph adjacency matrix.

[0015] In an optional implementation of the first aspect, before constructing and obtaining skeletal input information based on the skeletal information to be identified and the target global point information, the method further includes: acquiring a skeletal sample dataset, the skeletal sample dataset including multiple skeletal sample data, each skeletal sample data including skeletal point samples and corresponding behavior labels; normalizing the multiple skeletal sample data to obtain multiple normalized skeletal sample data; constructing multiple skeletal sample input data based on preset global points and each normalized skeletal sample data; and training preset global points, preset graph convolutional layers, preset temporal vectors, and preset Transformer encoders based on the multiple skeletal sample input data to obtain target global points, target graph convolutional layers, target temporal vectors, and target Transformer encoders.

[0016] In an optional implementation of the first aspect, training a preset global point, a preset graph convolutional layer, a preset temporal vector, and a preset Transformer encoder based on multiple skeletal sample input data includes: selecting a set of skeletal sample input data as the current skeletal sample input set from the multiple skeletal sample input data; inputting the current skeletal sample input set into a preset graph convolutional layer to obtain first feature information corresponding to each current skeletal sample input data; extracting preset global point feature information from the first feature information corresponding to each current skeletal sample input data, and generating encoded sample input data based on the preset global point feature information and the preset temporal vector; inputting the encoded sample input data into a preset Transformer encoder to obtain second feature information corresponding to each encoded sample input data, and obtaining the corresponding classification result based on the second feature information and a preset multilayer perceptron; calculating the training loss corresponding to the current skeletal sample input set using a contrastive loss function based on the first feature information corresponding to each skeletal sample input data, the second feature information corresponding to each encoded sample input data, and the corresponding behavior label; and so on. The training loss, backpropagation algorithm, and optimization algorithm update and iterate the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder, respectively. It then determines whether the current cumulative iteration count exceeds the preset iteration count, or whether the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder have converged. If the current cumulative iteration count exceeds the preset iteration count or the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder have converged, then the target global point, target graph convolutional layer, target temporal vector, and target Transformer encoder are obtained. If the current cumulative iteration count does not exceed the preset iteration count and the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder have not converged, then the process returns to the step of selecting a set of bone sample input data from multiple bone sample input data as the current bone sample input set.

[0017] In an optional implementation of the first aspect, skeletal input information is constructed based on the skeletal information to be identified and the target global point information, including: acquiring the skeletal information to be identified and the target global point information; normalizing the skeletal information to be identified; and constructing skeletal input information based on the target global point information and the normalized skeletal information to be identified.

[0018] Secondly, the present invention provides a behavior recognition device, which includes a construction module, a graph convolution module, and a recognition module. The construction module is used to construct and obtain skeletal input information based on the skeletal information to be recognized and target global point information. The target global point refers to a point in the skeletal information to be recognized where the information obtained from each skeletal point is proportionally consistent. The graph convolution module is used to extract skeletal features from the skeletal input information through a target graph convolutional layer to obtain first feature information. The first feature information represents the information after the skeletal information to be recognized and the target reference point information are combined. The recognition module is used to recognize the behavior corresponding to the skeletal information to be recognized based on a target Transformer encoder and the first feature information.

[0019] The behavior recognition device designed above combines graph convolutional layers and the encoder in the Transformer algorithm to perform behavior recognition on skeletal data. Thus, a small model based on graph convolutional layers and a Transformer encoder can recognize the behavior of skeletal data. Graph convolutional layers can fully analyze skeletal data, and although the encoder using Transformer has only two layers, it can effectively identify the correlation of skeletal data in the time dimension. Moreover, the small two-layer structure can have a faster recognition and inference speed, so the designed recognition model is small and has high recognition accuracy, thereby further improving the accuracy of this solution.

[0020] In an optional implementation of the second aspect, the recognition module is specifically used to extract target global point feature information from the first feature information, and generate encoded input data based on the target global point feature information and the target time vector; extract time features from the encoded input data through a target Transformer encoder to obtain second feature information, the second feature information representing the information after the target global point feature information and the target time vector are fully integrated; and recognize the behavior corresponding to the skeleton information to be recognized based on the second feature information.

[0021] In an optional implementation of the second aspect, the recognition module is further specifically used to extract the target time vector feature information from the second feature information, input the target time vector feature information into a multilayer perceptron, obtain the probability corresponding to each behavior type output by the multilayer perceptron, and determine the behavior corresponding to the skeleton information to be recognized based on the probability corresponding to each behavior type.

[0022] In an optional implementation of the second aspect, the identification module is further specifically used to concatenate the target global point feature information and the target time vector to obtain first concatenated data; and to add position encoding to the first concatenated data to obtain encoded input data.

[0023] In an optional implementation of the second aspect, the graph convolution module is specifically used to perform feature upscaling on the skeletal input information through a target graph convolutional layer; and to perform information fusion on the feature-upgraded skeletal input information to obtain first feature information.

[0024] In an optional implementation of the second aspect, the graph convolution module is further specifically used to generate the first feature information based on the skeletal input information after feature dimensionality upscaling and the target adjacency matrix, wherein the target adjacency matrix is ​​obtained through pre-training.

[0025] In an optional embodiment of the second aspect, the device further includes an acquisition module for acquiring a skeleton sample dataset, the skeleton sample dataset including multiple skeleton sample data, each skeleton sample data including a skeleton point sample and a corresponding behavior label; a normalization module for normalizing the multiple skeleton sample data to obtain multiple normalized skeleton sample data; the construction module is further used to construct multiple skeleton sample input data based on preset global points and each normalized skeleton sample data; and a training module for training preset global points, preset graph convolutional layers, preset temporal vectors, and preset Transformer encoders based on the multiple skeleton sample input data to obtain target global points, target graph convolutional layers, target temporal vectors, and target Transformer encoders.

[0026] In an optional implementation of the second aspect, the training module is specifically configured to: select a set of bone sample input data from multiple bone sample input data as the current bone sample input set; input the current bone sample input set into a preset graph convolutional layer to obtain first feature information corresponding to each current bone sample input data; extract preset global point feature information from the first feature information corresponding to each current bone sample input data, and generate encoded sample input data based on the preset global point feature information and a preset time vector; input the encoded sample input data into a preset Transformer encoder to obtain second feature information corresponding to each encoded sample input data; calculate the training loss corresponding to the current bone sample input set using the first feature information corresponding to each bone sample input data, the second feature information corresponding to each encoded sample input data, and the corresponding behavior label, through a contrastive loss function; and adjust the preset global points, preset graph convolutional layer, preset time vector, and... The preset Transformer encoder updates and iterates on each parameter. It then determines whether the current cumulative iteration count exceeds the preset iteration count, or whether the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder parameters have converged. If the current cumulative iteration count exceeds the preset iteration count or the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder parameters have converged, then the target global point, target graph convolutional layer, target temporal vector, and target Transformer encoder are obtained. If the current cumulative iteration count does not exceed the preset iteration count and the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder parameters have not converged, then the process returns to the step of selecting a set of bone sample input data from multiple bone sample input data as the current bone sample input set.

[0027] In an optional implementation of the second aspect, the construction module is specifically used to acquire the skeleton information to be identified and the target global point information; to perform normalization processing on the skeleton information to be identified; and to construct skeleton input information based on the target global point information and the normalized skeleton information to be identified.

[0028] Thirdly, this application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method described in the first aspect or any optional implementation thereof.

[0029] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method described in the first aspect or any optional implementation thereof.

[0030] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect or any optional implementation thereof.

[0031] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A first flowchart of the behavior recognition method provided in the embodiments of this application;

[0034] Figure 2 This is a schematic diagram of global points and skeleton points provided in the embodiments of this application;

[0035] Figure 3 A second flowchart of the behavior recognition method provided in the embodiments of this application;

[0036] Figure 4 A third flowchart of the behavior recognition method provided in the embodiments of this application;

[0037] Figure 5 A fourth flowchart of the behavior recognition method provided in the embodiments of this application;

[0038] Figure 6 This is a schematic diagram of the model structure provided in the embodiments of this application;

[0039] Figure 7 This is a schematic diagram of the structure of the behavior recognition device provided in the embodiments of this application;

[0040] Figure 8 A schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0041] Icons: 700 - Building module; 710 - Graph convolution module; 720 - Recognition module; 730 - Acquisition module; 740 - Normalization module; 750 - Training module; 8 - Electronic device; 801 - Processor; 802 - Memory; 803 - Communication bus. Detailed Implementation

[0042] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0044] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0047] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0048] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0049] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0050] Currently, intelligent recognition methods based on skeletal data analysis of target behavior are gradually becoming mainstream. Behavior recognition can play a key role in analyzing and judging the behavioral purpose and behavior patterns of targets.

[0051] The inventors have discovered that currently commonly used behavior recognition methods based on skeletal data generally employ deep learning models. However, current deep learning models are large and difficult to deploy on conventional computing devices. Furthermore, the computation process of current deep learning models is complex and slow.

[0052] Based on the above, the applicant provides a behavior recognition method, device, electronic device, and computer-readable storage medium. This solution combines graph convolutional layers and encoders in the Transformer algorithm to perform behavior recognition on skeletal data. Thus, a small model based on graph convolutional layers and Transformer encoders can recognize the behavior of skeletal data. Graph convolutional layers can fully analyze skeletal data. Although the encoder using Transformer has only two layers, it can effectively identify the correlation of skeletal data in the time dimension. The small two-layer structure can achieve faster recognition and inference speed, resulting in a small recognition model with high recognition accuracy. In addition, the graph convolutional layer network is optimized using global points, so that the features extracted by the graph convolutional layer network represent the global features of the skeleton. The introduction of target time vector class-token improves the recognition of time dimension features, thereby further improving the accuracy of this solution.

[0053] This application provides a behavior recognition method, which can be applied to computing devices, including but not limited to computers, servers, etc. Figure 1 As shown, this behavior recognition method can be implemented in the following way:

[0054] Step S100: Construct and obtain skeletal input information based on the skeletal information to be identified and the global point information of the target.

[0055] Step S110: Extract skeletal features from the skeletal input information through the target image convolutional layer to obtain the first feature information.

[0056] Step S120: Identify the behavior corresponding to the skeleton information to be identified based on the target Transformer encoder and the first feature information.

[0057] In the above embodiments, the skeletal information to be identified may include one or more skeletal data to be identified. Specifically, the skeletal data to be identified may include five dimensions: frame number T, skeletal point V, number of skeletal groups M, number of channels C, and sample size N, forming a five-dimensional matrix (N, C, T, V, M). Among them, the frame number T can represent the length of the action; the skeletal point V represents skeletal points such as the nose, jaw, neck, and thumb; the number of skeletal groups M represents the number of people identified; the number of channels C can vary according to the dimension of the skeletal data. For example, if the skeletal information to be identified is spatial skeletal data, then the number of channels C is 3, representing the x-axis, y-axis, and z-axis respectively; if the skeletal information to be identified is planar skeletal data, then the number of channels C is 2, representing the x-axis and y-axis respectively.

[0058] Target global points refer to points in the skeletal information to be identified where the proportion of information obtained from each skeletal point is consistent, such as... Figure 2 The global point shown is equidistant from the information of each skeletal point, meaning each skeletal point acquires the same proportion of information. Based on this, this scheme can obtain the information of each skeletal point through this target global point, and each skeletal point can also obtain the information of the target global point. This target global point can be obtained through training on preset global points.

[0059] For step S100, this scheme constructs skeletal input information based on the skeletal information to be identified and the target global point information. As one possible implementation, since the skeletal information to be identified contains a batch of skeletal data, in order to accelerate the convergence speed of the model, this scheme can first normalize the skeletal information to be identified, and then construct the skeletal input information based on the target global point information and the normalized skeletal information to be identified. Specifically, the normalization process can be batch normalization.

[0060] Furthermore, since this scheme recognizes the behavior of each individual, in addition to normalizing the skeletal information to be recognized, it can also concatenate the number of samples N and the number of bone groups T in the skeletal information to be recognized before constructing the skeletal input information. For example, assuming the skeletal information to be recognized is (N, C, T, V, M), then the skeletal information to be recognized after batch normalization and concatenation of the number of samples N and the number of bone groups T can be (N*M, C, T, V). Then, the skeletal input information constructed by combining this information with the target global points can be (N*M, C, T, V+1).

[0061] After obtaining the skeletal input information through the above method, this scheme can extract skeletal features from the skeletal input information through a target image convolutional layer to obtain the first feature information. This target image convolutional layer is obtained through pre-training, and the first feature information represents the information after the skeletal information to be identified and the target global point information have been fully integrated.

[0062] As one possible implementation, the target image convolutional layer can first perform feature upscaling on the skeletal input information, and then perform information fusion on the feature-upgraded skeletal input information to obtain the first feature information.

[0063] Specifically, the target image convolutional layer can first perform feature upscaling on the number of channels in the skeletal input information, thereby increasing the dimension of the number of channels in the skeletal input information and thus increasing the features of the skeletal data. For example, if the original number of channels C (e.g., 3) is C1 (e.g., 32, 64, 128), the target image convolutional layer can upscale the number of channels C to C1 (e.g., 32, 64, 128), and obtain the skeletal input information (N*M, T, C1, V+1) after feature upscaling, thereby increasing the dimension of the number of channels in the skeletal input information and increasing the features of the skeletal data.

[0064] Based on the above, the target graph convolutional layer uses the skeletal input information after feature dimensionality enhancement and the target adjacency matrix to generate the first feature information. Specifically, the target graph convolutional layer can multiply the matrix (N*M, T, C1, V+1) corresponding to the skeletal input information after feature dimensionality enhancement with the target graph adjacency matrix to obtain the first feature information. Among them, the target adjacency matrix is ​​obtained by pre-training a preset graph adjacency matrix. Because the dimension of V is increased by 1, the graph adjacency matrix should be changed from (V,V) to (V+1,V+1) as an M matrix. The graph adjacency matrix operation is to first assign M[0] to 1 / (V+1), assign M[1:V+1,0] to 0.3, p(i,j) indicates whether point i and point j are adjacent, 1 if yes, 0 if no, 1 if i and j are the same, D(i) represents the degree of i, then M[i+1,j+1]=p(i,j)*(D(i)+1)*0.7.

[0065] After obtaining the first feature information through the above method, this scheme identifies the behavior corresponding to the skeleton information to be identified based on the target Transformer encoder and the first feature information.

[0066] As one possible implementation method, such as Figure 3 As shown, step S120 can be implemented in the following ways:

[0067] Step S300: Extract the target global point feature information from the first feature information.

[0068] Step S310: Generate encoded input data based on the target global point feature information and the target time vector.

[0069] Step S320: Extract time features from the encoded input data using the target Transformer encoder to obtain the second feature information.

[0070] Step S330: Identify the behavior corresponding to the skeleton information to be identified based on the second feature information.

[0071] In the above implementation, since the target map convolutional layer combines the information of the skeleton to be identified with the target global point information to obtain the first feature information, the target global point feature information in the first feature information corresponding to the target global point information can characterize the feature information of each skeleton point. Therefore, this scheme can extract the target global point feature information only from the first feature information. Since the extracted target global point feature information is only 1 in number, the matrix of the target global point feature information can be simplified to (N*M, C1, T).

[0072] Based on the above, this scheme can generate encoded input data based on the target global point feature information and the target time vector. The target time vector is also called the class-token, which can be obtained by training a preset time vector, which can be a random matrix.

[0073] Specifically, this scheme can first concatenate the target time vector with the target global point feature information to obtain concatenated data. For example, assuming the target time vector is (N*M, C1, 1), (N*M, C1, 1) can be concatenated with (N*M, C1, T) to obtain concatenated data (N*M, C1, T+1).

[0074] Furthermore, since the second dimension of the target time vector is generally the time dimension and the third dimension is the channel dimension, that is, the target time vector is (N*M, 1, C1), based on this, this scheme can first transpose the target global point feature information, that is, transpose (N*M, C1, T) to (N*M, T, C1), and then concatenate it with the target time vector (N*M, 1, C1) to obtain the concatenated data (N*M, T+1, C1).

[0075] Since this solution requires the recognition of multiple frames of temporally ordered skeletal data, and the Transformer encoder alone cannot distinguish the temporal sequence, this solution adds positional encoding to the stitched data to obtain encoded input data. This allows the Transformer encoder to recognize the temporal sequence of the encoded input data, thereby identifying the temporal sequence of multiple frames of skeletal data and the correlation between preceding and subsequent actions, thus improving the accuracy of behavior recognition.

[0076] Specifically, this scheme concatenates the spliced ​​data with a trainable matrix to obtain the encoded input data. For example, the matrix output after concatenating the spliced ​​data (N*M, T+1, C1) with the trainable matrix P (1, T+1, C1) can be used as the encoded input data. Furthermore, it should be noted that, in addition to generating encoded input data using the above method, this scheme can also directly add positional encoding to the first feature information without introducing the target time vector, thereby obtaining the encoded input data.

[0077] This scheme inputs the aforementioned encoded input data into a Transformer encoder. The Transformer encoder can extract the temporal features from the encoded input data to obtain the second feature information. That is, the Transformer encoder only extracts features from the time dimension, i.e., the number of frames T, in the encoded input data to obtain the second feature information. The second feature information represents the information after the target global point feature information and the target time vector are fully integrated.

[0078] Finally, the behavior corresponding to the skeleton information to be identified can be recognized based on the second feature information. As one possible implementation method, such as... Figure 4 As shown, this solution can be implemented in the following ways:

[0079] Step S400: Extract the target time vector feature information from the second feature information.

[0080] Step S410: Input the target time vector feature information into the multilayer perceptron to obtain the probability corresponding to each behavior type output by the multilayer perceptron.

[0081] Step S420: Determine the behavior type corresponding to the skeleton information to be identified based on the probability corresponding to each behavior type.

[0082] In the above implementation, when a target time vector is introduced into the encoded input data, since the target time vector is combined with the time dimension of the target global point feature information, only the feature information corresponding to the target time vector needs to be extracted to represent the time dimension feature information of all skeletal points. For example, if the Transformer encoder output is (N*M, T+1, C1), this scheme only takes the first index data of the second dimension to obtain the second feature information, i.e., (N*M, C1).

[0083] In the above implementation, the multilayer perceptron can identify the probability of each skeleton data to be identified belonging to each behavior type based on the target time vector feature information, thereby determining the behavior type with the highest probability as the type corresponding to the skeleton information to be identified. For example, specifically, the multilayer perceptron can first convert the aforementioned second feature information (N*M,C1) into a (N,C1,M) matrix, then take the mean of the third dimension matrix to reduce the dimensionality to obtain (N,C1), and then pass it through two fully connected layers. The first fully connected layer expands (N,C1) to (N,512), and the second fully connected layer shrinks (N,512) to (N,S), where S is the number of categories, thereby determining the recognition type corresponding to the skeleton information to be identified.

[0084] It should be noted that when the target time vector is not included in the encoded input data, this scheme can input all the second feature information output by the Transformer encoder into the multilayer perceptron, thereby performing behavior recognition through the multilayer perceptron. For example, the (N*M, T+1, C1) matrix output by the Transformer encoder can be input into the multilayer perceptron.

[0085] The behavior recognition method described above combines graph convolutional layers and the encoder in the Transformer algorithm to perform behavior recognition on skeletal data. This allows for behavior recognition of skeletal data using a small model based on graph convolutional layers and a Transformer encoder. The graph convolutional layers provide a thorough analysis of the skeletal data, and the Transformer encoder, although only two layers, effectively identifies the temporal correlations of the skeletal data. The small two-layer structure also allows for faster recognition and inference speeds, resulting in a small recognition model with high accuracy. Furthermore, the graph convolutional layer network is optimized using global points, allowing the features extracted by the network to represent global skeletal features. Introducing a target temporal vector (class-token) enhances temporal feature recognition, further improving the accuracy of this approach.

[0086] In an optional implementation of this embodiment, the aforementioned target global point information, target graph convolutional layer, target temporal vector, target adjacency matrix, and Transformer encoder can all be obtained through pre-training. Specifically, such as... Figure 5 As shown, the training process can be implemented in the following way:

[0087] Step S500: Obtain a skeleton sample dataset, which includes multiple skeleton sample data, each of which includes a skeleton point sample and its corresponding behavior label.

[0088] Step S510: Normalize the multiple bone sample data to obtain multiple bone sample data after normalization.

[0089] Step S520: Construct multiple bone sample input data based on the preset global points and the normalized bone sample data of each bone sample.

[0090] Step S530: Train the preset global points, preset graph convolutional layers, preset temporal vectors and preset Transformer encoders based on multiple skeleton sample input data to obtain target global points, target graph convolutional layers, target temporal vectors and target Transformer encoders.

[0091] In the above implementation, this method first acquires a batch of bone sample data to form a bone sample dataset. Then, in order to accelerate the training and convergence speed of the model, this method performs batch normalization processing on multiple bone sample data to obtain multiple bone sample data after normalization. Then, it constructs multiple bone sample input data with preset global points (preset random matrix) and each bone sample data after normalization. Finally, it performs training to obtain target global points, target graph convolutional layer, target temporal vector and target Transformer encoder.

[0092] Specifically, for step S530, this scheme first selects a set of bone sample input data from multiple bone sample input data as the current bone sample input set; inputs the current bone sample input set into a preset graph convolutional layer to obtain the first feature information corresponding to each current bone sample input data; then extracts the preset global point feature information from the first feature information corresponding to each current bone sample input data, and generates encoded sample input data based on the preset global point feature information and the preset time vector; inputs the encoded sample input data into a preset Transformer encoder to obtain the second feature information corresponding to each encoded sample input data, and obtains the classification result of the corresponding encoded sample input data based on the second feature information. The methods for constructing global points and introducing time vectors are the same as those described above, and will not be repeated here.

[0093] Based on the above, this scheme calculates the training loss corresponding to the current bone sample input set by using the first feature information corresponding to each bone sample input data, the second feature information corresponding to each encoded sample input data, and the corresponding behavior label, through a comparative loss function; it updates and iterates the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder according to the training loss, backpropagation algorithm, and optimization algorithm; it then determines whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder are... Convergence; if the current cumulative number of iterations exceeds the preset number of iterations or the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder converge, then the target global point, target graph convolutional layer, target temporal vector, and target Transformer encoder are obtained; if the current cumulative number of iterations does not exceed the preset number of iterations and the parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder do not converge, then the process returns to the step of selecting a set of bone sample input data from multiple bone sample input data as the current bone sample input set. Additionally, it should be noted that if this scheme uses the aforementioned multilayer perceptron for behavior recognition, this scheme can also pre-train the multilayer perceptron using the second feature information and corresponding classification label corresponding to each encoded sample input data.

[0094] In an optional implementation of this embodiment, as a specific example, the behavior recognition method designed in this solution can be as follows: Figure 6 As shown, BN represents normalization; GCN represents graph convolutional layer; dimension alignment represents the introduction of target temporal vector and positional encoding; MLP represents multilayer perceptron.

[0095] Based on the above, the input skeleton data to be identified first enters the BN layer for normalization. Then, between the BN layer and the GCN layer, global points are introduced into the normalized skeleton data. The skeleton data with global points is then input into the GCN layer, which performs feature upscaling and information fusion, thereby extracting only the first index data of the skeleton point dimension to obtain the target global point feature information. Then, class-token and position encoding are introduced in the dimension alignment layer. After generating encoded input data, it is input into the Transformer encoder for temporal feature extraction, thereby extracting the first index data of the time dimension to obtain the second feature information. Finally, the Multilayer Perceptron (MLP) performs behavior recognition based on the second feature information to obtain the behavior type corresponding to the skeleton data to be identified.

[0096] Figure 7 A schematic structural block diagram of a behavior recognition device provided in this application is shown. It should be understood that this device is related to... Figures 1 to 6 The method implementation described above corresponds to the method embodiment and can perform the steps involved in the aforementioned method. The specific functions of this device can be found in the description above; to avoid repetition, detailed descriptions are omitted here. This device includes at least one software functional module that can be stored in a memory or embedded in the device's operating system (OS) in the form of software or firmware. Specifically, the device includes: a construction module 700, a graph convolution module 710, and a recognition module 720. The construction module 700 is used to construct and obtain skeletal input information based on the skeletal information to be recognized and target global point information; wherein, the target global point represents a point in the skeletal information to be recognized where the information obtained from each skeletal point is proportionally consistent. The graph convolution module 710 is used to extract skeletal features from the skeletal input information through a target graph convolutional layer to obtain first feature information; wherein, the first feature information represents the information after the skeletal information to be recognized and the target reference point information have been integrated. The recognition module 720 is used to recognize the behavior corresponding to the skeletal information to be recognized based on the target Transformer encoder and the first feature information.

[0097] The behavior recognition device designed above combines graph convolutional layers and the encoder in the Transformer algorithm to perform behavior recognition on skeletal data. Thus, a small model based on graph convolutional layers and a Transformer encoder can recognize the behavior of skeletal data. Graph convolutional layers can fully analyze skeletal data, and although the encoder using Transformer has only two layers, it can effectively identify the correlation of skeletal data in the time dimension. Moreover, the small two-layer structure can have a faster recognition and inference speed, so the designed recognition model is small and has high recognition accuracy, thereby further improving the accuracy of this solution.

[0098] In an optional embodiment of this example, the recognition module 720 is specifically used to extract the target global point feature information from the first feature information, and generate coded input data based on the target global point feature information and the target time vector; extract the time features of the coded input data through the target Transformer encoder to obtain the second feature information, which represents the information after the target global point feature information and the target time vector are fully integrated; and recognize the behavior corresponding to the skeleton information to be recognized based on the second feature information.

[0099] In an optional embodiment of this example, the recognition module 720 is further specifically used to extract the target time vector feature information from the second feature information, input the target time vector feature information into the multilayer perceptron, obtain the probability corresponding to each behavior type output by the multilayer perceptron, and determine the behavior corresponding to the skeleton information to be recognized based on the probability corresponding to each behavior type.

[0100] In an optional embodiment of this example, the identification module 720 is further specifically used to concatenate the target global point feature information and the target time vector to obtain first concatenated data; and to add position encoding to the first concatenated data to obtain encoded input data.

[0101] In an optional embodiment of this example, the graph convolution module 710 is specifically used to perform feature upscaling on the skeletal input information through the target graph convolution layer; and to perform information fusion on the feature-upgraded skeletal input information to obtain first feature information.

[0102] In an optional embodiment of this example, the graph convolution module 710 is further specifically used to generate the first feature information based on the skeletal input information after feature dimensionality upscaling and the target adjacency matrix, wherein the target adjacency matrix is ​​obtained through pre-training.

[0103] In an optional embodiment of this example, the device further includes an acquisition module 730 for acquiring a skeleton sample dataset, the skeleton sample dataset including multiple skeleton sample data, each skeleton sample data including a skeleton point sample and a corresponding behavior label; a normalization module 740 for normalizing the multiple skeleton sample data to obtain multiple normalized skeleton sample data; the construction module 700 is further used to construct multiple skeleton sample input data based on preset global points and each normalized skeleton sample data; and a training module 750 for training preset global points, preset graph convolutional layers, preset temporal vectors, and preset Transformer encoders based on the multiple skeleton sample input data to obtain target global points, target graph convolutional layers, target temporal vectors, and target Transformer encoders.

[0104] In an optional embodiment of this example, the training module 750 is specifically configured to: select a set of bone sample input data from multiple bone sample input data as the current bone sample input set; input the current bone sample input set into a preset graph convolutional layer to obtain first feature information corresponding to each current bone sample input data; extract preset global point feature information from the first feature information corresponding to each current bone sample input data, and generate encoded sample input data based on the preset global point feature information and a preset time vector; input the encoded sample input data into a preset Transformer encoder to obtain second feature information corresponding to each encoded sample input data; calculate the training loss corresponding to the current bone sample input set using a contrastive loss function based on the first feature information corresponding to each bone sample input data, the second feature information corresponding to each encoded sample input data, and the corresponding behavior label; and adjust the preset global points, preset graph convolutional layer, and preset time vector based on the training loss, backpropagation algorithm, and optimization algorithm. The process involves updating and iterating the parameters corresponding to the preset Transformer encoder; determining whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the parameters corresponding to the preset global point, preset graph convolutional layer, preset time vector, and preset Transformer encoder have converged; if the current cumulative number of iterations exceeds the preset number of iterations or the parameters corresponding to the preset global point, preset graph convolutional layer, preset time vector, and preset Transformer encoder have converged, then the target global point, target graph convolutional layer, target time vector, and target Transformer encoder are obtained; if the current cumulative number of iterations does not exceed the preset number of iterations and the parameters corresponding to the preset global point, preset graph convolutional layer, preset time vector, and preset Transformer encoder have not converged, then the process returns to the step of selecting a set of bone sample input data from multiple bone sample input data as the current bone sample input set.

[0105] In an optional implementation of this embodiment, the construction module 700 is specifically used to acquire the skeleton information to be identified and the target global point information; to perform normalization processing on the skeleton information to be identified; and to construct skeleton input information based on the target global point information and the normalized skeleton information to be identified.

[0106] According to some embodiments of this application, such as Figure 8As shown, this application provides an electronic device 8, including: a processor 801 and a memory 802. The processor 801 and the memory 802 are interconnected and communicate with each other through a communication bus 803 and / or other forms of connection mechanism (not shown). The memory 802 stores a computer program executable by the processor 801. When the computing device is running, the processor 801 executes the computer program to perform any optional implementation method, such as steps S100 to S120: constructing and obtaining skeletal input information based on the skeletal information to be identified and the target global point information; extracting skeletal features from the skeletal input information through a target map convolutional layer to obtain first feature information; and identifying the behavior corresponding to the skeletal information to be identified based on the target Transformer encoder and the first feature information.

[0107] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0108] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0109] This application provides a computer program product that, when run on a computer, causes the computer to perform a method in any of the optional implementations.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A behavior recognition method, characterized in that, include: Based on the bone information to be identified and the target global point information, bone input information is constructed and obtained; wherein, the target global point refers to a point in the bone information to be identified where the proportion of information obtained by each bone point is consistent. Skeletal features are extracted from the skeletal input information by a convolutional layer on the target image to obtain first feature information; wherein, the first feature information represents the information after the skeletal information to be identified and the target reference point information are fully integrated; The behavior corresponding to the skeleton information to be identified is identified based on the target Transformer encoder and the first feature information; The step of identifying the behavior corresponding to the skeleton information to be identified based on the target Transformer encoder and the first feature information includes: Extract the target global point feature information from the first feature information, and generate coded input data based on the target global point feature information and the target time vector; The target Transformer encoder extracts temporal features from the encoded input data to obtain second feature information; wherein, the second feature information represents the information after the target global point feature information and the target time vector complete information are combined. The behavior corresponding to the skeleton information to be identified is identified based on the second feature information; The step of identifying the behavior corresponding to the skeleton information to be identified based on the second feature information includes: Extract the target time vector feature information from the second feature information; The target time vector feature information is input into a multilayer perceptron to obtain the probability corresponding to each behavior type output by the multilayer perceptron. The behavior corresponding to the skeleton information to be identified is determined based on the probability corresponding to each behavior type.

2. The method according to claim 1, characterized in that, The step of generating coded input data based on target global point feature information and target time vector includes: The target global point feature information and the target time vector are concatenated to obtain the first concatenated data; Add position encoding to the first spliced ​​data to obtain the encoded input data.

3. The method according to claim 1, characterized in that, The step of extracting features from the skeletal input information through a target image convolutional layer to obtain first feature information includes: The dimensionality of the skeletal input information is increased by using a convolutional layer on the target image. The first feature information is obtained by integrating the skeletal input information after feature dimensionality enhancement.

4. The method according to claim 3, characterized in that, The process of integrating the skeletal input information after feature dimensionality enhancement to obtain the first feature information includes: The first feature information is generated based on the skeletal input information after feature dimensionality enhancement and the target adjacency matrix; wherein, the target adjacency matrix is ​​obtained through pre-training.

5. The method according to any one of claims 1-4, characterized in that, Before constructing and obtaining skeletal input information based on the skeletal information to be identified and the target global point information, the method further includes: Obtain a skeleton sample dataset; wherein, the skeleton sample dataset includes multiple skeleton sample data, and each skeleton sample data includes a skeleton point sample and a corresponding behavior label; The multiple bone sample data are normalized to obtain multiple bone sample data after normalization. Multiple bone sample input data are constructed based on the preset global points and the normalized data of each bone sample. The target global points, target graph convolutional layer, target time vector, and target Transformer encoder are trained based on multiple skeletal sample input data.

6. The method according to claim 5, characterized in that, The step of training a preset global point, a preset graph convolutional layer, a preset temporal vector, and a preset Transformer encoder based on multiple skeletal sample input data includes: Select a set of bone sample input data from the plurality of bone sample input data as the current bone sample input set; Input the current bone sample input set into a preset graph convolutional layer to obtain the first feature information corresponding to each current bone sample input data; Extract the preset global point feature information from the first feature information corresponding to each current bone sample input data, and generate coded sample input data based on the preset global point feature information and the preset time vector; The encoded sample input data is input into a preset Transformer encoder to obtain the second feature information corresponding to each encoded sample input data; The training loss for the current bone sample input set is calculated by using the first feature information corresponding to each bone sample input data, the second feature information corresponding to each encoded sample input data, and the corresponding behavior label, through the contrastive loss function. The parameters corresponding to the preset global point, preset graph convolutional layer, preset temporal vector, and preset Transformer encoder are updated and iterated according to the training loss, backpropagation algorithm, and optimization algorithm. Determine whether the current cumulative number of iterations exceeds the preset number of iterations, or whether the parameters corresponding to the preset global point, the preset graph convolutional layer, the preset time vector, and the preset Transformer encoder have converged; If the current cumulative number of iterations exceeds the preset number of iterations or the parameters corresponding to the preset global point, the preset graph convolutional layer, the preset time vector, and the preset Transformer encoder converge, then the target global point, the target graph convolutional layer, the target time vector, and the target Transformer encoder are obtained. If the current cumulative number of iterations does not exceed the preset number of iterations and the parameters corresponding to the preset global point, the preset graph convolutional layer, the preset time vector, and the preset Transformer encoder do not converge, then return to the step of selecting a set of bone sample input data from the multiple bone sample input data as the current bone sample input set.

7. A behavior recognition device, characterized in that, The device includes a construction module, a graph convolution module, and a recognition module; The construction module is used to construct and obtain skeletal input information based on the skeletal information to be identified and the target global point information; wherein, the target global point refers to a point in the skeletal information to be identified where the proportion of information obtained by each skeletal point is consistent. The graph convolution module is used to extract skeletal features from the skeletal input information through a target graph convolutional layer to obtain first feature information; wherein, the first feature information represents the information after the skeletal information to be identified and the target reference point information are fully integrated; The recognition module is used to recognize the behavior corresponding to the skeleton information to be recognized based on the target Transformer encoder and the first feature information; The recognition module is specifically used to extract the target global point feature information from the first feature information, and generate coded input data based on the target global point feature information and the target time vector; The target Transformer encoder extracts temporal features from the encoded input data to obtain second feature information; wherein, the second feature information represents the information after the target global point feature information and the target time vector complete information are combined; the target time vector feature information is extracted from the second feature information; the target time vector feature information is input into a multilayer perceptron to obtain the probability corresponding to each behavior type output by the multilayer perceptron; the behavior corresponding to the skeleton information to be identified is determined according to the probability corresponding to each behavior type.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images