Skeletal sequence behavior recognition method and apparatus

Abstract

The application provides a skeleton sequence behavior recognition method, comprising: obtaining only skeleton coding units corresponding to a topological structure in a to-be-processed skeleton coding frame group containing multiple skeleton coding frames, without detail key points of the topological structure; inputting the skeleton coding frame group as initial input information into a target processing model, the target processing model comprising at least two layers of processing modules, controlling a spatial dimension processing unit and a time dimension processing unit in each layer of processing modules to respectively perform processing on input information in a topological structure and between topological structures, to obtain a target processing result; and determining a behavior action corresponding to the skeleton coding frame group based on the target processing result, so that the behavior action corresponding to the skeleton coding frame group can distinguish between large movements and recognize detail movements.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and more particularly, to a skeleton sequence behavior recognition method and device. BACKGROUND

[0002] The behavior recognition model based on the transformer is a model with an encoding-decoding structure, which uses an attention mechanism to realize parallel capture of sequence dependence and simultaneously processes each position of the skeleton key point sequence, which greatly reduces the training time while achieving excellent performance.

[0003] The standard transformer uses a standard self-attention mechanism to calculate the similarity between the target key point and other key points, and then fuses the information of the other key points into the target key point according to the similarity, and the calculation complexity is the square of the number of key points n, i.e. O(n 2 ), so when all the key points of the human body are used as input, a large amount of computing power and memory will be consumed.

[0004] In order to reduce the consumption of computing power, the standard transformer can also use the skeleton sequence re-encoded based on part.

[0005] However, since the standard transformer only considers the information interaction between parts, it greatly reduces the accuracy of model discrimination while reducing the computing power, such as running and jumping, which can be distinguished through the linkage between parts, but for typing, eating and other movements that require hand detail movements, it is difficult to distinguish only by considering the linkage between parts. SUMMARY

[0006] Therefore, the present application provides a skeleton sequence behavior recognition method, as follows:

[0007] A skeleton sequence behavior recognition method, comprising:

[0008] Obtaining a skeleton encoding frame group to be processed, the skeleton encoding frame group to be processed comprising a plurality of skeleton encoding frames, each skeleton encoding frame comprising a plurality of skeleton encoding units, one skeleton encoding unit corresponding to a topological structure of an animal body, each skeleton encoding unit corresponding to a first array, the first array comprising at least one data;

[0009] Inputting the skeleton encoding frame group as initial input information into a target processing model, the target processing model comprising at least two processing modules, each processing module comprising a spatial dimension processing unit and a time dimension processing unit;

[0010] The space dimension processing unit and the time dimension processing unit in each processing module in the target processing model respectively perform topology-intra change and topology-inter change processing on input information to obtain a target processing result.

[0011] Based on the target processing result, a behavior action corresponding to the skeletal coding frame group is determined.

[0012] Optionally, the method described above, after the skeletal coding frame group to be processed is obtained, further includes:

[0013] The first array corresponding to the skeletal coding unit in the skeletal coding frame group is sequentially sorted to obtain a first skeletal coding sequence;

[0014] A first identifier is inserted in front of the first skeletal coding sequence to obtain a second skeletal coding sequence, and the second skeletal coding sequence is used as initial input information of the target processing model.

[0015] Optionally, the method described above, in which the space dimension processing unit and the time dimension processing unit in each processing module in the target processing model respectively perform topology-intra change and topology-inter change processing on input information to obtain a target processing result, includes:

[0016] Topology-intra change and topology-inter change processing are performed on input information input to a current layer in the space dimension and the time dimension respectively to obtain a processing result of the current layer;

[0017] The processing result of the current layer is used as input information of a next layer for processing in the space dimension and the time dimension until processing of each layer in the target processing model is completed, and a target processing result is output.

[0018] Optionally, the method described above, in which topology-intra change and topology-inter change processing are performed on input information input to a current layer in the space dimension and the time dimension respectively to obtain a processing result of the current layer, includes:

[0019] The space dimension processing unit is controlled to perform topology-intra change and topology-inter change processing on the input information of the current layer to obtain first output information;

[0020] The time dimension processing unit is controlled to perform topology-intra change and topology-inter change processing on the second skeletal coding sequence based on the first output information to obtain second output information, and the second output information is used as the processing result of the current layer.

[0021] Optionally, the method described above, in which the space dimension processing unit is controlled to perform topology-intra change and topology-inter change processing on the input information of the current layer to obtain first output information, includes:

[0022] The control space dimension processing unit obtains first target information from the input information of the current layer, the first target information corresponding to the position and length of the first identifier respectively;

[0023] In the space dimension, a first similarity of each second array in the first target information and first non-target information is calculated, a first result value is obtained based on the first similarity, the second array has the same length as the first array, and the first non-target information is the remaining information in the input information except the first target information;

[0024] A first change value between topologies in a plurality of skeletal coding frames is calculated in units of frames, and a second change value within each topology in the skeletal coding frame is calculated in units of topologies;

[0025] A second result value is obtained based on the first change value and the second change value;

[0026] The first output information is calculated based on the first result value and the second result value.

[0027] Optionally, the method described above controls the time dimension processing unit to obtain second output information based on the first output information for processing of intra-topology changes and inter-topology changes in the second skeletal coding sequence, including:

[0028] The time dimension processing unit adds the first output information and the second skeletal coding sequence to obtain a third skeletal coding sequence;

[0029] Second target information is obtained from the third skeletal coding sequence, the second target information corresponding to the position and length of the first identifier respectively;

[0030] In the time dimension, a third similarity of each third array in the second target information and second non-target information is calculated, a third result value is obtained based on the third similarity, the third array has the same length as the first array, and the second non-target information is the remaining information in the input information except the second target information;

[0031] A third change value between topologies in a plurality of skeletal coding frames is calculated in units of frames, and a fourth change value within each topology in the skeletal coding frame is calculated in units of topologies;

[0032] A fourth result value is obtained based on the third change value and the fourth change value;

[0033] The second output information is calculated based on the third result value and the fourth result value.

[0034] Optionally, the method described above, controlling the spatial dimension processing unit to perform intra-topology and inter-topology processing on the input information of the current layer to obtain first output information, comprises:

[0035] The spatial dimension processing unit calculates, in the spatial dimension, a fifth change value between topologies in the plurality of skeletal coding frames in units of frames, and calculates a sixth change value within each topology in the skeletal coding frame in units of topologies;

[0036] Based on the fifth change value and the sixth change value, the first output information is obtained.

[0037] Optionally, the method described above, controlling the time dimension processing unit to perform intra-topology and inter-topology processing on the second skeletal coding sequence based on the first output information to obtain second output information, comprises:

[0038] The time dimension processing unit adds the first output value to the second skeletal coding sequence to obtain a third skeletal coding sequence.

[0039] In the time dimension, a seventh change value between topologies in the plurality of skeletal coding frames is calculated in units of frames, and an eighth change value within each topology in the skeletal coding frame is calculated in units of topologies.

[0040] Based on the seventh change value and the eighth change value, the second output information is calculated.

[0041] Optionally, the method described above, based on the target processing result, obtains the behavior action corresponding to the skeletal coding frame group, comprising:

[0042] The third target information in the target processing result is obtained, and the third target information corresponds to the position and length of the first identifier, respectively.

[0043] Based on the type corresponding to the third target information, the behavior action corresponding to the skeletal coding frame group is determined.

[0044] A skeletal sequence recognition method, comprising:

[0045] receive input information, the input information being information sent by a previous structure to a current processing module in a target processing model, the target processing model comprising an input layer and at least two layers of transformer modules, each layer of transformer modules comprising a spatial dimension processing unit and a time dimension processing unit, the previous structure comprising any one of the input layer, the spatial dimension processing unit and the time dimension processing unit, the input information being related to a group of to-be-processed skeletal coding frames, the group of to-be-processed skeletal coding frames containing multiple skeletal coding frames, each skeletal coding frame comprising multiple skeletal coding units, one skeletal coding unit corresponding to one topological structure of an animal body, each skeletal coding unit corresponding to a first array, the first array comprising at least one data;

[0046] reorganize at least the input information based on a function of a current processing unit to obtain target information corresponding to the current processing unit;

[0047] convert the target information to obtain a K matrix, a Q matrix and a V matrix;

[0048] perform multi-head self-attention processing based on the K matrix, the Q matrix and the V matrix to obtain a first processing result, the first processing result representing changes between topological structures;

[0049] perform linear mapping based on the V matrix to obtain a second processing result, the second processing result representing changes within a topological structure;

[0050] obtain a third processing result based on the first processing result and the second processing result;

[0051] perform multi-head self-attention processing on the input information to obtain a fourth processing result;

[0052] reorganize the third processing result to obtain a fifth processing result, the fifth processing result corresponding to at least the input information;

[0053] obtain a processing result of the current processing unit based on the fourth processing result and the fifth processing result.

[0054] A skeletal sequence behavior recognition device comprises:

[0055] an acquisition module configured to acquire a group of to-be-processed skeletal coding frames, the group of to-be-processed skeletal coding frames containing multiple skeletal coding frames, each skeletal coding frame comprising multiple skeletal coding units, one skeletal coding unit corresponding to one topological structure of an animal body, each skeletal coding unit corresponding to a first array, the first array comprising at least one data;

[0056] a processing module configured to input the group of skeleton coding frames as initial input information into a target processing model, the target processing model comprising at least two layers of processing modules, each layer of processing modules comprising a spatial dimension processing unit and a time dimension processing unit; and control the spatial dimension processing unit and the time dimension processing unit in each layer of processing modules in the target processing model to respectively perform processing on the input information in terms of intra-topological structure change and inter-topological structure change, to obtain a target processing result.

[0057] a determination module configured to determine a behavior action corresponding to the group of skeleton coding frames based on the target processing result.

[0058] According to the technical solution described above, the present application provides a skeleton sequence behavior recognition method, which comprises: obtaining a group of skeleton coding frames to be processed, the group of skeleton coding frames to be processed comprising a plurality of skeleton coding frames, each skeleton coding frame comprising a plurality of skeleton coding units, one skeleton coding unit corresponding to one topological structure of an animal body, each skeleton coding unit corresponding to a first array, the first array comprising at least one data; inputting the group of skeleton coding frames as initial input information into a target processing model, the target processing model comprising at least two layers of processing modules, each layer of processing modules comprising a spatial dimension processing unit and a time dimension processing unit; controlling the spatial dimension processing unit and the time dimension processing unit in each layer of processing modules in the target processing model to respectively perform processing on the input information in terms of intra-topological structure change and inter-topological structure change, to obtain a target processing result; and determining a behavior action corresponding to the group of skeleton coding frames based on the target processing result. In this scheme, the group of skeleton coding frames to be processed only comprises skeleton coding units corresponding to topological structures, and does not have detailed key points of topological structures. The group of skeleton coding frames formed by the plurality of skeleton coding frames are processed in terms of intra-topological structure change and inter-topological structure change in two dimensions of spatial dimension and time dimension, and finally a behavior action corresponding to the group of skeleton coding frames is obtained, which realizes distinguishing between large movements and recognizing detailed movements. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0060] Figure 1 is a flowchart of a skeleton sequence behavior recognition method embodiment 1 provided by the present application;

[0061] Figure 2is a skeleton encoding frame schematic diagram in a skeleton sequence behavior recognition method embodiment 1 provided by the present application;

[0062] Figure 3 is a flowchart of a skeleton sequence behavior recognition method embodiment 2 provided by the present application;

[0063] Figure 4 is a flowchart of a skeleton sequence behavior recognition method embodiment 3 provided by the present application;

[0064] Figure 5 is a flowchart of a skeleton sequence behavior recognition method embodiment 4 provided by the present application;

[0065] Figure 6 is a flowchart of a skeleton sequence behavior recognition method embodiment 5 provided by the present application;

[0066] Figure 7 is a first skeleton encoding sequence X in a spatial dimension analysis and reorganization in a skeleton sequence behavior recognition method embodiment 5 provided by the present application; s a schematic diagram of a process;

[0067] Figure 8 is a process schematic diagram of a time dimension block in a layer of a transformer in a classifier model in a skeleton sequence behavior recognition method embodiment 5 provided by the present application;

[0068] Figure 9 is a process schematic diagram of a first layer of a transformer layer in a skeleton sequence behavior recognition method embodiment 4 provided by the present application;

[0069] Figure 10 is a flowchart of a skeleton sequence behavior recognition method embodiment 6 provided by the present application;

[0070] Figure 11 is a flowchart of a skeleton sequence behavior recognition method embodiment 7 provided by the present application;

[0071] Figure 12 is a flowchart of a skeleton sequence behavior recognition method embodiment 8 provided by the present application;

[0072] Figure 13 is a structural schematic diagram of a skeleton sequence behavior recognition device embodiment provided by the present application. DETAILED DESCRIPTION

[0073] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0074] As shown in Fig. 1, a flowchart of a skeletal sequence behavior recognition method embodiment 1 provided by the present application is shown, the method is applied to an electronic device, and the method comprises the following steps: Figure 1

[0075] Step S101: acquiring a skeletal coded frame group to be processed.

[0076] The skeletal coded frame group to be processed comprises a plurality of skeletal coded frames, each skeletal coded frame comprises a plurality of skeletal coding units, one skeletal coding unit corresponds to one topological structure of an animal body, each skeletal coding unit corresponds to a first array, and the first array comprises at least one data.

[0077] In the embodiments of the present application, the topological structure of a human body is taken as an example to describe the skeletal coded frame.

[0078] When the human body moves, each topological structure moves to generate a series of continuous skeletal motion information, which can be acquired by a somatosensory camera (such as a Kinect) sensor or extracted from a captured image video frame by frame by a human body key point algorithm to form a human body skeletal sequence. The re-encoding based on the organizational structure part means that the human body key points are divided into different parts according to the topological structure of the human body activity, so as to improve the calculation efficiency and the model robustness. The original human body key points are 25, and after re-encoding, there are 5 parts, each of which is a human body topological structure. Correspondingly, one skeletal coding unit in the skeletal coded frame corresponds to one re-encoded part.

[0079] The skeletal coding unit is represented by a first array composed of at least one data. Specifically, the skeletal coded frame arranges a plurality of skeletal coding units in sequence in units of frames.

[0080] For example, there are 10 skeletal coded frames, each frame has 5 skeletal coding units, and each skeletal coding unit has 2 data. Therefore, there are 100 data arranged in sequence in the skeletal coded frame group, wherein the first 10 data represent the first frame, the 11th to 20th data represent the second frame, and the 91st to 100th data represent the tenth frame.

[0081] ​As an example, each bone coding unit can correspond to a first array that includes only one data or two or more data, which can be an integer or a decimal, and the application does not limit the specific value of the data.

[0082] However, the first array corresponding to the bone coding unit contains less data relative to the original data obtained for a topology.

[0083] Figure 2 The figure shows a schematic diagram of a bone coding frame involved in the embodiment, which includes five topologies, including a torso, a left arm, a right arm, a left leg, and a right leg, which divides the head into the torso.

[0084] Specifically, the bone coding unit in the bone coding frame involved in the application is a bone coding unit obtained by dimensionally reorganizing and re-encoding the bone key points of the bone key point frame. For specific bone key point encoding methods, refer to the invention application file of the application No. 202210010474.9 invention name "bone key point encoding method, device, electronic equipment and storage medium" submitted by the applicant on January 5, 2022.

[0085] For the convenience of understanding, the subsequent embodiments of the application will be explained with human body as an example. It can be understood that the embodiments of the application can be applied not only to human body but also to other animal bodies, such as cats, monkeys, dogs, etc.

[0086] The bone coding frame group includes a plurality of bone coding frames, each frame of which refers to the Figure 2 It is shown that the bone coding frame group is generated based on a series of actions, so that any two adjacent bone coding frames are not completely the same.

[0087] Step S102: inputting the bone coding frame group as initial input information into a target processing model;

[0088] The target processing model includes at least two processing modules, each processing module including a spatial dimension processing unit and a time dimension processing unit.

[0089] In the bone coding frame group, the bone coding units in each frame have spatial correlation, and the corresponding same topology in the plurality of bone coding frames has time correlation.

[0090] The target processing model includes multiple layers of processing modules, each layer of processing module is capable of analyzing from two aspects of spatial dimension and time dimension, and the skeleton coded frame group is input as initial input information into the target processing model, and the spatial dimension and time dimension of the skeleton coded frame group are processed in each layer of processing module.

[0091] Step S103: controlling the spatial dimension processing unit and the time dimension processing unit in each layer of processing module in the target processing model to respectively perform topology-intra change and topology-inter change processing on the input information, to obtain a target processing result:

[0092] Each layer of processing module can adopt a transformer model, the transformer model includes a spatial dimension processing unit and a time dimension processing unit, which can respectively perform topology-intra change and topology-inter change processing on the input information input into the current layer of processing module, and the final target processing result is obtained through analysis of multiple layers of processing modules.

[0093] Specifically, the skeleton coded frame group is processed, self-attention between each skeleton coded unit is calculated, topology-inter change is determined, and value (feature) is locally interacted to obtain topology-intra change.

[0094] The spatial dimension processing unit analyzes and processes the topology-intra change and the topology-inter change of the input information input into the current processing unit, and the time dimension processing unit analyzes and processes the topology-intra change and the topology-inter change of the input information input into the current processing unit.

[0095] The spatial dimension processing unit and the time dimension processing unit can be arranged in any order, and preferably the same arrangement order is adopted in different layers of processing modules.

[0096] Specifically, the target processing result is a data sequence consistent with the length of the skeleton coded frame group.

[0097] The process of the processing will be described in detail in subsequent embodiments, and will not be described in detail in this embodiment.

[0098] Step S104: determining a behavior action corresponding to the skeleton coded frame group based on the target processing result.

[0099] The processing result can be mapped to multiple behavior actions, and based on specific data in the processing result, a behavior action corresponding to data satisfying a predetermined condition is selected as the behavior action corresponding to the skeleton coded frame group.

[0100] To sum up, the embodiment provides a bone sequence behavior recognition method, which comprises the following steps:

[0101] As shown in Figure 3 the flow chart of embodiment 2 of the bone sequence behavior recognition method provided by the application, the method comprises the following steps:

[0102] Step S301: acquiring a bone coding frame group to be processed;

[0103] In the embodiment, the step S301 is the same as the step S101 in the embodiment 1, and thus no further description is given.

[0104] Step S302: sequentially sorting the first arrays corresponding to the bone coding units in the bone coding frame group to obtain a first bone coding sequence;

[0105] In the embodiment, after each bone coding frame in the bone coding frame group is sorted according to the time sequence, the first arrays corresponding to the bone coding units in each bone coding frame are sorted according to the predetermined arrangement order of the bone coding units in the bone coding frame, and thus the first bone coding sequence is obtained.

[0106] In the embodiment, the first bone coding sequence comprises the spatial relationship (i.e., the association between different bone coding units in the same frame) and the time relationship (i.e., the association between the same bone coding unit in different frames).

[0107] Wherein, in order for the subsequent target processing model to correctly process the first skeletal coding sequence, a position parameter related to the target processing model is also added to the first array corresponding to each skeletal coding unit.

[0108] Wherein, the target processing model is a model that has been pre-trained.

[0109] Wherein, the position parameter is a series of parameters generated during the training of the target processing model, and the position parameter is added to the skeletal coding frame group according to the number of frames and the topology structure.

[0110] For example, there are 10 skeletal coding frames, each frame has 5 skeletal coding units, and each skeletal coding unit has 2 data, so there are 100 data arranged in order in the skeletal coding frame group, and the position parameter is added to each skeletal coding unit composed of two data.

[0111] Step S303: inserting a first identifier in front of the first skeletal coding sequence to obtain a second skeletal coding sequence;

[0112] Wherein, the second skeletal coding sequence is used as initial input information of the target processing model.

[0113] Wherein, the second skeletal coding sequence is used as initial input information of the target processing model in the next step S304.

[0114] Wherein, the first identifier is inserted in front of the first skeletal coding sequence, and the first identifier is an identifier in the same form as the skeletal coding unit, and the first identifier is an array with the same length as the first array of any skeletal coding unit.

[0115] For example, the first skeletal coding unit is composed of three data, and the first identifier is also composed of three data.

[0116] Specifically, the first identifier is a classification code, and cls_token is used to represent the first identifier in this application.

[0117] In this embodiment, the second skeletal coding sequence with the added first identifier is analyzed to obtain the behavior action corresponding to each coding frame group.

[0118] Wherein, the first identifier is inserted in front of the first skeletal coding sequence, and the first identifier is specially used as a classification identifier, so that the second skeletal coding sequence with the added first identifier is analyzed in the spatial dimension and the time dimension to obtain the behavior action corresponding to the skeletal coding frame group. Compared with the traditional space-time transformer model, the classification accuracy of the network can be improved.

[0119] Step S304: input the second skeleton coding sequence as initial input information into the target processing model;

[0120] Step S305: control the spatial dimension processing unit and the time dimension processing unit in each processing module in the target processing model to respectively perform intra-topology change and inter-topology change processing on the input information, to obtain a target processing result:

[0121] Step S306: based on the target processing result, determine the behavior action corresponding to the skeleton coding frame group.

[0122] Among them, steps S304-306 are consistent with steps S102-104 in embodiment 1, and will not be repeated here.

[0123] In summary, the skeleton sequence behavior recognition method provided in the embodiment further includes: sequentially sorting the first arrays corresponding to the skeleton coding units in the skeleton coding frame group to obtain a first skeleton coding sequence; and inserting a first identifier in front of the first skeleton coding sequence to obtain a second skeleton coding sequence, which is used as initial input information of the target processing model. In this scheme, by sorting the first arrays corresponding to the skeleton coding units in the skeleton coding frame group to obtain a first skeleton coding sequence, and inserting a first identifier in front of the first skeleton coding sequence, the first identifier is specially used as a classification identifier, so that the second skeleton coding sequence is input as initial input information into the target processing model for analysis, to obtain the behavior action corresponding to the skeleton coding frame group. Compared with the traditional space-time transformer model, the classification accuracy of the network can be improved.

[0124] As shown in Figure 4 , it is a flowchart of embodiment 3 of the skeleton sequence behavior recognition method provided in the application. The method includes the following steps:

[0125] Step S401: obtain a skeleton coding frame group to be processed;

[0126] Step S402: sequentially sort the first arrays corresponding to the skeleton coding units in the skeleton coding frame group to obtain a first skeleton coding sequence;

[0127] Step S403: insert a first identifier in front of the first skeleton coding sequence to obtain a second skeleton coding sequence;

[0128] Step S404: input the second skeleton coding sequence as initial input information into the target processing model;

[0129] Among them, steps S401-404 are consistent with steps S301-304 in embodiment 2, and will not be repeated here.

[0130] Step S405: For the input information of the current layer, the processing of intra-topological structure change and inter-topological structure change is performed in the spatial dimension and the time dimension respectively, to obtain the processing result of the current layer;

[0131] Step S406: The processing result of the current layer is taken as the input information of the next layer for processing in the spatial dimension and the time dimension, until each layer in the target processing model completes the processing, and the target processing result is output.

[0132] The processing result of each layer and the target output result are data sequences consistent with the length of the input second skeleton code sequence.

[0133] The second skeleton code sequence is taken as the initial input information input into the target processing model, so that the target processing model processes the initial input information to obtain the target processing result.

[0134] The target processing model in the present application is stacked by multiple processing modules, the processing module adopts a transformer, and each transformer in each layer includes a spatial attention block and a time attention block. The spatial attention block corresponds to the spatial dimension processing unit in the present application, which can analyze and process the intra-topological structure change and the inter-topological structure change of the input information in the spatial dimension. The time attention block corresponds to the time dimension processing unit in the present application, which can analyze and process the intra-topological structure change and the inter-topological structure change of the input information in the time dimension.

[0135] The target processing model is specifically a classifier model.

[0136] Specifically, in the classifier model for processing the second skeleton code sequence, each transformer model outputs the processing result to the next layer after one processing, until all layers are processed.

[0137] The target processing model includes multiple processing modules. The first layer processing module of the target processing model processes the initial input information (the second skeleton code sequence) as input information, and the non-first layer processing module processes the processing result of the previous layer processing module as input information.

[0138] Specifically, after receiving the input information, each processing module processes the input information in the spatial dimension and the time dimension to obtain the processing result of the current layer.

[0139] The data sequence of the processing result obtained by each processing module processing the input information is a data sequence consistent with the length of the second skeleton code sequence.

[0140] For example, the second skeleton coding sequence is a sequence of 100 numbers, and the processing result obtained by each processing module is also a sequence of 100 numbers, but the sequence is different from the value of the second skeleton coding sequence.

[0141] Wherein, after each layer processing module processes the input information inputted by it in spatial dimension and time dimension, the processing result of the last layer is taken as the target processing result of the target processing model.

[0142] Step S407: determining the action corresponding to the skeleton coding frame group based on the target processing result.

[0143] Wherein, step S407 is consistent with step S306 in embodiment 2, and will not be repeated here.

[0144] In summary, the skeleton sequence behavior recognition method provided in the embodiment includes: processing the input information inputted by the current layer in spatial dimension and time dimension respectively to obtain the processing result of the current layer; processing the processing result of the current layer in spatial dimension and time dimension as the input information of the next layer until each layer in the target processing model is processed, and outputting the target processing result. In the scheme, the multiple layers of processing modules in the target processing model respectively process the input information inputted by it in spatial dimension and time dimension, and input the processing result as the input information of the next layer, and the processing is executed in a loop until all layers are processed.

[0145] As shown in the flow chart of the skeleton sequence behavior recognition method provided in the embodiment 4 of the application, the method includes the following steps: Figure 5

[0146] Step S501: obtaining a skeleton coding frame group to be processed;

[0147] Step S502: sequentially sorting the first array corresponding to the skeleton coding unit in the skeleton coding frame group to obtain a first skeleton coding sequence;

[0148] Step S503: inserting a first identifier in front of the first skeleton coding sequence to obtain a second skeleton coding sequence;

[0149] Step S504: inputting the second skeleton coding sequence as initial input information into a target processing model;

[0150] Wherein, steps S501-504 are consistent with steps S401-404 in embodiment 3, and will not be repeated here.

[0151] ​Step S505: control the spatial dimension processing unit to perform intra-topology change and inter-topology change processing on the input information of the current layer to obtain first output information;

[0152] Step S506: control the temporal dimension processing unit to perform intra-topology change and inter-topology change processing on the second skeletal coded sequence based on the first output information to obtain second output information, which is the processing result of the current layer.

[0153] In this embodiment, the order of the two processing units in the processing module of each layer is the spatial dimension processing unit and the temporal dimension processing unit.

[0154] Correspondingly, after receiving the input information input into the processing module of the current layer, the spatial dimension processing unit is first controlled to perform intra-topology change and inter-topology change processing on the input information to obtain first output information.

[0155] Then, the first output information obtained by the spatial dimension processing unit is combined with the second skeletal coded sequence, specifically, the two data sequences are added to obtain a new data sequence, intra-topology change and inter-topology change processing is performed on the new data sequence to obtain second output information, and the second output information is taken as the processing result of the current layer.

[0156] It should be noted that if the order of the processing units in the processing module of each layer is the temporal dimension processing unit and the spatial dimension processing unit, the input information input into the current layer is first processed by the temporal dimension processing unit, the processing result of the temporal dimension processing unit is added to the second skeletal coded sequence, and the spatial dimension processing unit processes the obtained new data sequence to obtain the processing result of the current layer.

[0157] Step S507: perform spatial dimension and temporal dimension processing on the processing result of the current layer as input information of the next layer until the processing of each layer in the target processing model is completed, and output a target processing result.

[0158] Step S508: determine the behavior action corresponding to the skeletal coded frame group based on the target processing result.

[0159] Steps S507-508 are the same as steps S406-407 in Embodiment 3, and will not be described here.

[0160] In conclusion, the method for recognizing bone sequence behavior provided in the embodiment can process the input information of the current layer in the spatial dimension to obtain first output information, combine the first output information with the second bone coding sequence, process the combined information in the time dimension based on the time dimension processing unit of the processing module to obtain the processing result of the current layer, and process the input information of the current layer in the topology structure and between the topology structures in the spatial dimension and the time dimension, so that the accurate action containing the topology structure and the topology structure between the topology structures is recognized.

[0161] As shown in Figure 6 FIG. 5 is a flowchart of an embodiment 5 of the method for recognizing bone sequence behavior provided in the application. The method comprises the following steps:

[0162] Step S601: Obtain a bone coding frame group to be processed.

[0163] Step S602: Sort the first array corresponding to the bone coding unit in the bone coding frame group in sequence to obtain a first bone coding sequence.

[0164] Step S603: Insert a first identifier in front of the first bone coding sequence to obtain a second bone coding sequence.

[0165] Step S604: Input the second bone coding sequence as initial input information into a target processing model.

[0166] Steps S601-604 are the same as steps S501-504 in the embodiment 4, and will not be described herein.

[0167] Step S605: Control the spatial dimension processing unit to obtain first target information from the input information of the current layer.

[0168] The position and length of the first target information correspond to the position and length of the first identifier, respectively.

[0169] The first target information is obtained from the input information of the current layer, and the first target information has the same position and length as the first identifier inserted in front of the first bone coding sequence.

[0170] Specifically, the position of the first identifier is at the front of the second bone coding sequence, and the data length of the first identifier is consistent with the data length of each bone coding unit. Correspondingly, the first target information is the first several data in the input information of the current layer, and the number of the first target information is consistent with the data length of the first identifier.

[0171] Specifically, when the current layer is the first layer of the target processing model, the first target information is the first identifier, and when the current layer is a non-first layer of the target processing model, the first target information is the first several data in the input information input into the current layer, and the number of the first target information is consistent with the data length of the first identifier.

[0172] For example, the first skeletal coding sequence has N skeletal coding units (N is an integer greater than 1), each skeletal coding unit can be used as a token of a topological structure, the first identifier can be used as the N+1 token, and the second skeletal coding sequence has N+1 tokens.

[0173] In this way, since the first identifier is a token specially used for classification, in the analysis process, the processing module of each layer processes the first identifier and the target information at the corresponding position independently.

[0174] Step S606: Control the spatial dimension processing unit to calculate, in the spatial dimension, a first similarity between the first target information and each second array in the first non-target information, and obtain a first result value based on the first similarity.

[0175] The second array has the same length as the first array, and the first non-target information is the remaining information in the input information except the first target information.

[0176] The steps S606-609 are the processing procedures performed by the spatial dimension processing unit.

[0177] The input information is divided into two parts, including the first target information and the first non-target information, and the first non-target information includes a plurality of second arrays, each second array has a length corresponding to a first array of a skeletal coding unit, and each second array can represent the data of the skeletal coding unit in the current layer.

[0178] Specifically, based on the skeletal coding units in each skeletal coding frame, the second array is used as the array corresponding to the topological structure of the current layer, and the first non-target information is mapped into the K matrix, the Q matrix, and the V matrix of the attention model based on the frame as the mapping basis. The first target information X cls The K matrix and the V matrix use a multi-head self-attention mechanism to calculate the similarity between the first target information and other second arrays, and then calculate the weighted sum of each topological structure to obtain the first result value.

[0179] The following formula is used for calculation:

[0180]

[0181] K, V = linear(X p ) (2)

[0182] wherein cls represents the first target information, T represents taking transpose, p represents the pthtopology structure, and C represents the number of features of each topology structure (corresponding to the skeletal coding unit data).

[0183] The first result value calculated is

[0184] Step S607: Calculate the first change value between topology structures in the plurality of skeletal coding frames in units of frames, and calculate the second change value within each topology structure in the skeletal coding frame in units of topology structures.

[0185] Specifically, the first non-target information (corresponding to the first skeletal coding sequence after inserting the position coding) (using X s ) is reorganized, such as converting the first non-target information from BxNxC to BxCxAxP, wherein B represents the number of sequences, one sequence represents one action, one skeletal sequence coding can include multiple actions, N represents the number of skeletal coding units in the skeletal coding sequence, and C represents the number of features of each skeletal coding unit.

[0186] For example, the first skeletal coding sequence is Bx50xC, a total of 10 skeletal coding frames, each frame containing 5 skeletal coding units, and after reorganizing the first non-target information, the first non-target information is BxCx5x10.

[0187] Wherein the reorganization of the non-target information is related to the time dimension and the spatial dimension, the time dimension corresponds to the time sequence of each skeletal coding frame, and the spatial dimension corresponds to the same topology structure within each skeletal coding frame.

[0188] Further, based on the reorganized first non-target information, the first change value between skeletal coding units is calculated in the spatial dimension.

[0189] Specifically, based on the skeletal coding units in each skeletal coding frame, the second array is used as the topology structure corresponding to the skeletal coding unit data of the current layer, and the reorganized first non-target information sequence is mapped to the K matrix, the Q matrix, and the V matrix of the attention model according to the frame as the mapping basis, wherein the K matrix and the Q matrix and the multi-head attention algorithm are used to calculate the similarity between the topology structures and the weighted sum to obtain the first change value The V matrix is linearly mapped to obtain the second change value within the topology structure

[0190] The following formula is used,

[0191]

[0192]

[0193] K, Q, V = linear(X s ) (5)

[0194] The first change value and the second change value

[0195] Step S608: Based on the first change value and the second change value, a second result value is obtained.

[0196] Wherein, the first change value between the topological structures and the second change value within each topological structure are added to obtain the first result value.

[0197] For example, the first change value is represented by , and the second change value is represented by , and the two change values are added together, wherein the addition result is represented by ,

[0198] Wherein,

[0199] As shown in Figure 7 , the first non-target information X s after spatial dimension analysis and reorganization, wherein the first non-target information X s after analysis and reorganization is mapped to the K matrix, the Q matrix and the V matrix of the attention model; based on the K matrix, the Q matrix and the V matrix, multi-head self-attention calculation and weighted sum are performed to obtain the first change value between the topological structures and the linear mapping (linear) is performed on the V matrix to obtain the second change value , and the two are added to obtain the output result

[0200] Step S609: Based on the first result value and the second result value, the first output information is calculated.

[0201] Wherein, the first result value and the second result value are spliced together to obtain the final output result of the spatial attention block of the spatial dimension.

[0202] Wherein, the final output result is represented by , the first result value is represented by , the second result value is represented by , and the following formula is used for calculation:

[0203]

[0204] Step S610: control the time dimension processing unit to add the first output information and the second skeletal code sequence to obtain a third skeletal code sequence;

[0205] Wherein, the process of the time dimension for the second skeletal code sequence is similar to the process of the spatial dimension analysis, the difference between the two is only the input sequence, the process of the time dimension analysis of the second skeletal code sequence combines the first output value of the spatial dimension analysis.

[0206] Wherein, steps S610-615 are the processing procedures executed by the time dimension processing unit.

[0207] Wherein, the first output information processed by the last spatial attention block is added to the second skeletal code sequence to obtain the third skeletal code sequence.

[0208] Wherein, the length of the first output information is the same as that of the second skeletal code sequence, and each data in the first output information is added to each data in the second skeletal code sequence in sequence to obtain the third skeletal code sequence.

[0209] Step S611: obtain second target information from the third skeletal code sequence;

[0210] Wherein, the second target information corresponds to the position and length of the first identifier respectively.

[0211] Wherein, the second target information is obtained from the third skeletal code sequence, and the length of the first identifier and the length of each skeletal unit in the input information are the same.

[0212] Specifically, the position of the first identifier is at the front of the second skeletal code sequence, and the data length of the first identifier is consistent with the data length of each skeletal unit, and accordingly, the second target information is the first several data in the input information at the current layer, and the number of the second target information is consistent with the data length of the first identifier.

[0213] Step S612: in the time dimension, calculate the third similarity of each third array in the second target information and the second non-target information, and obtain a third result value based on the third similarity;

[0214] Wherein, the length of the third array is the same as that of the first array, and the second non-target information is the remaining information in the input information except the second target information.

[0215] The third skeleton coding sequence is divided into two parts, including second target information and second non-target information, wherein the second non-target information includes a plurality of third arrays, each third array has a length corresponding to a first array of a skeleton coding unit, and each third array can represent the topology data in the current layer.

[0216] Specifically, based on the skeleton coding units in each frame of the skeleton coding frame, the second array is used as the topology corresponding to the skeleton coding unit of the current layer, and each topology is used as a mapping basis to map the second non-target information into a K matrix, a Q matrix and a V matrix of an attention model. cls The second target information and the K matrix and the V matrix are calculated using a multi-head self-attention mechanism to calculate the similarity between the second target information and the topology represented by the third array, and then the similarity is used as a weight to calculate a third change value of each topology by weighted sum.

[0217] The calculation process involves formulas (1) and (2).

[0218] Step S613: Calculate a third change value between topologies in a plurality of frames of skeleton coding frames in a frame unit, and calculate a fourth change value within a topology in a topology unit.

[0219] Specifically, the second non-target information is first reorganized, and then the reorganized second non-target information is used to calculate the third change value between topologies and the fourth change value within each topology.

[0220] Specifically, based on the skeleton coding units in each frame of the skeleton coding frame, the second array is used as the topology corresponding to the skeleton coding unit of the current layer, and each topology is used as a mapping basis to map the second non-target information into a K matrix, a Q matrix and a V matrix of an attention model. The K matrix and the Q matrix and a multi-head attention algorithm are used to calculate the similarity between topologies and calculate a weighted sum to obtain a third change value (which can also be represented by ).

[0221] The calculation process involves formulas (3), (4) and (5).

[0222] Step S614: Obtain a second result value based on the third change value and the fourth change value.

[0223] The third change value between topologies and the fourth change value within each topology are added to obtain a fourth result value.

[0224] The formulas involved in this calculation process refer to formula (6).

[0225] Step S615: Calculate and obtain second output information based on the third result value and the fourth result value;

[0226] Among them, the third result value and the fourth result value are spliced ​​together to obtain the final output result of the time attention block of this time dimension.

[0227] The third result value can also be used Indicates that the fourth result value can also be used The formula reference (7) involved in the calculation process is shown.

[0228] The second output information is the processing result of the current layer.

[0229] Step S616: using the processing result of the current layer as the input information of the next layer to process the spatial dimension and the temporal dimension until each layer in the target processing model is processed and the target processing result is output;

[0230] Specifically, the second output information obtained in step S606 is used as the input information of the next layer, and the spatial dimension and time dimension processing are continued in the next layer until the last layer. The second output information output by the processing module in the last layer is the target processing result of the target processing model.

[0231] like Figure 8 The figure shows a schematic diagram of the process of processing the skeleton coding sequence by the time dimension block in a transformer layer in the classifier model provided in this embodiment, wherein the third skeleton coding sequence X is received. p , split out the second target information X cls After that, the remaining second non-target information is mapped into the K matrix, Q matrix, and V matrix of the attention model by column mapping, and the second target information X cls The result value is calculated using the multi-head self-attention algorithm with the K matrix and the V matrix The calculation process refers to formulas (1) and (2); the second target information is reorganized to obtain X s , the X s Input the IIPA processing module and get the output result value The result value After reorganization, the result value is obtained After the dimensions are consistent, the result value and the resulting value Concatenate them together to get the final output The specific splicing process refers to formula (7). Among them, the IIPA processing module processes X s Process Reference Figure 6 The specific process is as follows: s The column mapping is used as the K matrix, Q matrix, and V matrix of the attention model. The multi-head self-attention algorithm is used based on the K matrix, Q matrix, and V matrix to obtain the change value. Do linear mapping based on the V matrix to get the change value The calculation process refers to formulas (3), (4) and (5), and the result is obtained by adding the two together.

[0232] In the specific implementation, when the spatial dimension processing module of the transformer is used, the final output result value of the time dimension is converted to After adding the received second skeleton code sequence, the above Figure 7 Similar process.

[0233] like Figure 9 The figure shows the process of the first transformer layer. p , input the transformer layer, the transformer layer includes a spatial dimension processing unit and a temporal dimension processing unit, the second skeleton encoding sequence serves as the input of the spatial dimension processing unit, the output of the spatial dimension processing unit is added to the second skeleton encoding sequence as the input of the temporal dimension processing unit, the output of the temporal dimension processing unit is added to the second skeleton encoding sequence as the output of the transformer layer, and output to the next layer.

[0234] Step S617: Based on the target processing result, determine the behavior action corresponding to the skeleton encoding frame group.

[0235] Among them, steps S616-617 are consistent with steps S406-407 in Example 3 and are not repeated in this embodiment.

[0236] In summary, in a skeletal sequence behavior recognition method provided by this embodiment, the specific process of the spatial dimension unit performing intra-topological changes and inter-topological changes on the input information of the current layer to obtain the first output information is described in detail, and based on the first output information and the second skeletal coding sequence as input information, the process of the time dimension unit performing intra-topological changes and inter-topological changes on its input information to obtain the second output information is described in detail.

[0237] like Figure 10As shown, a flowchart of a skeletal sequence behavior recognition method embodiment 6 provided in the present application includes the following steps:

[0238] Step S1001: Obtain a skeletal coding frame group to be processed.

[0239] Step S1002: Sort the first array corresponding to the skeletal coding unit in the skeletal coding frame group in sequence to obtain a first skeletal coding sequence.

[0240] Step S1003: Insert a first identifier in front of the first skeletal coding sequence to obtain a second skeletal coding sequence.

[0241] Step S1004: Input the second skeletal coding sequence as initial input information into a target processing model.

[0242] Step S1005: Perform topology-intra-structure change and topology-inter-structure change processing on the input information input into the current layer in the spatial dimension and the time dimension, respectively, to obtain the processing result of the current layer.

[0243] Step S1006: Perform spatial dimension and time dimension processing on the processing result of the current layer as input information of the next layer until each layer in the target processing model is processed to output a target processing result.

[0244] Among them, steps S1001-1006 are consistent with steps S401-406 in embodiment 3, and this embodiment will not be described in detail.

[0245] Step S1007: Obtain third target information in the target processing result.

[0246] Among them, the third target information and the position and length of the first identifier correspond respectively.

[0247] Among them, the target processing result is a number sequence with the same length as the input second skeletal coding sequence.

[0248] Specifically, since the inserted first identifier is a cls-token dedicated to classification, accordingly, after processing by the multi-layer processing module, only the data corresponding to the first identifier needs to be analyzed.

[0249] Correspondingly, in the target processing result, the third target information corresponding to the position and length of the first identifier is obtained.

[0250] Step S1008: Determine the behavior action corresponding to the skeletal coding frame group based on the type corresponding to the third target information.

[0251] The target processing model further includes a fully connected layer (FC) disposed after a last processing module (transformer layer) of the target processing model. The FC analyzes third target information obtained from the target processing result and determines a corresponding action.

[0252] The third target information includes a probability value.

[0253] Specifically, the third target information indicates that the action corresponding to the input bone coding frame group belongs to 10 categories, and the third target information represents probability values of the 10 categories, each probability value representing a possibility of the action corresponding to the bone coding frame group.

[0254] The category with the largest probability value is determined as the action corresponding to the bone coding frame group.

[0255] In summary, the bone sequence action recognition method provided in this embodiment includes: obtaining third target information in a target processing result, the third target information corresponding to positions and lengths of the first identifiers, respectively; and determining an action corresponding to a bone coding frame group based on a type corresponding to the third target information. In this scheme, only part of the target processing result output by the target processing model is needed to determine the action corresponding to the bone coding frame group, the data processing amount is small, and the classification accuracy is high.

[0256] As shown in Figure 11 FIG. 7 is a flowchart of a bone sequence action recognition method embodiment 7 provided by the present application. The method includes the following steps:

[0257] Step S1101: obtaining a bone coding frame group to be processed;

[0258] Step S1102: sequentially sorting first arrays corresponding to bone coding units in the bone coding frame group to obtain a first bone coding sequence;

[0259] Steps S1101-S1102 are the same as steps S401-S402 in embodiment 3. For specific explanations, please refer to the corresponding steps in embodiment 2. This embodiment will not be described in detail.

[0260] Step S1103: inputting the first bone coding sequence as initial input information into a target processing model;

[0261] It should be noted that in this embodiment, the first bone coding sequence is not inserted with a first identifier, and the first bone coding sequence is directly inputted as initial input information into the target processing model for processing.

[0262] wherein the first skeletal coding sequence is a skeletal coding sequence coded by the insertion position.

[0263] Specifically, the first skeletal coding sequence is input as initial input information into the first layer processing module of the target processing model,

[0264] Step S1104: processing the input information of the current layer in the spatial dimension and the temporal dimension respectively to obtain the processing result of the current layer.

[0265] wherein the processing result of each layer and the target output result are data sequences consistent with the length of the input first skeletal coding sequence.

[0266] The first skeletal coding sequence is input as initial input information into the target processing model, so that the target processing model processes the initial input information to obtain the target processing result.

[0267] The target processing model in the present application is stacked by multiple layers of processing modules, the processing module adopts a transformer, and each layer of the transformer includes a spatial attention block and a temporal attention block. The spatial attention block corresponds to the spatial dimension processing unit in the present application, which can analyze and process the input information in the spatial dimension for topology intra-change and topology inter-change. The temporal attention block corresponds to the temporal dimension processing unit in the present application, which can analyze and process the input information in the temporal dimension for topology intra-change and topology inter-change.

[0268] wherein the target processing model is specifically a classifier model.

[0269] Specifically, in the classifier model for processing the first skeletal coding sequence, each layer of the transformer model outputs the processing result to the next layer after one processing, until all layers are processed.

[0270] wherein the target processing model includes multiple layers of processing modules, the first layer processing module of the target processing model processes the initial input information (the first skeletal coding sequence) as input information, and the non-first layer processing module processes the processing result of the previous layer as input information.

[0271] Specifically, after receiving the input information, each layer of the processing module processes it in the spatial dimension and the temporal dimension to obtain the processing result of the current layer.

[0272] The processing module of each layer processes the input information, and the data sequence of the processing result obtained is a data sequence consistent with the length of the first skeletal code sequence.

[0273] The processing result of the last layer is the target processing result of the target processing model after the input information input into each layer is processed in the spatial dimension and the time dimension.

[0274] In the embodiment, the order of the two processing units in the processing module of each layer is a spatial dimension processing unit and a time dimension processing unit.

[0275] It should be noted that if the order of the processing units in the processing module of each layer is a time dimension processing unit, a spatial dimension processing unit, the input information input into the current layer is first processed by the time dimension processing unit, the processing result of the time dimension processing unit is added to the second skeletal code sequence, and the spatial dimension processing unit processes the new data sequence to obtain the processing result of the current layer.

[0276] Specifically, the step S1104 includes:

[0277] Step S11041: control the spatial dimension processing unit to process the input information of the current layer in the topology to obtain first output information;

[0278] The step S11041 includes:

[0279] Step S110411: control the spatial dimension processing unit to calculate the fifth change value between topologies in the plurality of skeletal code frames in the spatial dimension in units of frames, and calculate the sixth change value within each topology in the skeletal code frame in units of topologies;

[0280] The input information input into the current layer is first divided into a second number corresponding to the first array of skeletal code units, and each second array can represent the data of the skeletal code unit in the current layer.

[0281] Specifically, the input information of the current layer (corresponding to the first skeletal code sequence after the insertion position coding) is reorganized (using X s is converted into BxCxAxP, where B represents the number of sequences, one sequence represents one action, and one skeletal code sequence can include multiple actions, N represents the number of skeletal code units in the skeletal code sequence, and C represents the number of features of each skeletal code unit.

[0282] For example, the first skeleton coding sequence is B×50×C, which is 10 skeleton coding frames in total. Each frame contains 5 skeleton coding units. After the input information is reorganized, it is B×C×5×10.

[0283] Furthermore, a first change value between skeletal encoding units is calculated in a spatial dimension based on the reorganized input information.

[0284] Specifically, based on the skeleton coding unit in each skeleton coding frame, correspondingly, the second array is used as the topological structure corresponding to the skeleton coding unit data of the current layer, and the frame is used as the mapping basis. The reorganized input information is mapped to the K matrix, Q matrix, and V matrix of the attention model, wherein the K matrix and Q matrix and the multi-head attention algorithm are used to calculate the weighted sum of the similarities between the topological structures to obtain the fifth change value Perform a linear mapping on the V matrix to obtain the sixth change value in the topological structure

[0285] Using the following formula,

[0286]

[0287]

[0288] K,Q,V=linear(X s ) (10)

[0289] The first change value is calculated based on the above formulas (8)-(10): and the second change value

[0290] Step S110412: Obtain first output information based on the fifth change value and the sixth change value.

[0291] The fifth change value between the topological structures and the sixth change value within each topological structure are added together to obtain the first output information.

[0292] For example, the first change value is Indicates that the second change value adopts Indicates that the two change values ​​are added together, where the output result of this spatial dimension processing unit is express,

[0293] in,

[0294] Step S11042: Control the time dimension processing unit to process the first skeleton coding sequence for intra-topological changes and inter-topological changes based on the first output information to obtain second output information, and the second output information is used as the processing result of the current layer.

[0295] Wherein, step S11042 includes:

[0296] Step S110421: controlling the time dimension processing unit to add the first output value to the first skeleton code sequence to obtain a fourth skeleton code sequence;

[0297] Among them, the first output information obtained by processing the previous spatial attention block and the first skeleton coding sequence are added together to obtain a fourth skeleton coding sequence.

[0298] Among them, the first output information has the same length as the first skeleton coding sequence, and each data in the first output information and each data in the first skeleton coding sequence are added in sequence according to the arrangement order to obtain the fourth skeleton coding sequence.

[0299] Step S110422: In the time dimension, based on the fourth skeleton coding sequence, a seventh change value between topological structures in multiple skeleton coding frames is calculated in units of frames, and an eighth change value within each topological structure in the skeleton coding frame is calculated in units of topological structures;

[0300] Specifically, the second array is reorganized first, and then the third change value between the topological structures and the fourth change value within each topological structure are calculated based on the reorganized second array.

[0301] Specifically, based on the skeleton coding unit in each skeleton coding frame, correspondingly, the second array is used as the topological structure corresponding to the skeleton coding unit of the current layer, and each topological structure is used as the mapping basis. The second array is mapped to the K matrix, Q matrix, and V matrix of the attention model, wherein the K matrix and Q matrix and the multi-head attention algorithm are used to calculate the similarity between each topological structure and calculate the weighted sum to obtain the seventh change value (which can also be used Represented), linear mapping is performed on the V matrix to obtain the eighth change value in the topological structure (it can also be used express).

[0302] The formulas involved in this calculation process refer to formulas (8)-(10).

[0303] Step S110423: Based on the seventh change value and the eighth change value, the second output information is calculated and used as the processing result of the current layer.

[0304] The seventh change value between the topologies and the eighth change value within each topology are added to obtain second output information.

[0305] The calculation process involves a formula as shown in formula (11).

[0306] Step S1105: The processing result of the current layer is taken as input information of the next layer for spatial dimension and time dimension processing until each layer processing in the target processing model is completed, and a target processing result is output.

[0307] The step S1105 is consistent with the step S406 in the embodiment 3, and the specific explanation can be referred to the corresponding step in the embodiment 3, which will not be repeated here.

[0308] Step S1106: Based on the target processing result, a behavior action corresponding to the skeleton coding frame group is determined.

[0309] The target processing model further has a full connection layer, which is specifically arranged after the last layer processing module (transformer layer) of the target processing model, and can analyze the target processing result to determine the corresponding action.

[0310] Specifically, the full connection layer converts the multi-dimensional data array (array with multiple rows and columns) output by the last layer processing module into a one-dimensional data array.

[0311] Based on the data of each predetermined number of positions in the one-dimensional data array, the action category corresponding to the skeleton coding frame group is determined, and based on the numerical value of each position in the one-dimensional data array, the probability of the corresponding action type is determined, and then the category with the maximum probability value is selected as the behavior action corresponding to the skeleton coding frame group.

[0312] In summary, the skeleton sequence behavior recognition method provided in the embodiment sorts the first array corresponding to the skeleton coding unit in the skeleton coding frame group to obtain a first skeleton coding sequence, inputs the first skeleton coding sequence as initial input information into a target processing model, and processes the input information of the current layer in the spatial dimension to obtain first output information based on the spatial dimension processing unit arranged in any layer processing module. The first output information and the first skeleton coding sequence are combined, and the combined information is processed in the time dimension to obtain the processing result of the current layer based on the time dimension processing unit arranged in the processing module. The spatial dimension and the time dimension are combined, the input information of the current layer is processed in the topology structure and the topology structure, and the accurate action containing the topology structure and the topology structure is realized for the final recognition action.

[0313] As shown in Figure 12 Figure 8 is a flowchart of a method embodiment 8 for recognizing a skeletal sequence provided by the present application, which comprises the following steps:

[0314] Step S1201: receiving input information;

[0315] Wherein the input information is information sent by a previous structure to a current processing module in a target processing model, the target processing model comprising an input layer and at least two layers of transformer modules, each layer of transformer modules comprising a spatial dimension processing unit and a time dimension processing unit.

[0316] Wherein the previous structure comprises any one of the input layer, the spatial dimension processing unit and the time dimension processing unit, the input information being related to a skeletal coded frame group to be processed, the skeletal coded frame group comprising a plurality of skeletal coded frames, each skeletal coded frame comprising a plurality of skeletal coded units, one skeletal coded unit corresponding to one topological structure of an animal body, each skeletal coded unit corresponding to a first array, the first array comprising at least one data.

[0317] Wherein the interpretation of the skeletal coded frame group is as explained in Embodiment 1, which is not repeated here.

[0318] Wherein the initial input information input into the target processing model can comprise a first skeletal coded sequence or a second skeletal coded sequence.

[0319] Correspondingly, when the structure executing the present scheme is the first unit (the spatial dimension processing unit or the time dimension processing unit) of the first layer of transformer modules of the target processing model, the input information is the first skeletal coded sequence or the second skeletal coded sequence.

[0320] Correspondingly, when the structure executing the present scheme is the first unit (the spatial dimension processing unit or the time dimension processing unit) of the non-first layer of transformer modules of the target processing model, the input information is the sum of the first skeletal coded sequence or the second skeletal coded sequence and the processing result of the previous layer of transformer modules.

[0321] Correspondingly, when the structure executing the present scheme is the second unit (the time dimension processing unit or the spatial dimension processing unit) of any layer of transformer modules of the target processing model, the input information is the sum of the first skeletal coded sequence or the second skeletal coded sequence and the first output information of the first unit of the layer of transformer modules.

[0322] ​​​​​​​​​​​​​​​​​​​​Step S1202: reorganize at least the input information based on the function of the current processing unit to obtain target information corresponding to the current processing unit;

[0323] If the target processing model takes the first skeletal coding sequence as initial input information, then the input information is directly reorganized based on the function of the current processing unit to obtain the target information.

[0324] If the target processing model takes the second skeletal coding sequence as initial input information, then the input information is first split to obtain the first target information or the second target information corresponding to the first identifier, and the first non-target information or the second non-target information is reorganized based on the function of the current processing unit to obtain the target information.

[0325] The reorganization process can refer to the explanation in the foregoing embodiments, which will not be repeated here.

[0326] Step S1203: convert the target information to obtain K matrix, Q matrix and V matrix;

[0327] Specifically, the first non-target information or the second non-target information (corresponding to the first skeletal coding sequence after the insertion position coding) is reorganized, such as being converted from BxNxC to BxCxAxP, where B represents the number of sequences, one sequence represents one action, one skeletal coding sequence can include multiple actions, N represents the number of skeletal coding units in the skeletal coding sequence, and C represents the number of features of each skeletal coding unit. s

[0328] Step S1204: perform multi-head self-attention processing based on the K matrix, the Q matrix and the V matrix to obtain a first processing result, which represents the change between topological structures;

[0329] Step S1205: perform linear mapping based on the V matrix to obtain a second processing result, which represents the change within a topological structure;

[0330] Specifically, based on the skeletal coding units in each skeletal coding frame, the reorganized first non-target information sequence or the second non-target information sequence is mapped to the K matrix, the Q matrix and the V matrix of the attention model as the topological structure corresponding to the skeletal coding unit data of the current layer based on the frame, where the K matrix and the Q matrix and the multi-head attention algorithm are used to calculate the similarity between topological structures and obtain the first processing result The V matrix is linearly mapped to obtain the second processing result within the topological structure ​

[0331] The specific formula is (3)-(5) or (8)-(10) in the foregoing embodiments.

[0332] Step S1206: obtaining a third processing result based on the first processing result and the second processing result.

[0333] The third processing result is obtained by adding the first processing result and the second processing result.

[0334] The specific formula is (6) in the foregoing embodiments.

[0335] Step S1207: performing multi-head self-attention processing on the input information to obtain a fourth processing result.

[0336] The target processing model is based on the first skeleton encoding sequence to convert to obtain the K matrix and the V matrix when the first skeleton encoding sequence is used as the initial input information, and the fourth processing result is obtained by performing multi-head self-attention processing based on the K matrix and the V matrix.

[0337] When the second skeleton encoding sequence is used as the initial input information, the input information is first split to obtain the first target information or the second target information corresponding to the first identifier, and the K matrix and the V matrix are converted from the first non-target information or the second non-target information, and the fourth processing result is obtained by performing multi-head self-attention processing based on the K matrix and the V matrix and the first target information / second target information.

[0338] The specific formula is (1) and (2) in the foregoing embodiments.

[0339] Step S1208: obtaining a fifth processing result by reorganizing the third processing result, and the fifth processing result at least corresponds to the input information.

[0340] The third processing result is reorganized so that the matrix dimension of the obtained fifth processing result is consistent with the input information before reorganization.

[0341] Step S1209: obtaining a processing result of the current processing unit based on the fourth processing result and the fifth processing result.

[0342] The first result value and the second result value are spliced together to obtain the final output result of the structure executing the embodiment.

[0343] The specific formula is (7) in the foregoing embodiments.

[0344] In summary, in a skeletal sequence behavior recognition method provided by this embodiment, the processing unit of the processing module of any layer in the target processing model receives input information, and reorganizes the input information at least based on the function of the current processing unit to obtain target information corresponding to the current processing unit; obtains a K matrix, a Q matrix and a V matrix based on the target information conversion; performs multi-head self-attention processing based on the K matrix, the Q matrix and the V matrix to obtain a first processing result, and the first processing result represents the changes between topological structures; performs linear mapping based on the V matrix to obtain a second processing result, and the second processing result represents the changes within the topological structure; obtains a third processing result based on the first processing result and the second processing result; performs multi-head self-attention processing on the input information to obtain a fourth processing result; obtains a fifth processing result obtained by reorganizing the third processing result, and the fifth processing result at least corresponds to the input information; obtains the processing result of the current processing unit based on the fourth processing result and the fifth processing result. In this solution, the process of processing the input received by the processing unit in each layer of the processing module in the target processing model is introduced in detail. Based on the transformer module adopted by the processing module and the improvement of the transformer module algorithm, the processing of the changes within the topological structure and between the topological structures of its input information is realized. Based on the target processing model, the precise actions corresponding to the re-encoded skeletal coding frame group can be identified, thereby improving the recognition accuracy.

[0345] Corresponding to the above-mentioned embodiment of a skeleton sequence behavior recognition method provided by this application, this application also provides an embodiment of a device applying the skeleton sequence behavior recognition method.

[0346] like Figure 13 The figure shows a schematic diagram of a skeleton sequence behavior recognition device embodiment 1 provided by the present application. The device includes the following structures: an acquisition module 1301, a processing module 1302 and a determination module 1303;

[0347] The acquisition module 1301 is configured to acquire a skeleton coding frame group to be processed, wherein the skeleton coding frame group to be processed includes multiple skeleton coding frames, each skeleton coding frame includes multiple skeleton coding units, each skeleton coding unit corresponds to a topological structure of an animal body, each skeleton coding unit corresponds to a first array, and the first array includes at least one data;

[0348] The processing module 1302 is configured to input the group of skeleton coding frames as initial input information into a target processing model, the target processing model including at least two layers of processing modules, each layer of processing modules including a spatial dimension processing unit and a time dimension processing unit; and control the spatial dimension processing unit and the time dimension processing unit in each layer of processing modules in the target processing model to respectively perform processing on the input information in a topology-internal change and a topology-interchange change, to obtain a target processing result.

[0349] The determining module 1303 is configured to determine a behavior action corresponding to the group of skeleton coding frames based on the target processing result.

[0350] Optionally, the method further includes:

[0351] The sequence module is configured to sequentially sort a first array corresponding to a skeleton coding unit in the group of skeleton coding frames, to obtain a first skeleton coding sequence.

[0352] The identification insertion module is configured to insert a first identification in front of the first skeleton coding sequence, to obtain a second skeleton coding sequence, the second skeleton coding sequence being used as initial input information of the target processing model.

[0353] Optionally, the processing module includes:

[0354] The input information of a current layer is processed in a topology-internal change and a topology-interchange change in a spatial dimension and a time dimension, to obtain a processing result of the current layer.

[0355] The processing result of the current layer is processed in a spatial dimension and a time dimension as input information of a next layer, until processing of each layer in the target processing model is completed, to output a target processing result.

[0356] Optionally, the processing module is specifically configured to:

[0357] The spatial dimension processing unit is controlled to perform processing on the input information of the current layer in a topology-internal change and a topology-interchange change, to obtain first output information.

[0358] The time dimension processing unit is controlled to perform processing on the second skeleton coding sequence in a topology-internal change and a topology-interchange change based on the first output information, to obtain second output information, the second output information being used as the processing result of the current layer.

[0359] Optionally, the processing module is specifically configured to:

[0360] The spatial dimension processing unit is controlled to obtain first target information from the input information of the current layer, the first target information corresponding to a position and a length of the first identification, respectively.

[0361] In the spatial dimension, a first similarity of the first target information and each second array in the first non-target information is calculated, a first result value is obtained based on the first similarity, the second array has the same length as the first array, and the first non-target information is remaining information in the input information except the first target information;

[0362] A first change value between topologies in the multiple frames of the skeletal coding frames is calculated in units of frames, and a second change value in each topology in the skeletal coding frames is calculated in units of topologies;

[0363] A second result value is obtained based on the first change value and the second change value;

[0364] The first output information is calculated based on the first result value and the second result value.

[0365] Optionally, the processing module is specifically configured to:

[0366] The first output information is added to the second skeletal coding sequence to obtain a third skeletal coding sequence by the time-dimension processing unit;

[0367] Second target information corresponding to the position and length of the first identifier is obtained from the third skeletal coding sequence;

[0368] In the time dimension, a third similarity of the second target information and each third array in the second non-target information is calculated, a third result value is obtained based on the third similarity, the third array has the same length as the first array, and the second non-target information is remaining information in the input information except the second target information;

[0369] A third change value between topologies in the multiple frames of the skeletal coding frames is calculated in units of frames, and a fourth change value in each topology in the skeletal coding frames is calculated in units of topologies;

[0370] A fourth result value is obtained based on the third change value and the fourth change value;

[0371] The second output information is calculated based on the third result value and the fourth result value.

[0372] Optionally, the processing module is specifically configured to:

[0373] The fifth change value between topologies in the multiple frames of the skeletal coding frames is calculated in units of frames in the spatial dimension by the spatial-dimension processing unit, and the sixth change value in each topology in the skeletal coding frames is calculated in units of topologies;

[0374] The first output information is obtained based on the fifth change value and the sixth change value.

[0375] Optionally, the processing module is specifically configured to:

[0376] The time-dimension processing unit adds the first output value to the second skeletal coding sequence to obtain a third skeletal coding sequence.

[0377] In the time dimension, a seventh change value between topological structures in a plurality of skeletal coding frames is calculated in a frame unit, and an eighth change value in each topological structure in the skeletal coding frame is calculated in a topological structure unit.

[0378] The second output information is obtained based on the seventh change value and the eighth change value.

[0379] Optionally, the determining module is specifically configured to:

[0380] Third target information in the target processing result is obtained, and the third target information corresponds to the position and length of the first identifier, respectively.

[0381] Based on the type corresponding to the third target information, a behavior action corresponding to the skeletal coding frame group is determined.

[0382] The specific function implementation of each structure of the device is referred to the explanation in the method embodiment, and will not be described here in the device embodiment.

[0383] In summary, in the skeletal sequence behavior recognition device provided in the embodiment, the skeletal coding frame group to be processed obtained only contains skeletal coding units corresponding to topological structures, and does not have detailed key points of the topological structures. According to the processing of the change in the topological structure and the change between the topological structures in the skeletal coding frame group composed of the plurality of skeletal coding frames in the space dimension and the time dimension, the behavior action corresponding to the skeletal coding frame group is finally obtained, which realizes the distinguishing of large-scale actions and the recognition of detailed actions.

[0384] Based on the skeletal sequence behavior recognition method provided in the above embodiment, the embodiment of the present application further provides an electronic device, which includes at least one memory and at least one processor. The memory stores a program, and the processor calls the program stored in the memory. The program is used to implement the skeletal sequence behavior recognition method.

[0385] Based on the skeletal sequence behavior recognition method provided in the above embodiment, the embodiment of the present application further provides a storage medium, which stores computer executable instructions. The computer executable instructions are used to execute the skeletal sequence behavior recognition method.

[0386] The above describes in detail the bone sequence behavior recognition method, device, electronic equipment and storage medium provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. Therefore, the content of the specification should not be understood as a limitation of the present application.

[0387] It should be noted that each of the embodiments in the specification adopts a progressive manner for description, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0388] It should also be noted that in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements, or including the elements inherent in the process, method, article or device, or further including the elements inherent in the process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0389] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A skeleton sequence behavior recognition method, characterized in that: include: Obtaining a skeleton coding frame group to be processed, wherein the skeleton coding frame group to be processed includes multiple skeleton coding frames, each skeleton coding frame includes multiple skeleton coding units, each skeleton coding unit corresponds to a topological structure of an animal body, each skeleton coding unit corresponds to a first array, and the first array includes at least one data; Sequentially sorting the first arrays corresponding to the skeleton coding units in the skeleton coding frame group to obtain a first skeleton coding sequence; inserting a first identifier in front of the first skeleton coding sequence to obtain a second skeleton coding sequence, wherein the second skeleton coding sequence is used as initial input information of the target processing model; Inputting the skeleton encoding frame group as initial input information into a target processing model, wherein the target processing model includes at least two layers of processing modules, each layer of processing modules includes a spatial dimension processing unit and a temporal dimension processing unit; Controlling the spatial dimension processing unit to process the input information of the current layer for intra-topological structure changes and inter-topological structure changes to obtain first output information; including: controlling the spatial dimension processing unit to calculate, in the spatial dimension, a fifth change value between topological structures in multiple skeleton encoding frames in units of frames, and to calculate, in units of topological structures, a sixth change value within each topological structure in the skeleton encoding frame; and obtaining the first output information based on the fifth change value and the sixth change value; The method comprises the steps of: controlling the time dimension processing unit to perform intra-topological structure changes and inter-topological structure changes on the second skeleton coding sequence based on the first output information to obtain second output information, and using the second output information as the processing result of the current layer; controlling the time dimension processing unit to add the first output information to the second skeleton coding sequence to obtain a third skeleton coding sequence; calculating, in the time dimension, a seventh change value between topological structures in multiple skeleton coding frames in units of frames, and an eighth change value within each topological structure in the skeleton coding frame in units of topological structures; and calculating the second output information based on the seventh change value and the eighth change value. The processing result of the current layer is used as the input information of the next layer to process the spatial dimension and the temporal dimension until each layer in the target processing model is processed and the target processing result is output; Based on the target processing result, the behavior action corresponding to the skeleton encoding frame group is determined.

2. The method according to claim 1, characterized in that The control space dimension processing unit performs intra-topological structure change and inter-topological structure change processing on the input information of the current layer to obtain first output information, further comprising: Controlling the spatial dimension processing unit to obtain first target information from the input information of the current layer, where the first target information corresponds to the position and length of the first identifier respectively; In a spatial dimension, calculating a first similarity between the first target information and each second array in the first non-target information, and obtaining a first result value based on the first similarity, wherein the second array has the same length as the first array, and the first non-target information is the remaining information in the input information excluding the first target information; Calculating a first change value between topological structures in multiple skeleton coding frames in units of frames, and calculating a second change value within each topological structure in the skeleton coding frame in units of topological structures; Obtaining a second result value based on the first change value and the second change value; The first output information is obtained by calculation based on the first result value and the second result value.

3. The method according to claim 1, characterized in that Controlling the time dimension processing unit to process the changes within the topological structure and between the topological structures of the second skeleton encoding sequence based on the first output information to obtain second output information further includes: Controlling the time dimension processing unit to add the first output information to the second skeleton encoding sequence to obtain a third skeleton encoding sequence; Acquire second target information from the third skeleton encoding sequence, where the second target information corresponds to the position and length of the first identifier respectively; In the time dimension, calculating a third similarity between the second target information and each third array in the second non-target information, and obtaining a third result value based on the third similarity, where the third array has the same length as the first array, and the second non-target information is the remaining information in the input information excluding the second target information; Calculating a third change value between topological structures in multiple skeleton coding frames in units of frames, and calculating a fourth change value within the topological structure in the skeleton coding frame in units of topological structures; Obtaining a fourth result value based on the third change value and the fourth change value; The second output information is obtained by calculation based on the third result value and the fourth result value.

4. The method according to claim 1, wherein Based on the target processing result, the behavior action corresponding to the skeleton encoding frame group is obtained, including: Acquire third target information in the target processing result, where the third target information corresponds to the position and length of the first identifier respectively; Based on the type corresponding to the third target information, the behavior action corresponding to the skeleton encoding frame group is determined.

5. A skeleton sequence recognition method, characterized in that: include: Receive input information, where the input information is information sent from a previous structure in a target processing model to a current processing module, the target processing model includes an input layer and at least two layers of transformer modules, each layer of transformer modules includes a spatial dimension processing unit and a temporal dimension processing unit, the previous structure includes any one of the input layer, the spatial dimension processing unit, and the temporal dimension processing unit, the input information is related to a skeleton coding frame group to be processed, the skeleton coding frame group to be processed includes multiple skeleton coding frames, each skeleton coding frame includes multiple skeleton coding units, a skeleton coding unit corresponds to a topological structure of an animal body, each skeleton coding unit corresponds to a first array, and the first array includes at least one data; reorganizing at least the input information based on the function of the current processing unit to obtain target information corresponding to the current processing unit; Based on the target information conversion, a K matrix, a Q matrix and a V matrix are obtained; Performing multi-head self-attention processing based on the K matrix, the Q matrix, and the V matrix to obtain a first processing result, where the first processing result represents a change between topological structures; Performing linear mapping based on the V matrix to obtain a second processing result, wherein the second processing result represents a change in the topological structure; Obtaining a third processing result based on the first processing result and the second processing result; Performing multi-head self-attention processing on the input information to obtain a fourth processing result; a fifth processing result obtained by reorganizing the third processing result, wherein the fifth processing result at least corresponds to the input information; Based on the fourth processing result and the fifth processing result, a processing result of the current processing unit is obtained.

6. A skeleton sequence behavior recognition device, characterized in that: include: An acquisition module is used to acquire a skeleton coding frame group to be processed, wherein the skeleton coding frame group to be processed includes multiple skeleton coding frames, each skeleton coding frame includes multiple skeleton coding units, each skeleton coding unit corresponds to a topological structure of an animal body, each skeleton coding unit corresponds to a first array, and the first array includes at least one data; A sequence module, configured to sequentially sort the first arrays corresponding to the skeleton coding units in the skeleton coding frame group to obtain a first skeleton coding sequence; an identifier insertion module, configured to insert a first identifier before the first skeleton encoding sequence to obtain a second skeleton encoding sequence, wherein the second skeleton encoding sequence is used as initial input information of a target processing model; a processing module, configured to input the skeleton encoding frame group as initial input information into a target processing model, wherein the target processing model comprises at least two layers of processing modules, each layer of processing modules comprising a spatial dimension processing unit and a temporal dimension processing unit; Controlling the spatial dimension processing unit to process the input information of the current layer for intra-topological structure changes and inter-topological structure changes to obtain first output information; including: controlling the spatial dimension processing unit to calculate, in the spatial dimension, a fifth change value between topological structures in multiple skeleton encoding frames in units of frames, and to calculate, in units of topological structures, a sixth change value within each topological structure in the skeleton encoding frame; and obtaining the first output information based on the fifth change value and the sixth change value; The control time dimension processing unit performs processing of intra-topological structure changes and inter-topological structure changes on the second skeleton coding sequence based on the first output information to obtain second output information, and the second output information is used as the processing result of the current layer; including: controlling the time dimension processing unit to add the first output information to the second skeleton coding sequence to obtain a third skeleton coding sequence; in the time dimension, calculating the seventh change value between the topological structures in multiple skeleton coding frames in units of frames, and calculating the eighth change value within each topological structure in the skeleton coding frame in units of topological structures; based on the seventh change value and the eighth change value, calculating the second output information; using the processing result of the current layer as the input information of the next layer to perform processing in the spatial dimension and the temporal dimension until each layer in the target processing model is processed and the target processing result is output; A determination module is used to determine the behavior action corresponding to the skeleton encoding frame group based on the target processing result.

Images