Spatiotemporal graph network interpretable method, visualization method and related device

Abstract

The embodiment of the present application proposes an interpretable method, a visualization method and related devices for a spatiotemporal graph network, which relates to the field of artificial intelligence technology, wherein the interpretable method for a spatiotemporal graph network includes: obtaining a skeleton sequence, a time series mask vector and a node sequence mask vector of key points of a human body, calculating an initial mask vector, and then calculating a motion sequence based on the initial mask vector and the skeleton sequence, inputting the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, and iteratively updating a reference value of an optimization function using the predicted behavior recognition result to obtain a target mask vector for interpreting the spatiotemporal graph network model. In this embodiment, the information of time and space is combined, and the target mask vector is used to represent the influence of time information and space information on the judgment result of the spatiotemporal graph network model, and the spatiotemporal graph network model is interpreted, so that the spatiotemporal graph network model can be optimized in a targeted manner to improve the accuracy of the behavior recognition result and the recognition efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a space-time graph network interpretable method, visualization method and related devices. Background Art

[0002] With the development of computer vision, the use of computer vision to identify human behavior has been widely used in many fields, such as intelligent video surveillance, public safety, behavior analysis, human-computer interaction, and intelligent robots. Most of the related technologies use highly accurate depth sensors to obtain human posture information, use human posture estimation algorithms to extract the positions of human skeleton joints, and structure the joint coordinate positions of the entire skeleton sequence into coordinate vectors. Then, based on deep learning, a spatiotemporal graph network model for behavior recognition is constructed, thereby learning the spatial and temporal information of the human skeleton sequence for behavior recognition.

[0003] However, the models used in these neural network recognition methods are all black box models, and their prediction results lack interpretability, that is, it is difficult to accurately explain under what circumstances the model will fail and be effective. In related technologies, there are methods such as the Laplace matrix to estimate the importance of model input features. However, this method can only roughly explain whether a certain model input feature pair is important to the model, and cannot explain the model more accurately, resulting in poor model interpretation. In the behavior recognition method, the human motion sequence has information at both the time and space levels. If the impact of these two levels of information on the judgment results of the spatiotemporal graph network model is not considered at the same time, it is difficult to optimize the spatiotemporal graph network model in a targeted manner to improve the accuracy of the behavior recognition results. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiments of the present application provide a space-time graph network interpretable method, a visualization method and related devices, which can use the target mask vector to explain the influence of information at both the time and space levels on the judgment results of the space-time graph network model, thereby specifically improving the recognition accuracy of the space-time graph network model.

[0006] In a first aspect, an embodiment of the present application provides a method for interpreting a spatiotemporal graph network, comprising:

[0007] Obtain skeleton sequence, time series mask vector and node sequence mask vector of key points of human body;

[0008] Calculating an initial mask vector according to the time series mask vector and the node sequence mask vector;

[0009] Calculating a motion sequence according to the initial mask vector and the skeleton sequence;

[0010] Inputting the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, wherein the spatiotemporal graph network model is used for behavior recognition;

[0011] The predicted behavior recognition result is used to iteratively update a reference value of the optimization function to obtain a target mask vector for explaining the spatiotemporal graph network model.

[0012] In an optional implementation, the calculating an initial mask vector according to the time series mask vector and the node sequence mask vector includes:

[0013] The initial mask vector is obtained by performing matrix multiplication on the time series mask vector and the node sequence mask vector, which is expressed as:

[0014] M=M T ×M V

[0015] Among them, M T represents the time series mask vector, M V represents the node sequence mask vector, and M represents the initial mask vector.

[0016] In an optional implementation, the calculating the motion sequence according to the initial mask vector and the skeleton sequence includes:

[0017] Calculating the Hadamard product between the skeleton sequence and the corresponding initial mask vector to obtain the motion sequence;

[0018] It is expressed as:

[0019] X C,T,V,M =M C,T,V,M ⊙G C,T,V,M

[0020] Among them, ⊙ represents the Hadamard product, G C,T,V,M represents the backbone sequence, M C,T,V,M represents the initial mask vector, X C,T,V,M represents the motion sequence, T represents the time sequence, C represents the motion three-dimensional coordinate information, V represents the number of key points of each character, and M represents the number of people in the motion sequence.

[0021] In an optional implementation, the iterative updating of the reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector for interpreting the spatiotemporal graph network model includes:

[0022] Calculating the reference value of the optimization function according to the predicted behavior recognition result and the initial mask vector;

[0023] The initial mask vector is adjusted according to the reference value until the optimization function reaches a convergence condition, thereby obtaining the target mask vector.

[0024] In an optional implementation, the adjusting the initial mask vector according to the reference value until the optimization function reaches a convergence condition to obtain the target mask vector includes:

[0025] Adjusting the initial mask vector to obtain an optimized mask vector;

[0026] Calculating an optimized motion sequence according to the optimized mask vector and the skeleton sequence;

[0027] Inputting the optimized motion sequence into a pre-trained spatiotemporal graph network model to obtain an optimized predicted behavior recognition result;

[0028] Calculating a reference value of the optimization function corresponding to the optimization prediction behavior recognition result;

[0029] It is determined whether the optimization function reaches a convergence condition according to the reference value. If the convergence condition is reached, the optimized mask vector is used as the target mask vector; otherwise, the above steps are repeated until the optimization function reaches a convergence condition.

[0030] In an optional implementation, the optimization function is a mutual information optimization function, which is expressed as:

[0031] MI(Y,M)=H(Y)-H(Y|M)

[0032] Wherein, Y represents the predicted behavior recognition result, M represents the initial mask vector, H(·) represents the entropy function, and MI(Y,M) represents the reference value.

[0033] In an optional implementation, the convergence condition of the optimization function is: maximizing a reference value of the optimization function.

[0034] In a second aspect, an embodiment of the present application provides a visualization method, including:

[0035] Acquire a motion sequence and a target mask vector, wherein the motion sequence and the target mask vector are calculated using the spatiotemporal graph network interpretable method according to any one of the first aspects;

[0036] Sampling the motion sequence frame by frame to obtain a display sequence;

[0037] Visualize the target mask vector of the displayed sequence between different frames.

[0038] In a third aspect, an embodiment of the present application provides a space-time graph network interpretable device, including:

[0039] The first acquisition module is used to acquire the skeleton sequence, time series mask vector and node sequence mask vector of the key points of the human body;

[0040] An initial mask vector calculation module, used to calculate an initial mask vector according to the time series mask vector and the node sequence mask vector;

[0041] A motion sequence calculation module, used for calculating a motion sequence according to the initial mask vector and the skeleton sequence;

[0042] A preliminary prediction module, used to input the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, wherein the spatiotemporal graph network model is used for behavior recognition;

[0043] The iterative optimization module is used to iteratively update the reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector for explaining the spatiotemporal graph network model.

[0044] In a fourth aspect, an embodiment of the present application provides a visualization device, including:

[0045] A second acquisition module, used to acquire a motion sequence and a target mask vector, wherein the motion sequence and the target mask vector are calculated using the spatiotemporal graph network interpretable method according to any one of the first aspects;

[0046] A skeleton sequence sampling module, used for sampling the motion sequence frame by frame to obtain a display sequence;

[0047] A visualization module is used to visualize the target mask vector of the display sequence between different frames.

[0048] A fifth aspect is a computer device, comprising a processor and a memory;

[0049] The memory is used to store programs;

[0050] The processor is used to execute the spatiotemporal graph network interpretable method as described in any one of the first aspects according to the program.

[0051] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the space-time graph network interpretable method described in any one of the first aspects.

[0052] The first aspect of the embodiment of the present application provides a method for interpreting a spatiotemporal graph network. Compared with the related art, the skeleton sequence, time series mask vector and node sequence mask vector of the key points of the human body are obtained, and the initial mask vector is calculated based on the time series mask vector and the node sequence mask vector, and then the motion sequence is calculated based on the initial mask vector and the skeleton sequence, and the motion sequence is input into the pre-trained spatiotemporal graph network model to obtain the predicted behavior recognition result, and the reference value of the optimization function is iteratively updated using the predicted behavior recognition result to obtain the target mask vector for interpreting the spatiotemporal graph network model. In this embodiment, the information at the two levels of time and space is combined, and the initial mask vector is generated using the time series mask vector and the node sequence mask vector. The target mask vector is obtained by iteratively updating the initial mask vector, and the target mask vector is used to characterize the influence of time information and space information on the judgment result of the spatiotemporal graph network model, and the spatiotemporal graph network model is interpreted, so that the spatiotemporal graph network model can be optimized in a targeted manner to improve the accuracy of the behavior recognition result and the recognition efficiency.

[0053] It can be understood that the beneficial effects of the second to sixth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 is a schematic diagram of an exemplary system architecture provided by an embodiment of the present application;

[0056] Figure 2 is a flow chart of a spatiotemporal graph network interpretable method provided by an embodiment of the present application;

[0057] Figure 3 is a schematic diagram of joints of a human skeleton provided by an embodiment of the present application;

[0058] Figure 4 It is another flow chart of a space-time graph network interpretable method provided by an embodiment of the present application;

[0059] Figure 5 It is another flow chart of a space-time graph network interpretable method provided by an embodiment of the present application;

[0060] Figure 6is a flowchart of a visualization method provided by an embodiment of the present application;

[0061] Figure 7 It is a visual schematic diagram provided by an embodiment of the present application;

[0062] Figure 8 It is a structural block diagram of a space-time graph network interpretable device provided by an embodiment of the present application;

[0063] Fig. 9 It is a structural block diagram of a visualization device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the embodiments of the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the embodiments of the present application.

[0065] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0066] It should also be understood that the references to "one embodiment" or "some embodiments" described in the specification of the embodiments of the present application mean that one or more embodiments of the embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0067] With the development of computer vision, the use of computer vision to identify human behavior has been widely used in many fields, such as intelligent video surveillance, public safety, behavior analysis, human-computer interaction, and intelligent robots. Most of the related technologies use highly accurate depth sensors to obtain human posture information, use human posture estimation algorithms to extract the positions of human skeleton joints, structure the joint coordinate positions of the entire skeleton sequence into coordinate vectors, and then build a spatiotemporal graph network model based on deep learning to learn the spatial and temporal information of the human skeleton sequence for behavior recognition.

[0068] However, the models used in these neural network recognition methods are all black box models, and their prediction results lack interpretability, that is, it is difficult to accurately explain under what circumstances the model will fail and be effective. If the relationship behind the prediction is not understood and inferred, these models cannot be understood and trusted, which hinders the application of these algorithms in key areas. Therefore, it is necessary to study the interpretability of neural network models. In related technologies, there are methods such as the Laplace matrix to estimate the importance of model input features. However, this method can only roughly explain whether a certain model input feature is important to the model, and cannot explain the model more accurately, resulting in poor model interpretation. In the behavior recognition method, the human motion sequence has information at both the time and space levels. If the impact of these two levels of information on the judgment results of the spatiotemporal graph network model is not considered at the same time, it is difficult to optimize the spatiotemporal graph network model in a targeted manner to improve the accuracy of the behavior recognition results.

[0069] Therefore, an embodiment of the present application provides an interpretable method for a spatiotemporal graph network, which obtains a skeleton sequence, a time series mask vector, and a node sequence mask vector of key points of a human body, calculates an initial mask vector based on the time series mask vector and the node sequence mask vector, and then calculates a motion sequence based on the initial mask vector and the skeleton sequence, and inputs the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, and iteratively updates the reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector for interpreting the spatiotemporal graph network model. In this embodiment, information at both the temporal and spatial levels is combined, and an initial mask vector is generated using a time series mask vector and a node sequence mask vector, and a target mask vector is obtained by iteratively updating the initial mask vector, and the target mask vector is used to characterize the influence of temporal information and spatial information on the judgment result of the spatiotemporal graph network model, and the spatiotemporal graph network model is interpreted, so that the spatiotemporal graph network model can be optimized in a targeted manner to improve the accuracy of behavior recognition results and recognition efficiency.

[0070] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0071] Figure 1A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0072] like Figure 1 As shown, the system architecture 100 may include a terminal device (such as Figure 1 The computer 101 includes one or more of the desktop computer 101, the tablet computer 102, and the portable computer 103, and of course, other terminal devices with display screens, etc.), a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0074] In one embodiment of the present invention, a user may utilize terminal device 101 (or terminal device 102 or 103) to upload a skeleton sequence collected by a collection device to server 105. A highly accurate depth sensor may be utilized to obtain human posture information, and a human posture estimation algorithm may be utilized to extract the positions of the human skeleton joints, and the joint coordinate positions of the entire skeleton sequence may be structured into coordinate vectors. After obtaining the skeleton sequence, the server 105 randomly generates a time series mask vector and a node sequence mask vector, and then calculates an initial mask vector based on the time series mask vector and the node sequence mask vector, and then calculates a motion sequence based on the initial mask vector and the skeleton sequence, and inputs the motion sequence into a pre-trained space-time graph network model to obtain a predicted behavior recognition result, and iteratively updates a reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector, and generates an initial mask vector using the time series mask vector and the node sequence mask vector, and obtains a target mask vector by iteratively updating the initial mask vector, and uses the target mask vector to characterize the influence of time information and spatial information on the judgment result of the space-time graph network model, and interprets the space-time graph network model, so that the space-time graph network model can be optimized in a targeted manner to improve the accuracy of the behavior recognition results and the recognition efficiency.

[0075] It should be noted that the space-time graph network interpretable method provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the space-time graph network interpretable device is generally set in the server 105. However, in other embodiments of the embodiment of the present application, the terminal device may also have similar functions as the server, so as to execute the space-time graph network interpretable method provided in the embodiment of the present application.

[0076] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will appreciate that, with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. Those skilled in the art will appreciate that Figure 1 The system architecture shown in the figure does not constitute a limitation on the embodiments of the present application, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0077] Based on the above system architecture, various embodiments of the spatiotemporal graph network interpretable method of the embodiments of the present application are proposed.

[0078] like Figure 2 As shown, Figure 2 It is a flowchart of a spatiotemporal graph network interpretable method provided in one embodiment of the present application, including but not limited to steps S110 to S150.

[0079] Step S110, obtaining a skeleton sequence, a time series mask vector, and a node sequence mask vector of key points of the human body.

[0080] In one embodiment, a highly accurate depth sensor can be used to extract human posture information from a video frame, or a Kinect depth camera can be used to directly shoot and obtain human posture information, or a 3D posture estimation algorithm can be used to obtain human posture information from an ordinary video. After obtaining the human posture information, the positions of key points of the human body (for example, the key points of the human body can be joints of the human skeleton) are extracted, and the coordinate positions of the key points of the entire skeleton sequence are structured into coordinate vectors, which constitute the coordinate information in the skeleton sequence in this embodiment.

[0081] In one embodiment, 15 joints of the human skeleton are selected as key points of the human body, and the relative distances between the joints and the three-dimensional coordinates of the joints are calculated. Figure 3 , is a schematic diagram of the joints of the human skeleton in this embodiment, and the 15 key points 300 of the human body in the figure include: head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, waist, left hip, right hip, left knee, right knee, left ankle, and right ankle. It can be understood that the joints in this embodiment are only for illustration, and it does not mean that only the 15 key points 300 of the human body can be selected in this embodiment. More key points of the human body can also be selected according to the calculation accuracy requirements and the computing power of the computing device, such as selecting 25 key points of the human body, etc., which are not specifically limited here.

[0082] In one embodiment, the skeleton sequence contains motion information of more than one human body. If multiple human bodies are included, local feature area segmentation is performed on all human body images in turn. If necessary, tilt correction is performed on the local feature areas of all human bodies to obtain the skeleton sequence corresponding to each human body, and then the skeleton sequence corresponding to each human body is generated using the skeleton sequence corresponding to each human body.

[0083] In one embodiment, since each joint point in the skeleton sequence corresponding to different movement behaviors of the human body contains time and space information, the time information is reflected in the different weights of different joint points in the same time frame, and the space information is reflected in the different weights of the same joint point in different time frames. Therefore, in this embodiment, the time series mask vector is used to block the time series related information in the motion sequence to represent the time information, and the node sequence mask vector is used to block the space sequence related information in the motion sequence to represent the space information, so as to simultaneously consider the impact of these two levels of information on the judgment result of the spatiotemporal graph network model.

[0084] In one embodiment, NTU-RGB-D skeleton sequence is used as a data set for illustration. The data set contains 60 types of actions (see Table 1), with a total of 56,880 samples, of which 40 types are daily behavior actions, 9 types are health-related actions, and 11 types are two-person interactions. These actions are performed by 40 people aged from 10 to 35 years old. The data set is collected by Microsoft Kinectv2 sensor and uses three cameras at different angles. The collected data forms include depth information, 3D skeleton information, RGB frames, and infrared sequences. In this embodiment, the skeleton sequence is represented by G C,T,V,M ;

[0085] Wherein, C represents the number of features, such as the three-dimensional coordinate information of the motion of each key point (such as a joint point) of the human body contained in the skeleton sequence. A three-dimensional coordinate system can be set, and the origin of the coordinates can be selected to obtain the coordinate information of each joint point contained in the skeleton sequence, which is expressed as (x, y, z);

[0086] T represents the length of the time series, that is, the data information of the time series needs to be collected to form a skeleton sequence;

[0087] V represents the number of key points (e.g., joint points) of each character, e.g., V=15 or V=25;

[0088] M represents the number of people in the motion sequence, for example, M=1 or M=2.

[0089] In one embodiment, the skeleton sequence may be 100-300 frames of collected data, each frame of the skeleton sequence is composed of M characters and V human key points (eg, V=25), and each human key point has features of C dimensions (eg, C=3).

[0090] In one embodiment, it is assumed that a time series (eg, a frame) is represented as: T = {T1, T2, T3, ... T s}, the spatial sequence (in this embodiment: the human body key point sequence) is expressed as: V = {V1, V2, ... V z}, then in this embodiment, the time series mask vector is expressed as: M C,T,M , reflecting the weights of different joints in the same time frame, the node sequence mask vector is expressed as: M C,V,M , reflecting the weight of the same joint point in different time frames. In this embodiment, a time series mask vector M can be randomly generated first C,T,M and node sequence mask vector M C,V,M For example, the time series mask vector and the node sequence mask vector can be all-1 matrices, which are not specifically limited here.

[0091] Step S120, calculating an initial mask vector according to the time series mask vector and the node sequence mask vector.

[0092] In one embodiment, according to the decoupling concept, the time series related information and the node sequence related information are independent and unrelated, so the time series mask vector and the node sequence mask vector can be matrix multiplied to expand the dimension of the initial mask vector to the same dimension as the skeleton sequence G. C,T,V,M The state with the same dimension obtains the initial mask vector, which is used to comprehensively explain the weight of the time series (such as frames) and the weight of the spatial series (such as human key points), expressed as:

[0093] M=M T ×M V

[0094] M C,T,V,M =M C,T,M ×M C,V,M

[0095] Among them, M T =M C,V,M represents the time series mask vector, M V =M C,V,M represents the node sequence mask vector, M = M C,T,V,M Represents the initial mask vector.

[0096] Step S130, calculating a motion sequence according to the initial mask vector and the skeleton sequence.

[0097] In one embodiment, in order to apply the above initial mask vector to the skeleton sequence for subsequent behavior recognition, the motion sequence is obtained by calculating the Hadamard product between the skeleton sequence and the corresponding initial mask vector, which is expressed as:

[0098] X C,T,V,M =M C,T,V,M ⊙G C,T,V,M

[0099] Among them, ⊙ represents the Hadamard product, G C,T,V,M represents the skeleton sequence, M C,T,V,M represents the initial mask vector, X C,T,V,M represents the motion sequence, T represents the time sequence, C represents the three-dimensional coordinate information of the motion, V represents the number of key points of each character, and M represents the number of people in the motion sequence.

[0100] In one embodiment, the Hadamard product is a type of matrix operation. If two matrices of the same order: The dimension is m×n, and A={a ij}, B={b ij}, then the Hadamard product of matrix A and matrix B is expressed as:

[0101]

[0102] It can be seen from the above formula that the Hadamard product of matrix A and matrix B is obtained by multiplying the corresponding elements of matrix A and matrix B to obtain the value of the corresponding position.

[0103] In one embodiment, if G C,k,V,M is a matrix of all 1s, the motion sequence is the same as the skeleton sequence.

[0104] Step S140, inputting the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, and the spatiotemporal graph network model is used for behavior recognition.

[0105] In one embodiment, after the motion sequence X is obtained in the above steps, the motion sequence X is input into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result Y.

[0106] In one embodiment, the spatiotemporal graph network model is a spatiotemporal graph network model, which is a type of graph neural network model (expressed as f(·)), and is pre-trained using a training data set consisting of motion vectors generated by a large number of human skeleton sequences. The training goal is to perform behavior recognition, and the motion vector is input and the behavior recognition result is output. It can be understood that the spatiotemporal graph network model in this embodiment can also be other types of models, as long as it can realize behavior recognition based on motion sequences, and the model structure is not limited here.

[0107] Step S150, using the predicted behavior recognition result to iteratively update the reference value of the optimization function to obtain a target mask vector for explaining the spatiotemporal graph network model.

[0108] In one embodiment, referring to Figure 4 Step S150 includes but is not limited to steps S151 to S152:

[0109] Step S151, calculating a reference value of the optimization function according to the predicted behavior recognition result and the initial mask vector.

[0110] Step S152, adjusting the initial mask vector according to the reference value until the optimization function reaches a convergence condition, thereby obtaining a target mask vector.

[0111] In one embodiment, referring to Figure 5 Step S152 includes but is not limited to steps S1521 to S1525:

[0112] Step S1521, adjusting the initial mask vector to obtain an optimized mask vector.

[0113] In one embodiment, a subgraph of the initial mask vector M is generated by a random graph vector as the optimization mask vector.

[0114] Step S1522, calculating and obtaining an optimized motion sequence according to the optimized mask vector and the skeleton sequence.

[0115] Step S1523, input the optimized motion sequence into the pre-trained spatiotemporal graph network model to obtain the optimized predicted behavior recognition result.

[0116] Step S1524, calculating a reference value of the optimization function corresponding to the optimization prediction behavior recognition result.

[0117] The method of calculating the reference value in the above steps S1522 to S1524 is the same as that in steps S120 to S140, except that the optimized mask vector is used instead of the initial mask vector.

[0118] Step S1525, judging whether the optimization function has reached the convergence condition according to the reference value, if so, taking the optimized mask vector as the target mask vector; otherwise, repeating the above steps until the optimization function has reached the convergence condition.

[0119] In one embodiment, if the convergence condition is not met, the mask vector is continuously adjusted and optimized according to the above step S1521 until the convergence condition is met.

[0120] In one embodiment, an optimization function is constructed based on a mutual information method. Mutual information is used to indicate whether there is a relationship between two variables A and B (variables A and B here are only examples and have no real meaning) and the strength of the relationship, which is expressed as:

[0121]

[0122] Among them, MI(A,B) represents the value of mutual information, P(A,B) represents the joint probability density between variables A and B, P(A) represents the probability density of variable A, and P(B) represents the probability density of variable B. From the above formula, we can see that if variables A and B are independent, then P(A,B)=P(A)P(B), and MI(A,B) is 0, which means that variables A and B are unrelated.

[0123] Derived deformation:

[0124] MI(A,B)=H(B)-H(B|A)

[0125] H(B)=-∫ B P(B)logP(B)

[0126] Among them, H(B) represents the entropy of variable B, which is used to measure the uncertainty of variable B. The more discrete the distribution of variable B, the higher the value. H(B|A) represents the uncertainty of variable B when variable A is known. In this embodiment, MI(A,B) can be interpreted as the amount by which the uncertainty of variable B is reduced due to the introduction of variable A. This reduction is expressed as H(B|A). Therefore, if the relationship between variables A and B is closer, MI(A,B) is larger. At this time, H(B|A) is 0, indicating that variables A and B are completely related. When variable A is determined, variable B is a fixed value, and there is no probability of other uncertain situations. Therefore, H(B|A) is 0. When MI(A,B) takes the value of 0, it means that variables A and B are independent. At this time, H(B)=H(B|A), indicating that the appearance of variable A does not affect variable B.

[0127] In one embodiment, an optimization function is constructed according to the meaning of the above mutual information. Since the purpose of this embodiment is to obtain the correlation between the predicted behavior recognition result and the time series related information of the skeleton sequence and the spatial sequence related information, the optimization function is expressed as:

[0128] MI(Y,M)=H(Y)-H(Y|M)

[0129] Among them, Y represents the predicted behavior recognition result, M represents the initial mask vector, H(·) represents the entropy function, and MI(Y,M) represents the reference value.

[0130] Since the initial mask vector M is based on the time series mask vector M T and node sequence mask vector M V It is obtained by matrix multiplication, so the optimization function is further expressed as:

[0131] MI(Y,(M T ,M V))=H(Y)-H(Y|(M T ,M V ))

[0132] According to the purpose of the optimization function, the convergence condition represents finding a suitable time series mask vector M T and node sequence mask vector M V , and obtain the initial mask vector M, so that there is the strongest correlation between the predicted behavior recognition result Y and the initial mask vector M. The strongest correlation is expressed as M (i.e., M T ,M V ) relative to Y has the largest mutual information value, that is, the reference value is the largest.

[0133] Therefore, in the above embodiment, the convergence condition of the optimization function is: maximizing the reference value of the optimization function, which is expressed as:

[0134]

[0135] When the reference value of the optimization function is the largest, it corresponds to the value of a time series T and a space series V. For example, in one embodiment, when T=T s ,V=V z When the convergence condition is reached, the convergence condition can be further expressed as:

[0136]

[0137] In one embodiment, H(Y) represents the uncertainty of the predicted behavior recognition result Y when the skeleton sequence is known. Since the relationship between the skeleton sequence and the predicted behavior recognition result Y is known, H(Y) is a constant. In this embodiment, the solution of the maximum mutual information of the above convergence condition can be equivalent to the solution of the minimum conditional entropy, which is expressed as:

[0138]

[0139] According to the definition of conditional entropy, the above formula is further transformed into:

[0140]

[0141] Among them, f(·) represents the pre-trained spatiotemporal graph network model. For example, given an input skeleton sequence G, the predicted behavior recognition result Y is expressed as: Y = f(G), and E represents the expectation.

[0142] In one embodiment, iterative optimization is performed in a gradient descent manner to obtain a target mask vector to explain the degree of influence (ie, weight) of the time series related information in the skeleton sequence and the related information in the spatial sequence on the decision result.

[0143] The convergence condition of the above formula can be rewritten as:

[0144]

[0145] in, Represents the target mask vector M C,T,V,M Under the condition of C,T,V,M The motion sequence is obtained by performing Hadamard product operation with the skeleton sequence, and the motion sequence is input into the pre-trained spatiotemporal graph network model to obtain the predicted behavior recognition result Y. The reference value of the optimization function is calculated, and the reference value is adjusted using the gradient descent method until the convergence condition is reached, and the target mask vector can be obtained.

[0146] The embodiment of the present application provides an interpretable method for a spatiotemporal graph network model, which obtains a skeleton sequence, a time series mask vector, and a node sequence mask vector of key points of a human body, calculates an initial mask vector based on the time series mask vector and the node sequence mask vector, and then calculates a motion sequence based on the initial mask vector and the skeleton sequence, inputs the motion sequence into a pre-trained spatiotemporal graph network model, obtains a predicted behavior recognition result, and uses the predicted behavior recognition result to iteratively update the reference value of the optimization function to obtain a target mask vector for interpreting the spatiotemporal graph network model. In this embodiment, information at both the temporal and spatial levels is combined, and an initial mask vector is generated using a time series mask vector and a node sequence mask vector. A target mask vector is obtained by iteratively updating the initial mask vector, and the target mask vector is used to characterize the influence of temporal information and spatial information on the judgment result of the spatiotemporal graph network model, and the spatiotemporal graph network model is interpreted, so that the spatiotemporal graph network model can be optimized in a targeted manner to improve the accuracy of behavior recognition results and recognition efficiency.

[0147] In addition, an embodiment of the present application proposes a visualization method based on obtaining a target mask vector for explaining the spatiotemporal graph network model, referring to Figure 6 , is a flowchart of the visualization method in this embodiment, including but not limited to steps S210 to S230.

[0148] Step S210, obtaining a motion sequence and a target mask vector.

[0149] In one embodiment, the motion sequence and the target mask vector are calculated using an interpretable method of a spatiotemporal graph network model as described in any of the above embodiments.

[0150] Step S220: sampling the motion sequence frame by frame to obtain a display sequence.

[0151] Step S230 , visually displaying the target mask vector of the display sequence between different frames.

[0152] In one embodiment, referring to Figure 7, which is a schematic diagram for visualization of this embodiment.

[0153] First, select the minimum coordinate in the motion sequence (including human skeleton motion information) and set it as the drawing coordinate origin, that is, the coordinates of each point in the motion sequence are subtracted from the minimum coordinate and then drawn in the three-dimensional coordinate system; then sample the motion sequence frame by frame. For example, if the motion sequence is 300 frames of data, the display sequence is obtained by sampling every 5 frames. You can use drawing tools (such as matplotlib) to draw the skeleton motion vector, and get the following Figure 7 Schematic diagram of the display sequence is shown.

[0154] Figure 7 As shown in , the key points 300 of the human body (such as skeleton nodes) and the connections of the skeleton in the display sequence are plotted on the three-dimensional coordinates according to the coordinate and time sequence, and the action interval is set to 1.5 times the action width of the previous frame to improve the visualization effect; finally, the generated target mask vector (i.e., the weight M obtained in the above embodiment) is visualized to display the influence of the target mask vector on the judgment result, and the influence can be presented in the form of a heat map. Figure 7 , using color depth to represent the different weights of the time series (such as frames) and the spatial series (such as key points of the human body) in the display sequence, so as to explain the spatiotemporal graph network model, visualize the weights of different joints in the same time frame and the different weights of the same joint in different time frames, and the influence of different weights on the judgment results of the spatiotemporal graph network model. Further, according to the weight, the spatiotemporal graph network model can be optimized in a targeted manner to improve the accuracy and efficiency of behavior recognition results.

[0155] It is understandable that the above Figure 7 It is only a visual schematic diagram of this embodiment and does not limit this embodiment.

[0156] In addition, an embodiment of the present application further provides a space-time graph network interpretable device, referring to Figure 8 , the device comprises:

[0157] A first acquisition module 810 is used to acquire a skeleton sequence, a time series mask vector and a node sequence mask vector of key points of a human body;

[0158] An initial mask vector calculation module 820, used to calculate an initial mask vector according to the time series mask vector and the node sequence mask vector;

[0159] A motion sequence calculation module 830, configured to calculate a motion sequence according to an initial mask vector and a skeleton sequence;

[0160] A preliminary prediction module 840 is used to input the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, and the spatiotemporal graph network model is used for behavior recognition;

[0161] The iterative optimization module 850 is used to iteratively update the reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector for explaining the spatiotemporal graph network model.

[0162] In one embodiment, the iterative optimization module 850 is further used to calculate a reference value of the optimization function based on the predicted behavior recognition result and the initial mask vector, and adjust the initial mask vector according to the reference value until the optimization function reaches a convergence condition to obtain a target mask vector.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] It should be noted that the space-time graph network model interpretable device in this embodiment can execute the following steps: Figure 2 The space-time graph network interpretable method in the embodiment shown. That is, the space-time graph network interpretable device in this embodiment and Figure 2 The space-time graph network interpretable methods in the illustrated embodiments all belong to the same inventive concept, and therefore these embodiments have the same implementation principles and technical effects, which will not be described in detail here.

[0165] In addition, an embodiment of the present application further provides a visualization device, referring to Fig. 9 , the device comprises:

[0166] A second acquisition module 910 is used to acquire a motion sequence and a target mask vector, where the motion sequence and the target mask vector are calculated using a spatiotemporal graph network interpretable method as in any one of the above embodiments;

[0167] The skeleton sequence sampling module 920 is used to sample the motion sequence frame by frame to obtain a display sequence;

[0168] The visualization module 930 is used to visualize the target mask vector of the display sequence between different frames.

[0169] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] It should be noted that the visualization device in this embodiment can perform the following operations: Figure 6 That is, the visualization device in this embodiment and the visualization method in the embodiment Figure 6 The visualization methods in the illustrated embodiments all belong to the same inventive concept, and therefore these embodiments have the same implementation principles and technical effects, which will not be described in detail herein.

[0171] In addition, one embodiment of the present application further provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0172] The processor and the memory may be connected via a bus or other means.

[0173] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0174] The non-transient software program and instructions required to implement the space-time graph network interpretable method or visualization method of the above embodiment are stored in the memory. When executed by the processor, the space-time graph network interpretable method or visualization method in the above embodiment is executed, for example, the above described Figure 2 Steps S110 to S150 of the method, Figure 4 Steps S151 to S152 of the method, Figure 5 Method steps S1521 to S1525, etc.

[0175] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by a processor in the above-mentioned computer device embodiment, so that the above-mentioned processor can execute the space-time graph network interpretable method or visualization method in the above-mentioned embodiment, for example, to execute the above-mentioned Figure 2 Steps S110 to S150 of the method, Figure 4 Steps S151 to S152 of the method, Figure 5 Method steps S1521 to S1525, etc.

[0176] For example, when executed by a processor in the above-mentioned computer device embodiment, the above-mentioned processor may execute the space-time graph network interpretable method or visualization method in the above-mentioned embodiment, for example, executing the above-mentioned Figure 2 Steps S110 to S150 of the method, Figure 4 Steps S151 to S152 of the method, Figure 5 Method steps S1521 to S1525, etc.

[0177] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0178] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the embodiments of the present application. These equivalent deformations or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A spatiotemporal graph network interpretable method, characterized in that: include: Obtain skeleton sequence, time series mask vector and node sequence mask vector of key points of human body; Performing matrix multiplication on the time series mask vector and the node sequence mask vector to obtain an initial mask vector; Calculating the Hadamard product between the skeleton sequence and the corresponding initial mask vector to obtain a motion sequence; Inputting the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, wherein the spatiotemporal graph network model is used for behavior recognition; Iteratively updating a reference value of an optimization function using the predicted behavior recognition result to obtain a target mask vector for explaining the spatiotemporal graph network model; The initial mask vector is expressed as: in, represents the time series mask vector, represents the node sequence mask vector, represents the initial mask vector; The motion sequence is expressed as: in, represents the Hadamard product, represents the backbone sequence, represents the initial mask vector, represents the motion sequence, T represents the time sequence, C represents the motion three-dimensional coordinate information, V represents the number of key points of each character, and M represents the number of people in the motion sequence.

2. The method for interpreting a spatiotemporal graph network according to claim 1, characterized in that: The iterative updating of the reference value of the optimization function using the predicted behavior recognition result to obtain a target mask vector for explaining the spatiotemporal graph network model includes: Calculating the reference value of the optimization function according to the predicted behavior recognition result and the initial mask vector; The initial mask vector is adjusted according to the reference value until the optimization function reaches a convergence condition, thereby obtaining the target mask vector.

3. The method for interpreting a spatiotemporal graph network according to claim 2, characterized in that: The step of adjusting the initial mask vector according to the reference value until the optimization function reaches a convergence condition to obtain the target mask vector includes: Adjusting the initial mask vector to obtain an optimized mask vector; Calculating an optimized motion sequence according to the optimized mask vector and the skeleton sequence; Inputting the optimized motion sequence into a pre-trained spatiotemporal graph network model to obtain an optimized predicted behavior recognition result; Calculating a reference value of the optimization function corresponding to the optimization prediction behavior recognition result; It is determined whether the optimization function reaches a convergence condition according to the reference value. If the convergence condition is reached, the optimized mask vector is used as the target mask vector; otherwise, the above steps are repeated until the optimization function reaches a convergence condition.

4. The method for interpreting a spatiotemporal graph network according to any one of claims 2 or 3, characterized in that: The optimization function is a mutual information optimization function, which is expressed as: in, represents the predicted behavior recognition result, represents the initial mask vector, represents the entropy function, Indicates the reference value.

5. The method for interpreting a spatiotemporal graph network according to any one of claim 4, characterized in that: The convergence condition of the optimization function is: maximizing the reference value of the optimization function.

6. A visualization method, characterized in that: include: Acquire a motion sequence and a target mask vector, wherein the motion sequence and the target mask vector are calculated using the spatiotemporal graph network interpretable method according to any one of claims 1 to 5; Sampling the motion sequence frame by frame to obtain a display sequence; Visualize the target mask vector of the displayed sequence between different frames.

7. A spatiotemporal graph network interpretable device, characterized in that: include: The first acquisition module is used to acquire the skeleton sequence, time series mask vector and node sequence mask vector of the key points of the human body; An initial mask vector calculation module, used for performing matrix multiplication on the time series mask vector and the node sequence mask vector to obtain an initial mask vector; A motion sequence calculation module, used for calculating the Hadamard product between the skeleton sequence and the corresponding initial mask vector to obtain a motion sequence; A preliminary prediction module, used to input the motion sequence into a pre-trained spatiotemporal graph network model to obtain a predicted behavior recognition result, wherein the spatiotemporal graph network model is used for behavior recognition; An iterative optimization module, used to iteratively update a reference value of an optimization function using the predicted behavior recognition result to obtain a target mask vector for explaining the spatiotemporal graph network model; The initial mask vector is expressed as: in, represents the time series mask vector, represents the node sequence mask vector, represents the initial mask vector; The motion sequence is expressed as: in, represents the Hadamard product, represents the backbone sequence, represents the initial mask vector, represents the motion sequence, T represents the time sequence, C represents the motion three-dimensional coordinate information, V represents the number of key points of each character, and M represents the number of people in the motion sequence.

8. A visualization device, characterized in that: include: A second acquisition module, used to acquire a motion sequence and a target mask vector, wherein the motion sequence and the target mask vector are calculated using the spatiotemporal graph network interpretable method according to any one of claims 1 to 5; A skeleton sequence sampling module, used for sampling the motion sequence frame by frame to obtain a display sequence; A visualization module is used to visualize the target mask vector of the display sequence between different frames.

9. A computer device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor is used to execute the spatiotemporal graph network interpretable method according to any one of claims 1 to 5, or the visualization method according to claim 6 according to the program.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the spatiotemporal graph network interpretable method described in any one of claims 1 to 5, or the visualization method described in claim 6.

Images