A behavior understanding method based on dynamic multimodal spatiotemporal graph attention network

Abstract

The present invention discloses a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network, including: obtaining multi-channel audio information and processing it to construct a global attention artificial potential field; obtaining multi-channel deep video information and inputting it into a target detection network to obtain a four-channel subgraph as a state description of a single target; updating the external features of the global attention artificial potential field and the four-channel subgraph; and processing the four-channel subgraph using an HRNet network and a three-dimensional convolutional network to construct a target view graph and generate a spatial adjacency matrix; and processing using a dynamic multimodal spatiotemporal graph attention network to obtain a multimodal behavior understanding result of an indoor scene. This solution is suitable for various indoor scenes with different target behavior understanding needs. By fully considering the target state characteristics and group environment characteristics through audio and video fusion and graph neural networks, the target behavior understanding speed can be optimized, and the multimodal behavior in indoor scenes can be understood more accurately.

Description

Technical Field

[0001] The present invention relates to the field of behavior understanding, and more specifically to a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network. Background Art

[0002] Intelligent systems' understanding of human behavior has very wide applications in all walks of life, such as virtual reality, intelligent monitoring, advanced human-computer interaction, etc. These applications rely on the intelligent system's perception and analysis of people and the surrounding environment.

[0003] Currently, most behavior understanding methods only focus on understanding behavior in videos or single-frame images. They use deep learning methods to input video clips into a three-dimensional convolutional neural network or input images into a two-dimensional convolutional neural network, use the neural network to extract input features, and then output the corresponding classification prediction results.

[0004] The behavioral intention of an individual is not only unique, but also affected by the environment and other individuals. Although the general method of understanding behavior based on individual spatiotemporal characteristics can effectively extract and understand individual behavioral characteristics, it is difficult to correctly understand the various behaviors in the interaction between people, people and the environment, and people and groups. In addition, human behavior itself is a multimodal process. The method based on video stream lacks descriptions of other dimensions, so it is impossible to correctly infer the behavioral intention of a person when key information is missing at the image level.

[0005] Therefore, how to design a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network that can make full use of the multimodal information of the environment to more accurately understand human behavior is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network, which can comprehensively utilize multimodal information, consider spatiotemporal relationships, and automatically learn key information, thereby achieving accurate and comprehensive behavior understanding of indoor scenes.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] The present invention provides a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network, which is applied to indoor scenes, including:

[0009] Step 1: Obtain and process multi-channel audio information, determine the strength of the artificial potential field for different sound sources based on the virtual connection level, and construct a global attention artificial potential field;

[0010] Step 2: Obtain multi-channel depth video information and input it into the target detection network to obtain a four-channel sub-image as a state description of a single target;

[0011] Step 3: Update the external features of the global attention artificial potential field and the four-channel sub-graph; and use the HRNet network and the three-dimensional convolutional network to process the four-channel sub-graph, construct the target view graph and generate the spatial adjacency matrix;

[0012] Step 4: Based on the updated four-channel subgraph and global attention artificial potential field after external features, and the spatial adjacency matrix, a dynamic multimodal spatiotemporal graph attention network is used for processing to obtain a multimodal behavior understanding result of the indoor scene.

[0013] Wherein, in said step 1, multi-channel audio information processing includes:

[0014] For the ambient noise reduction frequency band in a certain channel of the multi-channel audio information within a preset time interval, median filtering is used to remove abnormal signals;

[0015] Extracting the intensity peak in the ambient noise reduction frequency band after removing the abnormal signal, and performing peak location on the audio information of other channels in the multi-channel audio information;

[0016] The arrival time difference equations of the intensity peaks on multiple channels are solved to obtain the azimuth coordinates of the sound source and transform them into the world coordinate system.

[0017] Furthermore, the peak location includes:

[0018] Convert the intensity peak in the ambient noise reduction frequency band after removing abnormal signals into a spectrum waterfall diagram within the time period;

[0019] The spectrum waterfall diagram is standardized to serve as a two-dimensional image template;

[0020] Based on the two-dimensional image template, the peak location is completed by using a sliding window to perform template matching on the audio information of other channels in the multi-channel audio information.

[0021] Furthermore, in the step 2, the four-channel sub-image is used as a state description of a single target based on the target detection network, including:

[0022] Inputting the multi-channel depth video information into a YOLO detection network to obtain multiple three-dimensional detection frames containing the target;

[0023] The color point cloud in the multi-channel depth video information is transformed into the world coordinate system, and PL-ICP registration and fusion are performed, and duplicate points in the overlapping area are removed by downsampling;

[0024] The three-dimensional detection frame is clustered and fused using the Kmeans method so that the single target has a unique three-dimensional detection frame in the world coordinate system;

[0025] The image is intercepted based on the three-dimensional detection frame after cluster fusion, and the four-channel sub-image is used as a state description of a single target.

[0026] Furthermore, in the step three, the four-channel sub-image is processed using the HRNet network and the three-dimensional convolutional network, including:

[0027] Input the four-channel sub-image into the HRNet network to obtain the corresponding human body posture key point information;

[0028] Smoothing the corresponding human body posture key point information, including taking the average as an estimate of the true value of the human body posture key point based on the point cloud mapping result;

[0029] Based on the three-dimensional convolutional network, the key point information of the smoothed human body posture is recognized to obtain the corresponding behavior code and behavior intensity code;

[0030] The corresponding behavior coding and behavior intensity coding are concatenated with the timestamp and the target posture to obtain comprehensive feature information of the single target.

[0031] Furthermore, in step 4, the dynamic multimodal spatiotemporal graph attention network is used for processing, including: i Update, the updated target node feature is recorded as h i ':

[0032]

[0033] Where σ represents the activation function, i, j and k represent the target node, the first-order real-connected neighbor node of the target node and the first-order virtual-connected neighbor node of the target node respectively; h i 、h j and h k Respectively represent the target node characteristics, the target node first-order real connection neighbor node characteristics and the target node first-order virtual connection neighbor node characteristics; α ii Represents the attention parameter of the target node and the previous frame feature, α ij represents the normalized weight parameter of the first-order real connection neighbor node of the target node, α ik N represents the normalized weight parameter of the first-order virtual link neighbor node of the target node; (i) Represents the first-order real connection neighbor node index set of the target node, Represents the target node's first-order virtual link neighbor node index set.

[0034] Furthermore, the processing using a dynamic multimodal spatiotemporal graph attention network also includes:

[0035] The attention score e of the target node i to the first-order real connected neighbor node j of the target node ij , represented as the target node feature after linear transformation Features of first-order real-connected neighbor nodes of the target node The function of the visual attention matrix E is obtained as:

[0036] E={e ij}

[0037]

[0038] Among them, LeakyReLU represents the activation function, b represents the learning attention score vector, Indicates the previous frame feature of the target node 'h i Features after linear transformation, double vertical lines || represent the concatenation of the two vectors before and after;

[0039] The viewing attention matrix E is normalized to obtain the normalized viewing attention matrix E':

[0040] E'={e ij '=softmax j (e ij )}

[0041] Among them, softmax represents the normalization function.

[0042] Furthermore, the processing using a dynamic multimodal spatiotemporal graph attention network also includes:

[0043] According to the first-order virtual connection neighbor node characteristics h of the target node k The coordinates (x k ,y k ) determines its sound intensity level S(x k ,y k ), forming a sound intensity matrix S;

[0044] S={s kk =S(x k ,y k )}

[0045] Among them, S(x k ,y k ) represents the function of the field intensity in the acoustic artificial potential field with respect to the position;

[0046] Based on the sound intensity matrix S, the sound intensity gravity matrix P of all target nodes is calculated:

[0047]

[0048] Among them, d ik Represents the Euclidean distance between the target node i and the first-order virtual connection neighbor node k of the target node;

[0049] Based on the sound intensity gravity matrix P, the sound intensity gravity is introduced as the attention coefficient in the weight calculation process to obtain the sound intensity attention matrix

[0050]

[0051] Sure Values, including: is the mean value of the elements in the sound intensity gravity matrix P:

[0052]

[0053]

[0054] For the sound intensity attention matrix Perform normalization operation to obtain the normalized sound intensity gravity matrix P':

[0055]

[0056] Among them, softmax represents the normalization function.

[0057] Furthermore, the normalized view attention matrix E' is added to the normalized sound intensity attention matrix P' to obtain the attention matrix

[0058]

[0059] The attention matrix Input the neural network layer in the dynamic multimodal spatiotemporal graph attention network, and the updated feature is h i ”:

[0060]

[0061] Among them, h v represents the first-order neighbor node features of the target node, W represents the learnable weight parameter matrix, W∈R F ' ×F , R F ' ×F represents a real space with dimensions F'×F, N' (u) Represents the first-order neighbor node index set of the target node.

[0062] Furthermore, the neural network layer takes the directed graph target node feature matrix H as a parameter input and takes the updated target node feature matrix H' as an output, which is recorded as:

[0063] H'=F(H)

[0064] The neural network layers are stacked, and the output of the previous neural network layer is used as the input of the subsequent neural network layer. The subsequent neural network layer is expressed as:

[0065] H”=F 2 (H')

[0066] The cross entropy loss function is used to calculate the error between the predicted class label and the true class label:

[0067]

[0068] Among them, y M represents a set of nodes with class labels, F represents the number of categories, y f The valid bits for one-hot classification encoding represent the actual behavior class labels in the data set;

[0069] When the node belongs to the kth type of behavior:

[0070]

[0071] When the prediction results deviate greatly, the L value is large, and the network training parameters are penalized.

[0072] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0073] Beneficial effects:

[0074] 1. By constructing a global attention artificial potential field by determining the strength of the artificial potential field for different sound sources based on the virtual connection level, contextual information can be introduced in the processing of multi-channel audio information, which helps to better focus on specific sound sources and improve the accuracy and robustness of behavior understanding.

[0075] 2. Combining the four-channel subgraph and the spatial adjacency matrix with the global attention artificial potential field, integrating audio information and video information in the spatiotemporal dimension, and processing them using the dynamic multimodal spatiotemporal graph attention network can better model the behavioral characteristics and spatiotemporal relationships in indoor scenes, and improve the effect and accuracy of behavior understanding.

[0076] 3. Based on the fusion of multi-channel audio information and multi-channel deep video information based on the dynamic multimodal spatiotemporal graph attention network, multimodal information and spatiotemporal relationships are used to understand behavior, thereby improving the ability to understand complex behaviors in indoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0078] Figure 1 A flow chart of a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network provided by an embodiment of the present invention;

[0079] Figure 2 A framework diagram of a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network provided by an embodiment of the present invention;

[0080] Figure 3 A schematic diagram of multi-channel audio sound intensity peak matching provided by an embodiment of the present invention;

[0081] Figure 4 A schematic diagram of a global attention artificial potential field provided by an embodiment of the present invention;

[0082] Figure 5 A target view diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Embodiment 1:

[0085] The behavior understanding method of this technical solution in indoor scenes is specifically explained in combination with the teacher's lecture scene:

[0086] A multi-channel audio sensor and a depth camera are installed in a classroom, and a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is applied to improve the teaching effect. There is a teacher and several students in the classroom.

[0087] like Figure 1 and Figure 2 As shown, this embodiment provides a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network, including:

[0088] Step 1: Obtain multi-channel audio information and process it. The multi-channel audio sensors in the classroom capture the teacher's voice during the lecture and the students' answers and discussions. Through virtual connection level processing, the artificial potential field strength of different sound sources is determined to construct a global attention artificial potential field. This step can help the system focus on important sound sources, such as the teacher's explanation and the students' questions.

[0089] Step 2: Obtain multi-channel depth video information The depth camera captures real-time video of teachers and students in the classroom. The object detection network identifies and extracts teachers and students in different positions, and generates a four-channel sub-image as a state description of a single object. This helps the system understand the state and location information of each object in the classroom.

[0090] Step 3: External feature update and spatial adjacency matrix generation The global attention artificial potential field and the four-channel sub-graph are updated with external features, and the four-channel sub-graph is processed using the HRNet network and the three-dimensional convolutional network to construct the target view graph and generate the spatial adjacency matrix. This process helps the system capture the spatial relationship between objects in the classroom more accurately.

[0091] Step 4: Dynamic multimodal spatiotemporal graph attention network processing Based on the updated four-channel subgraph and global attention artificial potential field after external features, as well as the spatial adjacency matrix, the dynamic multimodal spatiotemporal graph attention network is used for processing to obtain the multimodal behavior understanding results of indoor scenes. Through this step, the system can better understand the interactions and behaviors of people in the classroom, and identify information such as the teacher's explanation focus and the students' participation level.

[0092] This technical solution can achieve real-time multimodal behavior understanding in classroom teaching scenarios and improve teaching quality. For example, when a teacher is giving a lecture, the system can automatically identify the status of the teacher and students, analyze the students' participation and listening effect, provide timely feedback and guidance to teachers, help teachers optimize classroom teaching strategies, and improve students' learning effects.

[0093] Embodiment 2:

[0094] In this embodiment, multi-channel audio information processing includes:

[0095] For the ambient noise reduction frequency band t in a certain channel of the multi-channel audio information within a preset time interval s , use median filtering to remove abnormal signals and obtain the noise reduction frequency t s ';

[0096] like Figure 3 As shown, the ambient noise reduction frequency band t in the time domain sUse a sliding window to obtain local kurtosis, extract the intensity peak in the ambient noise reduction frequency band after removing abnormal signals, and locate the peak of other channel audio information in the multi-channel audio information; including:

[0097] Convert the intensity peak in the ambient noise reduction frequency band after removing abnormal signals into a spectrum waterfall diagram within the time period;

[0098] The spectrum waterfall diagram is standardized and used as a two-dimensional image template;

[0099] Based on the two-dimensional image template, the peak location is completed by using sliding window to perform template matching on the audio information of other channels in the multi-channel audio information. Specifically:

[0100] For the g-th sliding window, assuming m n is the n-order central moment of the data in the window, then the kurtosis bk g :

[0101]

[0102] For the sliding window region r where the kurtosis is greater than 0 k , take the point where the peak is located as the key point, and take points at a certain interval, and use the difference between these sampling points and the window mean as the feature of this peak. Use the template matching method on channel ω to find the optimal matching area

[0103] Assuming that the sound source is located at L, the acquisition equipment of channel 1 and channel 2 are installed at M1 and M2. Based on the principle of TDoA technology, the arrival time difference of the same peak on each channel τ = τ1-τ2:

[0104]

[0105] Where c is the speed of sound.

[0106] like Figure 4 As shown, the arrival time difference equations of the intensity peaks on multiple channels are solved to obtain the azimuth coordinates of the sound source and convert them to the world coordinate system. The artificial potential field strength of different sound sources is determined according to their sound intensity levels:

[0107] For the detected peak point p k (x g ,y g ), whose sound intensity satisfies:

[0108]

[0109] Where p(t) is the instantaneous sound pressure, v r (t) is the instantaneous velocity component of the sound source in the direction r of the audio acquisition device. Let the sound quality at this point be S(x g,y g ), then the distance from the point d g The sound intensity gravitational force p on the target at g for:

[0110]

[0111] Where G is the sound intensity coefficient.

[0112] In this embodiment, the four-channel sub-image is used as the state description of a single target based on the target detection network, including:

[0113] The multi-channel depth video information c o :c o ={Point o |Point o =[pos o color o ]};

[0114] Input the YOLO detection network trained with the WiderPerson dataset to obtain multiple two-dimensional pixel point sets containing target individuals Tar2D_List = {Tar2D os = {pix o (m,n)}}, and then map it to three-dimensional space through the pixel-point cloud relationship to obtain the three-dimensional detection box of each target:

[0115] Tar3D_List = {Tar3D os =[point1(x,y,z),point2(x,y,z),point3(x,y,z)]}

[0116] The color point cloud in the multi-channel depth video information is transformed into the world coordinate system, and PL-ICP registration and fusion are performed, and duplicate points in the overlapping area are removed by downsampling;

[0117] For the depth video information c of the oth channel o , and proceed as above.

[0118] Select point cloud c o1 , c o2 The overlapping area is registered, assuming that the overlapping area is Sc, then:

[0119]

[0120] For point cloud c' o1 、c' o2 Use the PLICP algorithm for registration, and take c' o1 The reference surface S ref , for c' o2 Point p inoi , transform q k =(t,θ):

[0121]

[0122] Let the error be in the normal vector The projection on is the smallest, and the required transformation q can be obtained oz .

[0123] By processing the depth video information of each channel in turn as above, the depth video information of all channels can be transformed into the world coordinate system and fused into a complete point cloud of the indoor scene:

[0124]

[0125] The Kmeans method is used to cluster and fuse the 3D detection frame, so that a single target has a unique 3D detection frame in the world coordinate system;

[0126] For the three-dimensional detection box Tar3D os , take its center point: The center points of all detection frames are clustered using the K-Means method, and those within a certain distance are fused so that a single target has a unique detection frame in the world coordinate system:

[0127]

[0128] The image is intercepted based on the 3D detection frame after clustering fusion, and the four-channel sub-image is used as the state description of a single target.

[0129] Through the above steps, the perception information of the same sensor in different ranges can be fused together, reducing the inaccurate estimation of group interaction behavior caused by missed target detection in indoor situations.

[0130] In this embodiment, the four-channel sub-image is processed using the HRNet network and the three-dimensional convolutional network, including:

[0131] Input the four-channel sub-image into the HRNet network to obtain the corresponding human posture key point information;

[0132] Smoothing the corresponding human posture key point information, including taking the average as the estimate of the true value of the human posture key point based on the point cloud mapping result;

[0133] Based on the three-dimensional convolutional network, the key point information of the smoothed human body posture is recognized to obtain the corresponding behavior code and behavior intensity code;

[0134] The corresponding behavior coding and behavior intensity coding are concatenated with the timestamp and target posture to obtain the comprehensive feature information of a single target.

[0135] like Figure 5 As shown in the figure, a target view graph is constructed based on the comprehensive feature information of a single target. Each target is taken as a node. When the view angle is specified as ±60 degrees of the attitude angle, all nodes within the view angle have edges with an in-degree of 1 with the current target node, forming a graph; all target nodes are traversed to obtain a directed graph containing all targets, and then a spatial adjacency matrix is ​​generated.

[0136] In this embodiment, a dynamic multimodal spatiotemporal graph attention network is used for processing, including: i Update, the updated target node feature is recorded as h i ':

[0137]

[0138] Where σ represents the activation function, i, j and k represent the target node, the first-order real-connected neighbor node of the target node and the first-order virtual-connected neighbor node of the target node respectively; h i 、h j and h k Respectively represent the target node characteristics, the target node first-order real connection neighbor node characteristics and the target node first-order virtual connection neighbor node characteristics; α ii Represents the attention parameter of the target node and the previous frame feature, α ij represents the normalized weight parameter of the first-order real connection neighbor node of the target node, α ik N represents the normalized weight parameter of the first-order virtual link neighbor node of the target node; (i) Represents the first-order real connection neighbor node index set of the target node, Represents the target node's first-order virtual link neighbor node index set.

[0139] Processed using a dynamic multimodal spatiotemporal graph attention network, also includes:

[0140] The attention score e of the target node i to the first-order real connected neighbor node j of the target node ij , represented as the target node feature after linear transformation Features of first-order real-connected neighbor nodes of the target node The function of the visual attention matrix E is obtained as:

[0141] E={e ij}

[0142]

[0143] Among them, LeakyReLU represents the activation function, b represents the learning attention score vector, Indicates the previous frame feature of the target node 'hi Features after linear transformation, double vertical lines || represent the concatenation of the two vectors before and after;

[0144] Normalize the view attention matrix E to obtain the normalized view attention matrix E':

[0145] E'={e ij '=softmax j (e ij )}

[0146] Among them, softmax represents the normalization function.

[0147] Processed using a dynamic multimodal spatiotemporal graph attention network, also includes:

[0148] According to the first-order virtual connection neighbor node characteristics h of the target node k The coordinates (x k ,y k ) determines its sound intensity level S(x k ,y k ), forming a sound intensity matrix S;

[0149] S={s kk =S(x k ,y k )}

[0150] Among them, S(x k ,y k ) represents the function of the field intensity in the acoustic artificial potential field with respect to the position;

[0151] Based on the sound intensity matrix S, calculate the sound intensity gravity matrix P of all target nodes:

[0152]

[0153] Among them, d ik Represents the Euclidean distance between the target node i and the first-order virtual connection neighbor node k of the target node;

[0154] Based on the sound intensity gravity matrix P, the sound intensity gravity is introduced as the attention coefficient in the weight calculation process to obtain the sound intensity attention matrix

[0155]

[0156] Sure Values, including: is the mean value of the elements in the sound intensity gravity matrix P:

[0157]

[0158]

[0159] Attention Matrix for Sound Intensity Perform normalization operation to obtain the normalized sound intensity gravity matrix P':

[0160]

[0161] Among them, softmax represents the normalization function.

[0162] In this embodiment, the normalized viewing direction attention matrix E' and the normalized sound intensity attention matrix P' are added to obtain the attention matrix

[0163]

[0164] The attention matrix Input the neural network layer in the dynamic multimodal spatiotemporal graph attention network, and the updated feature is h i ”:

[0165]

[0166] Among them, h v represents the first-order neighbor node features of the target node, W represents the learnable weight parameter matrix, W∈R F ' ×F , R F ' ×F represents a real space with dimensions F'×F, N' (u) Represents the first-order neighbor node index set of the target node.

[0167] The neural network layer takes the directed graph target node feature matrix H as a parameter input and the updated target node feature matrix H' as output, which is recorded as:

[0168] H'=F(H)

[0169] The neural network layers are stacked, and the output of the previous neural network layer is used as the input of the subsequent neural network layer. The subsequent neural network layer is expressed as:

[0170] H”=F 2 (H')

[0171] The cross entropy loss function is used to calculate the error between the predicted class label and the true class label:

[0172]

[0173] Among them, y M represents a set of nodes with class labels, F represents the number of categories, y fThe valid bits for one-hot classification encoding represent the actual behavior class labels in the data set;

[0174] When the node belongs to the kth type of behavior:

[0175]

[0176] When the prediction result deviates greatly, the L value is large, and the network training parameters are penalized to make the prediction result closer to the true class label.

[0177] The technical solution of the above embodiment integrates multi-channel audio information and multi-channel deep video information, introduces a global attention artificial potential field, and applies a dynamic multimodal spatiotemporal graph attention network for behavior understanding. It can comprehensively utilize multimodal information, consider spatiotemporal relationships, and automatically learn key information, thereby achieving accurate and comprehensive behavior understanding of indoor scenes. It has broad application prospects in the fields of intelligent indoor monitoring, smart home, human-computer interaction, etc.

[0178] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0179] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A behavior understanding method based on dynamic multimodal spatiotemporal graph attention network, applied to indoor scenes, characterized by: include: Step 1: Acquire and process multi-channel audio information, determine the strength of the artificial potential field for different sound sources based on the virtual connection level, and construct a global attention artificial potential field; wherein the virtual connection level is the sound intensity level; Step 2: Obtain multi-channel depth video information and input it into the target detection network to obtain a four-channel sub-image as a state description of a single target; Step 3: Update the external features of the global attention artificial potential field and the four-channel sub-graph; and use the HRNet network and the three-dimensional convolutional network to process the four-channel sub-graph, construct the target view graph and generate the spatial adjacency matrix; Step 4: Based on the updated four-channel subgraph and global attention artificial potential field after external features, and the spatial adjacency matrix, a dynamic multimodal spatiotemporal graph attention network is used for processing to obtain a multimodal behavior understanding result of the indoor scene.

2. According to claim 1, a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is characterized in that: In the step 1, the multi-channel audio information processing includes: For the ambient noise reduction frequency band in a certain channel of the multi-channel audio information within a preset time interval, median filtering is used to remove abnormal signals; Extracting the intensity peak in the ambient noise reduction frequency band after removing the abnormal signal, and performing peak location on the audio information of other channels in the multi-channel audio information; The arrival time difference equations of the intensity peaks on multiple channels are solved to obtain the azimuth coordinates of the sound source and transform them into the world coordinate system.

3. According to claim 2, a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is characterized in that: The peak location comprises: Convert the intensity peak in the ambient noise reduction frequency band after removing abnormal signals into a spectrum waterfall diagram within the time period; The spectrum waterfall diagram is standardized to serve as a two-dimensional image template; Based on the two-dimensional image template, the peak location is completed by using a sliding window to perform template matching on the audio information of other channels in the multi-channel audio information.

4. According to claim 1, a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is characterized in that: In the step 2, the four-channel sub-image is used as a state description of a single target based on the target detection network, including: Inputting the multi-channel depth video information into a YOLO detection network to obtain multiple three-dimensional detection frames containing the target; The color point cloud in the multi-channel depth video information is transformed into the world coordinate system, and PL-ICP registration and fusion are performed, and duplicate points in the overlapping area are removed by downsampling; The three-dimensional detection frame is clustered and fused using the Kmeans method so that the single target has a unique three-dimensional detection frame in the world coordinate system; The image is intercepted based on the three-dimensional detection frame after cluster fusion, and the four-channel sub-image is used as a state description of a single target.

5. According to claim 1, a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is characterized in that: In the step 3, the four-channel sub-image is processed using the HRNet network and the three-dimensional convolutional network, including: Input the four-channel sub-image into the HRNet network to obtain the corresponding human body posture key point information; Smoothing the corresponding human body posture key point information, including taking the average as an estimate of the true value of the human body posture key point based on the point cloud mapping result; Based on the three-dimensional convolutional network, the key point information of the smoothed human body posture is recognized to obtain the corresponding behavior code and behavior intensity code; The corresponding behavior coding and behavior intensity coding are concatenated with the timestamp and the target posture to obtain comprehensive feature information of the single target.

6. According to claim 1, a behavior understanding method based on a dynamic multimodal spatiotemporal graph attention network is characterized in that: In step 4, a dynamic multimodal spatiotemporal graph attention network is used for processing, including: i Update, the updated target node feature is recorded as h i ': Where σ represents the activation function, i, j and k represent the target node, the first-order real-connected neighbor node of the target node and the first-order virtual-connected neighbor node of the target node respectively; h i 、h j and h k Respectively represent the target node characteristics, the target node first-order real connection neighbor node characteristics and the target node first-order virtual connection neighbor node characteristics; α ii Represents the attention parameter of the target node and the previous frame feature, α ij represents the normalized weight parameter of the first-order real connection neighbor node of the target node, α ik N represents the normalized weight parameter of the first-order virtual link neighbor node of the target node; (i) Represents the first-order real connection neighbor node index set of the target node, Represents the target node's first-order virtual link neighbor node index set.

7. The behavior understanding method based on dynamic multimodal spatiotemporal graph attention network according to claim 6 is characterized in that: The processing using a dynamic multimodal spatiotemporal graph attention network also includes: The attention score e of the target node i to the first-order real connected neighbor node j of the target node ij , represented as the target node feature after linear transformation Features of first-order real-connected neighbor nodes of the target node The function of , the visual attention matrix E is obtained as: And={and ij } Among them, LeakyReLU represents the activation function, b represents the learning attention score vector, Indicates the previous frame feature of the target node 'h i Features after linear transformation, double vertical lines || represent the concatenation of the two vectors before and after; The viewing attention matrix E is normalized to obtain the normalized viewing attention matrix E': E'={e ij '=softmax j (e ij )} Among them, softmax represents the normalization function.

8. The behavior understanding method based on dynamic multimodal spatiotemporal graph attention network according to claim 7 is characterized in that: The processing using a dynamic multimodal spatiotemporal graph attention network also includes: According to the first-order virtual connection neighbor node characteristics h of the target node k The coordinates (x k ,y k ) determines its sound intensity level S(x k ,y k ), forming a sound intensity matrix S; S={s kk =S(x k ,y k )} Among them, S(x k ,y k ) represents the function of the field intensity in the acoustic artificial potential field with respect to the position; Based on the sound intensity matrix S, the sound intensity gravity matrix P of all target nodes is calculated: Among them, d ik represents the Euclidean distance between the target node i and the first-order virtual connection neighbor node k of the target node, and G represents the sound intensity coefficient; Based on the sound intensity gravity matrix P, the sound intensity gravity is introduced as the attention coefficient in the weight calculation process to obtain the sound intensity attention matrix Sure Values, including: is the mean value of the elements in the sound intensity gravity matrix P: Among them, M 2 represents the number of elements in the sound intensity gravity matrix P; For the sound intensity attention matrix Perform normalization operation to obtain the normalized sound intensity gravity matrix P': Among them, softmax represents the normalization function.

9. The behavior understanding method based on dynamic multimodal spatiotemporal graph attention network according to claim 8 is characterized in that: Add the normalized view attention matrix E' and the normalized sound intensity attention matrix P' to get the attention matrix The attention matrix Input the neural network layer in the dynamic multimodal spatiotemporal graph attention network, and the updated feature is h i ”: Among them, h v represents the first-order neighbor node features of the target node, W represents the learnable weight parameter matrix, W∈R F'×F , R F'×F represents a real space with dimensions F'×F, N' (u) Represents the first-order neighbor node index set of the target node.

10. The behavior understanding method based on dynamic multimodal spatiotemporal graph attention network according to claim 9 is characterized in that: The neural network layer takes the directed graph target node feature matrix H as a parameter input and takes the updated target node feature matrix H' as an output, which is recorded as: H'=F(H) The neural network layers are stacked, and the output of the previous neural network layer is used as the input of the subsequent neural network layer. The subsequent neural network layer is expressed as: H”=F 2 (H') The cross entropy loss function is used to calculate the error between the predicted class label and the true class label: Among them, y M represents a set of nodes with class labels, F represents the number of categories, y f The effective bit of the one-hot classification code represents the actual behavior class label in the data set, and Ln(·) represents the natural logarithm function; When the node belongs to the kth type of behavior: When the prediction results deviate greatly, the L value is large, and the network training parameters are penalized.

Images