A behavior recognition method based on space-time relationship and electronic equipment

Abstract

The application is suitable for the technical field of device management, and provides a behavior recognition method based on space-time relationship and an electronic device. The method comprises the following steps: receiving target video data to be recognized; introducing the target video data into a preset inter-frame action extraction network to obtain inter-frame action feature data; introducing the inter-frame action feature data into a feature extraction network to output sparse feature data corresponding to the target video data; the feature extraction network is generated by performing sparse constraint processing on each convolution kernel in a pooling fusion network through selected weights; introducing the target video data into a context attention network to determine gait behavior data of a target object in the target video data; and obtaining a behavior category of the target object according to the gait behavior data and the sparse feature data. The above method can greatly reduce the calculation cost of video data in the behavior recognition process, thereby improving the operation efficiency.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a behavior recognition method and electronic device based on spatiotemporal relationships. Background Technology

[0002] With the continuous development of artificial intelligence technology, computers can assist users in performing various types of recognition operations to improve processing efficiency. For example, when users analyze video data, artificial intelligence algorithms can determine the behavior type of a target person in the video data, making it easier for users to analyze the target person. For example, when tracking the behavior of a target person or monitoring dangerous actions in key areas, the behavior recognition of artificial intelligence will greatly reduce the user's workload, thereby improving analysis efficiency.

[0003] Existing behavior recognition technologies often use optical flow information to determine the temporal and spatial information of a target object in a video, thereby determining the type of behavior of that target object. However, extracting optical flow frame by frame to determine the optical flow information of the entire video data requires building a large extraction network, which requires high computing power and large storage of the neural network, thus greatly increasing the computing cost of the computing device and reducing the computing efficiency. Summary of the Invention

[0004] This application provides a behavior recognition method, apparatus, electronic device, and storage medium based on spatiotemporal relationships. It can solve the problem that existing behavior recognition technologies often use optical flow information to determine the temporal and spatial information of a target object in a video, thereby determining the behavior type of the target object. However, extracting optical flow frame by frame to determine the optical flow information of the entire video data requires the establishment of a large extraction network. The device requires high computing power and large storage of the neural network, which greatly increases the computing cost of the computing device and reduces the computing efficiency.

[0005] In a first aspect, embodiments of this application provide a behavior recognition method based on spatiotemporal relationships, including:

[0006] Receive the target video data to be identified;

[0007] The target video data is imported into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data.

[0008] The inter-frame motion feature data is imported into the feature extraction network, which outputs sparse feature data corresponding to the target video data. The feature extraction network is generated by selecting weights and applying sparsity constraints to each convolutional kernel in the pooling fusion network.

[0009] The target video data is imported into a contextual attention network to determine the gait behavior data of the target object in the target video data; the contextual attention network is used to extract the positional relationship between the target object and environmental objects in the target video data.

[0010] The behavior category of the target object is obtained based on the gait behavior data and the sparse feature data.

[0011] In one possible implementation of the first aspect, before importing the inter-frame motion feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes:

[0012] The selection weight with a value of 0 is configured for at least one convolutional kernel to be identified in the pooling fusion network to obtain the network to be corrected;

[0013] Multiple preset training feature data are input into the network to be corrected to generate a first training result, and multiple training feature data are input into the pooling fusion network to generate a second training result;

[0014] Based on the first training result and the second training result, the loss value of the network to be corrected is determined;

[0015] If the loss value is less than or equal to the loss threshold, the convolution to be identified with the selected weights configured will be identified as a redundant convolution kernel.

[0016] If the loss value is greater than the preset loss threshold, the convolutional kernel to be identified with the selected weights will be identified as a necessary convolutional kernel.

[0017] Return to the operation of configuring the selection weight with a value of 0 for at least one unidentified convolutional kernel in the pooling fusion network to obtain the network to be corrected, until all the unidentified convolutional kernels in the pooling fusion network have been classified;

[0018] The feature extraction network is generated based on all the necessary convolutional kernels.

[0019] In one possible implementation of the first aspect, the feature training data is associated with a baseline action label; the feature extraction network generated based on the feature training data is associated with the baseline action label.

[0020] Before importing the inter-frame motion feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes:

[0021] Multiple candidate action labels are determined based on the inter-frame motion data;

[0022] Based on the multiple candidate action labels and the baseline action labels corresponding to each candidate extraction network, the matching degree between each candidate extraction network is calculated.

[0023] The candidate extraction network with the highest matching degree is selected as the feature extraction network.

[0024] In one possible implementation of the first aspect, the step of importing the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data includes:

[0025] Determine the image tensors of any two consecutive video image frames within the target video data;

[0026] Based on the key positions of the target object in the video image frame, the coordinates of multiple feature points are determined; the coordinates of the feature points are determined based on the gait behavior of the target object.

[0027] In the image tensor, the tensor representation of the coordinates of each feature point is determined, and the feature vector of the target object in the video image frame is generated based on the coordinate representation of all the feature points.

[0028] Based on the feature vectors of any two consecutive video image frames, a displacement correlation matrix is ​​constructed; the displacement correlation matrix is ​​used to determine the displacement correlation score between the coordinates of each feature point in one video image frame and the coordinate points of another video image frame.

[0029] The maximum displacement distance between the coordinates of each feature point and the two consecutive video image frames is determined based on the displacement correlation matrix, and the displacement matrix of the target object is determined based on all the maximum displacement distances.

[0030] The displacement matrix is ​​imported into a preset feature transformation model to generate motion feature sub-data for any two consecutive video image frames.

[0031] The inter-frame motion feature data is obtained based on the motion feature sub-data of all the video image frames.

[0032] In one possible implementation of the first aspect, prior to receiving the target video data to be identified, the method further includes:

[0033] Acquire sample video data for training the behavior recognition module; the behavior recognition module includes the inter-frame action extraction network, the feature extraction network, and the context attention network;

[0034] Positive sample data and negative sample data are generated based on the sample video data; the positive sample data is obtained by interfering with the background information in the sample video data; the negative sample data is obtained by interfering with the frame sequence of sample video frames in the sample video data.

[0035] First spatial information and first optical flow information are generated using the positive sample data, and second spatial information and second optical flow information are generated using the negative sample data;

[0036] Spatial enhancement information is obtained based on the first spatial information and the second spatial information;

[0037] Optical flow enhancement information is obtained based on the second optical flow information and the first optical flow information;

[0038] The spatial enhancement information and the optical flow enhancement information are imported into the behavior recognition module to obtain the training recognition results of the sample video data;

[0039] The position learning parameters within the initial recognition module are pre-trained based on the training results of all the sample video data to obtain the behavior recognition module.

[0040] In one possible implementation of the first aspect, generating positive sample data and negative sample data based on the sample video data includes:

[0041] The sample video data is divided into multiple video segments according to the preset action time duration; the duration of each video segment is no longer than the action time duration.

[0042] According to the preset disorder processing algorithm, the frame sequence number of the sample video frames in each video segment is updated respectively;

[0043] The negative sample data is obtained by encapsulating each sample video frame based on the updated frame number.

[0044] In one possible implementation of the first aspect, the step of importing the target video data into a contextual attention network to determine the gait behavior data of the target object in the target video data further includes:

[0045] Determine the target object and at least one environment object within each video image frame of the target video data;

[0046] A first context feature is determined based on the first position coordinates of each key feature point of the target object in all the video image frames; the key feature points are human key points related to the gait of the target object.

[0047] The second context feature is determined based on the relative positional relationship between the target object and the environment object in each of the video frames;

[0048] The first contextual feature and the second contextual feature are imported into the contextual attention network to generate the gait behavior data.

[0049] Secondly, embodiments of this application provide a behavior recognition device based on spatiotemporal relationships, including:

[0050] The target video data receiving unit is used to receive the target video data to be identified.

[0051] The inter-frame motion feature data extraction unit is used to import the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data.

[0052] The sparse feature data unit is used to import the inter-frame action feature data into the feature extraction network and output the sparse feature data corresponding to the target video data; the feature extraction network is generated by selecting weights to perform sparsity constraint processing on each convolution kernel in the pooling fusion network.

[0053] A gait behavior data recognition unit is used to import the target video data into a contextual attention network to determine the gait behavior data of the target object in the target video data; the contextual attention network is used to extract the positional relationship between the target object and environmental objects in the target video data.

[0054] The behavior recognition unit is used to obtain the behavior category of the target object based on the gait behavior data and the sparse feature data.

[0055] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects above.

[0056] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0057] Fifthly, embodiments of this application provide a computer program product that, when run on a server, causes the server to execute the method described in any one of the first aspects above.

[0058] The beneficial effects of this application embodiment compared with the prior art are as follows: After receiving the target video data that needs to be recognized for behavior, the target video data is imported into the inter-frame action extraction network to extract the action feature information between each video image frame, and action feature data is generated based on the action feature information between all video image frames. Then, the action feature data is imported into the feature extraction network for feature extraction to obtain the corresponding sparse feature data. Since the feature extraction network is obtained by selecting weights to perform sparsity constraint processing on the convolution kernels in the pooling fusion network, the entire feature extraction network will reduce unnecessary convolution kernels, thereby reducing the network size, reducing not only the amount of computation but also the network resource consumption. At the same time, in order to consider the relationship between action behaviors in the global dimension, a context attention network is introduced to determine the gait behavior data of the target object in the entire target video data. Finally, by extracting two types of data, the behavior category of the target object in the target video data is determined, thus achieving the purpose of automatically recognizing the behavior category. Compared with existing behavior recognition technologies, the embodiments of this application do not require calculating the optical flow information of the entire video data. Instead, they determine the action feature information between each video frame through a plug-and-play inter-frame action retrieval network, thereby greatly reducing the computing cost of computing devices and reducing the amount of computation. Furthermore, the pooling fusion network is subjected to sparsity constraint processing, which reduces the network size, thereby reducing resource consumption and further improving the recognition efficiency. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram illustrating the implementation of a behavior recognition method based on spatiotemporal relationships according to an embodiment of this application;

[0061] Figure 2 This is a schematic diagram of the structure of an inter-frame action extraction network provided in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of a pooling fusion network provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the structure of a context attention network provided in an embodiment of this application;

[0064] Figure 5This is a schematic diagram of an implementation of S102 of a behavior recognition method based on spatiotemporal relationship provided in the second embodiment of this application;

[0065] Figure 6 This is a schematic diagram of an implementation of S102 of a behavior recognition method based on spatiotemporal relationship provided in the three embodiments of this application;

[0066] Figure 7 This is a schematic diagram of an implementation of S102 of a behavior recognition method based on spatiotemporal relationships provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram illustrating an implementation of a behavior recognition method based on spatiotemporal relationships, provided in another embodiment of this application.

[0068] Figure 9 This is a schematic diagram illustrating an implementation of a behavior recognition method S104 based on spatiotemporal relationships provided in an embodiment of this application;

[0069] Figure 10 This is a schematic diagram of the behavior recognition device based on spatiotemporal relationships provided in an embodiment of this application;

[0070] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0071] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0072] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0073] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0074] The behavior recognition method based on spatiotemporal relationships provided in this application can be applied to electronic devices capable of performing behavior recognition on video data, such as smartphones, servers, tablets, laptops, ultra-mobile personal computers (UMPCs), and netbooks. This application does not impose any limitations on the specific type of electronic device.

[0075] Please see Figure 1 , Figure 1 The illustration shows an implementation diagram of a behavior recognition method based on spatiotemporal relationships provided in an embodiment of this application. The method includes the following steps:

[0076] In S101, the target video data to be identified is received.

[0077] In this embodiment, the electronic device may be configured with a video database containing multiple video data sets. When behavior recognition is required for a specific video data set within the video database, the electronic device will identify that video data as the target video data and perform subsequent processing. Each video data set in the video database may be configured with a behavior identifier. For video data whose behavior category has been identified, its behavior identifier contains the identified behavior category; for video data whose behavior category has not been identified, the behavior identifier is empty. In this case, the electronic device can read whether the behavior identifier is empty and identify the video data with an empty behavior identifier as the target video data.

[0078] In one possible implementation, the target video data can specifically be a video server. When a user needs to perform behavior recognition on a video, they can install the corresponding client program on their local user terminal, import the target video data to be recognized into the client program, and initiate a recognition request. After receiving the recognition request, the user terminal can establish a communication connection with the video server through the client program, send the target video data to the video server, and perform behavior recognition through the recognition server.

[0079] In one possible implementation, to improve the efficiency of behavior recognition, the electronic device can be set with a corresponding video duration threshold. If the video duration of the original video data is greater than the aforementioned video duration threshold, the original video data can be divided into two or more video segments, with the video duration of each video segment not exceeding the aforementioned video duration threshold. The divided video segments are then identified as target video data, and subsequent behavior recognition operations are performed.

[0080] In S102, the target video data is imported into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data.

[0081] In this embodiment, to reduce the computational burden of behavior recognition, the action behavior recognition module of the electronic device is equipped with an inter-frame action extraction network. This network specifically determines the action feature information between any two adjacent video image frames. That is, the focus of the inter-frame action extraction network is not the user's behavior globally, but rather the action changes between every two frames. By sorting through all the action changes between frames, the complete action of the entire video can be obtained, facilitating subsequent behavior recognition. Compared to global optical flow information, the inter-frame action extraction network provided in this embodiment is plug-and-play. The amount of data input to the network each time is specifically the data volume of two video image frames, rather than importing the entire target video data into the recognition network to extract optical flow information. This reduces the cache space usage and also lowers the computational requirements of the computer.

[0082] In one possible implementation, the method for determining the motion feature information between the aforementioned video image frames can be as follows: by using the aforementioned inter-frame motion extraction network, the object region of the target object is identified, and then the area deviation between the two object regions is identified. Based on the direction, position, and size of the deviation area, the motion feature information of the target object is determined. Then, based on the frame number of each video image frame, the number of each motion feature information is determined, and all motion feature information is encapsulated based on the number to generate the aforementioned motion feature data.

[0083] For example, Figure 2 A schematic diagram of the structure of an inter-frame action extraction network provided in an embodiment of this application is shown. See also... Figure 2 As shown, the input data of this inter-frame action extraction network consists of two video image frames, namely image t and image t+1. These two video image frames are adjacent in frame number. The electronic device can perform vector transformation on these two video image frames through a vector transformation module, then perform dimensionality reduction processing through a pooling layer, and determine the displacement information between the vector identifiers corresponding to the two video image frames through an activation layer and a displacement calculation module. Subsequently, the action recognition unit determines the action information between the two video image frames. Specifically, the action recognition unit can be composed of multiple convolutional layers, as shown in the figure, including a first convolutional layer based on a 1*7*7 convolutional kernel, a second convolutional layer based on a 1*3*3 convolutional kernel, a third convolutional layer based on a 1*3*3 convolutional kernel, and a fourth convolutional layer based on a 1*3*3 convolutional kernel.

[0084] In S103, the inter-frame motion feature data is imported into the feature extraction network, and the sparse feature data corresponding to the target video data is output. The feature extraction network is generated by selecting weights to perform sparsity constraint processing on each convolutional kernel in the pooling fusion network.

[0085] In this embodiment, before performing sparsity constraint processing, the electronic device can be configured with a corresponding pooling fusion network. Since the action feature information in the aforementioned inter-frame action extraction module is discrete, feature extraction is required to determine continuous actions for subsequent action recognition. Based on this, the electronic device can import the inter-frame action feature data into the aforementioned pooling fusion network, perform pooling dimensionality reduction processing, and fuse features to output the corresponding fused feature data. The fused feature data can be represented as follows:

[0086]

[0087] Where Maxpool is the fused feature data; Actioni is the inter-frame action information corresponding to the i-th video image frame; N is the total number of frames in the target video data; and T is the feature transpose.

[0088] Furthermore, as another embodiment of this application, the pooling fusion network is specifically a homologous bilinear pooling network. Homologous bilinear pooling generates a symmetric matrix by calculating the outer product of features at different spatial locations, and then performs average pooling on this matrix to obtain bilinear features. It can provide stronger feature representations than linear models and can be optimized in an end-to-end manner. Traditional global average pooling (GAP) only captures first-order statistical information, ignoring more refined details useful for behavior recognition. To address this issue, this paper proposes to borrow the bilinear pooling method used in fine-grained classification and fuse it with the GAP method, so that more refined features can be extracted for behaviors with high similarity, thereby obtaining better recognition results.

[0089] For example, Figure 3 A schematic diagram of the pooling fusion network provided in an embodiment of this application is shown. See also... Figure 3As shown, the pooling fusion network combines bilinear pooling and first-order pooling. Features extracted by the last convolutional layer are processed by a bilinear pooling module inserted before global average pooling to capture the second-order statistics of the spatial feature map, thus obtaining a second-order classification output. This second-order output is then added to the first-order feature vector obtained from global average pooling to obtain the classification output vector. By combining first-order and second-order vectors, large contextual cues and fine-grained behavioral information can be captured, enriching the classification layer of existing behavior recognition networks. Simultaneously, the original GAP branch is crucial for backpropagation during end-to-end training, reducing the training difficulty of the bilinear pooling module.

[0090] As can be seen, the aforementioned pooling fusion network contains a large number of convolutional and pooling layers. To process inter-frame action feature data, each convolutional and pooling layer can be configured with one or more convolutional kernels to extract features from different action dimensions. However, some convolutional kernels may be redundant for action dimension feature extraction, meaning they do not contribute to the result, thus greatly increasing the amount of data in the entire network. Therefore, electronic devices can optimize the aforementioned pooling fusion network by determining the contribution of each convolutional kernel to the subsequent output result through different selection weights, and identifying convolutional kernels that contribute little to the output result, or even convolutional kernels that have no impact on the result, i.e., redundant convolutional kernels. These redundant convolutional kernels are removed from the aforementioned pooling fusion network, thereby reducing the overall network size and eliminating unnecessary data in the output sparse feature data, thus improving the accuracy of recognition.

[0091] In one possible implementation, the above-mentioned sparsity constraint processing can be as follows: the electronic device contains multiple redundant network templates, each containing at least one selection weight with a weight value of 0. The redundant network templates are superimposed on the pooling fusion network, and the loss value between the verification data of the superimposed pooling fusion network and the baseline data of the pooling fusion network before superposition is identified. If the loss value is less than a preset loss threshold, the pooling fusion network after superimposing the redundant network templates is identified as the above-mentioned feature extraction network.

[0092] In S104, the target video data is imported into a contextual attention network to determine the gait behavior data of the target object in the target video data; the contextual attention network is used to extract the relative positional relationship between the target object and the environment object in the target video data.

[0093] In this embodiment, since the inter-frame action extraction network mainly focuses on local action changes, a contextual attention network is introduced in the electronic device to ensure recognition accuracy, enabling the recognition of global action changes. Specifically, this contextual attention network determines the changes in the positional relationship between the target object and the environment, thus identifying global action changes. Therefore, within the contextual attention network, each video frame in the target video data is labeled with both the target object and the environment, and the positional change vector between the target object and the environment in each video frame is identified. Based on the positional change vectors between each video frame, feature extraction and contextual attention recognition are performed to obtain the aforementioned gait behavior data.

[0094] For example, Figure 4 A schematic diagram of the structure of a context attention network provided in an embodiment of this application is shown. See also Figure 4 As shown, this contextual attention network can extract features from target video data, perform object detection, key node detection, and human detection. Object detection is used to identify environmental objects, human detection is used to identify target objects, and key node detection is used to determine gait changes of the human body. Finally, it uses a graph neural network convolutional layer to perform contextual attention, thereby outputting the corresponding gait behavior data.

[0095] In S105, the behavior category of the target object is obtained based on the gait behavior data and the sparse feature data.

[0096] In this embodiment, after obtaining gait behavior data and sparse feature data, the electronic device can import them into a fully connected layer, determine the confidence level between each candidate behavior category, and select the candidate behavior category with the highest confidence level as the behavior category of the target object, so as to achieve the purpose of behavior recognition of the target object.

[0097] In one possible implementation, the target video data is quite long, so the target object may contain multiple types of actions throughout the entire video length. In this case, the electronic device can output a sequence of actions based on the order in which the actions occur. This sequence of actions contains multiple elements, each of which corresponds to a behavior category.

[0098] As can be seen from the above, the behavior recognition method based on spatiotemporal relationship provided in this application, after receiving the target video data that needs to be recognized for behavior, imports the target video data into an inter-frame action extraction network to extract action feature information between each video image frame, and generates action feature data based on the action feature information between all video image frames. Then, the action feature data is imported into a feature extraction network for feature extraction to obtain the corresponding sparse feature data. Since the feature extraction network is obtained by selecting weights to perform sparsity constraint processing on the convolution kernels in the pooling fusion network, the entire feature extraction network will reduce unnecessary convolution kernels, thereby reducing the network size, reducing not only the amount of computation but also the network resource consumption. At the same time, in order to consider the relationship between action behaviors in the global dimension, a context attention network is introduced to determine the gait behavior data of the target object in the entire target video data. Finally, by extracting two types of data, the behavior category of the target object in the target video data is determined, thus achieving the purpose of automatically recognizing the behavior category. Compared with existing behavior recognition technologies, the embodiments of this application do not require calculating the optical flow information of the entire video data. Instead, they determine the action feature information between each video frame through a plug-and-play inter-frame action retrieval network, thereby greatly reducing the computing cost of computing devices and reducing the amount of computation. Furthermore, the pooling fusion network is subjected to sparsity constraint processing, which reduces the network size, thereby reducing resource consumption and further improving the recognition efficiency.

[0099] Figure 5 A flowchart illustrating the specific implementation of a behavior recognition method based on spatiotemporal relationships according to a second embodiment of the present invention is shown. See also... Figure 5 In contrast Figure 1 In the embodiment provided in this embodiment, before importing the inter-frame action feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes: S501 to S507, which are detailed below:

[0100] In S501, the selection weights with a value of 0 are configured for at least one convolutional kernel to be identified in the pooling fusion network to obtain the network to be corrected.

[0101] In this embodiment, the electronic device may store a pooling fusion network, which may be obtained through a cloud server or obtained by training a native network based on preset training data. As described in S103, the pooling fusion network is specifically used for feature extraction to determine continuous actions so as to perform subsequent action recognition.

[0102] In this embodiment, in order to identify redundant convolutional kernels contained in the pooling fusion network, the electronic device can adjust the contribution of each convolutional kernel to the output by selecting weights. Therefore, a selection weight with a weight value of 0 can be set and assigned to any convolutional kernel to be identified in order to shield the influence of the convolutional kernel to be identified on the subsequent data output.

[0103] In this embodiment, the number of convolutional kernels with a weight of 0 selected each time can be one or more, depending on the actual situation. The method of selecting the convolutional kernels to be identified can be random selection or sequential selection based on preset rules, which is not limited here.

[0104] In S502, multiple preset training feature data are input into the network to be corrected to generate a first training result, and multiple training feature data are input into the pooling fusion network to generate a second training result.

[0105] In this embodiment, the electronic device can store multiple training feature data. The process of generating training feature data is the same as the process of generating inter-frame motion feature data, which is based on the training video being imported into the inter-frame motion extraction network. The electronic device can import the training feature data into the adjusted network to be calibrated to obtain a first training result. At the same time, in order to determine the impact of selecting a weight of 0 on the output, the electronic device can import the same training feature data into a pooling fusion network to obtain a second training result.

[0106] In S503, the loss value of the network to be corrected is determined based on the first training result and the second training result.

[0107] In this embodiment, since the first training result and the second training result are generated based on the same training feature data, if selecting a convolutional kernel with weight 0 has a small impact on the result, the similarity between the two training results is high (i.e., the loss value is small); conversely, if selecting a convolutional kernel with weight 0 has a large impact on the result, the similarity between the two training results is low (i.e., the loss value is large). Therefore, the electronic device can determine whether the convolutional kernel to be identified is a redundant convolutional kernel by using the loss value between the training results corresponding to multiple training data.

[0108] In S504, if the loss value is less than or equal to the loss threshold, the convolution to be identified with the selected weights configured is identified as a redundant convolution kernel.

[0109] In S505, if the loss value is greater than a preset loss threshold, the convolutional kernel to be identified with the selected weights is identified as a necessary convolutional kernel.

[0110] In this embodiment, if the loss value corresponding to all training feature data is less than or equal to the loss threshold, the convolutional kernel to be identified with a selection weight of 0 can be identified as a redundant convolutional kernel; conversely, if the loss value corresponding to any training feature data is greater than the above-mentioned loss threshold, the convolutional kernel to be identified with a selection weight of 0 can be identified as a necessary convolutional kernel, and the type of the convolutional kernel to be identified is determined by the magnitude of the loss value.

[0111] In S506, the operation of configuring the selection weights with a value of 0 for at least one convolutional kernel to be identified in the pooling fusion network is returned to obtain the network to be corrected, until all the convolutional kernels to be identified in the pooling fusion network have been classified.

[0112] In this embodiment, after determining the type of convolution kernel to be identified, the electronic device can return to execute the operation of S501 until all convolution kernels to be identified have been classified, that is, identified as redundant convolution kernels or necessary convolution kernels.

[0113] In S507, the feature extraction network is generated based on all the necessary convolutional kernels.

[0114] In this embodiment, all redundant convolutional kernels in the pooling fusion network are removed, and all remaining necessary convolutional kernels are used to generate the corresponding feature extraction network, thereby reducing the network size of the feature extraction network.

[0115] In this embodiment, by configuring and selecting convolution kernels with weights of 0, the influence of some convolution kernels on the output results can be shielded, thereby identifying the convolution kernels that have a greater impact on the output, thus simplifying the overall size of the convolution kernel and reducing the computational load of behavior recognition.

[0116] Figure 6 A flowchart illustrating a specific implementation of a behavior recognition method based on spatiotemporal relationships provided in the third embodiment of this application is shown. See also Figure 6 In contrast Figure 1 In the embodiment provided in this embodiment, before importing the inter-frame action feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes: S601 to S603, which are detailed below:

[0117] Furthermore, the feature training data is associated with a baseline action label; the feature extraction network generated based on the feature training data is associated with the baseline action label;

[0118] Before importing the inter-frame motion feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes:

[0119] In S601, multiple candidate action tags are determined based on the inter-frame action data.

[0120] In S602, the matching degree between each candidate extraction network is calculated based on the multiple candidate action labels and the baseline action labels corresponding to each candidate extraction network.

[0121] In S603, the candidate extraction network with the highest matching degree is selected as the feature extraction network.

[0122] In this embodiment, by selecting weights to impose sparsity constraints, the network size of the feature extraction network can be reduced, but some computational loss may inevitably be introduced. To improve the accuracy of subsequent behavior recognition, during training, each training feature data can be associated with a baseline action label. The feature extraction network generated based on the training feature data corresponding to the same baseline action label can be used to identify the action category of a specific action label. That is, different baseline action labels will correspond to different feature extraction networks to achieve specialized training and improve recognition accuracy. Therefore, the database of the electronic device can store action extraction networks associated with different baseline action labels (which can be identified as candidate extraction networks before selection).

[0123] In this embodiment, after generating inter-frame motion feature data, the electronic device can determine multiple candidate motion labels and determine the matching degree between the inter-frame motion feature data and the candidate extraction network by the label correlation degree between the multiple candidate motion labels and the baseline motion label. Based on the matching degree, the candidate extraction network most relevant to the inter-frame motion feature data is selected as the feature extraction network for subsequent sparse feature data output.

[0124] In this embodiment of the application, by configuring different feature extraction networks for different action labels, the occurrence of a decrease in recognition accuracy due to a reduction in convolutional kernels can be reduced, thus ensuring recognition efficiency while also improving recognition accuracy.

[0125] Figure 7 A flowchart illustrating the specific implementation of a behavior recognition method S102 based on spatiotemporal relationships provided in the third embodiment of the present invention is shown. See also... Figure 7 In contrast Figure 1 In the embodiment described herein, S102 of the behavior recognition method based on spatiotemporal relationships includes: S1021 to S1027, which are detailed below:

[0126] Further, the step of importing the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data includes:

[0127] In S1021, the image tensors of any two consecutive video image frames within the target video data are determined.

[0128] In this embodiment, before extracting motion feature information between two video image frames, the electronic device needs to preprocess the video image frames, converting them from graphical representations into vector tensors. The image tensor corresponding to each video image frame is determined based on the image size of that frame. For example, the image size can be a tensor of size H*W*C, where H is determined based on the image length, W is determined based on the image width, H*W represents the spatial resolution of the video image frame, and C identifies the spatial location of the target object. For example, two consecutive video image frames can be identified as F(t) and F(t+1), i.e., the image tensors corresponding to the t-th and t+1-th video image frames, respectively.

[0129] In S1022, multiple feature point coordinates are determined based on the key positions of the target object in the video image frame; the feature point coordinates are determined based on the gait behavior of the target object.

[0130] In this embodiment, the electronic device can mark the location of the target object, i.e., the aforementioned key location, in each video image frame. Specifically, the target object is a physical person. In this case, the electronic device can use a human body template to perform a sliding bounding box operation within the video image frame, calculate the matching degree between the human body template and the bounding box area, and thus identify the area where the human body is located, i.e., the area where the target object is located. Alternatively, a face recognition algorithm can be used to locate the face region contained in the video image frame, and based on the face region, identify the area where the target object is located, thereby determining the key location of the target object.

[0131] In this embodiment, after determining the key location, the electronic device can use that key location as a reference to identify multiple key points within the target object, with each key point corresponding to a feature point coordinate. For example, key points related to gait behavior include: the knee joint, the center of the thigh, the center of the calf, the center of the torso, the head, the left arm, and the right arm. After marking each key point, the coordinates of that key point within a video image frame can be determined, i.e., the aforementioned feature point coordinates can be determined.

[0132] In S1023, the tensor representation of the coordinates of each feature point is determined in the image tensor, and the feature vector of the target object in the video image frame is generated based on the coordinate representation of all the feature points.

[0133] In this embodiment, after determining the coordinates of multiple feature points, the electronic device can locate the element where each feature coordinate point is located in the image tensor, thereby obtaining the expression of each feature coordinate point through the tensor, i.e. the tensor expression mentioned above. Finally, the tensor expressions of all feature coordinate points are encapsulated to obtain the feature vector related to the gait of the target object.

[0134] In S1024, a displacement correlation matrix is ​​constructed based on the feature vectors of any two consecutive video image frames; the displacement correlation matrix is ​​used to determine the displacement correlation score between the coordinates of each feature point in one video image frame and the coordinates of each coordinate point in another video image frame.

[0135] In this embodiment, after determining the tensor representation corresponding to the feature point coordinates of the key points and obtaining the feature vector composed of the tensor representations of all key points, the electronic device can calculate the vector deviation between two video image frames. Thus, based on the vector deviation, the displacement corresponding to each key point of the target object between the two video image frames can be determined, thereby obtaining the aforementioned displacement correlation matrix.

[0136] In this embodiment, since a large displacement is unlikely to occur at a certain position between two adjacent frames of the video, the displacement can be restricted to a specific region. Assuming that the region is centered at X and contains P2 feature points, the correlation score matrix between position X and all features in the candidate region of the adjacent video frame can be obtained by performing a dot product operation between the feature at position X and the feature in the corresponding candidate region. The dimension of this matrix is ​​H×W×P2, which is the displacement correlation matrix mentioned above, reflecting the relationship between the positions of adjacent frames.

[0137] In S1025, the maximum displacement distance between the coordinates of each feature point and the two consecutive video image frames is determined according to the displacement correlation matrix, and the displacement matrix of the target object is determined based on all the maximum displacement distances.

[0138] In this embodiment, after determining the correlation scores between the coordinates of each feature point and the coordinates of each coordinate point in the key region of another video image frame, the electronic device can select the value with the largest correlation score to determine the maximum displacement distance corresponding to the feature point coordinates. That is, the coordinate point associated with the feature point coordinates is located in the other video image frame. Since the correlation score determines the correlation between two coordinate points, if the correlation score between the two coordinate points is the largest, it means that the two coordinate points belong to the same coordinate point. Therefore, the coordinate point with the largest correlation score can be used to determine the maximum displacement distance corresponding to the key point coordinates. Thus, based on the maximum displacement distance of all key point coordinates, the displacement matrix of the target object can be determined.

[0139] Furthermore, as another embodiment of this application, the above-described S1025 specifically includes the following steps:

[0140] Step 1: Determine the displacement correlation array corresponding to the coordinates of each feature point in the displacement correlation matrix;

[0141] Step 2: Determine the parameter value with the largest correlation coefficient from the displacement correlation array as the maximum displacement distance of the feature coordinate point;

[0142] Step 3: Construct the displacement field of the target object in two-dimensional space based on the maximum displacement distance of all the feature coordinate points;

[0143] Step 4: The displacement field is pooled and reduced in dimensionality using the activation function softmax to obtain a one-dimensional confidence tensor;

[0144] Step 5: Fuse the displacement field and the one-dimensional confidence tensor to construct a displacement matrix for representing three-dimensional space.

[0145] In this embodiment, based on the correlation score matrix, by finding the point in another video image frame corresponding to the maximum score of each feature point in the correlation score matrix of the video image frame, the displacement field of motion information can be estimated. Since the correlation score is used to determine the correlation between two coordinate points, the correlation score between each feature point coordinate in another video image frame can be separated based on the displacement correlation matrix, i.e., the displacement correlation array. The parameter value with the largest correlation coefficient is determined to determine the corresponding coordinate point of the feature point coordinate in the other video image frame, and the distance between the two points is taken as the maximum displacement distance, thereby constructing the displacement field of the target object in two-dimensional space. Since the video image frame is a one- or two-dimensional image, the constructed displacement field is also two-dimensional. Specifically, a softmax layer can be added to extract features from the two-dimensional field, i.e., max pooling is performed, to obtain the confidence map of the target object. Finally, the two-dimensional displacement field and the one-dimensional confidence map are combined to form a displacement matrix with three-dimensional features.

[0146] In this embodiment, the motion of the target object is determined by constructing a two-dimensional displacement field, and the confidence level of each point within the displacement field is determined by pooling dimensionality reduction. This facilitates effective evaluation of the displacement, thereby enabling subsequent action recognition and improving the accuracy of action recognition.

[0147] In S1026, the displacement matrix is ​​imported into a preset feature transformation model to generate motion feature sub-data for any two consecutive video image frames.

[0148] In this embodiment, to match the features of downstream layers, the displacement tensor needs to be converted into a motion feature matrix matching the dimensions of the downstream layers. This can be fed into four depthwise separable convolutional layers—one 1×7 layer and three 1×3 layers—to convert it into motion features with the same number of channels C as the original input F(t). This is then fed into the next layer of the network.

[0149] In S1027, the inter-frame motion feature data is obtained based on the motion feature sub-data of all the video image frames.

[0150] In this embodiment, after determining the motion feature sub-data corresponding to each video image frame relative to the next video image frame, the electronic device can encapsulate the data according to the frame number of each video image frame to obtain inter-frame motion feature data about the entire target video data.

[0151] In this embodiment of the application, by marking the coordinates of multiple key points related to gait in the target object, and constructing the corresponding displacement matrix by the displacement of the key point coordinates, the motion feature sub-data of the target object can be determined by the displacement of the key points. This can reduce the number of points that need to be calculated, thereby further reducing the amount of computation and improving the computational efficiency.

[0152] Figure 8 A flowchart illustrating the specific implementation of a behavior recognition method based on spatiotemporal relationships according to a third embodiment of the present invention is shown. See also... Figure 8 In contrast Figure 1-7 In any of the embodiments described above, the behavior recognition method based on spatiotemporal relationships provided in this embodiment further includes, before receiving the target video data to be identified, steps S801 to S807, which are detailed below:

[0153] Furthermore, before receiving the target video data to be identified, the method further includes:

[0154] In S801, sample video data for training the behavior recognition module is acquired; the behavior recognition module includes the inter-frame action extraction network, the feature extraction network, and the context attention network.

[0155] In this embodiment, before performing behavior recognition on the target video data, the electronic device can train its local behavior recognition module to improve the accuracy of subsequent behavior recognition. This behavior recognition module specifically includes three networks: an inter-frame action extraction network for extracting inter-frame action motion data, a pooling fusion network for extracting and fusing features from the inter-frame action motion data, and a contextual attention network for determining the relative position between the target object and its environment. This allows the electronic device to determine the behavior category of the target object globally, enabling it to acquire sample video data from a video library. It should be noted that the sample video data refers to video data that has not been labeled with behavior categories, or weakly labeled video data. This training method utilizes adversarial learning, reducing the time spent on user labeling and improving both training efficiency and accuracy.

[0156] This embodiment introduces a deep bidirectional converter to better utilize position embedding and multi-head attention mechanisms to automatically select key information in videos. It designs a sequence self-supervised learning method for video understanding, and makes full use of massive Internet big data and existing public datasets to continuously optimize and train the behavior pre-training model, thereby obtaining a robust behavior pre-training model with domain universality and task sharing capabilities.

[0157] In S802, positive sample data and negative sample data are generated based on the sample video data; the positive sample data is obtained by interfering with the background information in the sample video data; the negative sample data is obtained by interfering with the frame sequence of sample video frames in the sample video data.

[0158] In this embodiment, after acquiring any sample video data, the electronic device can convert it into two different types of sample data: positive sample data obtained by interfering with background information (i.e., interfering with the spatial dimension) and negative sample data obtained by interfering with the frame sequence (i.e., interfering with the temporal dimension). This decouples action from spatial scene, further enhancing the network's sensitivity to action. This method of constructing positive and negative samples requires the network to pay attention to global statistical information in order to distinguish between positive and negative samples.

[0159] The process of generating positive samples can specifically include the following steps:

[0160] Step 1.1 Mark the sample objects in each sample video frame of the sample video data, and identify other areas besides the sample objects as background areas.

[0161] Step 1.2 Interpolate the background area using a preset thin plate spline to obtain a spatial interference image frame.

[0162] Step 1.3 Encapsulate each spatial interference image frame in the sample video data according to its frame number to obtain the positive sample data.

[0163] In this embodiment, the electronic device can locate the sample object in the sample video data using an object recognition algorithm (such as a face recognition algorithm or a human key point recognition algorithm). The sample object can also be a real person. After marking the sample object in the sample video data, other areas besides the area where the sample object is located can be identified as background areas. Since spatial interference is required, the electronic device can perform interpolation processing in the background area using a thin plate spline to partially occlude the background area, thereby eliminating the spatial correlation between sample video frames. The spatially interfered image frames after adding the thin plate spline are then re-encapsulated according to the frame sequence number to obtain positive sample data.

[0164] In this embodiment, the background area is interpolated using thin plate splines to destroy local scene information and construct positive samples, which can improve the sensitivity of subsequent recognition to user actions and thus improve the accuracy of training.

[0165] The process of generating negative samples can specifically include the following steps:

[0166] Step 2.1 Divide the sample video data into multiple video segments according to the preset action time duration; the duration of each video segment is not greater than the action time duration.

[0167] Step 2.2 Update the frame number of the sample video frames in each video segment according to the preset disorder processing algorithm.

[0168] Step 2.3 Encapsulate each of the sample video frames based on the updated frame sequence number to obtain the negative sample data.

[0169] In this embodiment, to achieve interference in the time dimension, the electronic device can divide the sample video data into multiple video segments, and then shuffle the video image frames within each video segment. Since an action has a certain duration, dividing the video into segments allows for the separation of different actions, thereby improving the sensitivity of subsequent action recognition. The aforementioned action duration is determined based on the average duration of an action determined through big data analysis. The electronic device then reconfigures the frame sequence numbers of each sample video frame within a video segment using a random algorithm, and encapsulates the sample video frames based on the updated frame sequence numbers to obtain negative sample data.

[0170] Typically, contrastive learning uses other videos as negative samples. However, using other videos, besides differing motion information, may introduce many features that make it easier for the network to distinguish between them. Therefore, this method of selecting negative samples cannot guarantee that the network will focus on motion. Based on this, this project proposes to use local temporal perturbation to disrupt optical flow information in order to construct negative samples. This method of constructing positive and negative samples forces the network to pay attention to global statistical information in order to distinguish between positive and negative samples.

[0171] In S803, first spatial information and first optical flow information are generated using the positive sample data, and second spatial information and second optical flow information are generated using the negative sample data.

[0172] In this embodiment, the electronic device can convert positive sample data using an encoding algorithm to obtain encoded data for each image frame in the positive sample data, i.e., obtain multiple feature maps. Then, the learned positional encoding is added to the extracted feature maps. After fusing the positional encoding, a depth bidirectional converter is used to model the temporal information. From the temporal information of the positive sample data, i.e., the first optical flow information, the spatial information is modeled to obtain the spatial information of the positive sample data, i.e., the first spatial information. Correspondingly, negative sample data is also processed accordingly to obtain the second spatial information and the second optical flow information.

[0173] In S804, spatial enhancement information is obtained based on the first spatial information and the second spatial information.

[0174] In this embodiment, since the first spatial information interferes with the background area, it is spatially unrelated, while the second spatial information does not interfere with the background area. Moreover, the two sample data are both from the same sample video data. Therefore, fusing the two spatial information can improve the sensitivity of spatial information capture, thereby obtaining spatial enhancement information.

[0175] In S805, optical flow enhancement information is obtained based on the second optical flow information and the first optical flow information.

[0176] In this embodiment, since the first optical flow information does not interfere with the time series, it is correlated in the time dimension, while the second optical flow information interferes with the time series. Since the two sample data are both from the same sample video data, fusing the two optical flow information can improve the sensitivity of time information capture, thereby obtaining optical flow enhancement information.

[0177] In S806, the spatial enhancement information and the optical flow enhancement information are imported into the behavior recognition module to obtain the training recognition result of the sample video data.

[0178] In S807, the position learning parameters within the initial recognition module are pre-trained based on the training results of all the sample video data to obtain the behavior recognition module.

[0179] In this embodiment, behavior recognition includes two key pieces of information: spatial information and temporal information. Spatial information refers to static information within the scene, such as objects and contextual information, which is easily captured within a single frame of the video. Temporal information primarily captures the dynamic characteristics of actions and is obtained by integrating spatial information between frames. For behavior recognition, how well action information is captured is crucial to model performance. The global average pooling layer used at the end of existing 3D convolutional neural networks hinders the enrichment of temporal information. To address this issue, a depthwise bidirectional transformer (Transformer) is proposed to replace global average pooling. K frames sampled from the input video are encoded by a 3D convolutional encoder. The resulting feature map is not processed by global average pooling at the end of the network; instead, the feature vector is segmented into a fixed-length sequence of tokens. Then, to preserve positional information, the learned positional encoding is added to the extracted features. After fusing the positional encoding, the temporal information is modeled using a Transformer block in a deep bidirectional transformer. The feature vectors obtained through the multi-head attention mechanism of the deep bidirectional transformer incorporate temporal information. These vectors are then concatenated and transformed through a multilayer perceptron. End-to-end training is completed by calculating contrastive loss, resulting in a pre-trained model with good generalization performance.

[0180] In this embodiment of the application, by determining positive sample data and negative sample data, the sensitivity to action and spatiotemporal information recognition can be improved, thereby enabling training of behavior categories without the need for annotation, thus improving the effect of pre-training.

[0181] Figure 9 A flowchart illustrating the specific implementation of a behavior recognition method S104 based on spatiotemporal relationships according to the fourth embodiment of the present invention is shown. See also... Figure 9 In contrast Figure 1-7 In any of the embodiments described above, the behavior recognition method S104 based on spatiotemporal relationships provided in this embodiment includes: S1041 to S1044, which are detailed below:

[0182] In S1041, the target object and at least one environment object within each video image frame of the target video data are determined.

[0183] In S1042, a first context feature is determined based on the first position coordinates of each key feature point of the target object in all the video image frames; the key feature points are human body key points related to the gait of the target object.

[0184] In S1043, a second context feature is determined based on the relative positional relationship between the target object and the environment object in each of the video frames.

[0185] In S1044, the first context feature and the second context feature are imported into the context attention network to generate the gait behavior data.

[0186] In this embodiment, deep convolutional neural networks can extract texture and appearance features from RGB images. They can directly or indirectly use pre-trained deep learning models trained on large-scale data from other visual tasks, thereby effectively transferring image feature representation knowledge. However, they are susceptible to interference from scenes and objects. In contrast, behavior recognition data based on high-level semantic human keypoints or other relationship modeling is relatively lightweight and unaffected by scenes and objects, but lacks texture and appearance information. It cannot effectively utilize the scene and object information upon which the behavior depends, and can only be used for behavior recognition of human-centered related actions. Therefore, it is necessary to fuse feature representations based on RGB images with information based on high-level contextual relationship modeling to better explore the temporal relationships between spatiotemporal features and the interaction patterns between people and people / objects. Simultaneously, it fully utilizes the abstract extraction capabilities of convolutional neural networks for low-level visual feature information and the reasoning capabilities of spatiotemporal graph neural networks for high-level semantic relationships. Specifically, attention-based 3D convolutional neural networks are used to extract video features of human body regions. These features are used for both RGB image-based behavior recognition and as input to the sub-network for human keypoint prediction. Human key node estimation is performed by outputting multiple frames of human key nodes from the network. The key node sequence images and video images are then fed into a graph convolutional contextual neural network (GCNN) model for behavior recognition based on human key nodes. Furthermore, a target detection model is used to detect people and objects in the scene in real time. Then, the representations of other human features and target features surrounding the target human are fed into the GCNN model for joint optimization and training. This integrates the detected target features, surrounding related human features, and human key nodes as contextual information for the behavior of the target human into the model via a graph neural network. This reduces the inconsistency gap between low-level visual features and high-level semantic information, while enhancing the model's ability to model and represent relationships between people and between people and objects. This improves the model's ability to learn and model key information in various complex and common situations.

[0187] In this embodiment of the application, by identifying environmental objects and determining the relationship between environmental objects and target objects, the accuracy of action type identification can be improved.

[0188] Figure 10 This diagram illustrates a structural block diagram of a behavior recognition device based on spatiotemporal relationships according to an embodiment of the present invention. The device includes units for performing various functions. Figure 1 The corresponding embodiments illustrate the steps implemented by the encryption device. Please refer to [link / reference] for details. Figure 1 and Figure 1 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown.

[0189] See Figure 10 The behavior recognition device based on spatiotemporal relationships includes:

[0190] The target video data receiving unit 11 is used to receive the target video data to be identified.

[0191] The inter-frame motion feature data extraction unit 12 is used to import the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data.

[0192] The sparse feature data unit 13 is used to import the inter-frame action feature data into the feature extraction network and output the sparse feature data corresponding to the target video data; the feature extraction network is generated by selecting weights to perform sparsity constraint processing on each convolution kernel in the pooling fusion network.

[0193] Gait behavior data recognition unit 14 is used to import the target video data into a contextual attention network to determine the gait behavior data of the target object in the target video data; the contextual attention network is used to extract the mutual positional relationship between the target object and the environmental object in the target video data.

[0194] The behavior recognition unit 15 is used to obtain the behavior category of the target object based on the gait behavior data and the sparse feature data.

[0195] Optionally, the behavior recognition device further includes:

[0196] A network to be corrected generation unit is used to configure the selection weight with a value of 0 for at least one convolutional kernel to be identified in the pooling fusion network to obtain the network to be corrected.

[0197] The training result determination unit is used to input multiple preset training feature data into the network to be corrected to generate a first training result, and to input multiple training feature data into the pooling fusion network to generate a second training result;

[0198] The loss value determination unit is used to determine the loss value of the network to be corrected based on the first training result and the second training result.

[0199] The first category identification unit is used to identify the convolution to be identified with the selected weights as a redundant convolution kernel if the loss value is less than or equal to the loss threshold.

[0200] The second category identification unit is used to identify the convolutional kernel to be identified with the selected weights as a necessary convolutional kernel if the loss value is greater than a preset loss threshold.

[0201] The loop execution unit is used to return to the operation of configuring the selection weight with a value of 0 for at least one unidentified convolutional kernel in the pooling fusion network to obtain the network to be corrected, until all the unidentified convolutional kernels in the pooling fusion network have been classified.

[0202] A network generation unit is used to generate the feature extraction network based on all the necessary convolutional kernels.

[0203] Optionally, the feature training data is associated with a baseline action label; the feature extraction network generated based on the feature training data is associated with the baseline action label; the behavior recognition device further includes:

[0204] A candidate tag generation unit is used to determine multiple candidate action tags based on the inter-frame action data;

[0205] The label matching unit is used to calculate the matching degree between each candidate extraction network based on the multiple candidate action labels and the baseline action labels corresponding to each candidate extraction network.

[0206] The network selection unit is used to select the candidate extraction network with the highest matching degree as the feature extraction network.

[0207] Optionally, the inter-frame motion feature data extraction unit 12 includes:

[0208] An image tensor conversion unit is used to determine the image tensor of any two consecutive video image frames within the target video data;

[0209] The feature point coordinate determination unit is used to determine the coordinates of multiple feature points based on the key positions of the target object in the video image frame; the feature point coordinates are determined based on the gait behavior of the target object.

[0210] The feature vector generation unit is used to determine the tensor representation of the coordinates of each feature point in the image tensor, and generate the feature vector of the target object in the video image frame based on the coordinate representation of all the feature points.

[0211] The displacement correlation matrix construction unit is used to construct a displacement correlation matrix based on the feature vectors of any two consecutive video image frames; the displacement correlation matrix is ​​used to determine the displacement correlation score between the coordinates of each feature point in one video image frame and the coordinate points in another video image frame.

[0212] The displacement matrix construction unit is used to determine the maximum displacement distance between the coordinates of each feature point and the two consecutive video image frames based on the displacement correlation matrix, and to determine the displacement matrix of the target object based on all the maximum displacement distances.

[0213] The motion feature sub-data determination unit is used to import the displacement matrix into a preset feature transformation model to generate motion feature sub-data for any two consecutive video image frames.

[0214] An action feature sub-data encapsulation unit is used to obtain the inter-frame action feature data based on the action feature sub-data of all the video image frames.

[0215] Optionally, the displacement matrix construction unit includes:

[0216] The displacement correlation array determination unit is used to determine the displacement correlation array corresponding to the coordinates of each feature point in the displacement correlation matrix;

[0217] The maximum displacement distance determination unit is used to determine the parameter value with the largest correlation coefficient from the displacement correlation array as the maximum displacement distance of the feature coordinate point;

[0218] The displacement field determination unit is used to construct the displacement field of the target object in two-dimensional space based on the maximum displacement distance of all the feature coordinate points;

[0219] The displacement field pooling unit is used to pool and reduce the dimensionality of the displacement field through the activation function softmax to obtain a one-dimensional confidence tensor.

[0220] The displacement field fusion unit is used to fuse the displacement field and the one-dimensional confidence tensor to construct a displacement matrix for representing three-dimensional space.

[0221] Optionally, the behavior recognition device further includes:

[0222] The sample video data acquisition unit is used to acquire sample video data for training the behavior recognition module; the behavior recognition module includes the inter-frame action extraction network, the feature extraction network, and the context attention network.

[0223] The sample data conversion unit is used to generate positive sample data and negative sample data based on the sample video data; the positive sample data is obtained by interfering with the background information in the sample video data; the negative sample data is obtained by interfering with the frame sequence of sample video frames in the sample video data.

[0224] An information extraction unit is used to generate first spatial information and first optical flow information from the positive sample data, and to generate second spatial information and second optical flow information from the negative sample data.

[0225] A spatial enhancement information generation unit is used to obtain spatial enhancement information based on the first spatial information and the second spatial information;

[0226] An optical flow enhancement information extraction unit is used to obtain optical flow enhancement information based on the second optical flow information and the first optical flow information;

[0227] The training recognition result output unit is used to import the spatial enhancement information and the optical flow enhancement information into the behavior recognition module to obtain the training recognition result of the sample video data;

[0228] The module training unit is used to pre-train the position learning parameters within the initial recognition module based on the training results of all the sample video data, so as to obtain the behavior recognition module.

[0229] Optionally, the sample data conversion unit includes:

[0230] The background region identification unit is used to mark the sample objects in each sample video frame of the sample video data, and identify other regions besides the sample objects as background regions.

[0231] The background region processing unit is used to interpolate the background region using a preset thin plate spline to obtain a spatial interference image frame.

[0232] The positive sample generation unit is used to encapsulate each spatial interference image frame in the sample video data according to its frame number to obtain the positive sample data.

[0233] Optionally, the sample data conversion unit includes:

[0234] A video segmentation unit is used to divide the sample video data into multiple video segments according to a preset action time duration; the duration of each video segment is not greater than the action time duration.

[0235] The out-of-order processing unit is used to update the frame sequence number of the sample video frames in each of the video segments according to a preset out-of-order processing algorithm.

[0236] The negative sample generation unit is used to encapsulate each of the sample video frames based on the updated frame sequence number to obtain the negative sample data.

[0237] Optionally, the gait behavior data recognition unit 14 includes:

[0238] An environment object recognition unit is used to determine the target object and at least one environment object within each video image frame of the target video data.

[0239] The first context feature generation unit is used to determine a first context feature based on the first position coordinates of each key feature point of the target object in all the video image frames; the key feature points are human key points related to the gait of the target object;

[0240] The second context feature generation unit is used to determine the second context feature based on the relative positional relationship between the target object and the environment object in each video frame;

[0241] The gait behavior data determination unit is used to import the first context feature and the second context feature into the context attention network to generate the gait behavior data.

[0242] Therefore, the behavior recognition device based on spatiotemporal relationships provided in this embodiment of the invention can also, after receiving the target video data that needs to be recognized for behavior, import the target video data into an inter-frame action extraction network to extract action feature information between each video image frame, and generate action feature data based on the action feature information between all video image frames. Then, the action feature data is imported into a feature extraction network for feature extraction to obtain the corresponding sparse feature data. Since the feature extraction network is obtained by selecting weights to perform sparsity constraint processing on the convolution kernels in the pooling fusion network, the entire feature extraction network will reduce unnecessary convolution kernels, thereby reducing the network size, reducing not only the amount of computation but also the network resource consumption. At the same time, in order to consider the relationship between action behaviors in the global dimension, a context attention network is introduced to determine the gait behavior data of the target object in the entire target video data. Finally, by extracting two types of data, the behavior category of the target object in the target video data is determined, thus achieving the purpose of automatically recognizing the behavior category. Compared with existing behavior recognition technologies, the embodiments of this application do not require calculating the optical flow information of the entire video data. Instead, they determine the action feature information between each video frame through a plug-and-play inter-frame action retrieval network, thereby greatly reducing the computing cost of computing devices and reducing the amount of computation. Furthermore, the pooling fusion network is subjected to sparsity constraint processing, which reduces the network size, thereby reducing resource consumption and further improving the recognition efficiency.

[0243] It should be understood that, Figure 10 In the structural block diagram of the behavior recognition device based on spatiotemporal relationships shown, each module is used to perform... Figures 1 to 9 The steps in the corresponding embodiments, and for Figures 1 to 9 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figures 1 to 9 as well as Figures 1 to 9 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0244] Figure 11 This is a structural block diagram of an electronic device provided in another embodiment of this application. For example... Figure 11 As shown, the electronic device 1100 of this embodiment includes: a processor 1110, a memory 1120, and a computer program 1130 stored in the memory 1120 and executable on the processor 1110, such as a program for a behavior recognition method based on spatiotemporal relationships. When the processor 1110 executes the computer program 1130, it implements the steps in the various embodiments of the behavior recognition methods based on spatiotemporal relationships described above, for example... Figure 1 S101 to S105 are shown. Alternatively, the processor 1110 implements the above when executing the computer program 1130. Figure 11 The functions of each module in the corresponding embodiments, for example, Figure 10 For details on the functions of units 11 to 15 shown, please refer to [link / reference]. Figure 10 The relevant descriptions in the corresponding embodiments.

[0245] For example, computer program 1130 may be divided into one or more modules, one or more of which are stored in memory 1120 and executed by processor 1110 to complete this application. One or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 1130 in electronic device 1100. For example, computer program 1130 may be divided into various unit modules, each with the specific functions described above.

[0246] Electronic device 1100 may include, but is not limited to, processor 1110 and memory 1120. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 1100 and does not constitute a limitation on electronic device 1100. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0247] The processor 1110 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0248] The memory 1120 can be an internal storage unit of the electronic device 1100, such as a hard disk or memory of the electronic device 1100. The memory 1120 can also be an external storage device of the electronic device 1100, such as a plug-in hard disk, smart memory card, flash memory card, etc. equipped on the electronic device 1100. Furthermore, the memory 1120 can include both internal storage units and external storage devices of the electronic device 1100.

[0249] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A behavior recognition method based on spatiotemporal relationships, characterized in that, include: Receive the target video data to be identified; The target video data is imported into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data. The inter-frame motion feature data is imported into a feature extraction network, which outputs sparse feature data corresponding to the target video data. The feature extraction network is generated by selecting weights and applying sparsity constraints to each convolutional kernel in the pooling fusion network. The target video data is imported into a contextual attention network to determine the gait behavior data of the target object in the target video data. The context attention network is used to extract the positional relationship between the target object and the environment object in the target video data; The behavior category of the target object is obtained based on the gait behavior data and the sparse feature data; Before importing the inter-frame motion feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes: The selection weight with a value of 0 is configured for at least one convolutional kernel to be identified in the pooling fusion network to obtain the network to be corrected; Multiple preset training feature data are input into the network to be corrected to generate a first training result, and multiple training feature data are input into the pooling fusion network to generate a second training result; Based on the first training result and the second training result, the loss value of the network to be corrected is determined; If the loss value is less than or equal to the loss threshold, the convolution to be identified with the selected weights configured will be identified as a redundant convolution kernel. If the loss value is greater than the preset loss threshold, the convolutional kernel to be identified with the selected weights will be identified as a necessary convolutional kernel. Return to the operation of configuring the selection weight with a value of 0 for at least one unidentified convolutional kernel in the pooling fusion network to obtain the network to be corrected, until all the unidentified convolutional kernels in the pooling fusion network have been classified; The feature extraction network is generated based on all the necessary convolutional kernels.

2. The behavior recognition method according to claim 1, characterized in that, The training feature data is associated with a baseline action label; the feature extraction network generated based on the training feature data is associated with the baseline action label; Before importing the inter-frame motion feature data into the feature extraction network and outputting the sparse feature data corresponding to the target video data, the method further includes: Multiple candidate action labels are determined based on the inter-frame motion data; Based on the multiple candidate action labels and the baseline action labels corresponding to each candidate extraction network, the matching degree between each candidate extraction network is calculated. The candidate extraction network with the highest matching degree is selected as the feature extraction network.

3. The behavior recognition method according to claim 1, characterized in that, The step of importing the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data includes: Determine the image tensors of any two consecutive video image frames within the target video data; Based on the key positions of the target object in the video image frame, the coordinates of multiple feature points are determined; the coordinates of the feature points are determined based on the gait behavior of the target object. In the image tensor, the tensor representation of the coordinates of each feature point is determined, and the feature vector of the target object in the video image frame is generated based on the coordinate representation of all the feature points. Based on the feature vectors of any two consecutive video image frames, a displacement correlation matrix is ​​constructed; the displacement correlation matrix is ​​used to determine the displacement correlation score between the coordinates of each feature point in one video image frame and the coordinates of each coordinate point in another video image frame. The maximum displacement distance between the coordinates of each feature point and the two consecutive video image frames is determined based on the displacement correlation matrix, and the displacement matrix of the target object is determined based on all the maximum displacement distances. The displacement matrix is ​​imported into a preset feature transformation model to generate motion feature sub-data for any two consecutive video image frames. The inter-frame motion feature data is obtained based on the motion feature sub-data of all the video image frames.

4. The behavior recognition method according to any one of claims 1-3, characterized in that, Before receiving the target video data to be identified, the method further includes: Acquire sample video data for training the behavior recognition module; the behavior recognition module includes the inter-frame action extraction network, the feature extraction network, and the context attention network; Positive sample data and negative sample data are generated based on the sample video data; the positive sample data is obtained by interfering with the background information in the sample video data; the negative sample data is obtained by interfering with the frame sequence of sample video frames in the sample video data. First spatial information and first optical flow information are generated using the positive sample data, and second spatial information and second optical flow information are generated using the negative sample data; Spatial enhancement information is obtained based on the first spatial information and the second spatial information; Optical flow enhancement information is obtained based on the second optical flow information and the first optical flow information; The spatial enhancement information and the optical flow enhancement information are imported into the behavior recognition module to obtain the training recognition results of the sample video data; The position learning parameters within the initial recognition module are pre-trained based on the training and recognition results of all the sample video data to obtain the behavior recognition module.

5. The behavior recognition method according to claim 4, characterized in that, The step of generating positive and negative sample data based on the sample video data includes: The sample video data is divided into multiple video segments according to the preset action time duration; the duration of each video segment is no longer than the action time duration. According to the preset disorder processing algorithm, the frame sequence number of the sample video frames in each video segment is updated respectively; The negative sample data is obtained by encapsulating each sample video frame based on the updated frame number.

6. The behavior recognition method according to any one of claims 1-3, characterized in that, The step of importing the target video data into a context attention network to determine the gait behavior data of the target object in the target video data further includes: Determine the target object and at least one environment object within each video image frame of the target video data; A first context feature is determined based on the first position coordinates of each key feature point of the target object in all the video image frames; the key feature points are human key points related to the gait of the target object. The second context features are determined based on the relative positional relationship between the target object and the environment object in each of the video image frames; The first contextual feature and the second contextual feature are imported into the contextual attention network to generate the gait behavior data.

7. A behavior recognition device based on spatiotemporal relationships, characterized in that, include: The target video data receiving unit is used to receive the target video data to be identified. The inter-frame motion feature data extraction unit is used to import the target video data into a preset inter-frame motion extraction network to obtain inter-frame motion feature data; the inter-frame motion feature data is used to determine the motion feature information between adjacent video image frames in the target video data. The sparse feature data unit is used to import the inter-frame motion feature data into the feature extraction network and output the sparse feature data corresponding to the target video data. The feature extraction network is generated by selecting weights and applying sparsity constraints to each convolutional kernel in the pooling fusion network. A gait behavior data recognition unit is used to import the target video data into a contextual attention network to determine the gait behavior data of the target object in the target video data. The context attention network is used to extract the positional relationship between the target object and the environment object in the target video data; A behavior recognition unit is used to obtain the behavior category of the target object based on the gait behavior data and the sparse feature data; The behavior recognition device further includes: A network to be corrected generation unit is used to configure the selection weight with a value of 0 for at least one convolutional kernel to be identified in the pooling fusion network to obtain the network to be corrected. The training result determination unit is used to input multiple preset training feature data into the network to be corrected to generate a first training result, and to input multiple training feature data into the pooling fusion network to generate a second training result; The loss value determination unit is used to determine the loss value of the network to be corrected based on the first training result and the second training result. The first category identification unit is used to identify the convolution to be identified with the selected weights as a redundant convolution kernel if the loss value is less than or equal to the loss threshold. The second category identification unit is used to identify the convolutional kernel to be identified with the selected weights as a necessary convolutional kernel if the loss value is greater than a preset loss threshold. The loop execution unit is used to return to the operation of configuring the selection weight with a value of 0 for at least one unidentified convolutional kernel in the pooling fusion network to obtain the network to be corrected, until all the unidentified convolutional kernels in the pooling fusion network have been classified. A network generation unit is used to generate the feature extraction network based on all the necessary convolutional kernels.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images