Behavior recognition method, model training method, device, and computer medium

Abstract

The application discloses a behavior recognition method, a model training method, a device and a computer medium, which can be applied to various scenes such as behavior recognition, video detection and intelligent transportation. The behavior recognition method comprises the following steps: acquiring a video segment to be recognized; determining a corresponding RGB image sequence and an optical flow image sequence according to the video segment; determining first feature information based on the RGB image sequence, wherein the first feature information comprises first spatial domain feature information and first time domain feature information of the RGB image sequence; determining second feature information based on the optical flow image sequence, wherein the second feature information comprises second spatial domain feature information and second time domain feature information of the optical flow image sequence; and determining a behavior recognition result corresponding to the video segment by using the first feature information and the second feature information. The time domain feature and the spatial domain feature in the video frame are analyzed, and the behavior of a target object in the video frame is analyzed, so that the accuracy of target object behavior recognition is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a behavior recognition method, a model training method, an apparatus, and a computer medium. Background Technology

[0002] In related technologies, when analyzing videos to determine the behavioral information of target objects in the videos, the feature extraction process of image frames in the videos is relatively crude and cannot effectively obtain the motion features of moving videos. When identifying the behavioral categories of target objects in the videos, the accuracy of identification is low. Summary of the Invention

[0003] This application provides a behavior recognition method, model training method, device, and computer medium that can analyze the behavior of target objects in video frames based on temporal and spatial features, thereby improving the accuracy of behavior recognition.

[0004] On one hand, embodiments of this application provide a behavior recognition method, the method comprising:

[0005] Obtain the video segment to be identified;

[0006] Determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0007] First feature information is determined based on the RGB image sequence, and the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence;

[0008] A second feature information is determined based on the optical flow image sequence, the second feature information including a second spatial domain feature information and a second temporal domain feature information for the optical flow image sequence;

[0009] The behavior recognition result corresponding to the video segment is determined using the first feature information and the second feature information.

[0010] On the other hand, embodiments of this application provide a model training method, the method comprising:

[0011] Obtain sample video segments and the corresponding target behavior category labels for the sample video segments;

[0012] Determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0013] The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence.

[0014] The classification module in the behavior recognition model uses the first sample feature information and the second sample feature information to determine the predicted behavior category corresponding to the sample video segment.

[0015] The behavior recognition model is trained using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model.

[0016] On the other hand, embodiments of this application provide a behavior recognition method, including:

[0017] When a recognition instruction is triggered by an object to perform behavior recognition on the selected video segment to be recognized, the video segment is input into the trained behavior recognition model, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment;

[0018] Display the behavior recognition results;

[0019] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0020] On the other hand, embodiments of this application provide a behavior recognition device, characterized in that it includes:

[0021] The first acquisition unit is used to acquire the video segment to be identified;

[0022] The first determining unit is used to determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0023] A first feature extraction unit is used to determine first feature information based on the RGB image sequence, wherein the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence.

[0024] And for determining second feature information based on the optical flow image sequence, the second feature information including second spatial domain feature information and second temporal domain feature information for the optical flow image sequence;

[0025] The first recognition unit is used to determine the behavior recognition result corresponding to the video segment using the first feature information and the second feature information.

[0026] On the other hand, a model training device is provided, comprising:

[0027] The second acquisition unit is used to acquire sample video segments and target behavior category labels corresponding to the sample video segments;

[0028] The second determining unit is used to determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0029] The second feature extraction unit is used to determine first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence through the feature extraction module in the behavior recognition model; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence.

[0030] The second identification unit is used to determine the predicted behavior category corresponding to the sample video segment by using the classification module in the behavior recognition model and the first sample feature information and the second sample feature information.

[0031] The training unit is used to train the behavior recognition model using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model.

[0032] On the other hand, embodiments of this application provide a behavior recognition device, including:

[0033] The detection unit is used to input the video segment into the trained behavior recognition model when a recognition instruction for behavior recognition is triggered by an object and the selected video segment to be recognized is detected, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment.

[0034] The display unit is used to display the behavior recognition results;

[0035] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0036] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to perform steps in the behavior recognition method or model training method as described in any of the above embodiments.

[0037] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the processor executing steps in the behavior recognition method or model training method as described in any of the above embodiments by calling the computer program stored in the memory.

[0038] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the behavior recognition method or model training method as described in any of the above embodiments.

[0039] This application embodiment obtains a video segment to be identified; determines the corresponding RGB image sequence and optical flow image sequence based on the video segment; determines first feature information based on the RGB image sequence, the first feature information including first spatial domain feature information and first temporal domain feature information for the RGB image sequence; determines second feature information based on the optical flow image sequence, the second feature information including second spatial domain feature information and second temporal domain feature information for the optical flow image sequence; and uses the first feature information and the second feature information to determine the behavior recognition result corresponding to the video segment. This scheme can analyze the behavior of the target object in the video frame based on the temporal and spatial domain features in the video frame, thereby improving the accuracy of behavior recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the structure of the behavior recognition system provided in the embodiments of this application.

[0042] Figure 2a This is a flowchart illustrating the behavior recognition method provided in an embodiment of this application.

[0043] Figure 2b This is a schematic diagram of the process for determining the first associated feature information provided in an embodiment of this application.

[0044] Figure 2c This is a schematic diagram of the process for determining the feature information of the first spatial domain provided in an embodiment of this application.

[0045] Figure 2d This is a flowchart illustrating the process of determining first time-domain feature information provided in an embodiment of this application.

[0046] Figure 2e This is a schematic diagram of the process for determining the second associated feature information provided in an embodiment of this application.

[0047] Figure 2f This is a flowchart illustrating the process of determining the summation result based on the first feature information and the second feature information, as provided in an embodiment of this application.

[0048] Figure 2g This is a schematic diagram of the target attention mechanism network provided in an embodiment of this application.

[0049] Figure 3a This is a flowchart illustrating the model training method provided in an embodiment of this application.

[0050] Figure 3b This is a schematic diagram illustrating the process of determining behavior recognition results based on a trained behavior recognition model, as provided in an embodiment of this application.

[0051] Figure 4 This is a flowchart illustrating the behavior recognition method provided in an embodiment of this application.

[0052] Figure 5 This is a schematic diagram of the structure of the behavior recognition device provided in the embodiments of this application.

[0053] Figure 6 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application.

[0054] Figure 7 This is a schematic diagram of the structure of the behavior recognition device provided in the embodiments of this application.

[0055] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] The embodiments of this application can be applied to various scenarios such as behavior recognition, video detection, and intelligent transportation.

[0058] This application provides a behavior recognition method, a model training method, a device, and a computer medium. Specifically, the behavior recognition method and model training method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart voice interaction device, smart home appliance, wearable smart device, aircraft, smart vehicle terminal, etc. The terminal can also include a client, which can be a video client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] First, some of the nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0060] Differential Augmentation Network: The extracted image frame is compared with the surrounding frames, and the stacked features are extracted using a two-dimensional convolutional neural network.

[0061] Attention: By retargeting the feature map, the weight of effective regions in the feature map is increased, thereby improving the accuracy of action recognition.

[0062] Behavior recognition: Given a segmented video clip, classify it according to the behavior of the target object in it, such as playing ball, running, eating, etc.

[0063] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0064] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0065] Deep Learning (DL) is a branch of machine learning that attempts to perform high-level abstractions of data using multiple processing layers with complex structures or multiple nonlinear transformations. Deep learning learns the inherent patterns and hierarchical representations of training sample data; the information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities similar to humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm, and its performance in speech and image recognition far surpasses previous related technologies.

[0066] Neural Network (NN): A deep learning model in the fields of machine learning and cognitive science that mimics the structure and function of biological neural networks.

[0067] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a behavior recognition system provided in an embodiment of this application. The system may include an image acquisition device 10 and a behavior recognition device 20, wherein the image acquisition device 10 and the behavior recognition device 20 are connected via a network, such as a wired or wireless network.

[0068] The image acquisition device 10 is used to acquire video segments to be identified, and the behavior recognition device 20 can be a terminal or a server. The terminal can be used to display a graphical user interface. Specifically, the terminal is used to interact with the user through the graphical user interface, such as downloading and installing a corresponding client and running it, calling and running a corresponding app, or displaying a corresponding graphical user interface by logging into a website.

[0069] Optionally, terminal 10 can be used to display the behavior recognition results corresponding to the video segment.

[0070] Specifically, the behavior recognition device 20 can be used for:

[0071] Obtain the video segment to be identified;

[0072] Determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0073] First feature information is determined based on the RGB image sequence, and the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence;

[0074] A second feature information is determined based on the optical flow image sequence, the second feature information including a second spatial domain feature information and a second temporal domain feature information for the optical flow image sequence;

[0075] The behavior recognition result corresponding to the video segment is determined using the first feature information and the second feature information.

[0076] Optionally, the behavior recognition device 20 described above can also be used for model training. When the behavior recognition device 20 is used for model training, it can be used to execute model training methods, such as:

[0077] Obtain sample video segments and the corresponding target behavior category labels for the sample video segments;

[0078] Determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0079] The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence.

[0080] The classification module in the behavior recognition model uses the first sample feature information and the second sample feature information to determine the predicted behavior category corresponding to the sample video segment.

[0081] The behavior recognition model is trained using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model.

[0082] Optionally, the device used for model training can also be other devices, that is, the system above also includes a model training device, which is used to perform the steps in the aforementioned model training method. The model training device can be a terminal or a server.

[0083] Optionally, specifically as described above, the behavior recognition device 20 can also be used for:

[0084] When a recognition instruction is triggered by an object to perform behavior recognition on the selected video segment to be recognized, the video segment is input into the trained behavior recognition model, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment;

[0085] Display the behavior recognition results;

[0086] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0087] Optionally, the aforementioned video segment contains several video frames, which are a sequence of video images arranged in a certain time order.

[0088] Optionally, the aforementioned video segment can be a segment of a complete video or a segment of a video that has been cut. That is, when performing behavior recognition on a video segment, the video data of the complete video can be divided into several segments to obtain the video data of each video segment.

[0089] Optionally, when the aforementioned video segment to be identified is one of multiple video segments obtained by cutting a complete video, all video segments can be identified before determining the behavior recognition result of the complete video segment, or each video segment can correspond to a behavior recognition result.

[0090] The following provides a detailed description of the solution proposed in this application. It should be noted that the order in which the embodiments are described below is not intended to limit the priority of the embodiments.

[0091] This application provides a behavior recognition method, which can be executed by a terminal or a server, or by both a terminal and a server. This application uses the example of the method being executed by a terminal for illustration. Figure 2a This is a flowchart illustrating the behavior recognition method provided in an embodiment of this application. The method includes the following steps S201-S205:

[0092] S201. Obtain the video segment to be identified;

[0093] The video segment can be a video segment received from the image acquisition device.

[0094] For example, the video segment can contain motion information of the target object, which can be a person, animal, moving object, etc.

[0095] The video segments can be those captured by image acquisition devices along the highway, or those used by users to observe experimental results during the experiment.

[0096] Optionally, a video segment can be a video segment containing a preset number of image frames.

[0097] S202. Determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0098] The RGB image sequence includes multiple RGB images, and the optical flow image sequence includes multiple optical flow images.

[0099] The aforementioned multiple RGB images can be frame images in a video segment. When determining the corresponding RGB image sequence based on the video segment, the process also includes aligning the sizes of the multiple RGB images in the RGB image sequence, that is, adjusting the sizes of the multiple RGB images in the RGB image sequence to a preset size, such as m*n.

[0100] Optionally, when determining the optical flow image sequence, it is also necessary to align the sizes of multiple optical flow images in the optical flow image sequence, that is, adjust the size of multiple optical flow images in the optical flow image sequence to a preset size, such as m*n, so that the size of the optical flow image is the same as that of the RGB image.

[0101] Optionally, in this application, bilinear interpolation can be used when adjusting the size of RGB or optical flow images.

[0102] Optionally, the optical flow image sequence can be determined based on the RGB image sequence. The RGB images correspond one-to-one with the optical flow images. When determining the optical flow image, the corresponding RGB image can be first converted into a grayscale image, and then the grayscale image can be converted into an optical flow image based on the TV-L1 algorithm.

[0103] Optionally, the pixel values ​​of each RGB image and the pixel values ​​of each optical flow image can be normalized to a preset range. The preset range can be [-1, 1]. Normalizing the pixel values ​​of the RGB images and the optical flow images facilitates subsequent data processing.

[0104] S203. Determine first feature information based on the RGB image sequence, wherein the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence;

[0105] Among them, the first spatial domain feature information represents the dependency relationship between each pixel in the first associated feature information sequence space and other pixels, and the first temporal domain feature information represents the positional relationship between a pixel in an image frame at different times and the pixel in an image frame at other times.

[0106] Optionally, in S203, determining the first feature information based on the RGB image sequence includes the following S2031-S2033:

[0107] S2031. For each RGB image among the plurality of RGB images, obtain the previous frame RGB image and the next frame RGB image of the RGB image;

[0108] Optionally, since the first RGB image in a series of RGB images does not have a preceding RGB image, and the last RGB image does not have a following frame, the first and last RGB images in a video segment can be filtered out when determining the RGB image sequence.

[0109] S2032. Based on the RGB image, the previous frame RGB image, and the next frame RGB image, determine the first associated feature information corresponding to the RGB image to obtain the first associated feature information sequence corresponding to the RGB image sequence. The dimension information of the first associated feature information sequence includes: the number of channels of the first associated feature information sequence, the number of image frames of the first associated feature information sequence, the feature length information of the first associated feature information sequence, and the feature width information of the first associated feature information sequence.

[0110] Specifically, in determining the RGB image I i When the first associated feature information (where i takes values ​​from 2 to N-1) is obtained, the image I is simultaneously input. i Image frames before and after I i-1 and I i-1 I i and I i-1 and I t+1 Perform the difference calculation separately, and then apply a 1x1 convolution kernel to image I. i Perform channel number transformation. Transform the two differences using a 1x1 convolution kernel to obtain the channel-dimension adjusted features.

[0111] In some optional embodiments of this application, the process for determining the corresponding first associated feature information based on the RGB image, the previous frame RGB image, and the next frame RGB image can be specifically referred to [reference needed]. Figure 2bAs shown, in the aforementioned S2032, determining the first associated feature information corresponding to the RGB image based on the RGB image, the previous frame RGB image, and the next frame RGB image may include:

[0112] The RGB image is subjected to a 1×1 convolution to transform the number of channels of the RGB image, resulting in a first result;

[0113] Perform a difference operation between the RGB image and the previous frame's RGB image to obtain the first image;

[0114] Perform a difference operation between the RGB image and the subsequent RGB image to obtain a second image;

[0115] Perform a 1×1 convolution on the first image to transform the number of channels in the first image and obtain the second result;

[0116] The second image is subjected to a 1×1 convolution to transform the number of channels of the second image, resulting in a third result;

[0117] The first result, the second result, and the third result are stacked along the channel dimension to obtain the first feature information to be processed;

[0118] The first feature information to be processed is processed based on a preset first residual network to obtain the first associated feature information corresponding to the RGB image; wherein, the first residual network can be a ResNet network, or it can be DenseNet, SwinTransformer, etc.

[0119] The aforementioned dimensional information of the RGB image includes: the number of channels in the RGB image, the image length information of the RGB image, and the image width information of the RGB image.

[0120] In this application, a 2D convolution kernel is used to determine the first associated feature information, which results in faster data processing speed.

[0121] Optionally, the number of channels in the first result, the second result, and the third result mentioned above are the same.

[0122] The aforementioned dimensional information of the first associated feature information includes: the number of channels of the first associated feature information, the feature length information of the first associated feature information, and the feature width information of the first associated feature information.

[0123] When the first associated feature information is image information, the feature length information of the first associated feature information is the image length information of the first associated feature information and the image width information of the first associated feature information.

[0124] S2033. Determine the corresponding first feature information based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule.

[0125] Optionally, in S2033, the corresponding first feature information is determined according to the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, including the following S01-S02:

[0126] S01. Determine the first spatial domain feature information based on the first associated feature information sequence and the first preset spatial domain feature extraction rule;

[0127] Optionally, the first preset spatial domain feature extraction rule includes: a first convolutional kernel group, a second convolutional kernel group, a first dimension transformation function, and a second dimension transformation function, wherein the second dimension transformation function is the inverse of the first dimension transformation function. In S01, the first spatial domain feature information is determined according to the first associated feature information sequence and the first preset spatial domain feature extraction rule, including the following S011-S012:

[0128] S011. Based on the first convolutional kernel group and the first dimension transformation function, the first associated feature information sequence is processed to obtain the first spatial relationship information corresponding to the first associated feature information sequence.

[0129] Optionally, such as Figure 2c As shown, the first convolutional kernel group includes a first convolutional kernel and a second convolutional kernel. Both the first and second convolutional kernels can be 1×1×1 convolutional kernels. In the aforementioned S011, the step of processing the first associated feature information sequence based on the first convolutional kernel group and the first dimension transformation function to obtain the first spatial relationship information corresponding to the first associated feature information sequence includes the following S0111-S0115:

[0130] S0111. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the first convolution kernel to obtain the first convolution result X. 11 (c1,t,h,w) to adjust the number of channels in the first associated feature information sequence;

[0131] Where c is the number of channels in the first associated feature information sequence X0(c,t,h,w), t is the number of image frames in the first associated feature information sequence X0(c,t,h,w), h is the feature length information of the first associated feature information sequence X0(c,t,h,w), and w is the feature width information of the first associated feature information sequence X0(c,t,h,w). c1 is the first convolution result X 11The number of channels in (c1,t,h,w).

[0132] Among them, the first convolution result X 11 The dimensional information of (c1,t,h,w) may include: the first convolution result X 11 The sequence X0(c1,t,h,w) comprises the number of channels c1, the number of image frames t of the first associated feature information sequence X0(c,t,h,w), the feature length h of the first associated feature information sequence X0(c,t,h,w), and the feature width w of the first associated feature information sequence X0(c,t,h,w). The first convolution result X... 11 The number of image frames in (c1,t,h,w) is the same as the number of image frames in the first associated feature information sequence X0(c,t,h,w), and the number of image frames in the first convolution result X 11 The feature length information of (c1,t,h,w) is the same as the feature length information of the first associated feature information sequence X0(c,t,h,w), and the first convolution result X 11 The feature width information of (c1,t,h,w) is the same as the feature width information of the first associated feature information sequence X0(c,t,h,w).

[0133] S0112. Apply the first convolution result X to the first dimension transformation function. 11 Perform a dimensional transformation on (c1,t,h,w) to obtain the first transformation result X. 21 (hw,c1t), the first transformation result X 21 (hw,c1t) can be 2D information, and the first transformation result X 21 The dimensional information of (hw,c1t) includes: the first convolution result X 11 The feature length information h of (c1,t,h,w) and the first convolution result X 11 The product of the feature width information w of (c1,t,h,w) and the first convolution result X 11 The number of channels c1 in (c1,t,h,w) and the result of the first convolution X 11 The product of the number of image frames t of (c1,t,h,w);

[0134] In other embodiments of this application, the first transformation result may also be 3D information, and the dimensional information of the first transformation result may further include: the first convolution result X 11 The feature length information h of (c1,t,h,w) and the first convolution result X 11 The product of the feature width information w of (c1,t,h,w) and the first convolution result X 11 The number of channels c1 in (c1,t,h,w), and the first convolution result X 11The number of image frames t in (c1,t,h,w).

[0135] S0113. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the second convolution kernel to obtain the second convolution result X. 12 (c1,t,h,w) is used to adjust the number of channels in the first associated feature information sequence X0(c,t,h,w), and the second convolution result X 12 The number of channels in (c1,t,h,w) and the first convolution result X 11 (c1,t,h,w) have the same number of channels;

[0136] S0114. Apply the first dimension transformation function to the second convolution result X. 12 (c1,t,h,w) undergoes a dimensional transformation to obtain a second transformation result, the dimensional information of the second transformation result being the same as that of the first transformation result;

[0137] S0115. Calculate the first transformation result X. 21 (hw,c1t) and the transpose X of the second transformation result 22 The product of (c1t,hw) yields the first spatial relationship information X3(hw,hw) corresponding to the first associated feature information sequence X0(c,t,h,w). The first spatial relationship information is 2-dimensional information. Specifically, the dimensional information of the first spatial relationship information includes: the product of the feature length information of the first transformation result and the feature width information of the first transformation result, and the product of the feature length information of the second transformation result and the feature width information of the second transformation result.

[0138] S012. Determine the first spatial domain feature information based on the second convolution kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information.

[0139] Optionally, such as Figure 2c As shown, the second convolutional kernel group includes a third convolutional kernel and a fourth convolutional kernel, wherein the aforementioned third and fourth convolutional kernels can be 1×1×1 convolutional kernels. In S012, the first spatial domain feature information is determined based on the second convolutional kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information, including the following S0121-S0127:

[0140] S0121. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the third convolution kernel to obtain the third convolution result X. 13(c1,t,h,w) is used to adjust the number of channels in the first associated feature information sequence X0(c,t,h,w), and the third convolution result X 13 The number of channels in (c1,t,h,w) is the same as the number of channels in the second convolution result;

[0141] S0122, Apply the first dimension transformation function to the third convolution result X. 13 (c1,t,h,w) undergoes dimensional transformation to obtain a third transformation result, wherein the dimensional information of the third transformation result is the same as that of the first transformation result.

[0142] S0123. Normalize the first spatial relationship information X3(hw,hw) to obtain the first normalization result;

[0143] Specifically, the first spatial relationship information can be normalized using the Softmax function to obtain the first normalization result.

[0144] S0124. Calculate the transpose X of the first normalization result and the third transformation result. 23 The product of (c1t,hw) yields the first product result X4(c1t,hw), and the dimension information of the first product result X4(c1t,hw) is the same as the dimension information of the transpose of the third transformation result.

[0145] S0125. The first product result X4(c1t,hw) is transformed using the second dimension transformation function to obtain the fourth transformation result X5(c1,t,h,w). The dimension information of the fourth transformation result X5(c1,t,h,w) is combined with the dimension information of the third convolution result X. 13 (c1,t,h,w) have the same dimensional information;

[0146] S0126. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the fourth convolution kernel to obtain the fourth convolution result X. 14 (c1,t,h,w) is used to adjust the number of channels in the first associated feature information sequence X0(c,t,h,w), and the fourth convolution result X 14 The number of channels in (c1,t,h,w) and the third convolution result X 13 (c1,t,h,w) have the same number of channels;

[0147] S0127. The fourth transformation result X5(c1,t,h,w) and the fourth convolution result X 14 Summing (c1,t,h,w) yields the first spatial domain feature information Y1(c1,t,h,w).

[0148] It should be noted that, Figure 2c In this context, t represents the number of image frames in the first associated feature information sequence X0(c,t,h,w), h represents the feature length of the first associated feature information sequence X0(c,t,h,w), and w represents the feature width of the first associated feature information sequence X0(c,t,h,w). c1 represents the first convolution result X. 11 The number of channels (c1,t,h,w). The dimensionality information of the first, second, third, and fourth convolution results is the same. The aforementioned method of determining the first spatial domain feature information Y1(c1,t,h,w) through the first spatial relationship information X3(hw,hw) can be used to treat the fourth transformation result X5(c1,t,h,w) as the weights of the spatial domain features of the first associated feature information sequence X0(c,t,h,w), superimposed on the fourth convolution result X. 14 (c1,t,h,w), and then the first spatial domain feature information Y1(c1,t,h,w) is determined.

[0149] S02. Determine the first time domain feature information based on the first associated feature information sequence and the first preset time domain feature extraction rule.

[0150] Optionally, the first preset time-domain feature extraction rule includes: a third convolution kernel group, a fourth convolution kernel group, a third-dimensional transformation function, and a fourth-dimensional transformation function, wherein the fourth-dimensional transformation function is the inverse of the third-dimensional transformation function. In S02, determining the first time-domain feature information based on the first associated feature information sequence and the first preset time-domain feature extraction rule includes the following S021-S022:

[0151] S021. Based on the third convolutional kernel group and the third dimension transformation function, the first associated feature information sequence is processed to obtain the first time relationship information corresponding to the first associated feature information sequence.

[0152] Optionally, such as Figure 2d As shown, the third convolutional kernel group includes a fifth convolutional kernel and a sixth convolutional kernel. Both the fifth and sixth convolutional kernels can be 1×1×1 convolutional kernels. In S021, the step of processing the first associated feature information sequence based on the third convolutional kernel group and the third-dimensional transformation function to obtain the first temporal relationship information corresponding to the first associated feature information sequence includes the following S0211-S0215:

[0153] S0211. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the fifth convolution kernel to obtain the fifth convolution result X. 31(c2,t,h,w) to adjust the number of channels in the first associated feature information sequence;

[0154] Where c2 is the result of the five convolutions X 31 The number of channels in (c2,t,h,w). The fifth convolution result X. 31 The dimensional information of (c2,t,h,w) may include: the result of the fifth convolution X 31 The number of channels c2 in (c2,t,h,w), and the result of the fifth convolution X 31 The number of image frames t (c2,t,h,w) and the result of the fifth convolution X 31 The feature length information h of (c2,t,h,w) and the fifth convolution result X 31 The feature width w of (c2,t,h,w) is given. The fifth convolution result X... 31 The number of image frames in (c2,t,h,w) is the same as the number of image frames in the first associated feature information sequence X0(c,t,h,w), and the fifth convolution result X 31 The feature length information of (c2,t,h,w) is the same as the feature length information of the first associated feature information sequence X0(c,t,h,w), and the fifth convolution result X 31 The feature width information of (c2,t,h,w) is the same as the feature width information of the first associated feature information sequence X0(c,t,h,w).

[0155] S0212, Apply the third-dimensional transformation function to the fifth convolution result X. 31 Perform a dimensional transformation on (c2,t,h,w) to obtain the fifth transformation result X. 41 (t,c2hw), the fifth transformation result is 2D information, the fifth transformation result X 41 The dimensional information of (t,c2hw) includes: the fifth convolution result X 31 The number of image frames t (c2,t,h,w) and the first product, where the first product is the fifth convolution result X. 31 The number of channels in (c2,t,h,w), and the result of the fifth convolution X 31 The feature length information h of (c2,t,h,w) and the fifth convolution result X 31 The product of the feature width information w of (c2,t,h,w);

[0156] In other embodiments of this application, the fifth transformation result can also be 3D information, and the dimensional information of the fifth transformation result can further include: the fifth convolution result X. 31 The number of image frames (c2,t,h,w), and the fifth convolution result X 31 The number of channels in (c2,t,h,w) and the result of the fifth convolution X31 The feature length information h of (c2,t,h,w) and the fifth convolution result X 31 The product of the feature width information w of (c2,t,h,w).

[0157] In other embodiments of this application, the fifth transformation result can also be 4-dimensional information, and the dimensional information of the fifth transformation result can further include: the fifth convolution result X. 31 The number of image frames (c2,t,h,w), and the fifth convolution result X 31 The number of channels in (c2,t,h,w) and the result of the fifth convolution X 31 The feature length information h of (c2,t,h,w) and the fifth convolution result X 31 The feature width information w of (c2,t,h,w) is...

[0158] S0213. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the sixth convolution kernel to obtain the sixth convolution result X. 32 (c2,t,h,w) is used to adjust the number of channels in the first associated feature information sequence X0(c,t,h,w), and the sixth convolution result X 32 The number of channels in (c2,t,h,w) and the fifth convolution result X 31 (c2,t,h,w) have the same number of channels;

[0159] S0214. Apply the third-dimensional transformation function to the sixth convolution result X. 32 A dimensional transformation is performed on (c2,t,h,w) to obtain the sixth transformation result. The dimensional information of the sixth transformation result is the same as that of the fifth transformation result X. 41 The dimensional information of (t,c2hw) is the same;

[0160] S0215. Calculate the fifth transformation result X. 41 (t,c2hw) and the transpose X of the sixth transformation result 42 The product of (c2hw,t) yields the first temporal relationship information X6(t,t) corresponding to the first associated feature information sequence X0(c,t,h,w). The dimension information of the first temporal relationship information X6(t,t) includes: the number of image frames in the fifth transformation result and the number of image frames in the sixth transformation result.

[0161] S022. Determine the first time domain feature information based on the fourth convolution kernel group, the third dimension transformation function, the fourth dimension transformation function, and the first time relationship information X6(t,t).

[0162] Optionally, such as Figure 2dAs shown, the fourth convolutional kernel group includes a seventh convolutional kernel and an eighth convolutional kernel. The aforementioned seventh and eighth convolutional kernels can be 1×1×1 convolutional kernels. In S022, the first time domain feature information is determined based on the fourth convolutional kernel group, the third-dimensional transformation function, the fourth-dimensional transformation function, and the first time relationship information, including the following S0221-S0227:

[0163] S0221. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the seventh convolution kernel to obtain the seventh convolution result X. 33 (c2,t,h,w) is used to adjust the number of channels in the first associated feature information sequence X0(c,t,h,w), and the seventh convolution result X 33 The number of channels in (c2,t,h,w) and the sixth convolution result X 32 (c2,t,h,w) have the same number of channels;

[0164] S0222, Apply the third-dimensional transformation function to the seventh convolution result X. 33 A dimensional transformation is performed on (c2,t,h,w) to obtain the seventh transformation result. The dimensional information of the seventh transformation result is the same as that of the fifth transformation result X. 41 The dimensional information of (t,c2hw) is the same;

[0165] S0223. Normalize the first time relationship information to obtain the second normalization result;

[0166] Specifically, the first time relationship information can be normalized using the Softmax function to obtain the second normalization result.

[0167] S0224. Calculate the transpose X of the second normalization result and the seventh transformation result. 43 The product of (c2hw,t) yields the second product result X7(c2hw,t), and the dimension information of the second product result X7(c2hw,t) is the same as the dimension information of the transpose of the seventh transformation result.

[0168] S0225. The second product result X7(c2hw,t) is transformed using the fourth dimension transformation function to obtain the eighth transformation result X8(c2,t,h,w). The dimension information of the eighth transformation result X8(c2,t,h,w) is combined with the seventh convolution result X. 33 (c2,t,h,w) have the same dimensional information;

[0169] S0226. Perform convolution processing on the first associated feature information sequence X0(c,t,h,w) using the eighth convolution kernel to obtain the eighth convolution result X.34 (c2,t,h,w) is used to adjust the number of channels in the first associated feature sequence, and the eighth convolution result X 34 The number of channels in (c2,t,h,w) and the seventh convolution result X 33 (c2,t,h,w) have the same number of channels;

[0170] S0227. The eighth transformation result X8(c2,t,h,w) and the eighth convolution result X 34 Summing (c2,t,h,w) yields the first time-domain feature information Y2(c2,t,h,w).

[0171] It should be noted that, Figure 2d In the sequence X0, t represents the number of image frames in the first associated feature information sequence X0(c,t,h,w), h represents the feature length of the first associated feature information sequence X0(c,t,h,w), and w represents the feature width of the first associated feature information sequence X0(c,t,h,w). C2 represents the fifth convolution result X. 31 The number of channels in (c2,t,h,w). The dimensionality information of the fifth, sixth, seventh, and eighth convolution results is the same.

[0172] The aforementioned method of determining the first time-domain feature information Y2(c2,t,h,w) through the first time-relationship information X6(t,t) can be used to treat the eighth transformation result X8(c2,t,h,w) as the weights of the time-domain features of the first associated feature information sequence X0(c,t,h,w), which are then superimposed on the eighth convolution result X. 34 (c2,t,h,w), and then the first time domain feature information Y2(c2,t,h,w) is determined.

[0173] S204. Determine second feature information based on the optical flow image sequence, wherein the second feature information includes second spatial domain feature information and second temporal domain feature information for the optical flow image sequence;

[0174] Wherein, the optical flow image sequence includes multiple optical flow images, and in S204, the determination of the second feature information based on the optical flow image sequence includes the following S2041-S2043:

[0175] S2041. For each optical flow image in the plurality of optical flow images, obtain the previous frame optical flow image and the next frame optical flow image of the optical flow image;

[0176] S2042. Based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image, determine the second associated feature information corresponding to the optical flow image to obtain the second associated feature information sequence corresponding to the optical flow image sequence. The dimension information of the second associated feature information sequence includes: the number of channels of the second associated feature information sequence, the number of image frames of the second associated feature information sequence, the feature length information of the second associated feature information sequence, and the feature width information of the second associated feature information sequence.

[0177] In some optional embodiments of this application, the process of determining the corresponding second associated feature information based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image can be specifically referred to Figure 2e As shown, in the aforementioned S2042, determining the second associated feature information corresponding to the optical flow image based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image may include:

[0178] The optical flow image is subjected to a 1×1 convolution to transform the number of channels in the optical flow image, resulting in a fourth result;

[0179] A third image is obtained by performing a difference operation between the optical flow image and the previous frame optical flow image;

[0180] Perform a difference operation between the optical flow image and the subsequent frame optical flow image to obtain the fourth image;

[0181] The third image is subjected to a 1×1 convolution to transform the number of channels of the third image, resulting in the fifth result;

[0182] The fourth image is subjected to a 1×1 convolution to transform the number of channels of the fourth image, resulting in the sixth result;

[0183] The fourth result, the fifth result, and the sixth result are stacked along the channel dimension to obtain the second feature information to be processed;

[0184] The second feature information to be processed is processed based on a preset second residual network to obtain the second associated feature information corresponding to the optical flow image. The second residual network can be a ResNet (Residual Neural Network), or a DenseNet (Densely connected convolutional networks), a Swin Transformer, etc. The Swin Transformer is a network with a clear hierarchical structure; the bottom layer processes more and more localized data, while the top layer processes less data but has richer semantic information.

[0185] The aforementioned dimensional information of the optical flow image includes: the number of channels in the optical flow image, the image length information of the optical flow image, and the image width information of the optical flow image.

[0186] Optionally, the fourth, fifth, and sixth results mentioned above have the same number of channels.

[0187] In this application, adjusting the number of channels can improve data processing speed, and convolution of the data can also play a role in the preliminary fusion of data features.

[0188] The aforementioned dimensional information of the second associated feature information includes: the number of channels of the first associated feature information, the feature length information of the second associated feature information, and the feature width information of the second associated feature information.

[0189] When the second associated feature information is image information, the feature length information of the second associated feature information is the image length information of the second associated feature information and the image width information of the second associated feature information.

[0190] S2043. Determine the corresponding second feature information based on the second associated feature information sequence, the second preset spatial domain feature extraction rule, and the second preset temporal domain feature extraction rule.

[0191] It should be noted that in S2043, the method of determining the corresponding second feature information based on the second associated feature information sequence, the second preset spatial domain feature extraction rule, and the second preset temporal domain feature extraction rule is similar to the method of determining the corresponding first feature information based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, and will not be repeated here. Specifically, the aforementioned first associated feature information sequence can be replaced with the second associated feature information sequence, the first preset spatial domain feature extraction rule can be replaced with the second preset spatial domain feature extraction rule, the first preset temporal domain feature extraction rule can be replaced with the second preset temporal domain feature extraction rule, and the first spatial relationship information can be replaced with the second spatial relationship information, and the first temporal relationship information can be replaced with the second temporal relationship information, etc. Among them, the data processing flow for determining the second spatial domain feature information based on the second preset spatial domain feature extraction rule is the same as the data processing flow for determining the first spatial domain feature information based on the first preset spatial domain feature extraction rule; and the data processing flow for determining the second temporal domain feature information based on the second preset temporal domain feature extraction rule is the same as the data processing flow for determining the first temporal domain feature information based on the first preset temporal domain feature extraction rule.

[0192] Optionally, the first feature information is the result of splicing the first spatial domain feature information and the first temporal domain feature information in the dimension of channel number; and the second feature information is the result of splicing the second spatial domain feature information and the second temporal domain feature information in the dimension of channel number.

[0193] For example, when the number of channels for the first spatial domain feature information is c1 and the number of channels for the first time domain feature information is c2, the number of channels for the first feature information is c3, where c3 = c1 + c2.

[0194] S205. Use the first feature information and the second feature information to determine the behavior recognition result corresponding to the video segment.

[0195] Optionally, in S205, determining the behavior recognition result corresponding to the video segment using the first feature information and the second feature information includes the following S2051-S2052:

[0196] S2051. Stack the first feature information and the second feature information in the dimension of channel number to obtain a stacking result;

[0197] The stacking result can be regarded as the result of splicing the first feature information and the second feature information in the channel dimension.

[0198] See details Figure 2fAs shown, the first feature information U1 and the second feature information U2 are concatenated to obtain U1U2. The dimension information of the first feature information U1 can be the number of channels of the first feature information U1, the number of image frames of the first feature information U1, the feature length information of the first feature information U1, and the feature width information of the first feature information U1.

[0199] Wherein, the number of channels of the first feature information U1 is the sum of the number of channels of the first spatial domain feature information and the number of channels of the first time domain feature information; the number of channels of the second feature information U2 is the sum of the number of channels of the second spatial domain feature information and the number of channels of the second time domain feature information.

[0200] Optionally, if the number of channels of the first feature information U1 is c3 and the number of channels of the second feature information U2 is also c3, then the number of channels of the stacked result is 2c3.

[0201] For example, after concatenating the first feature information U1(c3,t,h,w) and the second feature information U2(c3,t,h,w), and stacking them in the dimension of channel number, we get the stacked result U1U2(2c3,t,h,w).

[0202] S2052. Perform dimensional transformation on the stacking result based on the fifth dimension transformation function to obtain a first target transformation result. The dimensional information of the first target transformation result includes: the number of channels of the stacking result and a second product, wherein the second product is the product of the feature length information of the stacking result, the feature width information of the stacking result and the number of image frames of the stacking result.

[0203] Specifically, after performing a dimensional transformation on the stacking result based on the fifth-dimensional transformation function, the first target transformation result R1(2c3,hwt) can be obtained.

[0204] S2053. Based on the first target transformation result and the target attention mechanism network, determine the behavior recognition result corresponding to the video segment.

[0205] Optionally, in S2053, based on the first target transformation result and the target attention mechanism network, the behavior recognition result corresponding to the video segment is determined, including S20531-S20532:

[0206] S20531. Input the first target transformation result into the target attention mechanism network to obtain the output result;

[0207] S20532. Perform a dimensional transformation on the output result based on the sixth-dimensional transformation function to obtain a second target transformation result. The dimensional information of the second target transformation result is the same as the dimensional information of the stacked result. The sixth-dimensional transformation function is the inverse of the fifth-dimensional transformation function.

[0208] like Figure 2f As shown, the first target transformation result R1(2c3,hwt) is input into the target attention mechanism network to obtain the output result; then, the output result is transformed based on the sixth dimension transformation function to obtain the second target transformation result R2(2c3,t,h,w).

[0209] S20533. The second target transformation result is split to obtain the third feature information and the fourth feature information corresponding to the RGB image sequence and the optical flow image sequence, respectively.

[0210] like Figure 2f As shown, the third feature information is U3(c3,t,h,w), and the fourth feature information is U4(c3,t,h,w).

[0211] S20534. Summing the third feature information and the fourth feature information to obtain a summation result;

[0212] S20535. Determine the behavior recognition result corresponding to the video segment based on the summation result.

[0213] Optionally, the behavior recognition result can be determined based on the fully connected layer between the summation result and the behavior recognition result.

[0214] Optionally, the behavior recognition result can be the behavior type information of the target object in the video segment. The target object can be a person, animal, moving object, etc., and the behavior type information can be: speeding while driving, littering, etc.

[0215] Optionally, the target object can also be an experimental subject, such as a mouse or bacteria, and the behavior type information can be: eating, moving towards direction A, etc.

[0216] In some optional embodiments of this application, a schematic diagram of the target attention mechanism network structure can be found. Figure 2g As shown, the first target transformation result can be subjected to a three-dimensional linear transformation using different coefficient weights to obtain the first transformation information Q, the second transformation information K, and the third transformation information V, respectively. The product of the first transformation information and the transpose of the second transformation information is calculated, and the product result is normalized by softmax to obtain the processed result. The product of the processed result and the third transformation information is then calculated to obtain the output result.

[0217] This application embodiment obtains a video segment to be identified; determines the corresponding RGB image sequence and optical flow image sequence based on the video segment; determines first feature information based on the RGB image sequence, the first feature information including first spatial domain feature information and first temporal domain feature information for the RGB image sequence; determines second feature information based on the optical flow image sequence, the second feature information including second spatial domain feature information and second temporal domain feature information for the optical flow image sequence; and uses the first feature information and the second feature information to determine the behavior recognition result corresponding to the video segment. This scheme can analyze the behavior of the target object in the video frame based on the temporal and spatial domain features in the video frame, thereby improving the accuracy of the behavior recognition of the target object.

[0218] This application also provides a model training method, which can be executed by a terminal or a server, or by both a terminal and a server. This application embodiment uses the execution of the method by a terminal as an example for illustration. Figure 3a This is a flowchart illustrating the model training method provided in an embodiment of this application. The method includes steps S301-S305:

[0219] S301. Obtain sample video segments and the target behavior category labels corresponding to the sample video segments;

[0220] The sample video segment can be a video segment received from the image acquisition device.

[0221] Optionally, the sample video segment can be a video segment containing a preset number of image frames.

[0222] Specifically, the target behavior category label can be a text or image tag used to indicate the target behavior category corresponding to the sample video segment.

[0223] S302. Determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0224] The sample RGB image sequence includes multiple sample RGB images, and the sample optical flow image sequence includes multiple sample optical flow images;

[0225] The aforementioned multiple sample RGB images can be frame images in the sample video segment. When determining the corresponding sample RGB image sequence based on the sample video segment, the method also includes size alignment of multiple sample RGB images in the sample RGB image sequence, that is, adjusting the size of multiple sample RGB images in the sample RGB image sequence to a preset size, such as m1*n1.

[0226] Optionally, when determining the sample optical flow image sequence, it is also necessary to align the sizes of multiple sample optical flow images in the sample optical flow image sequence, that is, adjust the size of multiple sample optical flow images in the sample optical flow image sequence to a preset size, such as m1*n1, so that the sample optical flow image and the sample RGB image have the same size.

[0227] Optionally, in this application, when adjusting the size of the sample RGB or sample optical flow image, bilinear interpolation may be used.

[0228] Optionally, the sample optical flow image sequence can be determined based on the sample RGB image sequence. The sample RGB image and the sample optical flow image correspond one-to-one. When determining the sample optical flow image, the corresponding sample RGB image can be first transformed into a grayscale image, and then the grayscale image can be transformed into the sample optical flow image based on the TV-L1 algorithm.

[0229] Optionally, the pixel values ​​of each sample RGB image and the pixel values ​​of each sample optical flow image can be normalized to a preset range. The preset range can be [-1, 1]. Normalizing the pixel values ​​of the sample RGB images and the sample optical flow images facilitates subsequent data processing and improves the training speed of the behavior recognition model.

[0230] S303. Using the feature extraction module in the behavior recognition model, determine first sample feature information based on the sample RGB image sequence, and determine second sample feature information based on the sample optical flow image sequence; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence;

[0231] Optionally, the aforementioned feature extraction module includes: a difference enhancement module, a spatial domain feature extraction module, and a temporal domain feature extraction module, wherein the sample RGB image sequence includes multiple sample RGB images, and the sample optical flow image sequence includes multiple sample optical flow images;

[0232] In S303, the determination of first sample feature information based on the sample RGB image sequence and the determination of second sample feature information based on the sample optical flow image sequence by the feature extraction module in the behavior recognition model includes the following S3031-S3033:

[0233] S3031. Using the differential enhancement module, for each sample RGB image in the plurality of sample RGB images, obtain the previous frame sample RGB image and the next frame sample RGB image of the sample RGB image; determine the third associated feature information corresponding to the sample RGB image based on the sample RGB image, the previous frame sample RGB image, and the next frame sample RGB image, to obtain the third associated feature information sequence corresponding to the sample RGB image sequence; and for each sample optical flow image in the plurality of sample optical flow images, obtain the previous frame sample optical flow image and the next frame sample optical flow image of the sample optical flow image; determine the fourth associated feature information corresponding to the sample optical flow image based on the sample optical flow image, the previous frame sample optical flow image, and the next frame sample optical flow image, to obtain the fourth associated feature information sequence corresponding to the sample optical flow image sequence;

[0234] Among them, once the behavior recognition model is trained, the differential augmentation module in the trained behavior recognition model is: Figure 2b The document includes a functional module that determines the first associated feature information corresponding to the RGB image based on the RGB image, the previous frame RGB image, and the next frame RGB image; and... Figure 2e The system includes a functional module that determines the second associated feature information corresponding to the optical flow image based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image. Specifically, in the differential enhancement module of the trained behavior recognition model, the values ​​of the 1×1 convolution kernels used to determine the first result, the second result, the third result, the fourth result, the fifth result, and the sixth result are the values ​​obtained during training.

[0235] Optionally, in the differential enhancement module of the trained behavior recognition model, the parameters of the first residual network and the parameters of the second residual network are also the values ​​obtained during training.

[0236] That is, the aforementioned Figure 2b In the corresponding embodiment, determining the first associated feature information corresponding to the RGB image based on the RGB image, the previous frame RGB image, and the next frame RGB image can be considered as: determining the first associated feature information corresponding to the RGB image based on the RGB image, the previous frame RGB image, and the next frame RGB image through the differential enhancement module in the trained behavior recognition model. Furthermore, Figure 2eIn a corresponding embodiment, determining the second associated feature information corresponding to the optical flow image based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image can be regarded as: determining the second associated feature information corresponding to the optical flow image based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image through the differential enhancement module in the trained behavior recognition model.

[0237] S3032. The spatial domain feature extraction module determines the third spatial domain feature information based on the third associated feature information sequence and the third preset spatial domain feature extraction rule; and determines the fourth spatial domain feature information based on the fourth associated feature information sequence and the fourth preset spatial domain feature extraction rule.

[0238] Among them, once the behavior recognition model is trained, the spatial domain feature extraction module in the trained behavior recognition model is: Figure 2c The diagram includes a functional module for determining the first spatial domain feature information based on the first associated feature information sequence and the first preset spatial domain feature extraction rule. It also includes a functional module (not shown) for determining the second spatial domain feature information based on the second associated feature information sequence and the second preset spatial domain feature extraction rule.

[0239] That is, the aforementioned Figure 2c The determination of the first spatial domain feature information based on the first associated feature information sequence and the first preset spatial domain feature extraction rule can be considered as follows: the first spatial domain feature information is determined by the spatial domain feature extraction module in the trained behavior recognition model, based on the first associated feature information sequence and the first preset spatial domain feature extraction rule. Furthermore, the determination of the second spatial domain feature information based on the second associated feature information sequence and the second preset spatial domain feature extraction rule (not shown in the figure) can be considered as follows: the second spatial domain feature information is determined by the spatial domain feature extraction module in the trained behavior recognition model, based on the second associated feature information sequence and the second preset spatial domain feature extraction rule.

[0240] Among them, the values ​​of the first, second, third, and fourth convolutional kernels in the spatial domain feature extraction module of the trained behavior recognition model are the values ​​obtained during training.

[0241] S3033. The time domain feature extraction module determines the third time domain feature information based on the third associated feature information sequence and the third preset time domain feature extraction rule; and determines the fourth time domain feature information based on the fourth associated feature information sequence and the fourth preset time domain feature extraction rule.

[0242] Among them, once the behavior recognition model is trained, the time-domain feature extraction module in the trained behavior recognition model is: Figure 2d The diagram includes a functional module for determining the first time-domain feature information based on the first associated feature information sequence and the first preset time-domain feature extraction rule, and a functional module (not shown) for determining the second time-domain feature information based on the second associated feature information sequence and the second preset time-domain feature extraction rule.

[0243] That is, the aforementioned Figure 2d In the corresponding embodiments, determining the first time-domain feature information based on the first associated feature information sequence and the first preset time-domain feature extraction rule can be considered as: determining the first time-domain feature information through the time-domain feature extraction module in the trained behavior recognition model, based on the first associated feature information sequence and the first preset time-domain feature extraction rule. Similarly, determining the second time-domain feature information based on the second associated feature information sequence and the second preset time-domain feature extraction rule (not shown in the figure) can be considered as: determining the second time-domain feature information through the time-domain feature extraction module in the trained behavior recognition model, based on the second associated feature information sequence and the second preset time-domain feature extraction rule.

[0244] Among them, the values ​​of the fifth, sixth, seventh, and eighth convolutional kernels in the time-domain feature extraction module of the trained behavior recognition model are the values ​​obtained during training.

[0245] S304. Using the classification module in the behavior recognition model, the predicted behavior category corresponding to the sample video segment is determined by the first sample feature information and the second sample feature information.

[0246] Once the behavior recognition model is trained, the classification module in the trained behavior recognition model is: Figure 2f The module in the middle uses the first feature information and the second feature information to determine the behavior recognition result corresponding to the video segment.

[0247] That is, the aforementioned Figure 2f In a corresponding embodiment, determining the behavior recognition result corresponding to the video segment using the first feature information and the second feature information can be regarded as: determining the behavior recognition result corresponding to the video segment using the first feature information and the second feature information through the classification module in the trained behavior recognition model.

[0248] Optionally, in the classification module of the trained behavior recognition model, the parameters (such as the weights of each coefficient) of the target attention mechanism are the values ​​obtained during training.

[0249] S305. Using the predicted behavior category and the target behavior category label, train the behavior recognition model to obtain a trained behavior recognition model.

[0250] In the training of the behavior recognition model, the optimization algorithm can employ stochastic gradient descent or adaptive gradient descent, and overfitting can be prevented through strategies such as Dropout; the loss function can be the cross-entropy loss function. Dropout refers to temporarily removing neural network units from the network with a certain probability during the training process.

[0251] In some optional embodiments of this application, see [link to relevant documentation]. Figure 3b As shown, after the behavior recognition model is trained, when performing behavior recognition on video segments based on the trained behavior recognition model, the video segments can be preprocessed first to obtain RGB image sequences and optical flow image sequences.

[0252] The first associated feature information sequence corresponding to the RGB image sequence is determined by the differential enhancement unit 1 in the differential enhancement module of the trained behavior recognition model, and the second associated feature information sequence corresponding to the optical flow image sequence is determined by the differential enhancement unit 2 in the differential enhancement module of the trained behavior recognition model.

[0253] The training behavior recognition model uses a spatial domain feature extraction unit 1 in its spatial domain feature extraction module to determine first spatial domain feature information based on the first associated feature information sequence and the first preset spatial domain feature extraction rule. It also uses a spatial domain feature extraction unit 2 in its spatial domain feature extraction module to determine second spatial domain feature information based on the second associated feature information sequence and the second preset spatial domain feature extraction rule. Furthermore, the training behavior recognition model uses a temporal domain feature extraction unit 1 in its temporal domain feature extraction module to determine first temporal domain feature information based on the first associated feature information sequence and the first preset temporal domain feature extraction rule. Finally, the training behavior recognition model uses a temporal domain feature extraction unit 2 in its temporal domain feature extraction module to determine second temporal domain feature information based on the second associated feature information sequence and the second preset temporal domain feature extraction rule.

[0254] Then, using the classification module in the trained behavior recognition model, the behavior recognition result corresponding to the video segment is determined by using the first feature information and the second feature information.

[0255] In this application, the aforementioned spatial domain feature extraction module can extract the dependency relationship between each pixel and other pixels in the first associated feature information sequence space. Furthermore, the contribution of image frames at different time points to behavior recognition may vary, and the aforementioned temporal domain feature extraction module can uncover the importance of image frames at different time points. The aforementioned classification module can adaptively fuse feature information with different image modalities (i.e., RGB images and optical flow images), improving the accuracy of behavior recognition (i.e., behavior classification). The model training method in this application employs a deep learning training scheme. Since relevant labeled data may be lacking in practical applications, the training scheme in this application can use self-supervised learning to pre-train the behavior recognition model for better application in real-world scenarios.

[0256] This application proposes an action recognition method based on a differential enhancement network, a spatial domain feature extraction module, and a temporal domain feature extraction module. The method uses the differential enhancement network to construct the basic module for action recognition, extracting local motion features from video frames to improve the accuracy of action recognition. Secondly, the spatial and temporal feature extraction modules proposed in this application can perform weighted operations on the spatial and temporal domains based on the feature sequences corresponding to the video frames (i.e., the first or second associated feature information sequence), thereby increasing the weight of keyframes and effective spatial regions, and thus improving the accuracy of action recognition. Furthermore, the image analysis in this application involves RGB images and optical flow images; by analyzing information from these two modalities, the accuracy of action recognition is further improved.

[0257] This application also provides a behavior recognition method, which can be executed by a terminal or a server, or by both a terminal and a server. This application embodiment uses the execution of the method by a terminal as an example for illustration. Figure 4 This is a flowchart illustrating the behavior recognition method provided in an embodiment of this application. The method includes S401-S402:

[0258] S401. When a recognition instruction is triggered by an object to perform behavior recognition on the selected video segment to be recognized, the video segment is input into the trained behavior recognition model, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment.

[0259] The terminal interface can display multiple alternative video segments for the object (which can be a user) to select. The object can select the video segment to be identified by the button corresponding to each alternative video segment. When the object clicks the button corresponding to the video segment to be identified, it can be regarded as a recognition instruction triggered by the object to perform behavior recognition on the selected video segment to be identified.

[0260] S402. Display the behavior recognition results;

[0261] Among them, the behavior recognition result can be the behavior type information of the target object in the video segment. The target object can be a person, animal, moving object, etc., and the behavior type information can be: speeding while driving, littering, etc.

[0262] Optionally, the target object can also be an experimental subject, such as a mouse or bacteria, and the behavior type information can be: eating, moving towards direction A, etc.

[0263] The behavior recognition method in this application can also be used to determine a scene containing behavior type information of the patient's physical condition based on a video segment containing the patient's action information.

[0264] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0265] Optionally, the object can select the video segment to be identified using controls displayed on the terminal interface.

[0266] This application also provides a behavior recognition device. Figure 5 This is a schematic diagram of the structure of the behavior recognition device 50, which may include:

[0267] The first acquisition unit 51 is used to acquire the video segment to be identified;

[0268] The first determining unit 52 is used to determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0269] The first feature extraction unit 53 is used to determine first feature information based on the RGB image sequence, the first feature information including first spatial domain feature information and first temporal domain feature information for the RGB image sequence, and to determine second feature information based on the optical flow image sequence, the second feature information including second spatial domain feature information and second temporal domain feature information for the optical flow image sequence;

[0270] The first identification unit 54 is used to determine the behavior identification result corresponding to the video segment using the first feature information and the second feature information.

[0271] Optionally, the RGB image sequence includes multiple RGB images, and the first feature extraction unit 53, when determining the first feature information based on the RGB image sequence, is specifically used for:

[0272] For each of the multiple RGB images, obtain the previous frame RGB image and the next frame RGB image of the RGB image;

[0273] Based on the RGB image, the previous frame RGB image, and the next frame RGB image, the first associated feature information corresponding to the RGB image is determined, and the first associated feature information sequence corresponding to the RGB image sequence is obtained. The dimension information of the first associated feature information sequence includes: the number of channels of the first associated feature information sequence, the number of image frames of the first associated feature information sequence, the feature length information of the first associated feature information sequence, and the feature width information of the first associated feature information sequence.

[0274] Based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, the corresponding first feature information is determined.

[0275] Optionally, the first feature extraction unit 53, when determining the corresponding first feature information based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, is specifically used for:

[0276] The first spatial domain feature information is determined based on the first associated feature information sequence and the first preset spatial domain feature extraction rule;

[0277] The first time domain feature information is determined based on the first associated feature information sequence and the first preset time domain feature extraction rule.

[0278] Optionally, the first preset spatial domain feature extraction rule includes: a first convolutional kernel group, a second convolutional kernel group, a first dimension transformation function, and a second dimension transformation function, wherein the second dimension transformation function is the inverse of the first dimension transformation function. The first feature extraction unit 53, when used to determine the first spatial domain feature information based on the first associated feature information sequence and the first preset spatial domain feature extraction rule, is specifically used for:

[0279] Based on the first convolutional kernel group and the first dimension transformation function, the first associated feature information sequence is processed to obtain the first spatial relationship information corresponding to the first associated feature information sequence;

[0280] The first spatial domain feature information is determined based on the second convolution kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information.

[0281] Optionally, the first convolutional kernel group includes a first convolutional kernel and a second convolutional kernel. The first feature extraction unit 53, when processing the first associated feature information sequence based on the first convolutional kernel group and the first dimension transformation function to obtain the first spatial relationship information corresponding to the first associated feature information sequence, is specifically used for:

[0282] The first convolution kernel is used to perform convolution processing on the first associated feature information sequence to obtain the first convolution result, thereby adjusting the number of channels of the first associated feature information sequence;

[0283] The first convolution result is transformed by the first dimension transformation function to obtain the first transformation result. The dimension information of the first transformation result includes: the product of the feature length information and the feature width information of the first convolution result, and the product of the number of channels of the first convolution result and the number of image frames.

[0284] The first associated feature information sequence is convolved by the second convolution kernel to obtain the second convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the second convolution result is the same as the number of channels of the first convolution result.

[0285] The second convolution result is transformed by the first dimension transformation function to obtain a second transformation result, and the dimension information of the second transformation result is the same as that of the first transformation result.

[0286] Calculate the product of the first transformation result and the transpose of the second transformation result to obtain the first spatial relationship information corresponding to the first associated feature information sequence. The dimension information of the first spatial relationship information includes: the product of the feature length information of the first transformation result and the feature width information of the first transformation result, and the product of the feature length information of the second transformation result and the feature width information of the second transformation result.

[0287] Optionally, the second convolutional kernel group includes a third convolutional kernel and a fourth convolutional kernel. The first feature extraction unit 53, when used to determine the first spatial domain feature information based on the second convolutional kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information, is specifically used for:

[0288] The first associated feature information sequence is convolved by the third convolution kernel to obtain the third convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the third convolution result is the same as the number of channels of the second convolution result.

[0289] The third convolution result is transformed by the first dimension transformation function to obtain the third transformation result, and the dimension information of the third transformation result is the same as that of the first transformation result.

[0290] The first spatial relationship information is normalized to obtain the first normalized result;

[0291] Calculate the product of the first normalization result and the transpose of the third transformation result to obtain the first product result. The dimensional information of the first product result is the same as the dimensional information of the transpose of the third transformation result.

[0292] The first product result is transformed by the second dimension transformation function to obtain a fourth transformation result, and the dimension information of the fourth transformation result is the same as that of the third convolution result.

[0293] The first associated feature information sequence is convolved by the fourth convolution kernel to obtain the fourth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the fourth convolution result is the same as the number of channels of the third convolution result.

[0294] The first spatial domain feature information is obtained by summing the fourth transformation result and the fourth convolution result.

[0295] Optionally, the first preset time-domain feature extraction rule includes: a third convolution kernel group, a fourth convolution kernel group, a third-dimensional transformation function, and a fourth-dimensional transformation function, wherein the fourth-dimensional transformation function is the inverse of the third-dimensional transformation function. The first feature extraction unit 53, when used to determine the first time-domain feature information based on the first associated feature information sequence and the first preset time-domain feature extraction rule, is specifically used for:

[0296] Based on the third convolutional kernel group and the third dimension transformation function, the first associated feature information sequence is processed to obtain the first time relationship information corresponding to the first associated feature information sequence;

[0297] The first time-domain feature information is determined based on the fourth convolution kernel group, the third-dimensional transformation function, the fourth-dimensional transformation function, and the first time relationship information.

[0298] Optionally, the third convolutional kernel group includes a fifth convolutional kernel and a sixth convolutional kernel. The first feature extraction unit 53, when processing the first associated feature information sequence based on the third convolutional kernel group and the third-dimensional transformation function to obtain the first temporal relationship information corresponding to the first associated feature information sequence, is specifically used for:

[0299] The first associated feature information sequence is convolved by the fifth convolution kernel to obtain the fifth convolution result, thereby adjusting the number of channels in the first associated feature information sequence;

[0300] The fifth convolution result is obtained by performing a dimensional transformation on the fifth convolution result using the third dimensional transformation function. The dimensional information of the fifth transformation result includes: the number of image frames of the fifth convolution result and a first product, wherein the first product is the product of the number of channels of the fifth convolution result, the feature length information of the fifth convolution result, and the feature width information of the fifth convolution result.

[0301] The first associated feature information sequence is convolved by the sixth convolution kernel to obtain the sixth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the sixth convolution result is the same as the number of channels of the fifth convolution result.

[0302] The sixth convolution result is obtained by performing a dimensional transformation on the third dimensional transformation function, and the dimensional information of the sixth transformation result is the same as that of the fifth transformation result.

[0303] Calculate the product of the fifth transformation result and the transpose of the sixth transformation result to obtain the first temporal relationship information corresponding to the first associated feature information sequence. The dimension information of the first temporal relationship information includes: the number of image frames in the fifth transformation result and the number of image frames in the sixth transformation result.

[0304] Optionally, the fourth convolutional kernel group includes a seventh convolutional kernel and an eighth convolutional kernel. The first feature extraction unit 53, when used to determine the first time-domain feature information based on the fourth convolutional kernel group, the third-dimensional transformation function, the fourth-dimensional transformation function, and the first temporal relationship information, is specifically used for:

[0305] The first associated feature information sequence is convolved by the seventh convolution kernel to obtain the seventh convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the seventh convolution result is the same as the number of channels of the sixth convolution result.

[0306] The seventh convolution result is obtained by performing a dimensional transformation on the third dimensional transformation function, and the dimensional information of the seventh transformation result is the same as that of the fifth transformation result.

[0307] The first time relationship information is normalized to obtain the second normalization result;

[0308] Calculate the product of the second normalized result and the transpose of the seventh transformation result to obtain the second product result. The dimensional information of the second product result is the same as the dimensional information of the transpose of the seventh transformation result.

[0309] The second product result is transformed by the fourth dimension transformation function to obtain the eighth transformation result, and the dimension information of the eighth transformation result is the same as that of the seventh convolution result.

[0310] The first associated feature information sequence is convolved by the eighth convolution kernel to obtain the eighth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the eighth convolution result is the same as the number of channels of the seventh convolution result.

[0311] The first time-domain feature information is obtained by summing the eighth transformation result and the eighth convolution result.

[0312] Optionally, the optical flow image sequence includes multiple optical flow images, and the first feature extraction unit 53, when determining the second feature information based on the optical flow image sequence, is specifically used for:

[0313] For each optical flow image in the plurality of optical flow images, obtain the previous frame optical flow image and the next frame optical flow image of the optical flow image;

[0314] Based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image, the second associated feature information corresponding to the optical flow image is determined, and the second associated feature information sequence corresponding to the optical flow image sequence is obtained. The dimension information of the second associated feature information sequence includes: the number of channels of the second associated feature information sequence, the number of image frames of the second associated feature information sequence, the feature length information of the second associated feature information sequence, and the feature width information of the second associated feature information sequence.

[0315] The corresponding second feature information is determined based on the second associated feature information sequence, the second preset spatial domain feature extraction rule, and the second preset temporal domain feature extraction rule.

[0316] Optionally, when the first recognition unit 54 determines the behavior recognition result corresponding to the video segment using the first feature information and the second feature information, it is specifically used for:

[0317] The first feature information and the second feature information are stacked in the dimension of channel number to obtain the stacking result;

[0318] The stacking result is transformed using the fifth-dimensional transformation function to obtain a first target transformation result. The dimensional information of the first target transformation result includes: the number of channels of the stacking result and a second product, where the second product is the product of the feature length information of the stacking result, the feature width information of the stacking result, and the number of image frames of the stacking result.

[0319] Based on the first target transformation result and the target attention mechanism network, the behavior recognition result corresponding to the video segment is determined.

[0320] Optionally, when the first recognition unit determines the behavior recognition result corresponding to the video segment based on the first target transformation result and the target attention mechanism network, it is specifically used for:

[0321] The first target transformation result is input into the target attention mechanism network to obtain the output result;

[0322] The output result is transformed using the sixth-dimensional transformation function to obtain the second target transformation result. The dimensional information of the second target transformation result is the same as that of the stacked result.

[0323] The second target transformation result is split to obtain third and fourth feature information corresponding to the RGB image sequence and the optical flow image sequence, respectively;

[0324] The third feature information and the fourth feature information are summed to obtain the summation result;

[0325] The behavior recognition result corresponding to the video segment is determined based on the summation result.

[0326] Each unit of the aforementioned behavior recognition device 50 can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0327] The behavior recognition device 50 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the behavior recognition device 50 can be the terminal or server.

[0328] This application also provides a model training device. Figure 6 This is a schematic diagram of the structure of a model training device 60, which may include:

[0329] The second acquisition unit 61 is used to acquire sample video segments and target behavior category labels corresponding to the sample video segments;

[0330] The second determining unit 62 is used to determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0331] The second feature extraction unit 63 is used to determine first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence through the feature extraction module in the behavior recognition model; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence.

[0332] The second identification unit 64 is used to determine the predicted behavior category corresponding to the sample video segment by using the classification module in the behavior recognition model and the first sample feature information and the second sample feature information.

[0333] Training unit 65 is used to train the behavior recognition model using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model.

[0334] Optionally, the feature extraction module includes: a difference enhancement module, a spatial domain feature extraction module, and a temporal domain feature extraction module, wherein the sample RGB image sequence includes multiple sample RGB images, and the sample optical flow image sequence includes multiple sample optical flow images;

[0335] The second feature extraction unit 63, when used by the feature extraction module in the behavior recognition model to determine the first sample feature information based on the sample RGB image sequence and the second sample feature information based on the sample optical flow image sequence, is specifically used for:

[0336] Through the differential enhancement module, for each sample RGB image in the plurality of sample RGB images, the previous frame sample RGB image and the next frame sample RGB image of the sample RGB image are obtained; based on the sample RGB image, the previous frame sample RGB image, and the next frame sample RGB image, the third associated feature information corresponding to the sample RGB image is determined, and the third associated feature information sequence corresponding to the sample RGB image sequence is obtained; and for each sample optical flow image in the plurality of sample optical flow images, the previous frame sample optical flow image and the next frame sample optical flow image of the sample optical flow image are obtained; based on the sample optical flow image, the previous frame sample optical flow image, and the next frame sample optical flow image, the fourth associated feature information corresponding to the sample optical flow image is determined, and the fourth associated feature information sequence corresponding to the sample optical flow image sequence is obtained;

[0337] The spatial domain feature extraction module determines the third spatial domain feature information based on the third associated feature information sequence and the third preset spatial domain feature extraction rule; and determines the fourth spatial domain feature information based on the fourth associated feature information sequence and the fourth preset spatial domain feature extraction rule.

[0338] The time-domain feature extraction module determines the third time-domain feature information based on the third associated feature information sequence and the third preset time-domain feature extraction rule; and determines the fourth time-domain feature information based on the fourth associated feature information sequence and the fourth preset time-domain feature extraction rule.

[0339] Each unit of the aforementioned model training device 60 can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0340] The model training device 60 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the model training device 60 can be the terminal or server.

[0341] This application also provides a behavior recognition device. Figure 7 This is a schematic diagram of the structure of a behavior recognition device 70, which may include:

[0342] The detection unit 71 is used to input the video segment into the trained behavior recognition model when a recognition instruction for behavior recognition is triggered by an object for the selected video segment to be recognized, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment.

[0343] Display unit 72 is used to display the behavior recognition results;

[0344] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0345] Each unit of the aforementioned behavior recognition device 70 can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0346] The behavior recognition device 70 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the behavior recognition device 70 can be the terminal or server.

[0347] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0348] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0349] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be the aforementioned behavior recognition device or model training device. For example... Figure 8 As shown, the computer device 800 may include: a communication interface 801, a memory 802, a processor 803, and a communication bus 804. The communication interface 801, memory 802, and processor 803 communicate with each other via the communication bus 804. The communication interface 801 is used for data communication between the computer device 800 and external devices. The memory 802 can be used to store software programs and modules, and the processor 803 executes the software programs and modules stored in the memory 802, such as the software programs for the corresponding operations in the aforementioned method embodiments.

[0350] Optionally, the processor 803 can invoke software programs and modules stored in the memory 802 to perform the following operations:

[0351] Obtain the video segment to be identified;

[0352] Determine the corresponding RGB image sequence and optical flow image sequence based on the video segment;

[0353] First feature information is determined based on the RGB image sequence, and the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence;

[0354] A second feature information is determined based on the optical flow image sequence, the second feature information including a second spatial domain feature information and a second temporal domain feature information for the optical flow image sequence;

[0355] The behavior recognition result corresponding to the video segment is determined using the first feature information and the second feature information.

[0356] Optionally, the processor 803 can also invoke software programs and modules stored in the memory 802 to perform the following operations:

[0357] Obtain sample video segments and the corresponding target behavior category labels for the sample video segments;

[0358] Determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment;

[0359] The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence.

[0360] The classification module in the behavior recognition model uses the first sample feature information and the second sample feature information to determine the predicted behavior category corresponding to the sample video segment.

[0361] The behavior recognition model is trained using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model.

[0362] Optionally, the processor 803 can also invoke software programs and modules stored in the memory 802 to perform the following operations:

[0363] When a recognition instruction is triggered by an object to perform behavior recognition on the selected video segment to be recognized, the video segment is input into the trained behavior recognition model, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment;

[0364] Display the behavior recognition results;

[0365] The trained behavior recognition model is a behavior recognition model trained based on the aforementioned model training method.

[0366] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the methods described in the embodiments of this application; for brevity, these will not be elaborated further here.

[0367] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described in the embodiments of this application. For brevity, these details will not be elaborated further here.

[0368] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding processes in the methods described in the embodiments of this application. For the sake of brevity, these details will not be elaborated further here.

[0369] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0370] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0371] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0372] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0373] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0374] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0375] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0376] In addition, the functional units in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0377] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0378] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A behavior recognition method, characterized in that, The method includes: Obtain the video segment to be identified; Determine the corresponding RGB image sequence and optical flow image sequence based on the video segment; First feature information is determined based on the RGB image sequence, and the first feature information includes first spatial domain feature information and first temporal domain feature information for the RGB image sequence; A second feature information is determined based on the optical flow image sequence, the second feature information including a second spatial domain feature information and a second temporal domain feature information for the optical flow image sequence; The behavior recognition result corresponding to the video segment is determined using the first feature information and the second feature information; The RGB image sequence includes multiple RGB images, and determining the first feature information based on the RGB image sequence includes: For each of the multiple RGB images, obtain the previous frame RGB image and the next frame RGB image of the RGB image; Based on the RGB image, the previous frame RGB image, and the next frame RGB image, the first associated feature information corresponding to the RGB image is determined, and the first associated feature information sequence corresponding to the RGB image sequence is obtained. The dimension information of the first associated feature information sequence includes: the number of channels of the first associated feature information sequence, the number of image frames of the first associated feature information sequence, the feature length information of the first associated feature information sequence, and the feature width information of the first associated feature information sequence. Based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, the corresponding first feature information is determined. The optical flow image sequence includes multiple optical flow images, and the determination of the second feature information based on the optical flow image sequence includes: For each optical flow image in the plurality of optical flow images, obtain the previous frame optical flow image and the next frame optical flow image of the optical flow image; Based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image, the second associated feature information corresponding to the optical flow image is determined, and the second associated feature information sequence corresponding to the optical flow image sequence is obtained. The dimension information of the second associated feature information sequence includes: the number of channels of the second associated feature information sequence, the number of image frames of the second associated feature information sequence, the feature length information of the second associated feature information sequence, and the feature width information of the second associated feature information sequence. The corresponding second feature information is determined based on the second associated feature information sequence, the second preset spatial domain feature extraction rule, and the second preset temporal domain feature extraction rule.

2. The method according to claim 1, characterized in that, The step of determining the corresponding first feature information based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule includes: The first spatial domain feature information is determined based on the first associated feature information sequence and the first preset spatial domain feature extraction rule; The first time domain feature information is determined based on the first associated feature information sequence and the first preset time domain feature extraction rule.

3. The method according to claim 2, characterized in that, The first preset spatial domain feature extraction rule includes: a first convolutional kernel group, a second convolutional kernel group, a first-dimensional transformation function, and a second-dimensional transformation function, wherein the second-dimensional transformation function is the inverse of the first-dimensional transformation function. Based on the first associated feature information sequence and the first preset spatial domain feature extraction rule, the first spatial domain feature information is determined, including: Based on the first convolutional kernel group and the first dimension transformation function, the first associated feature information sequence is processed to obtain the first spatial relationship information corresponding to the first associated feature information sequence; The first spatial domain feature information is determined based on the second convolution kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information.

4. The method according to claim 3, characterized in that, The first convolutional kernel group includes a first convolutional kernel and a second convolutional kernel. The step of processing the first associated feature information sequence based on the first convolutional kernel group and the first dimension transformation function to obtain the first spatial relationship information corresponding to the first associated feature information sequence includes: The first convolution kernel is used to perform convolution processing on the first associated feature information sequence to obtain the first convolution result, thereby adjusting the number of channels of the first associated feature information sequence; The first convolution result is transformed by the first dimension transformation function to obtain the first transformation result. The dimension information of the first transformation result includes: the product of the feature length information and the feature width information of the first convolution result, and the product of the number of channels of the first convolution result and the number of image frames. The first associated feature information sequence is convolved by the second convolution kernel to obtain the second convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the second convolution result is the same as the number of channels of the first convolution result. The second convolution result is transformed by the first dimension transformation function to obtain a second transformation result, and the dimension information of the second transformation result is the same as that of the first transformation result. Calculate the product of the first transformation result and the transpose of the second transformation result to obtain the first spatial relationship information corresponding to the first associated feature information sequence. The dimension information of the first spatial relationship information includes: the product of the feature length information of the first transformation result and the feature width information of the first transformation result, and the product of the feature length information of the second transformation result and the feature width information of the second transformation result.

5. The method according to claim 4, characterized in that, The second convolutional kernel group includes a third convolutional kernel and a fourth convolutional kernel. The step of determining the first spatial domain feature information based on the second convolutional kernel group, the first dimension transformation function, the second dimension transformation function, and the first spatial relationship information includes: The first associated feature information sequence is convolved by the third convolution kernel to obtain the third convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the third convolution result is the same as the number of channels of the second convolution result. The third convolution result is transformed by the first dimension transformation function to obtain the third transformation result, and the dimension information of the third transformation result is the same as that of the first transformation result. The first spatial relationship information is normalized to obtain the first normalized result; Calculate the product of the first normalization result and the transpose of the third transformation result to obtain the first product result. The dimensional information of the first product result is the same as the dimensional information of the transpose of the third transformation result. The first product result is transformed by the second dimension transformation function to obtain a fourth transformation result, and the dimension information of the fourth transformation result is the same as that of the third convolution result. The first associated feature information sequence is convolved by the fourth convolution kernel to obtain the fourth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the fourth convolution result is the same as the number of channels of the third convolution result. The first spatial domain feature information is obtained by summing the fourth transformation result and the fourth convolution result.

6. The method according to claim 2, characterized in that, The first preset time-domain feature extraction rule includes: a third convolution kernel group, a fourth convolution kernel group, a third-dimensional transformation function, and a fourth-dimensional transformation function, wherein the fourth-dimensional transformation function is the inverse of the third-dimensional transformation function. The step of determining the first time-domain feature information based on the first associated feature information sequence and the first preset time-domain feature extraction rule includes: Based on the third convolutional kernel group and the third dimension transformation function, the first associated feature information sequence is processed to obtain the first time relationship information corresponding to the first associated feature information sequence; The first time-domain feature information is determined based on the fourth convolution kernel group, the third-dimensional transformation function, the fourth-dimensional transformation function, and the first time relationship information.

7. The method according to claim 6, characterized in that, The third convolutional kernel group includes a fifth convolutional kernel and a sixth convolutional kernel. The processing of the first associated feature information sequence based on the third convolutional kernel group and the third-dimensional transformation function to obtain the first temporal relationship information corresponding to the first associated feature information sequence includes: The first associated feature information sequence is convolved by the fifth convolution kernel to obtain the fifth convolution result, thereby adjusting the number of channels in the first associated feature information sequence; The fifth convolution result is obtained by performing a dimensional transformation on the fifth convolution result using the third dimensional transformation function. The dimensional information of the fifth transformation result includes: the number of image frames of the fifth convolution result and a first product, wherein the first product is the product of the number of channels of the fifth convolution result, the feature length information of the fifth convolution result, and the feature width information of the fifth convolution result. The first associated feature information sequence is convolved by the sixth convolution kernel to obtain the sixth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the sixth convolution result is the same as the number of channels of the fifth convolution result. The sixth convolution result is obtained by performing a dimensional transformation on the third dimensional transformation function, and the dimensional information of the sixth transformation result is the same as that of the fifth transformation result. Calculate the product of the fifth transformation result and the transpose of the sixth transformation result to obtain the first temporal relationship information corresponding to the first associated feature information sequence. The dimension information of the first temporal relationship information includes: the number of image frames in the fifth transformation result and the number of image frames in the sixth transformation result.

8. The method according to claim 7, characterized in that, The fourth convolutional kernel group includes a seventh convolutional kernel and an eighth convolutional kernel. The step of determining the first time-domain feature information based on the fourth convolutional kernel group, the third-dimensional transformation function, the fourth-dimensional transformation function, and the first temporal relationship information includes: The first associated feature information sequence is convolved by the seventh convolution kernel to obtain the seventh convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the seventh convolution result is the same as the number of channels of the sixth convolution result. The seventh convolution result is obtained by performing a dimensional transformation on the third dimensional transformation function, and the dimensional information of the seventh transformation result is the same as that of the fifth transformation result. The first time relationship information is normalized to obtain the second normalization result; Calculate the product of the second normalized result and the transpose of the seventh transformation result to obtain the second product result. The dimensional information of the second product result is the same as the dimensional information of the transpose of the seventh transformation result. The second product result is transformed by the fourth dimension transformation function to obtain the eighth transformation result, and the dimension information of the eighth transformation result is the same as that of the seventh convolution result. The first associated feature information sequence is convolved by the eighth convolution kernel to obtain the eighth convolution result, thereby adjusting the number of channels of the first associated feature information sequence. The number of channels of the eighth convolution result is the same as the number of channels of the seventh convolution result. The first time-domain feature information is obtained by summing the eighth transformation result and the eighth convolution result.

9. The method according to claim 1, characterized in that, The step of determining the behavior recognition result corresponding to the video segment using the first feature information and the second feature information includes: The first feature information and the second feature information are stacked in the dimension of channel number to obtain the stacking result; The stacking result is transformed using the fifth-dimensional transformation function to obtain a first target transformation result. The dimensional information of the first target transformation result includes: the number of channels of the stacking result and a second product, where the second product is the product of the feature length information of the stacking result, the feature width information of the stacking result, and the number of image frames of the stacking result. Based on the first target transformation result and the target attention mechanism network, the behavior recognition result corresponding to the video segment is determined.

10. The method according to claim 9, characterized in that, The step of determining the behavior recognition result corresponding to the video segment based on the first target transformation result and the target attention mechanism network includes: The first target transformation result is input into the target attention mechanism network to obtain the output result; The output result is transformed using the sixth-dimensional transformation function to obtain the second target transformation result. The dimensional information of the second target transformation result is the same as the dimensional information of the stacked result. The second target transformation result is split to obtain third and fourth feature information corresponding to the RGB image sequence and the optical flow image sequence, respectively; The third feature information and the fourth feature information are summed to obtain the summation result; The behavior recognition result corresponding to the video segment is determined based on the summation result.

11. A model training method, characterized in that, The method includes: Obtain sample video segments and the corresponding target behavior category labels for the sample video segments; Determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment; The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence. The classification module in the behavior recognition model uses the first sample feature information and the second sample feature information to determine the predicted behavior category corresponding to the sample video segment. The behavior recognition model is trained using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model; The feature extraction module includes: a difference enhancement module, a spatial domain feature extraction module, and a temporal domain feature extraction module. The sample RGB image sequence includes multiple sample RGB images, and the sample optical flow image sequence includes multiple sample optical flow images. The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence, including: Through the differential enhancement module, for each sample RGB image in the plurality of sample RGB images, the previous frame sample RGB image and the next frame sample RGB image of the sample RGB image are obtained; based on the sample RGB image, the previous frame sample RGB image, and the next frame sample RGB image, the third associated feature information corresponding to the sample RGB image is determined, and the third associated feature information sequence corresponding to the sample RGB image sequence is obtained; and for each sample optical flow image in the plurality of sample optical flow images, the previous frame sample optical flow image and the next frame sample optical flow image of the sample optical flow image are obtained; based on the sample optical flow image, the previous frame sample optical flow image, and the next frame sample optical flow image, the fourth associated feature information corresponding to the sample optical flow image is determined, and the fourth associated feature information sequence corresponding to the sample optical flow image sequence is obtained; The spatial domain feature extraction module determines the third spatial domain feature information based on the third associated feature information sequence and the third preset spatial domain feature extraction rule; and determines the fourth spatial domain feature information based on the fourth associated feature information sequence and the fourth preset spatial domain feature extraction rule. The time-domain feature extraction module determines the third time-domain feature information based on the third associated feature information sequence and the third preset time-domain feature extraction rule; and determines the fourth time-domain feature information based on the fourth associated feature information sequence and the fourth preset time-domain feature extraction rule.

12. A behavior recognition method, characterized in that, include: When a recognition instruction is triggered by an object to perform behavior recognition on the selected video segment to be recognized, the video segment is input into the trained behavior recognition model, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment; Display the behavior recognition results; The trained behavior recognition model is a behavior recognition model trained based on the model training method described in claim 11.

13. A behavior recognition device, characterized in that, include: The first acquisition unit is used to acquire the video segment to be identified; The first determining unit is used to determine the corresponding RGB image sequence and optical flow image sequence based on the video segment; A first feature extraction unit is configured to determine first feature information based on the RGB image sequence, the first feature information including first spatial domain feature information and first temporal domain feature information for the RGB image sequence; and to determine second feature information based on the optical flow image sequence, the second feature information including second spatial domain feature information and second temporal domain feature information for the optical flow image sequence. The first identification unit is used to determine the behavior identification result corresponding to the video segment using the first feature information and the second feature information; The RGB image sequence includes multiple RGB images, and determining the first feature information based on the RGB image sequence includes: For each of the multiple RGB images, obtain the previous frame RGB image and the next frame RGB image of the RGB image; Based on the RGB image, the previous frame RGB image, and the next frame RGB image, the first associated feature information corresponding to the RGB image is determined, and the first associated feature information sequence corresponding to the RGB image sequence is obtained. The dimension information of the first associated feature information sequence includes: the number of channels of the first associated feature information sequence, the number of image frames of the first associated feature information sequence, the feature length information of the first associated feature information sequence, and the feature width information of the first associated feature information sequence. Based on the first associated feature information sequence, the first preset spatial domain feature extraction rule, and the first preset temporal domain feature extraction rule, the corresponding first feature information is determined. The optical flow image sequence includes multiple optical flow images, and the determination of the second feature information based on the optical flow image sequence includes: For each optical flow image in the plurality of optical flow images, obtain the previous frame optical flow image and the next frame optical flow image of the optical flow image; Based on the optical flow image, the previous frame optical flow image, and the next frame optical flow image, the second associated feature information corresponding to the optical flow image is determined, and the second associated feature information sequence corresponding to the optical flow image sequence is obtained. The dimension information of the second associated feature information sequence includes: the number of channels of the second associated feature information sequence, the number of image frames of the second associated feature information sequence, the feature length information of the second associated feature information sequence, and the feature width information of the second associated feature information sequence. The corresponding second feature information is determined based on the second associated feature information sequence, the second preset spatial domain feature extraction rule, and the second preset temporal domain feature extraction rule.

14. A model training device, characterized in that, include: The second acquisition unit is used to acquire sample video segments and target behavior category labels corresponding to the sample video segments; The second determining unit is used to determine the corresponding sample RGB image sequence and sample optical flow image sequence based on the sample video segment; The second feature extraction unit is used to determine first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence through the feature extraction module in the behavior recognition model; wherein, the first sample feature information includes third spatial domain feature information and third temporal domain feature information for the sample RGB image sequence; and the second sample feature information includes fourth spatial domain feature information and fourth temporal domain feature information for the sample optical flow image sequence. The second identification unit is used to determine the predicted behavior category corresponding to the sample video segment by using the classification module in the behavior recognition model and the first sample feature information and the second sample feature information. The training unit is used to train the behavior recognition model using the predicted behavior category and the target behavior category label to obtain a trained behavior recognition model. The feature extraction module includes: a difference enhancement module, a spatial domain feature extraction module, and a temporal domain feature extraction module. The sample RGB image sequence includes multiple sample RGB images, and the sample optical flow image sequence includes multiple sample optical flow images. The feature extraction module in the behavior recognition model determines first sample feature information based on the sample RGB image sequence and second sample feature information based on the sample optical flow image sequence, including: Through the differential enhancement module, for each sample RGB image in the plurality of sample RGB images, the previous frame sample RGB image and the next frame sample RGB image of the sample RGB image are obtained; based on the sample RGB image, the previous frame sample RGB image, and the next frame sample RGB image, the third associated feature information corresponding to the sample RGB image is determined, and the third associated feature information sequence corresponding to the sample RGB image sequence is obtained; and for each sample optical flow image in the plurality of sample optical flow images, the previous frame sample optical flow image and the next frame sample optical flow image of the sample optical flow image are obtained; based on the sample optical flow image, the previous frame sample optical flow image, and the next frame sample optical flow image, the fourth associated feature information corresponding to the sample optical flow image is determined, and the fourth associated feature information sequence corresponding to the sample optical flow image sequence is obtained; The spatial domain feature extraction module determines the third spatial domain feature information based on the third associated feature information sequence and the third preset spatial domain feature extraction rule; and determines the fourth spatial domain feature information based on the fourth associated feature information sequence and the fourth preset spatial domain feature extraction rule. The time-domain feature extraction module determines the third time-domain feature information based on the third associated feature information sequence and the third preset time-domain feature extraction rule; and determines the fourth time-domain feature information based on the fourth associated feature information sequence and the fourth preset time-domain feature extraction rule.

15. A behavior recognition device, characterized in that, include: The detection unit is used to input the video segment into the trained behavior recognition model when a recognition instruction for behavior recognition is triggered by an object and the selected video segment to be recognized is detected, so that the behavior recognition model determines the corresponding behavior recognition result based on the video segment; The display unit is used to display the behavior recognition results; The trained behavior recognition model is a behavior recognition model trained based on the model training method described in claim 11.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the behavior recognition method of any one of claims 1-10 or claim 12, or the model training method of claim 11.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the steps of the behavior recognition method according to any one of claims 1-10 or claim 12, or the model training method according to claim 11, by calling the computer program stored in the memory.

18. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of any one of claims 1-10 or the behavior recognition method of claim 12, or the model training method of claim 11.

Images