Video processing method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses a video processing method, device, computer equipment and storage medium. The video processing method includes: obtaining a target video frame sequence, the target video frame sequence includes a target key frame, and the target key frame includes a target object; identifying the target position information of the target object in the target key frame, and extracting the sequence features of the target video frame sequence, extracting the short-term interest region features of the target video frame sequence from the sequence features according to the target position information; obtaining a short-term interest region feature set of K associated video frame sequences; fusing the short-term interest region feature set into a long-term interest region feature; determining the behavior category of the target object in the target key frame according to the long-term interest region feature and the short-term interest region feature of the target video frame sequence. The present application can improve the efficiency and accuracy of video recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] Behavior detection involves detecting the actions of people in a video. The final output includes a human body detection box within the video frame and one or more behavioral labels corresponding to each person. Behavior detection has a wide range of applications in fields such as video surveillance and motion analysis. For example, in video surveillance, it can detect targets and identify abnormal events, ensuring the safety of people and property within the venue. In sports competitions, it can provide accurate data analysis and support, enhancing fairness in the arena.

[0003] Currently, videos are mainly analyzed manually based on past experience and knowledge to determine the detection frames and behavior categories of people in the video. However, manual analysis is not only inefficient, but also subject to significant subjective influences, resulting in inaccurate analysis results. Summary of the Invention

[0004] The embodiments of the present application provide a video processing method, apparatus, computer device, and storage medium, which can improve video recognition efficiency and recognition accuracy.

[0005] An embodiment of the present application provides a video processing method, including:

[0006] Acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1;

[0007] Identifying target position information of the target object in the target key frame, extracting sequence features of the target video frame sequence, and extracting short-term interest region features of the target video frame sequence from the sequence features according to the target position information;

[0008] Obtaining short-term region-of-interest feature sets of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer;

[0009] fusing the short-term region of interest feature set into a long-term region of interest feature;

[0010] The behavior category of the target object in the target key frame is determined according to the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence.

[0011] An embodiment of the present application provides a video processing device, including:

[0012] A first acquisition module is configured to acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1;

[0013] an identification module, configured to identify target position information of the target object in the target key frame, extract sequence features of the target video frame sequence, and extract short-term interest region features of the target video frame sequence from the sequence features according to the target position information;

[0014] A second acquisition module is configured to acquire a short-term region of interest feature set of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer;

[0015] A fusion module, configured to fuse the short-term region of interest feature set into a long-term region of interest feature;

[0016] A determination module is used to determine the behavior category of the target object in the target key frame according to the long-term interest region feature and the short-term interest region feature of the target video frame sequence.

[0017] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the methods in the above embodiments.

[0018] On one hand, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the methods in the above embodiments are executed.

[0019] On the one hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer instructions are executed by a processor of a computer device, the methods in the above embodiments are executed.

[0020] In the present application, the terminal device automatically identifies the position information and behavior category of the target object in the video frame without human intervention, thus avoiding the interference of subjective factors caused by manual analysis, improving the efficiency and accuracy of video recognition, and enriching the method of video recognition; furthermore, the present application extracts the long-term interest region features and short-term interest region features of the video frame sequence, and the long-term interest region features and short-term interest region features respectively represent the behavior characteristics of the target object in a long time period and the behavior characteristics in a short time period, which can enrich the feature expression method of the behavior category and further improve the recognition accuracy of the behavior category. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is a system architecture diagram of a video processing provided by an embodiment of the present application;

[0023] Figure 2a-2c This is a schematic diagram of a video processing scenario provided by an example of this application;

[0024] Figure 3 This is a flow chart of a video processing method provided by an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a pooling process provided in an embodiment of the present application;

[0026] Figure 5a-5b This is a schematic diagram of a recognition result provided by an embodiment of the present application;

[0027] Figure 6 This is a flow chart of a video processing method provided by an embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of decoupling long-term and short-term information provided by an embodiment of the present application;

[0029] Figure 8 This is a schematic diagram of a framework for atomic behavior detection provided by an embodiment of the present application;

[0030] Figure 9 This is a schematic diagram of an identification behavior category provided by an embodiment of the present application;

[0031] Figure 10 is a structural diagram of a video processing device provided in an embodiment of the present application;

[0032] Figure 11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0035] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, text processing, natural language processing, and machine learning / deep learning.

[0036] The solution provided in this application belongs to the field of artificial intelligence, which includes computer vision technology and machine learning / deep learning. This application uses a trained artificial intelligence model to automatically identify the location information of the target object in the video frame and the behavior category of the target object in the video frame. Subsequently, based on the recognition results, the target object's behavior quality can be evaluated, and abnormal behavior of the target object can be detected in a timely manner.

[0037] This application can be applied to video surveillance scenarios: a monitor can collect video of a certain area (for example, a certain street, a certain shopping mall entrance), and the solution of this application can be used to identify the location information and behavior category of the people contained in the video. When the behavior category of the person is identified as an abnormal category (for example, the abnormal category can be fighting, theft, etc.), an alarm message can be issued to alert security personnel. The security personnel can quickly locate the specific area where the abnormal behavior occurred based on the identified location information, ensuring the personal and property safety of people in the video surveillance area.

[0038] This application can also be applied to sports competition scenarios: record a video of an athlete competing in an arena, and use the solution of this application to identify the athlete's behavior category in the video. The identified behavior category can be used to determine the athlete's competitive state, which can be used for subsequent personnel selection to improve fairness in the competitive field; or the identified behavior category can be used to find out whether the athlete has made any illegal moves during the competition to improve the accuracy of the referee's judgment.

[0039] This application can also be applied to outdoor teaching scenarios: record students' outdoor sports videos, and use this application solution to identify the students' behavior categories in the video. The identified behavior categories can be used to determine the students' participation enthusiasm, providing data support for subsequent teaching effect evaluation and adjustment of teaching methods.

[0040] See Figure 1 , Figure 1 This is a system architecture diagram for video processing provided by an embodiment of the present application. The server 10f establishes a connection with a user terminal cluster through a switch 10e and a communication bus 10d. The user terminal cluster may include: user terminal 10a, user terminal 10b, ..., user terminal 10c. Taking user terminal 10a as an example, user terminal 10a obtains the current video to be identified and sends the video to server 10f. After receiving the video, server 10f divides the video into multiple video frame sequences, each of which contains a key frame. For any video frame sequence (referred to as a target video frame sequence), server 10f identifies the position information of the character in the key frame in the target video frame sequence, and extracts the sequence features of the target video frame sequence, and extracts the short-term interest region features of the target video frame sequence corresponding to the position information from the sequence features. Server 10f determines multiple video frame sequences adjacent to the target video frame sequence in multiple video frame sequences, and the server determines the short-term interest region features of each video frame sequence in the same manner, and fuses the short-term interest region features of adjacent video frame sequences into long-term interest region features. The server 10f determines the behavior category of the character in the key frame in the target video frame sequence according to the short-term interest region features and the long-term interest region features of the target video frame sequence.

[0041] The server 10f can identify the position information and behavior categories of the characters in the key frames in each video frame sequence, and send the position information and behavior categories of all identified video frame sequences to the user terminal 10a. The user terminal 10a can perform downstream tasks (for example, video content understanding) based on the received data, or directly display the received data on the screen.

[0042] Figure 1 The user terminals 10a, 10b, ..., and 10c shown in the figure can be smart devices with video processing capabilities, such as mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), and wearable devices. The user terminal cluster and the server 10f can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0043] The following Figure 2a-2c For example, this article explains how the server identifies the location information and behavior categories of people in the video. Figure 2a-2c , which is a schematic diagram of a video processing scenario provided by the example of this application. Figure 2a As shown, the server obtains the current video 20a to be identified, selects a key frame at a certain frame interval (for example, 20 frames), and selects the video frames before and after the key frame (for example, 16 frames before and after, a total of 32 frames) to form a video frame sequence with the key frame as the center. The composed video frame sequence includes the key frame itself. In this way, multiple video frame sequences can be determined from the video 20a. The following is explained using one of the video frame sequences 20b as an example. Assuming that the key frame in the video frame sequence 20b is image 20c, the server calls the trained human position frame detection model to identify the position information of the person in image 20c. The human position frame detection model can identify the position of the person in the image. As Figure 2a As shown in image 20d in FIG, the position of the person in image 20c is marked with a dotted rectangular frame.

[0044] The server inputs video frame sequence 20b into a trained feature extraction model 20e. Since video frame sequence 20b is composed of multiple video frames, feature extraction model 20e can be a 3D feature extraction model that can directly process the video frame sequence. Feature extraction model 20e includes multiple convolutional layers. After inputting video frame sequence 20b into feature extraction model 20e, feature extraction model 20e can output sequence features. The sequence features are composed of multiple feature maps, and there is a certain proportional relationship between the feature map and image 20c (for example, the size of the feature map is 1 / 4 the size of image 20c).

[0045] Based on the proportional relationship and the position of the person in the image 20c, the server extracts the partial sequence features corresponding to the position from the sequence features output by the feature extraction model 20e, and calls the extracted partial sequence features the short-term interest region features 20f of the video frame sequence 20b. Simply put, the short-term interest region features 20f are the features of the person in the video frame sequence 20b (most of the background features have been removed).

[0046] like Figure 2b As shown above, video 20a is divided into multiple video frame sequences. The video frame sequences before and after video frame sequence 20b are selected from the multiple video frame sequences, i.e., the video frame sequences adjacent to video frame sequence 20b are determined. Assume that the video frame sequences adjacent to video frame sequence 20b are video frame sequence 20g, video frame sequence 20h, video frame sequence 20i, and video frame sequence 20j. It should be noted that the video frame sequences adjacent to video frame sequence 20b do not include video frame sequence 20b itself.

[0047] Front Figure 2a The invention describes in detail how to extract the short-time region of interest features 20f of the video frame sequence 20b. The server can determine the short-time region of interest features 20k of the video frame sequence 20g, the short-time region of interest features 20l of the video frame sequence 20h, the short-time region of interest features 20m of the video frame sequence 20i, and the short-time region of interest features 20n of the video frame sequence 20j in the same way.

[0048] The server fuses the above-mentioned short-term region of interest feature 20k, short-term region of interest feature 20l, short-term region of interest feature 20m and short-term region of interest feature 20n into a long-term region of interest feature 20p, wherein the fusion method can be to directly superimpose the above-mentioned four short-term region of interest features, or to use the attention mechanism to determine the degree of correlation between the short-term region of interest feature 20f and the four short-term region of interest features. The degree of correlation is the weight coefficient of the four short-term region of interest features. The four short-term region of interest features are weighted and summed based on the weight coefficient to obtain the long-term region of interest feature 20p of the video frame sequence 20b.

[0049] The short-term region of interest feature 20f and the long-term region of interest feature 20p are both features extracted based on the position of the human body, and the time period corresponding to the short-term region of interest feature 20f is the time period corresponding to the video frame sequence 20b, and the time period corresponding to the long-term region of interest feature 20p is the time period corresponding to the video frame sequence 20g - the time period corresponding to the video frame sequence 20j, so one feature is a short-term feature and the other feature is a long-term feature.

[0050] At this point, the server has extracted the short-term interest region features 20f and the long-term interest region features 20p of the video frame sequence 20b. Figure 2c As shown, by inputting the short-term interest region feature 20f into the fully connected layer 20q, the matching probability with multiple behavior types can be obtained; similarly, by inputting the long-term interest region feature 20p into the fully connected layer 20r, the matching probability with multiple behavior types can also be obtained. Of course, the multiple behavior types in the fully connected layer 20q are exactly the same as the multiple behavior types in the fully connected layer 20r.

[0051] By superimposing and normalizing the matching probabilities output by the two fully connected layers, we can obtain the final matching probabilities between the person in video frame sequence 20b and the various behavior types. The server can use the behavior type corresponding to the maximum matching probability as the behavior type of video frame sequence 20b. Suppose the behavior type of video frame sequence 20b is running.

[0052] The server can mark the position of the person in the key frame 20c of the video frame sequence 20b with a dotted rectangle and add the recognized behavior type "running" to obtain Figure 2c Image 20s shown.

[0053] Subsequently, the server may determine the position and behavior type of the person in each video frame sequence in the same manner, and output the recognition results.

[0054] The specific process of obtaining the target video frame sequence (such as the video frame sequence 20b in the above embodiment), identifying the target position information of the target object in the target key frame (such as the position of the person in the above embodiment), extracting the short-term interest region feature of the target video frame sequence (such as the short-term interest region feature 20f in the above embodiment), and extracting the long-term interest region feature (such as the long-term interest region feature 20p in the above embodiment) can be referred to below. Figure 3-Figure 9 Corresponding embodiments.

[0055] See Figure 3 , which is a flowchart of a video processing provided by an embodiment of the present application. Since the video processing of the present application involves the recognition of behavior categories by an artificial intelligence model, the following steps are described with a server having better performance as the execution subject. The video processing includes the following steps:

[0056] Step S101 : obtaining a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1.

[0057] Specifically, the server obtains the video to be identified (called the target video, as mentioned above Figure 2a-2c In the example of video 20a), the target video includes multiple video frames. A key frame is selected at a predetermined interval (e.g., 20 frames). N key frames can be determined from the target video, where N is a positive integer. For example, with respect to a key frame, the video frames preceding and following it (e.g., 16 frames before and after it, for a total of 32 frames) are selected around it to form a video frame sequence, which includes the key frame.

[0058] The server may determine the video frame sequence corresponding to each key frame in the same manner, that is, may obtain N video frame sequences.

[0059] Select any video frame sequence from these N video frame sequences (called the target video frame sequence, as mentioned above Figure 2a-2c The video frame sequence 20b) in the corresponding embodiment, and the key frame in the target video frame sequence is called the target key frame, the target key frame contains the target object, the target object can be a person, a certain animal (for example, a cat, a dog, etc.), or a game character, etc.

[0060] The following takes the target video frame sequence as an example for detailed description.

[0061] Step S102 , identifying the target position information of the target object in the target key frame, extracting sequence features of the target video frame sequence, and extracting short-term region of interest features of the target video frame sequence from the sequence features according to the target position information.

[0062] Specifically, the server inputs the target keyframe into a pre-trained object location box recognition model. The object location box recognition model is used to identify the location information of the target object in the image. The location information can be the coordinates of the four vertices of a rectangular box in the image. Of course, this rectangular box contains the target object. After the server inputs the target keyframe into the pre-trained object location box recognition model, the object location box recognition model outputs the location information of the target object in the target keyframe (referred to as the target location information).

[0063] The server inputs the target video frame sequence into the trained 3D feature extraction model (as mentioned above). Figure 2a-2cIn the corresponding embodiment, the feature extraction model 20e) can be specifically a 3D CNN (Convolutional Neural Network). The 3D feature extraction model can extract features of sequence data. After the target video frame sequence is input into the 3D feature extraction model, the sequence features of the target video frame sequence can be extracted. The sequence features of the target video frame sequence include multiple feature maps of the same size, and the size of each feature map satisfies a preset proportional relationship with the size of the target key frame. For example, the length of each feature map = 1 / 4 of the length of the target key frame, and the width of each feature map = 1 / 4 of the width of the target key frame.

[0064] In other words, a pixel in the feature map can correspond to an image area in the target keyframe.

[0065] The server scales the target position information according to a preset proportional relationship to obtain the scaled target position information (called adjusted target position information). The server extracts the unit feature map contained in the rectangular box corresponding to the adjusted target position information from each feature map. Of course, the number of unit feature maps is consistent with the number of feature maps. Since the target position information corresponds to the position coordinates of the four vertices of a rectangular box, after scaling, the adjusted target position information also corresponds to the position coordinates of the four vertices of a rectangular box.

[0066] Each unit feature map is aligned (or pooled) to obtain multiple aligned unit feature maps. Each aligned unit feature map contains pixels (the same as the preset number or the size of the aligned unit feature map is the same as the preset size). For example, the unit feature map contains 4×4 pixels and the aligned unit feature map contains 2×2 pixels.

[0067] The server combines multiple aligned unit feature maps into short-term interest region features of the target video frame sequence (as mentioned above Figure 2a-2c The server aligns the unit feature maps because, in a target keyframe, multiple target position information of multiple target objects can be identified, and the size of the rectangular box corresponding to each target position information is different. Consequently, the size of the unit feature map corresponding to each target position information is also different. To make the feature dimensions consistent, the size of the unit feature map needs to be adjusted so that all adjusted unit feature maps have the same size.

[0068] See Figure 4 , Figure 4 This is a schematic diagram of a pooling process provided by an embodiment of the present application. After the target object is identified in the key video frame (the region corresponds to Figure 4The unit feature map corresponding to the region is determined from the feature map contained in the sequence feature. Figure 4 It can be seen that the pixel values ​​in the current unit feature map include 1, 3, 2, 4, 7, 3, 2, 0, and 6. The unit feature map needs to be aligned or pooled. Assume that the size of the feature map after alignment or pooling is 2×2. First, for the pixel values ​​1, 3, 4, and 7 of the four pixels in the upper left corner, interpolation can be used to determine a new pixel value, or average pooling (or maximum pooling) can be used to determine a new pixel value. Assume that average pooling is used here to determine the new pixel value as (1+3+4+7) / 4=4; for the pixel values ​​3, 2, 7, and 3 of the four pixels in the upper right corner, average pooling is used to determine the new pixel value as (3+2+7+3) / 4=4; for the pixel values ​​4, 7, 2, and 0 of the four pixels in the lower left corner, average pooling is used to determine the new pixel value as (4+7+2+0) / 4=3; for the pixel values ​​7, 3, 0, and 6 of the four pixels in the lower right corner, average pooling is used to determine the new pixel value as (7+3+0+6) / 4=4. In summary, the pixel values ​​of the aligned unit feature map are 4, 4, 3, and 4, that is, 4, 4, 3, and 4 can be combined into a short-term region of interest feature.

[0069] Step S103 : obtaining short-term region of interest feature sets of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer.

[0070] Specifically, the server determines K video frame sequences adjacent to the target video frame sequence from multiple video frame sequences (all referred to as associated video frame sequences, as described above). Figure 2a-2c In the corresponding embodiments (video frame sequence 20g, video frame sequence 20h, video frame sequence 20i, and video frame sequence 20j), K is a positive integer, and K is less than or equal to N.

[0071] It should be noted that the K associated video frame sequences do not include the target video frame sequence.

[0072] The server obtains the short-term region of interest features of each associated video frame sequence (all referred to as the short-term region of interest features to be fused), and combines the P short-term region of interest features to be fused of the K associated video frame sequences into a short-term region of interest feature set. The above step S102 describes in detail how to extract the short-term region of interest features of a video frame sequence (i.e., the target video frame sequence). The server can use the same method to extract the short-term region of interest features to be fused for each associated video frame sequence. The specific process can be as follows:

[0073] As can be seen from the above, each associated video frame sequence contains a key frame (called an associated key frame), and each associated key frame contains a target object. The server calls the object position box recognition model to respectively identify the position information of the target object in each associated key frame (called associated position information). The 3D feature extraction model is called to extract the sequence features of each associated video frame sequence (called associated sequence features). Similarly, the server scales the associated position information of each associated key frame according to a preset ratio, and extracts the unit feature map corresponding to the scaled associated position information from each associated sequence feature. After alignment processing, the short-term interest area features to be fused for each associated video frame sequence can be obtained.

[0074] It is worth noting that the number P of short-term interest region features to be fused contained in the short-term interest region feature set is not less than the number K of associated video frame sequences. This is because at least one associated position information can be identified in an associated key frame. Of course, if an associated key frame contains multiple target objects, multiple associated position information can be identified in this associated key frame, and then multiple short-term interest region features to be fused can be determined based on an associated video frame sequence.

[0075] It can be known that the short-time region of interest features to be fused in the short-time region of interest feature set of the associated video frame sequence are the short-time region of interest features of the video frame sequence adjacent to the target video frame sequence, and the short-time region of interest feature set of the associated video frame sequence does not include the short-time region of interest features of the target video frame sequence.

[0076] Step S104: fusing the short-term region of interest feature set into a long-term region of interest feature.

[0077] Specifically, the server can directly superimpose the short-term interest region features to be fused in the short-term interest region feature set to obtain the long-term interest region features of the target video frame sequence (as described above). Figure 2a-2c This corresponds to the long-term region of interest feature 20p in the embodiment.

[0078] The server may also determine the degree of correlation between the short-term region of interest feature of the target video frame sequence and each short-term region of interest feature to be fused based on the attention mechanism. The degree of correlation between each short-term region of interest feature to be fused may be determined by calculating the vector inner product between the short-term region of interest of the target video frame sequence and each short-term region of interest feature to be fused. The degree of correlation may represent the influence of each short-term region of interest feature to be fused on the short-term region of interest feature of the target video frame sequence.

[0079] The server uses the correlation degree of each short-term ROI feature to be fused as a weighting coefficient, performs a weighted sum operation on the P short-term ROI features to be fused, and obtains the long-term ROI feature of the target video frame sequence.

[0080] Step S105 : determining the behavior category of the target object in the target key frame according to the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence.

[0081] At this point, the server has determined the long-term region of interest features and short-term region of interest features of the target video frame sequence. The server can determine the behavior category based on the long-term region of interest features and short-term region of interest features of the target video frame sequence, and then determine the final behavior category based on the two determined behavior categories.

[0082] The server may combine the target position information of the identified target object in the target key frame and the behavior category of the target object in the target key frame into a recognition result of the target video frame sequence, and output the recognition result of the target video frame sequence.

[0083] Subsequently, the server may store the short-term region of interest features and the short-term region of interest feature set of the target video frame sequence for use in subsequently determining recognition results of other video frame sequences of the target video.

[0084] Optionally, as can be seen from the above, the target location information corresponds to a rectangular box, and the rectangular box contains the target object. The server can mark the rectangular box in the target key frame, that is, mark the area of ​​the target object in the target key frame. The server can also add the final behavior category identified in the target key frame, where the behavior category can be added in the target key frame in the form of text. The server outputs the area where the marked target object is located and the target key frame with the added behavior category (as mentioned above Figure 2a-2c Corresponding to image 20s in the embodiment).

[0085] See Figure 5a-5b , which is a schematic diagram of a recognition result provided by an embodiment of the present application, such as Figure 5a As shown, the location of the target object is marked with a dotted rectangle in the key frame, and the recognized behavior category is marked. Figure 5a The behavior categories identified in the are: standing, singing. Figure 5b As shown, similarly, in another key frame, the location of the target object is marked with a dotted rectangle. Figure 5b There are two target objects in the key frame, so there will be two rectangular boxes marking the locations of the two target objects. Figure 5b The behavior categories of the two target objects are identified as: standing and playing ball. Therefore, we need to Figure 5b The behavior categories of the two target objects are marked respectively.

[0086] From the above, it can be seen that the present application decouples the prediction of short-term information and long-term information, and independently extracts features and calculates them. This approach ensures that short-term features and long-term features can be completely decoupled and independently calculated, which can enrich the recognition methods of behavior categories and improve the recognition accuracy of behavior categories.

[0087] See Figure 6 , Figure 6 This is a flow chart of a video processing method provided by an embodiment of the present application. The video processing method includes the following steps:

[0088] Step S201 : obtaining a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1.

[0089] Step S202 , identifying the target position information of the target object in the target key frame, extracting sequence features of the target video frame sequence, and extracting short-term region of interest features of the target video frame sequence from the sequence features according to the target position information.

[0090] Step S203, obtaining a short-term region of interest feature set of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, K is a positive integer, and the short-term region of interest feature set is fused into a long-term region of interest feature.

[0091] The specific process of step S201 to step S203 can be found in the above Figure 3 This corresponds to steps S101 to S104 in the embodiment.

[0092] Step S204 : performing recognition processing on the short-term region of interest features of the target video frame sequence to obtain prediction information of the first behavior of the target object in the target key frame.

[0093] Specifically, the server inputs the short-term interest region features of the target video frame sequence into the first feature recognition model (as described above) Figure 2a-2c Corresponding to the fully connected layer 20q in the embodiment, the first behavior prediction information is obtained. The first feature recognition model can be composed of a fully connected layer and a softmax function. The first behavior prediction information may include the matching probabilities (referred to as first matching probabilities) of the M behavior categories included in the first feature recognition model, and the sum of all first matching probabilities is equal to 1.

[0094] Step S205 : performing recognition processing on the long-term region of interest features to obtain second behavior prediction information of the target object in the target key frame.

[0095] Specifically, the server inputs the long-term interest region feature into the second feature recognition model (as described above Figure 2a-2c The fully connected layer 20r) in the corresponding embodiment obtains the second behavior prediction information, and the second feature recognition model can also be composed of a combination of a fully connected layer + a softmax function.

[0096] It should be noted that the first feature recognition model and the second feature recognition model can both be composed of a combination of a fully connected layer + a softmax function, but the coefficients of the fully connected layers are different.

[0097] The second behavior prediction information may include the matching probabilities of the M behavior categories contained in the second feature recognition model (referred to as second matching probabilities), and the sum of all second matching probabilities is equal to 1, and the M behavior categories of the first feature recognition model and the M behavior categories of the second feature recognition model are the same.

[0098] Step S206 : determining the behavior category of the target object in the target key frame according to the first behavior prediction information and the second behavior prediction information.

[0099] Specifically, the server adds the M first matching probabilities and the M second matching probabilities to obtain M target matching probabilities, and uses the behavior category corresponding to the maximum target matching probability among the M target matching probabilities as the behavior category of the target object in the target key frame.

[0100] Optionally, the server can also add the M first matching probabilities and the M second matching probabilities in a positional manner and perform normalization processing to obtain M probability values. The behavior category corresponding to the probability value greater than the preset threshold among the M probability values ​​is used as the behavior category of the target object in the target key frame.

[0101] For example, the first feature extraction model outputs M first matching probabilities: the probability that the target object belongs to behavior category A is 0.3, the probability that the target object belongs to behavior category B is 0.5, and the probability that the target object belongs to behavior category C is 0.2. The second feature extraction model outputs M second matching probabilities: the probability that the target object belongs to behavior category A is 0.1, the probability that the target object belongs to behavior category B is 0.8, and the probability that the target object belongs to behavior category C is 0.1. Therefore, the result of superimposing the M first matching probabilities and the M second matching probabilities is: the probability that the target object belongs to behavior category A is 0.4, the probability that the target object belongs to behavior category B is 1.3, and the probability that the target object belongs to behavior category C is 0.3. Therefore, it can be determined that behavior category B corresponding to the maximum probability 1.3 is the behavior category of the target object in the target key frame.

[0102] Optionally, as can be seen from the foregoing, in order to determine the behavior category of the target object in the target key frame, a 3D feature extraction model, a first feature recognition model, and a second feature recognition model are involved, and the 3D feature extraction model is used to extract sequence features of the video frame sequence, the first feature extraction model is based on the recognition of short-term interest region features, and the second feature extraction model is based on the recognition of long-term interest region features. The training of the above two models is specifically described below:

[0103] A sample video for model training is obtained, and the video frames contained in the sample video are divided into multiple video frame sequences (all of which can be referred to as sample video frame sequences). Each sample video frame sequence also contains key frames (all of which can be referred to as sample key frames). Similarly, the position information of the target object in each sample key frame is first identified, and then the short-term region of interest features and the long-term region of interest features of each sample video frame sequence are respectively extracted in the aforementioned manner (the extraction of short-term region of interest features and long-term region of interest features involves a 3D feature extraction model, a first feature recognition model, and a second feature recognition model). The behavior category of the target object in each sample video frame sequence is predicted based on the short-term region of interest features and the long-term region of interest features of each sample video frame sequence, and a prediction error is determined based on the predicted behavior category and the actual behavior category. Based on the prediction error, the model parameters of the 3D feature extraction model, the model parameters of the first feature recognition model, and the model parameters of the second feature recognition model are reversely adjusted.

[0104] The model parameters of the 3D feature extraction model, the first feature recognition model and the second feature recognition model can be adjusted multiple times based on multiple sample videos. When the three adjusted models converge, it means that the training of the current 3D feature extraction model, the first feature recognition model and the second feature recognition model is completed.

[0105] The modeling process of the 3D feature extraction model, the first feature recognition model, and the second feature recognition model is shown in the following formula (1):

[0106]

[0107] Among them, the video frame sequence is extracted by the 3D feature extraction model, and the ROI (Region Of Interest) operation is performed on the feature map through the human body detection frame to obtain the features corresponding to each human body, which is expressed as V. The feature V here can correspond to the short-term region of interest feature of the video frame sequence in this application. The variables required for classification can be split into long-term information variables Z l and short-term information variable Z s Among them, the long-term information variable Z l It can correspond to the second behavior prediction information in this application, the short-term information variable Z s It can correspond to the first line of prediction information in this application. Assume that C is the category to be predicted. Since C is not directly related to the input image feature V, C, Z l and Z s The joint distribution modeling of the three is expressed as the above formula (1).

[0108] In formula (1), the identification behavior category is decomposed into three problems: P(Z l |V) and P(Z s |V) respectively models the long-term and short-term features of the input video features, while P(C|Z l ,Z s ) is the discriminant function that obtains the final behavior category based on long-term information and short-term information. The whole process can be expressed as Figure 7 The content shown, Figure 7 This is a schematic diagram of decoupling long-term and short-term information provided by the embodiment of the present application. Figure 7 The joint modeling process shown decouples behavior category recognition into long-term context dependency and short-term context dependency. The variable spaces of the two are independent of each other, and the behavior categories can be identified and predicted separately.

[0109] See Figure 8 , Figure 8It is a schematic diagram of a framework for atomic behavior detection provided by an embodiment of the present application, wherein atomic detection refers to detecting the behavior of a person in a video, and the final output result includes a detection frame of the human body and a label of one or more behaviors corresponding to each human body. The specific process of atomic behavior detection is as follows: first, the video is split into video frames, key frames are selected at a certain frame interval (for example, 20 frames), and the video frames before and after the key frame are selected as the center (for example, 16 frames before and after, a total of 32 frames) to form a video frame sequence as the input of the detection framework. For key frames, a pre-trained human detector is used (the human detector can correspond to the object position frame recognition model in this application) to obtain the human detection frame. The video frame sequence is then input into a 3D CNN for feature extraction to obtain the feature. Figure X , and then use the previously obtained human detection frame for the feature Figure X Perform ROI operation (that is, obtain the features within the human body detection frame), and the obtained features are processed by the short-term context module and the long-term context module respectively to obtain the output results Zl and Zs of the two. Zl and Zs are normalized to obtain the final category output classification result C.

[0110] Among them, the long-term context module will extract and fuse the features of the video frame sequences before and after the current video frame sequence. This requires first extracting the features of the video frame sequences corresponding to all key frames of the entire video in advance, and storing the features in all detection frames in the feature storage module based on the detection frame of each person as the basic unit.

[0111] Through the above framework design, the prediction of short-term information and long-term information is decoupled, and features are extracted and calculated independently. The feature source of short-term information is the feature after the ROI operation is performed on the features of the current image block, while the feature source of long-term information is the feature within the fixed time window before and after the video frame sequence is the center. Note that the features of the time window used here will exclude the features of the current video frame sequence itself. This approach ensures that short-term features and long-term features can be completely decoupled and calculated independently.

[0112] See Figure 9 , Figure 9 This is a schematic diagram of an identification behavior category provided by an embodiment of the present application, such as Figure 9As shown, multiple video frames before and after the current video frame can be combined into a video frame sequence, based on which short-term context clues can be inferred, and behaviors such as "sitting" and "writing" can be obtained. Based on multiple video frame sequences before and after the video frame sequence, long-term context clues can be inferred, and a category such as "talking to people" can be obtained. The above "sitting", "writing" and "talking to people" are combined into a behavior category identified based on a video frame sequence. It can be seen that the present application designs an atomic behavior detection algorithm based on the decoupling of long-term information and short-term information, decoupling the classification of behavior into two problems: long-term prediction and short-term prediction. The two problems are predicted separately, and finally the results are fused at the probability level obtained by the two, which effectively improves the detection accuracy of the algorithm.

[0113] From the above, it can be seen that the use of this application can finely model long-term information and short-term information, decouple long-term information and short-term information, and perform category prediction on the two separately, ultimately achieving higher behavior detection accuracy, and providing efficient and reliable technical support for applications such as video content understanding and analysis.

[0114] For further information, see Figure 10 , which is a structural diagram of a video processing device provided by an embodiment of the present application. Figure 10 As shown, the video processing device 1 can be applied to the above Figure 3-Figure 9 Specifically, the video processing device 1 may be a computer program (including program code) running in a computer device, for example, the video processing device 1 is an application software; the video processing device 1 may be used to execute the corresponding steps of the method provided in the embodiment of the present application.

[0115] The video processing device 1 may include: a first acquisition module 11 , a recognition module 12 , a second acquisition module 13 , a fusion module 14 and a determination module 15 .

[0116] A first acquisition module 11 is configured to acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1;

[0117] an identification module 12, configured to identify target position information of the target object in the target key frame, extract sequence features of the target video frame sequence, and extract short-term region of interest features of the target video frame sequence from the sequence features according to the target position information;

[0118] A second acquisition module 13 is configured to acquire a short-term region of interest feature set of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer;

[0119] A fusion module 14 is configured to fuse the short-term region of interest feature set into a long-term region of interest feature set;

[0120] The determination module 15 is configured to determine the behavior category of the target object in the target key frame according to the long-term region of interest features and the short-term region of interest features of the target video frame sequence.

[0121] In a possible implementation, the video processing device 1 may further include: a combining module 16 .

[0122] The combining module 16 is configured to combine the target position information of the target object in the target key frame and the behavior category of the target object in the target key frame into a recognition result of the target video frame sequence, and output the recognition result of the target video frame sequence.

[0123] In a possible implementation, when the determination module 15 is used to determine the behavior category of the target object in the target key frame based on the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence, it is specifically used to:

[0124] Identifying and processing the short-term region of interest features of the target video frame sequence to obtain prediction information of the first behavior of the target object in the target key frame;

[0125] performing recognition processing on the long-term region of interest feature to obtain second behavior prediction information of the target object in the target key frame;

[0126] A behavior category of the target object in the target key frame is determined according to the first behavior prediction information and the second behavior prediction information.

[0127] In one possible implementation, the first behavior prediction information includes first matching probabilities of M behavior categories, and the second behavior prediction information includes second matching probabilities of the M behavior categories, where M is a positive integer;

[0128] When the determining module 15 is used to determine the behavior category of the target object in the target key frame according to the first behavior prediction information and the second behavior prediction information, it is specifically used to:

[0129] Superimpose the M first matching probabilities and the M second matching probabilities to obtain M target matching probabilities;

[0130] The behavior category of the target object in the target key frame is determined according to the M target matching probabilities and the behavior category corresponding to each target matching probability.

[0131] In a possible implementation, when the recognition module 12 is used to extract sequence features of the target video frame sequence, it is specifically used to:

[0132] Calling a 3D feature extraction model to extract sequence features of the target video frame sequence;

[0133] The sequence features of the target video frame sequence include multiple feature maps of the same size, and the size of each feature map satisfies a preset proportional relationship with the size of the target key frame.

[0134] In a possible implementation, when the recognition module 12 is used to extract the short-term region of interest features of the target video frame sequence from the sequence features according to the target position information, it is specifically used to:

[0135] Scaling the target position information according to the preset proportional relationship to obtain adjusted target position information, and extracting a plurality of unit feature maps corresponding to the adjusted target position information from the plurality of feature maps;

[0136] Each unit feature map is aligned to obtain multiple aligned unit feature maps, and the multiple aligned unit feature maps are combined into the short-term interest region feature of the target video frame sequence.

[0137] In a possible implementation, each associated video frame sequence includes an associated key frame, and each associated key frame includes the target object;

[0138] When the second acquisition module 13 is used to acquire the short-term interest region feature set of K associated video frame sequences, it is specifically used to:

[0139] Identifying the associated position information of the target object in each associated key frame, and calling the 3D feature extraction model to extract associated sequence features of each associated video frame sequence;

[0140] According to the associated position information of each associated key frame, the short-term interest region feature to be fused of each associated video frame sequence is extracted from each associated sequence feature;

[0141] The short-term region of interest features to be fused of K associated video frame sequences are combined into the short-term region of interest feature set.

[0142] In a possible implementation, the short-term region of interest feature set includes P short-term region of interest features to be fused, where P is not less than K;

[0143] When the fusion module 14 is used to fuse the short-term region of interest feature set into the long-term region of interest feature, it is specifically used to:

[0144] Determine the correlation between the short-term region of interest feature of the target video frame sequence and each short-term region of interest feature to be fused based on the attention mechanism;

[0145] According to the correlation degree corresponding to each short-term region of interest feature to be fused, a weighted sum operation is performed on the P short-term region of interest features to be fused to obtain the long-term region of interest feature.

[0146] In a possible implementation, when the first acquisition module 11 is used to acquire the target video frame sequence, it is specifically configured to:

[0147] Acquire the target video, where the target video includes multiple video frames;

[0148] Selecting N key frames from the plurality of video frames, where N is a positive integer;

[0149] Combining each key frame and video frames adjacent to each key frame in a plurality of video frames included in the target video into video frame sequences, wherein each video frame sequence includes the same number of video frames;

[0150] A target video frame sequence is selected from the N video frame sequences, and a key frame in the target video frame sequence is used as a target key frame.

[0151] In a possible implementation, the video processing device 1 further includes: an adding module 17 .

[0152] The adding module 17 is used to mark the area where the target object is located in the target key frame according to the target position information, add the behavior category in the target key frame, and output the target key frame with the marked area where the target object is located and the added behavior category.

[0153] According to one embodiment of the present application, Figure 3-Figure 9 Each step involved in the method shown can be performed by Figure 10 The video processing device shown in FIG. 1 is executed by each module. For example, Figure 3 The steps S101-S105 shown in FIG can be respectively Figure 10 , the first acquisition module 11, the recognition module 12, the second acquisition module 13, the fusion module 14, the determination module 15, the combination module 16 and the addition module 17 shown in FIG. Figure 6 Steps S204 to S206 shown in FIG can be replaced by Figure 10 The determination module 15 shown in FIG.

[0154] The present application automatically identifies the location information and behavior category of the target object in the video frame by the terminal device, without the need for human intervention, avoiding the interference of subjective factors caused by manual analysis, improving the efficiency and accuracy of video recognition, and enriching the method of video recognition; furthermore, the present application extracts the long-term interest region features and short-term interest region features of the video frame sequence, and the long-term interest region features and short-term interest region features respectively represent the behavior characteristics of the target object in a long time period and the behavior characteristics in a short time period, which can enrich the feature expression method of the behavior category and further improve the accuracy of identifying the behavior category.

[0155] Further, see Figure 11 , is a structural diagram of a computer device provided in an embodiment of the present application. Figure 3-Figure 9 The server in the corresponding embodiment may be the computer device 1000. Figure 11 As shown, the computer device 1000 may include: a user interface 1002, a processor 1004, an encoder 1006 and a memory 1008. A signal receiver 1016 is used to receive or send data via a cellular interface 1010, a WIFI interface 1012, ..., or an NFC interface 1014. The encoder 1006 encodes the received data into a data format for computer processing. A computer program is stored in the memory 1008, and the processor 1004 is configured to perform the steps of any of the above method embodiments through the computer program. The memory 1008 may include a volatile memory (e.g., a dynamic random access memory DRAM) and may also include a non-volatile memory (e.g., a one-time programmable read-only memory OTPROM). In some instances, the memory 1008 may further include a memory remotely located relative to the processor 1004, and these remote memories may be connected to the computer device 1000 via a network. The user interface 1002 may include: a keyboard 1018 and a display 1020.

[0156] exist Figure 11 In the computer device 1000 shown, the processor 1004 can be used to call the computer program stored in the memory 1008 to implement:

[0157] Acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any one of N video frame sequences included in the target video, where N is a positive integer greater than 1;

[0158] Identifying target position information of the target object in the target key frame, extracting sequence features of the target video frame sequence, and extracting short-term interest region features of the target video frame sequence from the sequence features according to the target position information;

[0159] Obtaining short-term region-of-interest feature sets of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer;

[0160] fusing the short-term region of interest feature set into a long-term region of interest feature;

[0161] The behavior category of the target object in the target key frame is determined according to the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence.

[0162] In one embodiment, the processor 1004 further performs the following steps:

[0163] combining target position information of the target object in the target key frame and the behavior category of the target object in the target key frame into a recognition result of the target video frame sequence;

[0164] Output the recognition result of the target video frame sequence.

[0165] In one embodiment, when determining the behavior category of the target object in the target key frame based on the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence, the processor 1004 specifically performs the following steps:

[0166] Identifying and processing the short-term region of interest features of the target video frame sequence to obtain prediction information of the first behavior of the target object in the target key frame;

[0167] performing recognition processing on the long-term region of interest feature to obtain second behavior prediction information of the target object in the target key frame;

[0168] A behavior category of the target object in the target key frame is determined according to the first behavior prediction information and the second behavior prediction information.

[0169] In one embodiment, the first behavior prediction information includes first matching probabilities of M behavior categories, and the second behavior prediction information includes second matching probabilities of the M behavior categories, where M is a positive integer;

[0170] When the processor 1004 determines the behavior category of the target object in the target key frame according to the first behavior prediction information and the second behavior prediction information, the processor 1004 specifically performs the following steps:

[0171] Superimpose the M first matching probabilities and the M second matching probabilities to obtain M target matching probabilities;

[0172] The behavior category of the target object in the target key frame is determined according to the M target matching probabilities and the behavior category corresponding to each target matching probability.

[0173] In one embodiment, when extracting sequence features of the target video frame sequence, the processor 1004 specifically performs the following steps:

[0174] Calling a 3D feature extraction model to extract sequence features of the target video frame sequence;

[0175] The sequence features of the target video frame sequence include multiple feature maps of the same size, and the size of each feature map satisfies a preset proportional relationship with the size of the target key frame.

[0176] In one embodiment, when extracting the short-term region of interest features of the target video frame sequence from the sequence features according to the target position information, the processor 1004 specifically performs the following steps:

[0177] Scaling the target position information according to the preset proportional relationship to obtain adjusted target position information, and extracting a plurality of unit feature maps corresponding to the adjusted target position information from the plurality of feature maps;

[0178] Each unit feature map is aligned to obtain multiple aligned unit feature maps, and the multiple aligned unit feature maps are combined into the short-term interest region feature of the target video frame sequence.

[0179] In one embodiment, each associated video frame sequence includes an associated key frame, and each associated key frame includes the target object;

[0180] When the processor 1004 obtains the short-term interest region feature set of K associated video frame sequences, it specifically performs the following steps:

[0181] Identifying the associated position information of the target object in each associated key frame, and calling the 3D feature extraction model to extract associated sequence features of each associated video frame sequence;

[0182] According to the associated position information of each associated key frame, the short-term interest region feature to be fused of each associated video frame sequence is extracted from each associated sequence feature;

[0183] The short-term region of interest features to be fused of K associated video frame sequences are combined into the short-term region of interest feature set.

[0184] In one embodiment, the short-term region of interest feature set includes P short-term region of interest features to be fused, where P is not less than K;

[0185] When the processor 1004 executes the fusion of the short-term region of interest feature set into the long-term region of interest feature, the processor 1004 specifically performs the following steps:

[0186] Determine the correlation between the short-term region of interest feature of the target video frame sequence and each short-term region of interest feature to be fused based on the attention mechanism;

[0187] According to the correlation degree corresponding to each short-term region of interest feature to be fused, a weighted sum operation is performed on the P short-term region of interest features to be fused to obtain the long-term region of interest feature.

[0188] In one embodiment, when executing the acquisition of the target video frame sequence, the processor 1004 specifically performs the following steps:

[0189] Acquire the target video, where the target video includes multiple video frames;

[0190] Selecting N key frames from the plurality of video frames, where N is a positive integer;

[0191] Combining each key frame and video frames adjacent to each key frame in a plurality of video frames included in the target video into video frame sequences, wherein each video frame sequence includes the same number of video frames;

[0192] A target video frame sequence is selected from the N video frame sequences, and a key frame in the target video frame sequence is used as a target key frame.

[0193] In one embodiment, the processor 1004 further performs the following steps:

[0194] Marking the area where the target object is located in the target key frame according to the target position information, and adding the behavior category in the target key frame;

[0195] Outputs the target keyframes with the target object's region marked and the behavior category added.

[0196] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3-Figure 9 The description of the video processing method in the corresponding embodiment can also be performed as described above. Figure 10 The description of the video processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here either.

[0197] In addition, it should be noted that: the embodiment of the present application also provides a computer storage medium, and the computer storage medium stores the computer program executed by the video processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3-Figure 9 Therefore, the description of the video processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device for execution, or executed on multiple computer devices located at one location, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can be combined into a blockchain network.

[0198] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the above-mentioned Figures 3 to 9 The method in the corresponding embodiment will therefore not be described in detail here.

[0199] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0200] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video processing method, characterized in that: include: Acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any video frame sequence among N video frame sequences included in the target video, where N is a positive integer greater than 1; Identify the target position information of the target object in the target key frame, extract the sequence features of the target video frame sequence, and extract the short-term interest region features of the target video frame sequence from the sequence features according to the target position information; Acquire a short-term interest region feature set of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer; Merging the short-term region of interest feature set into a long-term region of interest feature; The behavior category of the target object in the target key frame is determined according to the long-term interest region feature and the short-term interest region feature of the target video frame sequence.

2. The method according to claim 1, characterized in that Also includes: combining the target position information of the target object in the target key frame and the behavior category of the target object in the target key frame into a recognition result of the target video frame sequence; Output the recognition result of the target video frame sequence.

3. The method according to claim 1, characterized in that The step of determining the behavior category of the target object in the target key frame according to the long-term region of interest feature and the short-term region of interest feature of the target video frame sequence comprises: Identify and process the short-term interest region features of the target video frame sequence to obtain prediction information of the first behavior of the target object in the target key frame; Performing recognition processing on the long-term interest region feature to obtain second behavior prediction information of the target object in the target key frame; The behavior category of the target object in the target key frame is determined according to the first behavior prediction information and the second behavior prediction information.

4. The method according to claim 3, characterized in that The first behavior prediction information includes first matching probabilities of M behavior categories, and the second behavior prediction information includes second matching probabilities of the M behavior categories, where M is a positive integer; The determining the behavior category of the target object in the target key frame according to the first behavior prediction information and the second behavior prediction information includes: Superimposing the M first matching probabilities and the M second matching probabilities into M target matching probabilities; According to the M target matching probabilities and the behavior category corresponding to each target matching probability, the behavior category of the target object in the target key frame is determined.

5. The method according to claim 1, characterized in that The extracting the sequence features of the target video frame sequence comprises: Calling a 3D feature extraction model to extract sequence features of the target video frame sequence; The sequence features of the target video frame sequence include a plurality of feature maps of the same size, and the size of each feature map satisfies a preset proportional relationship with the size of the target key frame.

6. The method according to claim 5, characterized in that The step of extracting the short-term region of interest feature of the target video frame sequence from the sequence feature according to the target position information comprises: According to the preset proportional relationship, the target position information is scaled to obtain adjusted target position information, and a plurality of unit feature maps corresponding to the adjusted target position information are extracted from the plurality of feature maps; Each unit feature map is aligned separately to obtain multiple aligned unit feature maps, and the multiple aligned unit feature maps are combined into the short-term interest region feature of the target video frame sequence.

7. The method according to claim 5, characterized in that Each associated video frame sequence includes an associated key frame, and each associated key frame includes the target object; The step of obtaining a short-term region of interest feature set of K associated video frame sequences includes: Identify the associated position information of the target object in each associated key frame, and call the 3D feature extraction model to extract the associated sequence features of each associated video frame sequence; According to the associated position information of each associated key frame, the short-term interest region feature to be fused of each associated video frame sequence is extracted from each associated sequence feature; The short-term region of interest features to be fused of K associated video frame sequences are combined into the short-term region of interest feature set.

8. The method according to claim 1, characterized in that The short-term interest region feature set includes P short-term interest region features to be fused, where P is not less than K; The step of fusing the short-term region of interest feature set into a long-term region of interest feature set includes: Determine the correlation between the short-term region of interest feature of the target video frame sequence and each short-term region of interest feature to be fused based on the attention mechanism; According to the correlation degree corresponding to each short-term region of interest feature to be fused, a weighted sum operation is performed on the P short-term region of interest features to be fused to obtain the long-term region of interest feature.

9. The method according to claim 1, characterized in that: The step of obtaining a target video frame sequence comprises: Acquire the target video, where the target video includes multiple video frames; Selecting N key frames from the plurality of video frames, where N is a positive integer; Combining each key frame and video frames adjacent to each key frame in a plurality of video frames included in the target video into video frame sequences, wherein the number of video frames included in each video frame sequence is the same; A target video frame sequence is selected from the N video frame sequences, and a key frame in the target video frame sequence is used as a target key frame.

10. The method according to any one of claims 1 to 9, characterized in that: Also includes: According to the target position information, marking the area where the target object is located in the target key frame, and adding the behavior category in the target key frame; Outputs a target keyframe that marks the area where the target object is located and the added behavior category.

11. A video processing device, characterized in that: include: A first acquisition module is used to acquire a target video frame sequence, wherein the target video frame sequence includes a target key frame, the target key frame includes a target object, and the target video frame sequence is any video frame sequence among N video frame sequences included in the target video, where N is a positive integer greater than 1; An identification module, used to identify the target position information of the target object in the target key frame, extract the sequence features of the target video frame sequence, and extract the short-term interest region features of the target video frame sequence from the sequence features according to the target position information; A second acquisition module is used to acquire a short-term interest region feature set of K associated video frame sequences, where the K associated video frame sequences are video frame sequences adjacent to the target video frame sequence in the N video frame sequences, and K is a positive integer; A fusion module, used for fusing the short-term interest region feature set into a long-term interest region feature; A determination module is used to determine the behavior category of the target object in the target key frame according to the long-term interest region feature and the short-term interest region feature of the target video frame sequence.

12. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 10.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, a computer device having the processor executes the method according to any one of claims 1 to 10.

Images