Action recognition method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a motion recognition method, device and equipment and a storage medium. The method comprises the following steps: dividing a video sequence to be recognized into T video segments; extracting image features corresponding to the T video segments respectively by using a feature extraction network; performing feature fusion processing on the image features corresponding to the T video segments respectively according to the differences between the image features corresponding to the T video segments respectively, to obtain target video features; and determining a motion category of a moving target in the video sequence according to the target video features. The method can ensure accurate recognition of the motion category of the moving target in the video sequence.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an action recognition method, apparatus, device and storage medium. Background Technology

[0002] Video-based action recognition tasks refer to identifying the category of actions performed by a moving target within a video sequence. Unlike ordinary image classification tasks, which only need to consider a single image, video-based action recognition tasks require more consideration of temporal information to accurately identify the action category of a moving target within a video sequence. For example, distinguishing between different swimming strokes often requires comprehensively considering information from multiple images within the time series to make an accurate judgment.

[0003] The relevant technologies are currently mainly based on the Temporal Shift Moduel (TSM) algorithm to realize video-based action recognition tasks. The TSM algorithm adds a temporal feature fusion strategy to the feature extraction network, which can perform feature fusion processing on image features between adjacent video segments, thereby realizing the exchange and fusion of feature information in the temporal dimension.

[0004] However, the inventors of this application have found that the above-mentioned implementation methods often fail to guarantee that the exchanged and fused feature information contains time information with high reference value, which will also affect the accuracy of the final action recognition result. Summary of the Invention

[0005] This application provides an action recognition method, apparatus, device, and storage medium that can ensure accurate identification of the action category of a moving target in a video sequence.

[0006] In view of the above, the first aspect of this application provides an action recognition method, the method comprising:

[0007] Obtain the video sequence to be identified;

[0008] The video sequence is divided into T video segments, where T is an integer greater than 1;

[0009] The T video segments are processed by a feature extraction network to obtain the image features corresponding to each of the T video segments.

[0010] Based on the degree of difference between the image features corresponding to each of the T video segments, feature fusion processing is performed on the image features corresponding to each of the T video segments to obtain the target video features;

[0011] Based on the target video features, determine the action category of the moving target in the video sequence.

[0012] A second aspect of this application provides an action recognition device, the device comprising:

[0013] The video acquisition module is used to acquire the video sequence to be identified;

[0014] A video segmentation module is used to divide the video sequence into T video segments, where T is an integer greater than 1;

[0015] The feature extraction module is used to perform feature extraction processing on the T video segments through a feature extraction network to obtain the image features corresponding to each of the T video segments.

[0016] The feature fusion module is used to perform feature fusion processing on the image features corresponding to the T video segments based on the difference between the image features corresponding to each of the T video segments, so as to obtain the target video features;

[0017] The action recognition module is used to determine the action category of the moving target in the video sequence based on the target video features.

[0018] A third aspect of this application provides an apparatus comprising a processor and a memory:

[0019] The memory is used to store computer programs;

[0020] The processor is configured to perform the steps of the action recognition method as described in the first aspect above, according to the computer program.

[0021] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program for performing the steps of the action recognition method described in the first aspect.

[0022] A fifth aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the action recognition method described in the first aspect.

[0023] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0024] This application provides an action recognition method that innovatively proposes an adaptive feature fusion strategy. When identifying the action category of a moving target in a video sequence based on this adaptive feature fusion strategy, the image features corresponding to each of the T video segments are fused according to the differences between their respective image features to obtain target video features. Then, based on these target video features, the action category of the moving target in the video sequence is identified. By performing feature fusion processing on the image features corresponding to each of the T video segments within the sub-image features based on the differences between their respective image features, sufficient and effective information exchange and fusion in the temporal dimension can be ensured for image features that are of greater interest in motion information, thereby ensuring accurate identification of the action category of the moving target in the video sequence. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the principle of the time offset strategy in the TSM algorithm of related technologies;

[0026] Figure 2 This is a schematic diagram illustrating an application scenario of the action recognition method provided in the embodiments of this application;

[0027] Figure 3 A flowchart illustrating the action recognition method provided in this application embodiment;

[0028] Figure 4 A schematic diagram illustrating the principle of generating image features corresponding to T video segments as provided in an embodiment of this application;

[0029] Figure 5 A schematic diagram illustrating the principle of dividing the image features into T video segments as provided in an embodiment of this application;

[0030] Figure 6 The schematic diagrams are for the forward offset processing, reverse offset processing, and no offset processing provided in the embodiments of this application.

[0031] Figure 7 This is a schematic diagram illustrating the principle of stitching together target sub-image features corresponding to X sub-image features, as provided in an embodiment of this application.

[0032] Figure 8 This is a schematic diagram illustrating the working principle of the adaptive channel offset module provided in the embodiments of this application;

[0033] Figure 9 This is a schematic diagram of the structure of the first motion recognition device provided in the embodiments of this application;

[0034] Figure 10 This is a schematic diagram of the structure of the second motion recognition device provided in the embodiments of this application;

[0035] Figure 11 This is a schematic diagram of the structure of the third motion recognition device provided in the embodiments of this application;

[0036] Figure 12 This is a schematic diagram of the structure of the fourth motion recognition device provided in the embodiments of this application;

[0037] Figure 13 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0038] Figure 14 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0040] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0044] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0045] The solutions provided in this application relate to computer vision technology in artificial intelligence, and are specifically illustrated through the following embodiments:

[0046] To facilitate understanding of the action recognition method provided in this application, the TSM algorithm in related technologies is first introduced below. When recognizing the action category of a moving target in a video sequence based on the TSM algorithm, the video sequence to be recognized needs to be divided into several video segments. Then, a frame image is randomly sampled from each video segment, and the sampled image is input into a feature extraction network for feature extraction. The TSM algorithm incorporates a time-shifting strategy into the feature extraction network, which can perform feature shifting operations between adjacent video segments to achieve information exchange and fusion in the time dimension.

[0047] Figure 1 This is a schematic diagram illustrating the principle of the time offset strategy in the TSM algorithm of related technologies. For example... Figure 1As shown, assuming the video sequence is divided into four video segments, the feature extraction network performs feature extraction processing on the images randomly sampled from each video segment to obtain the image features corresponding to each of the four video segments, namely the image features of video segment 1, the image features of video segment 2, the image features of video segment 3, and the image features of video segment 4. Each video segment image feature includes 6 feature channels (c = 1 to c = 6).

[0048] Assume a fixed time offset strategy: forward offset of image features in feature channel c=1, reverse offset of image features in feature channel c=2, and no feature offset of image features in other feature channels. Accordingly, when offsetting video segment image features according to this time offset strategy, in feature channel c=1, the image feature of segment t is forward-shifted along the time series to segment (t+1), for example, shifting the image feature of video segment 1 to the image feature of video segment 2, shifting the image feature of video segment 2 to the image feature of video segment 3, and so on. In feature channel c=2, the image feature of segment t is reverse-shifted along the time series to segment (t-1), for example, shifting the image feature of video segment 2 to the image feature of video segment 1, shifting the image feature of video segment 3 to the image feature of video segment 2, and so on. However, in feature channels c=3 to c=6, the image features of each video segment remain unchanged.

[0049] After the feature offset processing described above, the image features of each video segment are fused with the image features of its adjacent video segments. For example, the image features of video segment 2 include features shifted from the image features of video segment 1 in the c=1 feature channel, and features shifted from the image features of video segment 3 in the c=2 feature channel. In this way, the exchange and fusion of information between adjacent video segments is achieved in the time dimension.

[0050] However, the inventors of this application have discovered that the TSM algorithms in related technologies are currently based on the aforementioned fixed time offset strategy for feature offset processing. For example, assuming that the image features include C feature channels, the image features on the first C / 4 feature channels are fixedly offset forward, the image features on the C / 4 to C / 2 feature channels are offset backward, and the image features on the last C / 2 feature channels remain unchanged. Different feature channels in feature extraction networks often focus on different information. Some feature channels focus more on motion information in the image, capturing motion changes between different time points, while others focus more on background information. For features that model motion information, feature exchange and feature fusion in the time dimension are very meaningful, helping to exchange and fuse time-representative information between different video segments. However, for features that model background information, since they are relatively stable and remain basically unchanged across different video segments, feature exchange and feature fusion in the time dimension are often less meaningful.

[0051] As discussed above, the TSM algorithm in related technologies, based on a fixed time offset strategy, only performs feature offset processing on features in specific feature channels, which is clearly inappropriate. This approach often fails to guarantee that the feature channels undergoing feature offset processing are those that focus on motion information. In other words, it's difficult to ensure that feature exchange and fusion in the time dimension provide high reference value for the final action category recognition. When the feature channels undergoing feature offset processing are those that focus on background information, the advantages of feature offset processing will be significantly reduced, and it may even affect the accuracy of the final action recognition result.

[0052] To address the problems existing in the aforementioned related technologies, this application provides an action recognition method. This method innovatively proposes an adaptive channel offset strategy. This adaptive channel offset strategy can combine the degree of attention of the feature channel to motion information and use a feature offset method adapted to the feature channel to offset the image features on the feature channel, thereby ensuring accurate recognition of the action category of the moving target in the video sequence.

[0053] Specifically, in the action recognition method provided in this application embodiment, a video sequence to be recognized can first be acquired and divided into T (T is an integer greater than 1) video segments. Then, feature extraction processing is performed on these T video segments using a feature extraction network to obtain image features corresponding to these T video segments. These image features corresponding to these T video segments include C (C is an integer greater than 1) feature channels; and in the direction of the C feature channels, the image features corresponding to these T video segments are divided into X (X is an integer greater than 1) sub-image features, where each sub-image feature includes M (M is a positive integer less than C) feature channels. Furthermore, for each of the X sub-image features, feature shift processing is performed on the image features corresponding to the T video segments in the sub-image feature based on the difference between the image features corresponding to the T video segments in the sub-image feature, to obtain the target sub-image feature corresponding to the sub-image feature. Finally, the target sub-image features corresponding to the X sub-image features are concatenated to obtain the target video feature, and the action category of the moving target in the video sequence is determined based on the target video feature.

[0054] Compared to related technologies that use a fixed time offset strategy to offset image features, the method provided in this application allows for more flexible exchange and fusion of feature information in the time dimension. Since the difference between the image features corresponding to the T video segments in the sub-image features can objectively and realistically reflect the amount of motion information carried in the sub-image features, this difference can measure the degree of attention paid to motion information by the feature channels included in the sub-image features. Therefore, performing feature offset processing on the image features corresponding to the T video segments in the sub-image features based on this difference ensures that the feature offset processing method is adapted to the degree of attention paid to motion information by the feature channels in the sub-image features. This guarantees that image features on feature channels that are more concerned with motion information undergo sufficient and effective information exchange and fusion in the time dimension, thereby ensuring accurate identification of the action category of moving targets in the video sequence.

[0055] It should be understood that the action recognition method provided in this application embodiment can be applied to electronic devices with image processing capabilities, such as terminal devices or servers. Specifically, the terminal device can be a computer, smartphone, tablet computer, personal digital assistant (PDA), etc.; the server can be an application server or a web server. In actual deployment, it can be a standalone server, a cluster server, or a cloud server.

[0056] To facilitate understanding of the action recognition method provided in this application embodiment, the following uses a server as the execution subject of the action recognition method as an example to illustrate the applicable application scenarios of the action recognition method.

[0057] See Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the action recognition method provided in the embodiments of this application. For example... Figure 2 As shown, the application scenario includes a database 210 and a server 220; wherein, the database 210 is used to store the video sequence to be identified, and the server 220 is used to retrieve the video sequence to be identified from the database 210 and execute the action recognition method provided in this application embodiment to identify the action category of the moving target in the retrieved video sequence.

[0058] In practical applications, server 220 can retrieve the video sequence to be identified from database 210, and then divide the retrieved video sequence into T (T is an integer greater than 1) video segments. For example, the video sequence can be divided into 8 video segments. Next, server 220 can perform feature extraction processing on the T video segments through a feature extraction network to obtain the image features corresponding to these T video segments. If the feature extraction network includes C (C is an integer greater than 1) feature channels, the image features obtained after feature extraction processing will also include C feature channels accordingly. For example, server 220 can randomly sample one frame from each video segment to obtain T frames, and then input the T frames into the feature extraction network. The feature extraction network performs feature extraction processing on each frame to obtain the image features corresponding to that frame, which are used as the image features corresponding to the video segment to which that frame belongs. Then, the image features corresponding to each video segment are arranged according to the time order of the T video segments to obtain the image features corresponding to these T video segments.

[0059] After obtaining the image features corresponding to T video segments, server 220 can divide these image features into X (X is an integer greater than 1) sub-image features along the direction of C feature channels. Each sub-image feature includes M (M is a positive integer less than C) feature channels. Different sub-image features may have the same or different number of feature channels. Furthermore, server 220 can perform feature offset processing on each sub-image feature based on the adaptive channel offset strategy provided in this embodiment. Specifically, for each sub-image feature, server 220 can perform feature offset processing on the image features corresponding to the T video segments within that sub-image feature based on the degree of difference between them, to obtain the target sub-image feature corresponding to that sub-image feature.

[0060] Since the difference between the image features corresponding to the T video segments in the sub-image features can reflect the amount of motion information carried in the sub-image features, the degree of attention of the feature channels included in the sub-image features to motion information can be measured based on this difference. Therefore, performing feature shift processing on the image features corresponding to the T video segments in the sub-image features based on this difference can ensure that the feature shift processing method adopted is adapted to the degree of attention of the feature channels in the sub-image features to motion information. For image features on feature channels that pay more attention to motion information, they can be effectively exchanged and fused in the time dimension.

[0061] After obtaining the target sub-image features corresponding to each of the X sub-image features through the aforementioned feature offset processing, server 220 can stitch these target sub-image features together along the directions of C feature channels to obtain the target video features corresponding to the video sequence. Furthermore, server 220 can identify the action category of moving targets in the video sequence based on these target video features.

[0062] It should be noted that, Figure 2 The application scenarios shown are merely examples. In practical applications, the action recognition method provided in this application embodiment can also be applied to other application scenarios. For example, the server can also identify the action category of moving targets in video sequences uploaded by terminal devices, and so on. This application does not limit the application scenarios of this action recognition method in any way.

[0063] The action recognition method provided in this application will be described in detail below through method embodiments.

[0064] See Figure 3 , Figure 3 This is a flowchart illustrating the action recognition method provided in an embodiment of this application. For ease of description, the following embodiments still use a server as the executing entity of this action recognition method. Figure 3 As shown, the action recognition method includes the following steps:

[0065] Step 301: Obtain the video sequence to be identified.

[0066] In practical applications, when a server performs a video-based action recognition task, it can obtain the video sequence to be recognized from a database, a terminal device, or other relevant servers according to actual needs. This application does not limit the source of the video sequence to be recognized, nor does it limit the way the server obtains the video sequence to be recognized.

[0067] In one possible implementation, the server can retrieve the video sequence to be identified from a database. For example, the database can be used to store video sequences to be recommended to target users. After retrieving the video sequence from the database, the server can identify the action category of moving targets in the video sequence using the action recognition method provided in this application embodiment, and then determine whether to recommend the video sequence to the target user based on the action category of the moving targets in the video sequence. For example, the database can also be used to store video sequences to be published. After retrieving the video sequence from the database, the server can identify the action category of moving targets in the video sequence using the action recognition method provided in this application embodiment, and then determine whether the video sequence involves sensitive content based on the action category of the moving targets in the video sequence, and decide whether to continue publishing the video sequence accordingly.

[0068] It should be understood that, in practical applications, the action recognition method provided in this application embodiment can be applied not only to the aforementioned video recommendation and video approval application scenarios, but also to other application scenarios according to actual needs. In other application scenarios, the aforementioned database should be used to store video sequences adapted to that application scenario. This application does not impose any limitations on the application scenarios of this action recognition method.

[0069] In another possible implementation, the server can receive video sequences to be identified uploaded by the terminal device. For example, when a user needs to identify the action category of a moving target in a video sequence through the server, the user can select the video sequence to be identified from the video sequences stored locally on the terminal device, and then trigger the sending of the selected video sequence to the server through a video upload control in the relevant application.

[0070] It should be understood that when the execution subject of the action recognition method provided in this application embodiment is a terminal device, the terminal device can respond to the video selection operation triggered by the user, select the video sequence to be recognized from the video sequence stored locally, and then perform the action recognition method provided in this application embodiment on the selected video sequence to identify the action category of the moving target in the video sequence.

[0071] Step 302: Divide the video sequence into T video segments, where T is an integer greater than 1.

[0072] After the server obtains the video sequence to be identified, it can divide the video sequence into T video segments according to a preset video sequence segmentation rule, where T is an integer greater than 1. For example, assuming the video sequence segmentation rule is to divide the video sequence into 8 equal video segments, then if the length of the video sequence obtained by the server is L, after segmentation, 8 video segments of length L / 8 will be obtained.

[0073] It should be understood that in practical applications, the above video sequence partitioning rules can be set according to actual needs. These rules can involve dividing the video sequence equally into T video segments, or dividing it unequally into T video segments. This application does not impose any limitations on the video sequence partitioning rules, nor on the number of video segments to be divided.

[0074] Step 303: Perform feature extraction processing on the T video segments using a feature extraction network to obtain the image features corresponding to the T video segments.

[0075] After the server divides the video sequence into T video segments, it can use a feature extraction network to extract features from these T video segments to obtain the image features corresponding to each of the T video segments.

[0076] It should be understood that in practical applications, if a feature extraction network includes multiple feature channels, the image features extracted by the feature extraction network will include a corresponding number of feature channels. For example, if the feature extraction network includes C (C is an integer greater than 1) feature channels, the image features corresponding to the T video segments extracted by the feature extraction network will also include C feature channels.

[0077] In practice, after the server divides the video sequence into T video segments, it can randomly sample one frame from each video segment to obtain T frames. Then, the server can input these T frames into a feature extraction network. After feature extraction processing by the feature extraction network, the image features corresponding to each of the T frames will be obtained. The image features corresponding to each of the T frames are the image features corresponding to each of the T video segments.

[0078] Optionally, to facilitate subsequent feature shifting processing along the feature channel dimension, after the server obtains the image features corresponding to each of the T video segments, it can arrange these image features according to the chronological order of the T video segments to obtain the image features corresponding to the T video segments. Accordingly, if the feature extraction network includes C feature channels, the image features corresponding to the T video segments obtained by arranging the image features corresponding to the T video segments will also include C feature channels.

[0079] To facilitate understanding of the above implementation process, the following example illustrates how the server divides a video sequence into four video segments, and how the feature extraction network includes six feature channels. Figure 4 The schematic diagram shown illustrates the implementation process. For example... Figure 4As shown, after the server divides the acquired video sequence into 4 video segments, it can randomly extract one frame from each video segment, thus obtaining 4 frames (image 1, image 2, image 3, and image 4). Then, the server can input these 4 frames into a feature extraction network to obtain the image features corresponding to each of the 4 frames (image feature 1, image feature 2, image feature 3, and image feature 4). The image features corresponding to each frame are the image features corresponding to the video segment to which that frame belongs, and each video segment's image features include 6 feature channels. Furthermore, the server can arrange the image features corresponding to the 4 video segments according to their chronological order in the video sequence to obtain the image features corresponding to the 4 video segments.

[0080] It should be noted that, in order to better exchange and fuse feature information in the time dimension, the method provided in this application embodiment can incorporate the adaptive channel offset strategy provided in this application embodiment during the feature extraction process of the feature extraction network. That is, during the feature extraction process of the feature extraction network, feature offset processing is performed on the image features extracted by the feature extraction layer in the feature extraction network based on the adaptive channel offset strategy in this application embodiment.

[0081] In a feature extraction network comprising N (N being an integer greater than 1) feature extraction layers, the server can use the i-th feature extraction layer in this network to perform feature extraction processing on the (i-1)-th target image features corresponding to each of the T video segments, obtaining the i-th image features corresponding to these T video segments. Here, i is an integer greater than or equal to 1 and less than N. When i equals 1, the (i-1)-th target image feature should be the T video segments, and the i-th image feature corresponding to the T video segments includes C feature channels. Along the direction of the C feature channels, the i-th image feature corresponding to these T video segments is divided into X (X being an integer greater than 1) i-th sub-image features, and the i-th sub-image feature includes M (M being a positive integer less than C) feature channels. For each of the X i-th sub-image features, based on the difference between the i-th image features corresponding to the T video segments within that i-th sub-image feature, feature shifting is performed on the i-th image features corresponding to the T video segments within that i-th sub-image feature to obtain the i-th target sub-image feature corresponding to the i-th sub-image feature. Then, for each of the T video segments, the image features belonging to that video segment from the i-th target sub-image features corresponding to the X i-th sub-image features are concatenated to obtain the i-th target image feature corresponding to that video segment. Finally, the i-th target image features corresponding to the T video segments can be input into the (i+1)-th feature extraction layer of the feature extraction network.

[0082] Based on the adaptive channel offset strategy in this application embodiment, the specific implementation method of performing feature offset processing on the image features extracted by the feature extraction layer in the feature extraction network is similar to the specific implementation method of performing feature offset processing on the image features corresponding to T video segments described below. For details, please refer to the description of the implementation process of performing feature offset processing on the image features corresponding to T video segments described below.

[0083] The server performs feature offset processing on the i-th image features corresponding to the T video segments extracted by the i-th feature extraction layer in the feature extraction network, obtaining the i-th target sub-image features corresponding to each of the X i-th sub-image features. Based on these i-th target sub-image features, the server can determine the i-th target image features corresponding to each of the T video segments. Specifically, for each of the T video segments, the server can concatenate the image features belonging to that video segment from the X i-th target sub-image features, thereby obtaining the i-th target image feature corresponding to that video segment. Furthermore, the server can input the i-th target image features corresponding to each of the T video segments into the (i+1)-th feature extraction layer in the feature extraction network to perform the above processing on the (i+1)-th image features corresponding to the T video segments extracted by the (i+1)-th feature extraction layer.

[0084] It should be understood that in practical applications, the server can perform the above feature exchange and fusion processing on all image features extracted by each feature extraction layer in the feature extraction network, or it can select some feature extraction layers from the feature extraction network and perform the above feature exchange and fusion processing only on the image features extracted by these feature extraction layers.

[0085] Step 304: Based on the difference between the image features corresponding to each of the T video segments, perform feature fusion processing on the image features corresponding to each of the T video segments to obtain the target video features.

[0086] After the server obtains the image features corresponding to each of the T video segments output by the feature extraction network, it can calculate the difference between the image features corresponding to each of the T video segments. Then, based on the difference between the image features corresponding to each of the T video segments, feature fusion processing is performed on the image features corresponding to each of the T video segments to obtain the target video features.

[0087] In practice, the server can calculate the difference between image features corresponding to adjacent video segments, and then perform corresponding convolutional fusion processing on the image features corresponding to adjacent video segments based on this difference. It should be understood that the greater the difference between image features corresponding to adjacent video segments, the more sufficient the information exchange and fusion of the image features corresponding to adjacent video segments in the time dimension will be during convolutional fusion processing. Furthermore, the target video features are constructed based on the image features obtained after convolutional fusion processing.

[0088] Optionally, if before executing step 304, the server has already arranged the image features corresponding to the T video segments according to their time order, and obtained the image features corresponding to the T video segments (the image features include C feature channels), then the server can further perform feature offset processing in the dimension of the feature channels to achieve feature fusion between the above image features.

[0089] Specifically, the server can divide the image features corresponding to the T video segments into X sub-image features in the directions of the C feature channels, where X is an integer greater than 1; the sub-image features include M feature channels, where M is a positive integer less than C.

[0090] The server performs feature extraction processing on T video segments using a feature extraction network. After obtaining the image features corresponding to the T video segments, it can divide the image features corresponding to the T video segments into X (X is an integer greater than 1) sub-image features along the feature channels. That is, the server can divide the image features corresponding to the T video segments into multiple sub-image features along C feature channel directions. Each sub-image feature includes M (M is a positive integer less than C) feature channels. Different sub-image features can have the same or different number of feature channels.

[0091] Figure 5 This is a schematic diagram illustrating the principle of dividing image features into T video segments, as an example. Figure 5 As shown, the server can divide the image features corresponding to T video segments into two parts along the direction of the feature channels, resulting in sub-image features X1 and sub-image features X2. The number of feature channels included in sub-image features X1 and sub-image features X2 are both C / 2.

[0092] It should be understood that in practical applications, the server can also divide the image features corresponding to T video segments into X sub-image features along the direction of the feature channel according to other image feature division rules. This application does not specifically limit X, nor does it limit the image feature division rules.

[0093] Furthermore, for each of the X sub-image features, based on the difference between the image features corresponding to the T video segments in the sub-image features, feature shifting processing is performed on the image features corresponding to the T video segments in the sub-image features to obtain the target sub-image feature corresponding to the sub-image feature.

[0094] After the server divides the image features corresponding to T video segments into X sub-image features, it can further perform adaptive feature offset processing on the image features corresponding to the T video segments in each sub-image feature based on the adaptive channel offset strategy provided in this application embodiment. Specifically, for each sub-image feature among the X sub-image features, the server can perform feature offset processing on the image features corresponding to the T video segments in that sub-image feature according to the degree of difference between the image features corresponding to the T video segments in that sub-image feature. The sub-image feature obtained after feature offset processing is the target sub-image feature.

[0095] like Figure 5 As shown, after dividing the image features corresponding to T video segments into X sub-image features, each sub-image feature includes the image features of each of the T video segments in their respective feature channels. In this application, feature offset processing is performed on the image features corresponding to each of the T video segments in the sub-image features, which is essentially offset processing on the image features of each of the T video segments in their respective feature channels.

[0096] It should be noted that the feature offset processing in the embodiments of this application mainly includes forward offset processing, reverse offset processing, and no offset processing. Forward offset processing refers to shifting the image features of the t-th video segment to the image features of the (t+1)-th video segment on a certain feature channel c, as shown in formula (1); it should be understood that after performing forward offset processing on the image features of the 1st video segment on feature channel c (i.e., shifting the image features of the 1st video segment to the image features of the 2nd video segment on feature channel c), zero-padding can be performed on the image features of the 1st video segment on feature channel c. Reverse offset processing refers to shifting the image features of the t-th video segment to the image features of the (t-1)-th video segment on a certain feature channel c, as shown in formula (2). It should be understood that after performing reverse offset processing on the image features of the T-th video segment on feature channel c (i.e., shifting the image features of the T-th video segment to the image features of the (T-1)-th video segment on feature channel c), the image features of the T-th video segment on feature channel c can be zero-padding. Non-offset processing refers to keeping the image features of each video segment on a certain feature channel c unchanged.

[0097] forwardshift(X)→X[c,t+1,:,:]=X[c,t,:,:] (1)

[0098] backwardshift(X)→X[c,t,∶,:]=X[c,t+1,:,:] (2)

[0099] Figure 6 This is a schematic diagram illustrating the principles of forward offset processing, reverse offset processing, and no offset processing. For example... Figure 6 As shown in (a), assuming a positive offset is applied to sub-image feature X1 (which includes image features corresponding to each of the four video segments), then in the two feature channels included in sub-image feature X1, the image feature corresponding to the first video segment is shifted to the image feature corresponding to the second video segment, the image feature corresponding to the second video segment is shifted to the image feature corresponding to the third video segment, and so on; zero-padding can be applied to the image feature missing in the first video segment. Figure 6 As shown in (b), assuming that the sub-image feature X2 (which includes the image features corresponding to each of the four video segments) undergoes reverse offset processing, then in the two feature channels included in the sub-image feature X2, the image feature corresponding to the fourth video segment is shifted to the image feature corresponding to the third video segment, the image feature corresponding to the third video segment is shifted to the image feature corresponding to the second video segment, and so on; the image feature missing at the fourth video segment can be zero-padded. Figure 6 As shown in (c), if the sub-image feature X3 (which includes the image features corresponding to the four video segments) is processed without offset, then the sub-image feature X3 remains unchanged.

[0100] In some embodiments, the server can determine the feature offset method to be applied to the image features corresponding to each of the T video segments in the sub-image feature based on the inter-segment difference feature corresponding to the sub-image feature. That is, the server can determine the inter-segment difference feature corresponding to the sub-image feature based on the difference between the image feature corresponding to the (t+1)th video segment and the image feature corresponding to the tth video segment in the sub-image feature, where t is an integer greater than or equal to 1 and less than T; then, based on the inter-segment difference feature, feature offset processing is performed on the image features corresponding to each of the T video segments in the sub-image feature to obtain the target sub-image feature corresponding to the sub-image feature.

[0101] In practice, the server can determine the inter-segment difference feature D corresponding to the sub-image feature X using formula (3):

[0102] D=abs(X[:,t+1,:,:]-X[:,t,:,:]) (3)

[0103] Where abs() is the function for taking the absolute value; (X[:,t+1,:,:]-X[:,t,:,:]) means subtracting the image feature corresponding to the t-th video segment from the image feature corresponding to the (t+1)-th video segment in the sub-image feature X.

[0104] When the server divides the image features corresponding to T video segments into X sub-image features along the direction of C feature channels, the server can calculate the inter-segment difference feature corresponding to each of the X sub-image features using the above formula (3).

[0105] It should be noted that the magnitude of the inter-segment difference feature can characterize the amount of motion information contained in the corresponding sub-image features. For a feature channel c, if the feature channel models motion information, then the differences between the image features corresponding to adjacent video segments will be relatively large, and the inter-segment difference feature will also be relatively large. Conversely, if the feature channel models background information, then the differences between the image features corresponding to adjacent video segments will be relatively small, and the inter-segment difference feature will also be relatively small.

[0106] After the server determines the inter-segment difference feature corresponding to the sub-image feature, it can determine the channel offset coefficient corresponding to the sub-image based on the inter-segment difference feature. Then, based on the feature offset direction assigned to the sub-image, the channel offset coefficient, and the sub-image feature, the server performs offset processing on the image features corresponding to the T video segments in the sub-image feature on the M feature channels included in the sub-image feature, thereby obtaining the target sub-image feature corresponding to the sub-image feature.

[0107] When determining the channel offset coefficient, the server can use an objective function to determine the channel offset coefficient corresponding to the segment difference feature of the sub-image feature. The channel offset coefficient determined by the objective function is proportional to the segment difference feature, and the value of the determined channel offset coefficient is between 0 and 1.

[0108] For example, the server can process the inter-segment difference feature corresponding to the sub-image feature using the sigmoid function to obtain the channel offset coefficient α corresponding to the sub-image feature, as shown in formula (4). The specific form of the sigmoid function is shown in formula (5).

[0109] α=sigmoid(D) (4)

[0110]

[0111] The output range of the sigmoid function is between 0 and 1, and the output value is proportional to the input value. That is, the larger the inter-segment difference feature of the input, the larger the channel offset coefficient obtained after processing by the sigmoid function.

[0112] It should be understood that in practical applications, in addition to using the sigmoid function to process the inter-segment difference features corresponding to the sub-image features and obtain the channel offset coefficients corresponding to the sub-image features, the server can also use other functions to process the inter-segment difference features corresponding to the sub-image features and obtain the channel offset coefficients corresponding to the sub-image features. This application does not specifically limit the target function used when processing the inter-segment difference features.

[0113] Specifically, when performing feature offset processing on the image features corresponding to each of the T video segments in the sub-image features based on the channel offset coefficient, the server can first adjust the sub-image features based on the channel offset coefficient to obtain reference sub-image features; then, along the feature offset direction assigned to the sub-image features, feature offset processing is performed on the image features corresponding to each of the T video segments in the reference sub-image features on the M feature channels included in the sub-image features to obtain the reference sub-image features after feature offset processing; furthermore, based on the reference sub-image features after feature offset processing, the reference sub-image features, and the sub-image features, the target sub-image features corresponding to the sub-image features are determined.

[0114] Assume that the server divides the image features corresponding to T video segments into sub-image features X1 and sub-image features X2 along the direction of C feature channels, and the feature offset direction assigned to sub-image feature X1 is positive offset, and the feature offset direction assigned to sub-image feature X2 is negative offset. At this time, the server can calculate the target sub-image feature Output1 corresponding to sub-image feature X1 by Equation (6), and calculate the target sub-image feature Output2 corresponding to sub-image feature X2 by Equation (7).

[0115] Output1=forwardshift(α1X1)-α1X1+X1 (6)

[0116] Output2=backwardshift(α2X2)-α2X2+X2 (7)

[0117] Where α1 represents the channel offset coefficient corresponding to sub-image feature X1, α1X1 represents the reference sub-image feature corresponding to sub-image feature X1, and forwardshift(α1X1) represents the reference sub-image feature after feature shifting by performing forward shifting processing on the reference sub-image feature α1X1. Similarly, α2 represents the channel offset coefficient corresponding to sub-image feature X2, α2X2 represents the reference sub-image feature corresponding to sub-image feature X2, and backwardshift(α2X2) represents the reference sub-image feature after feature shifting processing by performing backward shifting processing on the reference sub-image feature α2X2.

[0118] It should be understood that if the feature channels included in a certain sub-image feature are mainly used to model motion information, the inter-segment difference feature corresponding to the sub-image feature will be relatively large. Furthermore, the channel offset coefficient α calculated based on the inter-segment difference feature will be close to 1. At this time, the above formula (6) will be approximately transformed into formula (8), and the above formula (7) will be approximately transformed into formula (9).

[0119] Output1=forwardshift(X1) (8)

[0120] Output2= backwardshift(X2) (9)

[0121] Conversely, if the feature channels included in a certain sub-image feature are mainly used to model background information, the inter-segment difference feature corresponding to the sub-image feature will be relatively small. Furthermore, the channel offset coefficient calculated based on the inter-segment difference feature will be close to 0. At this time, the above formula (6) will be approximately transformed into formula (10), and the above formula (7) will be approximately transformed into formula (11).

[0122] Output1 = X1 (10)

[0123] Output2 = X2 (11)

[0124] In other words, the larger the inter-segment difference feature corresponding to the sub-image feature, that is, the greater the difference between the image features corresponding to each of the T video segments in the sub-image feature, the larger the magnitude of the feature shift processing performed when performing feature shift processing on the image features corresponding to each of the T video segments in the sub-image feature. Conversely, the smaller the inter-segment difference feature corresponding to the sub-image feature, that is, the smaller the difference between the image features corresponding to each of the T video segments in the sub-image feature, the smaller the magnitude of the feature shift processing performed when performing feature shift processing on the image features corresponding to each of the T video segments in the sub-image feature, and the closer it is to no shift processing.

[0125] In other embodiments, the server can determine whether to perform feature offset processing on the image features corresponding to the T video segments in the sub-image feature based on the degree of difference between the image features corresponding to the T video segments in the sub-image feature. That is, the server can determine whether the degree of difference between the image features corresponding to the T video segments in the sub-image feature is greater than a preset degree of difference threshold; if so, then according to a preset feature offset method, feature offset processing is performed on the image features corresponding to the T video segments in the sub-image feature on M feature channels to obtain the target sub-image feature corresponding to the sub-image feature; if not, then the sub-image feature itself is used as its corresponding target sub-image feature.

[0126] In specific implementation, the server can still use the inter-segment difference feature to characterize the difference between the image features corresponding to the T video segments in the sub-image feature. That is, the server can first determine the inter-segment difference feature corresponding to the sub-image feature by the above formula (3) based on the difference between the image feature corresponding to the (t+1)th video segment and the image feature corresponding to the tth video segment in the sub-image feature. Then, it is determined whether the inter-segment difference feature corresponding to the sub-image feature is greater than the preset inter-segment difference threshold. If it is, it means that the feature channels included in the sub-image feature are mainly used to model motion information. If not, it means that the feature channels included in the sub-image feature are mainly used to model background information.

[0127] It should be understood that in practical applications, besides using inter-segment difference features to characterize the difference between the image features corresponding to the T video segments in the sub-image features, other parameters can also be used to characterize this difference. This application does not impose any limitations on the representation of this difference. Accordingly, when other forms are used to represent this difference, the aforementioned preset difference threshold should be a threshold parameter corresponding to that form.

[0128] If the difference between the image features corresponding to the T video segments in the sub-image feature is greater than a preset difference threshold, it indicates that the feature channels included in the sub-image feature are mainly used for modeling motion information. At this time, the server can perform feature offset processing on the image features corresponding to the T video segments in the sub-image feature on the M feature channels in the sub-image feature according to the feature offset direction pre-assigned to the sub-image feature, to obtain the target sub-image feature corresponding to the sub-image feature. For example, assuming that the feature offset direction assigned by the server to the sub-image feature is a positive offset, the server can translate the image feature corresponding to the t-th video segment in the sub-image feature to the image feature corresponding to the (t+1)-th video segment on the M feature channels in the sub-image feature.

[0129] If the difference between the image features corresponding to the T video segments in the sub-image feature is not greater than the preset difference threshold, it means that the feature channels included in the sub-image feature are mainly used to model background information. In this case, the server does not need to perform feature offset processing on the image features corresponding to the T video segments in the sub-image feature, and can directly use the sub-image feature itself as its corresponding target sub-image feature.

[0130] Then, the target video features are obtained by splicing the target sub-image features corresponding to each of the X sub-image features.

[0131] Based on the adaptive channel offset strategy in this application embodiment, the server performs corresponding feature offset processing on each of the X sub-image features to obtain the target sub-image features corresponding to each of the X sub-image features. Then, the server can stitch together the target sub-image features corresponding to each of the X sub-image features in the directions of the C feature channels to obtain the target video features corresponding to the video sequence obtained in step 301.

[0132] Figure 7 This is a schematic diagram illustrating the principle of stitching together target sub-image features corresponding to X sub-image features, as exemplified. Figure 7 As shown, assuming that in step 303 the server divides the image features corresponding to T video segments into sub-image features X1 and sub-image features X2 along the direction of C feature channels, after the server obtains the target sub-image feature Output1 corresponding to sub-image feature X1 and the target sub-image feature Output2 corresponding to sub-image feature X2 in step 304, the server can concatenate the target sub-image feature Output1 and the target sub-image feature Output2 along the direction of C feature channels to obtain the target video feature, which can be specifically expressed by formula (12).

[0133] Output=Concat(Output1,Output2) (12)

[0134] Concat() means concatenating the target sub-image features Output1 and Output2 along the direction of C feature channels.

[0135] It should be understood that the distribution positions of target sub-image features Output1 and Output2 in the target video features are the same as the distribution positions of sub-image features X1 and X2 in the image features corresponding to the T video segments.

[0136] Step 305: Determine the action category of the moving target in the video sequence based on the target video features.

[0137] Once the server obtains the target video features, it can identify the action category of the moving target in the video sequence based on these features. The specific implementation method for identifying the action category of a moving target based on video features is a mature technology and will not be elaborated here. It should be noted that the action categories that this application can identify can specifically include motion categories, behavior categories, event categories, etc.

[0138] In one possible implementation, the action recognition method provided in the above embodiments of this application can be applied to a video recommendation scenario. Specifically, the action recognition method is executed on a video sequence to be recommended to a target user to determine the action category of moving targets in the video sequence, and then to determine whether to recommend the video sequence to the target user. Specifically, the server can obtain the action categories of interest to the target user, and then determine whether the action category of the moving targets in the video sequence belongs to the target user's action categories of interest. If so, the video sequence is recommended to the target user.

[0139] It should be understood that in practical applications, the aforementioned target user's interest action categories can be determined based on the target user's historical video viewing records. For example, the server can determine video sequences with a viewing duration exceeding a preset duration based on the target user's historical video viewing records, and then determine the action categories of moving objects within these video sequences as the target user's interest action categories. Alternatively, the server can, based on the target user's historical video viewing records, count the number of times the target user has historically viewed video sequences containing that action category for each action category, and then determine the action categories with historical viewing counts exceeding a preset number of viewings as the target user's interest action categories.

[0140] For the video sequence to be recommended, the server can identify the action category of the moving target in it using the action recognition method described above. Then, it can determine whether the action category belongs to the action category of interest to the target user. If so, the video sequence is recommended to the target user; otherwise, the video sequence is not recommended to the target user.

[0141] In another possible implementation, the action recognition method provided in the above embodiments of this application can be applied to video review scenarios. Specifically, the action recognition method is executed on a video sequence to be published on a video platform to determine the action category of moving targets in the video sequence, and then to determine whether to continue publishing the video sequence on the video platform. Specifically, the server can obtain a set of sensitive action categories, and then determine whether the action category of the moving targets in the video sequence belongs to that set of sensitive action categories. If so, the publication of the video sequence is blocked.

[0142] It should be understood that in practical applications, the action categories included in the sensitive action category set are usually action categories that are not suitable for playback on video platforms, such as violent fights. The action categories in this sensitive action category set can be set by relevant technical personnel according to the actual business needs of the video platform.

[0143] For the video sequence to be published, the server can identify the action category of the moving target in it using the above action recognition method, and then determine whether the action category belongs to the sensitive action category set. If it does, the video sequence will be blocked from being published; otherwise, the video sequence can be published normally.

[0144] It should be understood that, in addition to applying the action recognition method provided in the embodiments of this application to video recommendation and video review scenarios, in practical applications, the action recognition method provided in the embodiments of this application can also be applied to other application scenarios according to actual needs.

[0145] Compared to related technologies that use a fixed time offset strategy to offset image features, the method provided in this application allows for more flexible exchange and fusion of feature information in the time dimension. Since the difference between the image features corresponding to the T video segments in the sub-image features can objectively and realistically reflect the amount of motion information carried in the sub-image features, this difference can measure the degree of attention paid to motion information by the feature channels included in the sub-image features. Therefore, performing feature offset processing on the image features corresponding to the T video segments in the sub-image features based on this difference ensures that the feature offset processing method is adapted to the degree of attention paid to motion information by the feature channels in the sub-image features. This guarantees that image features on feature channels that are more concerned with motion information undergo sufficient and effective information exchange and fusion in the time dimension, thereby ensuring accurate identification of the action category of moving targets in the video sequence.

[0146] To facilitate a further understanding of the action recognition method provided in the embodiments of this application, the following description will still take the server as the execution subject as an example to provide an overall exemplary introduction to the action recognition method provided in the embodiments of this application.

[0147] Assuming the server acquires a video sequence of length L, the server can first divide the video sequence into T (taking T=8 as an example) video segments, each segment being L / 8 in length. Then, the server can randomly sample one frame from each video segment, obtaining a total of T frames from the T video segments, which are then used as input to the feature extraction network.

[0148] The feature extraction network may include several adaptive channel shift modules, and the specific working principle of the adaptive channel shift module is as follows: Figure 8 As shown. Assume the input feature of this adaptive channel offset module is A, with feature dimensions C×T×H×W, where C represents the number of feature channels included, T represents the number of video segments, and H and W represent the spatial dimensions of the input feature. It should be understood that in practical applications, the above input feature can be the feature output of a feature extraction layer in the feature extraction network, or it can be the final output feature of the feature extraction network.

[0149] In the adaptive channel offset module, the input feature A can be divided into two parts along the direction of C feature channels: sub-image feature X1 and sub-image feature X2. Both sub-image feature X1 and sub-image feature X2 contain C / 2 feature channels. For sub-image feature X1 and sub-image feature X2, their respective inter-segment difference features D1 and D2 can be calculated. That is, for each sub-image feature, the image feature corresponding to the (t+1)th video segment is subtracted from the image feature corresponding to the tth video segment. The specific calculation formula is as follows:

[0150] D1=abs(X1[:,t+1,:,:]-X1[:,t,:,:])

[0151] D2=abs(X2[:,t+1,:,:]-X2[:,t,:,:])

[0152] It should be noted that the inter-segment difference feature can characterize the amount of motion information contained in the sub-image features. For a certain feature channel c, if the feature channel models motion information, then the differences between the image features corresponding to adjacent video segments will be relatively large, and the inter-segment difference feature will also be relatively large. Conversely, if the feature channel models background information, then the differences between the image features corresponding to adjacent video segments will be relatively small, and the inter-segment difference feature will also be relatively small.

[0153] Then, the adaptive channel offset module can use the sigmoid function to calculate the channel offset coefficient α1 corresponding to sub-image feature X1 and the channel offset coefficient α2 corresponding to sub-image feature X2:

[0154] α1 = sigmoid(D1)

[0155] α2 = sigmoid(D2)

[0156]

[0157] The output range of the sigmoid function is between 0 and 1. The larger the inter-segment difference feature corresponding to the sub-image feature, the closer the channel offset coefficient corresponding to the sub-image feature is to 1. Conversely, the smaller the inter-segment difference feature corresponding to the sub-image feature, the closer the channel offset coefficient corresponding to the sub-image feature is to 0.

[0158] Furthermore, the adaptive channel offset module applies both forward and reverse offset processing to the reference sub-image features αX, which have been adjusted by the channel offset coefficient α, as shown in the following formula:

[0159] Output1=forwardshift(α1X1)-α1X1+X1,

[0160] Output2=backwardshift(α1X2)-α1X2+X1

[0161] It should be understood that if the feature channels included in a certain sub-image feature are mainly used to model motion information, then the inter-segment difference feature corresponding to that sub-image feature will be relatively large. Furthermore, the channel offset coefficient α calculated based on this inter-segment difference feature will be close to 1. At this time, the above formula approximately becomes:

[0162] Output1 = forwardshift(X1)

[0163] Output2 = backwardshift(X2)

[0164] In this case, the adaptive channel offset module performs the corresponding feature offset processing according to the feature offset direction pre-assigned to each sub-image feature.

[0165] Conversely, if the feature channels included in a certain sub-image feature are mainly used to model background information, the inter-segment difference feature corresponding to that sub-image feature will be relatively small. Furthermore, the channel offset coefficient calculated based on this inter-segment difference feature will be close to 0. In this case, the above formula approximately becomes:

[0166] Output1 = X1

[0167] Output2 = X2

[0168] In this situation, the adaptive channel offset module performs the operation of keeping the original features unchanged.

[0169] After completing the above feature offset processing, the adaptive channel offset module can use the following formula to stitch together the target sub-image features obtained after feature offset processing in the directions of C feature channels to obtain the target video feature output:

[0170] Output=Concat(Output1,Output2)

[0171] Ultimately, the server can identify the action category of the moving target in the acquired video sequence based on the target video features.

[0172] In response to the action recognition method described above, this application also provides a corresponding action recognition device to enable the practical application and implementation of the above action recognition method.

[0173] See Figure 9 , Figure 9 For the above text Figure 3The diagram shows a structural schematic of an action recognition device 900 corresponding to the action recognition method shown. The action recognition device 900 includes:

[0174] Video acquisition module 901 is used to acquire the video sequence to be identified;

[0175] The video segmentation module 902 is used to divide the video sequence into T video segments, where T is an integer greater than 1;

[0176] The feature extraction module 903 is used to perform feature extraction processing on the T video segments through a feature extraction network to obtain the image features corresponding to each of the T video segments.

[0177] The feature fusion module 904 is used to perform feature fusion processing on the image features corresponding to the T video segments according to the difference between the image features corresponding to the T video segments respectively, so as to obtain the target video features;

[0178] The action recognition module 905 is used to determine the action category of the moving target in the video sequence based on the target video features.

[0179] Optional, in Figure 9 Based on the motion recognition device shown, see Figure 10 , Figure 10 This is a schematic diagram of another motion recognition device 1000 provided in an embodiment of this application. (See attached diagram.) Figure 10 As shown, the device further includes:

[0180] The feature arrangement module 1001 is used to arrange the image features corresponding to each of the T video segments according to the time order of the T video segments, so as to obtain the image features corresponding to the T video segments; the image features corresponding to the T video segments include C feature channels, where C is an integer greater than 1;

[0181] The feature segmentation module 1002 is used to divide the image features corresponding to the T video segments into X sub-image features in the directions of the C feature channels, where X is an integer greater than 1; the sub-image features include M feature channels, where M is a positive integer less than C;

[0182] The feature fusion module 904 includes:

[0183] The feature offset unit 1003 is used to perform feature offset processing on the image features corresponding to the T video segments in the sub-image features for each of the X sub-image features, based on the difference between the image features corresponding to the T video segments in the sub-image features, to obtain the target sub-image feature corresponding to the sub-image feature.

[0184] The feature stitching unit 1004 is used to stitch together the target sub-image features corresponding to the X sub-image features to obtain the target video features.

[0185] Optional, in Figure 10 Based on the action recognition device shown, the feature offset unit 1003 includes:

[0186] The segment difference determination subunit is used to determine the segment difference feature corresponding to the sub-image feature based on the difference between the image feature corresponding to the (t+1)th video segment and the image feature corresponding to the tth video segment in the sub-image feature; where t is an integer greater than or equal to 1 and less than T;

[0187] The feature offset subunit is used to perform feature offset processing on the image features corresponding to each of the T video segments in the sub-image features based on the inter-segment difference features, so as to obtain the target sub-image features.

[0188] Optional, in Figure 10 Based on the action recognition device shown, the feature offset subunit is specifically used for:

[0189] Based on the inter-segment difference feature corresponding to the sub-image feature, determine the channel offset coefficient corresponding to the sub-image feature;

[0190] Based on the feature offset direction assigned to the sub-image, the channel offset coefficient, and the sub-image features, the image features corresponding to the T video segments in the sub-image features are offset on the M feature channels to obtain the target sub-image features.

[0191] Optional, in Figure 10 Based on the action recognition device shown, the feature offset subunit is specifically used for:

[0192] The objective function determines the channel offset coefficient corresponding to the segment difference feature of the sub-image feature; the channel offset coefficient is proportional to the segment difference feature, and the channel offset coefficient is greater than or equal to 0 and less than or equal to 1.

[0193] Optional, in Figure 10 Based on the action recognition device shown, the feature offset subunit is specifically used for:

[0194] The sub-image features are adjusted based on the channel offset coefficient to obtain reference sub-image features;

[0195] Along the feature offset direction assigned to the sub-image features, feature offset processing is performed on the image features corresponding to each of the T video segments in the reference sub-image features on the M feature channels to obtain the reference sub-image features after feature offset processing.

[0196] The target sub-image features are determined based on the reference sub-image features after feature offset processing, the reference sub-image features, and the sub-image features.

[0197] Optional, in Figure 10 Based on the action recognition device shown, the feature offset unit 1003 is specifically used for:

[0198] If the difference is greater than a preset difference threshold, feature offset processing is performed on the image features corresponding to the T video segments in the sub-image features in the M feature channels according to a preset feature offset method to obtain the target sub-image features.

[0199] If the difference is less than or equal to the preset difference threshold, the sub-image feature is used as the target sub-image feature.

[0200] Optional, in Figure 10 Based on the action recognition device shown, and assuming the feature extraction network comprises N feature extraction layers, where N is an integer greater than 1; the feature offset unit 1003 is further used for:

[0201] Through the i-th feature extraction layer, feature extraction processing is performed on the (i-1)-th target image features corresponding to each of the T video segments to obtain the i-th image features corresponding to the T video segments; where i is an integer greater than or equal to 1 and less than N, and when i equals 1, the (i-1)-th target image feature is the T video segments; the i-th image features corresponding to the T video segments include the C feature channels;

[0202] The i-th image features corresponding to the T video segments are divided into X i-th sub-image features along the directions of the C feature channels; the i-th sub-image features include the M feature channels;

[0203] For each i-th sub-image feature among the X i-th sub-image features, based on the difference between the i-th image features corresponding to the T video segments in the i-th sub-image features, feature shifting processing is performed on the i-th image features corresponding to the T video segments in the i-th sub-image features to obtain the i-th target sub-image features corresponding to the i-th sub-image.

[0204] For each of the T video segments, the image features belonging to that video segment in the i-th target sub-image features corresponding to the i-th sub-image features of the X i-th sub-image features are stitched together to obtain the i-th target image feature corresponding to that video segment.

[0205] The i-th target image features corresponding to each of the T video segments are input into the (i+1)-th feature extraction layer.

[0206] Optional, in Figure 9 Based on the motion recognition device shown, see Figure 11 , Figure 11 This is a schematic diagram of another motion recognition device provided in an embodiment of this application. Figure 11 As shown, the device also includes:

[0207] The video recommendation module 1101 is used to obtain the target user's interest in action categories; and recommend the video sequence to the target user if the target's action category in the video sequence belongs to the interest in action categories.

[0208] Optional, in Figure 9 Based on the motion recognition device shown, see Figure 12 , Figure 12 This is a schematic diagram of another motion recognition device provided in an embodiment of this application. Figure 12 As shown, the device also includes:

[0209] The video review module 1201 is used to obtain a set of sensitive action categories; if the action category of the target running in the video sequence belongs to the set of sensitive action categories, the video sequence is blocked from being published.

[0210] The action recognition device provided in this application embodiment can more flexibly exchange and fuse feature information in the time dimension. Since the difference between the image features corresponding to the T video segments in the sub-image features can objectively and realistically reflect the amount of motion information carried in the sub-image features, this difference can measure the degree of attention paid to motion information by the feature channels included in the sub-image features. Therefore, performing feature shifting processing on the image features corresponding to the T video segments in the sub-image features based on this difference can ensure that the feature shifting processing method is adapted to the degree of attention paid to motion information by the feature channels in the sub-image features. That is, it ensures that image features on feature channels that are more concerned with motion information are fully and effectively exchanged and fused in the time dimension, thereby ensuring accurate identification of the action category of moving targets in the video sequence.

[0211] This application also provides a device for identifying the action category of a moving target in a video sequence. Specifically, the device can be a server or a terminal device. The server and terminal device provided in this application will be described below from the perspective of hardware implementation.

[0212] See Figure 13 , Figure 13 This is a schematic diagram of the structure of a server 1300 provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations stored in the storage media 1330 on the server 1300.

[0213] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0214] The steps performed by the server in the above embodiments can be based on this Figure 13 The server structure shown.

[0215] CPU 1322 is used to perform the following steps:

[0216] Obtain the video sequence to be identified;

[0217] The video sequence is divided into T video segments, where T is an integer greater than 1;

[0218] The T video segments are processed by a feature extraction network to obtain the image features corresponding to each of the T video segments.

[0219] Based on the degree of difference between the image features corresponding to each of the T video segments, feature fusion processing is performed on the image features corresponding to each of the T video segments to obtain the target video features;

[0220] Based on the target video features, determine the action category of the moving target in the video sequence.

[0221] Optionally, the CPU 1322 can also be used to execute any of the steps of the action recognition method provided in the embodiments of this application.

[0222] See Figure 14 , Figure 14 This is a schematic diagram of a terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device, including smartphones, computers, tablets, personal digital assistants, etc. Taking a computer as an example:

[0223] Figure 14 This is a block diagram illustrating a portion of the structure of a computer associated with the terminal provided in an embodiment of this application. (Reference) Figure 14 The computer includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 14 The computer architecture shown does not constitute a limitation on the computer and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0224] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various computer functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer (such as audio data, telephone directory, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0225] The processor 1480 is the control center of the computer, connecting various parts of the computer through various interfaces and lines. It performs various computer functions and processes data by running or executing software programs and / or modules stored in the memory 1420, and by calling data stored in the memory 1420, thereby providing overall monitoring of the computer. Optionally, the processor 1480 may include one or more processing units; preferably, the processor 1480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 1480.

[0226] In this embodiment of the application, the processor 1480 included in the terminal also has the following functions:

[0227] Obtain the video sequence to be identified;

[0228] The video sequence is divided into T video segments, where T is an integer greater than 1;

[0229] The T video segments are processed by a feature extraction network to obtain the image features corresponding to each of the T video segments.

[0230] Based on the degree of difference between the image features corresponding to each of the T video segments, feature fusion processing is performed on the image features corresponding to each of the T video segments to obtain the target video features;

[0231] Based on the target video features, determine the action category of the moving target in the video sequence.

[0232] Optionally, the processor 1480 is further configured to execute steps of any implementation of the action recognition method provided in the embodiments of this application.

[0233] This application also provides a computer-readable storage medium for storing a computer program that executes any one of the implementation methods of the action recognition method described in the foregoing embodiments.

[0234] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the implementation methods of the action recognition method described in the foregoing embodiments.

[0235] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0237] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0238] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0239] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing computer programs.

[0240] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0241] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An action recognition method, characterized in that, The method includes: Obtain the video sequence to be identified; The video sequence is divided into T video segments, where T is an integer greater than 1; The T video segments are processed by a feature extraction network to obtain the image features corresponding to each of the T video segments. Arrange the image features corresponding to each of the T video segments according to their time sequence to obtain the image features corresponding to the T video segments; the image features corresponding to the T video segments include C feature channels, where C is an integer greater than 1; In the directions of the C feature channels, the image features corresponding to the T video segments are divided into X sub-image features, where X is an integer greater than 1; each sub-image feature includes M feature channels, where M is a positive integer less than C; For each of the X sub-image features, based on the difference between the image features corresponding to the T video segments in the sub-image feature, feature shifting processing is performed on the image features corresponding to the T video segments in the sub-image feature to obtain the target sub-image feature corresponding to the sub-image feature; the feature shifting processing method is adapted to the degree of attention of the feature channels in the sub-image feature to motion information; The target video features are obtained by concatenating the target sub-image features corresponding to the X sub-image features; Based on the target video features, determine the action category of the moving target in the video sequence; The step of performing feature shifting processing on the image features corresponding to the T video segments in the sub-image features based on the difference between the image features corresponding to the T video segments in the sub-image features to obtain the target sub-image features corresponding to the sub-image features includes: Based on the difference between the image feature corresponding to the (t+1)th video segment and the image feature corresponding to the tth video segment in the sub-image features, the inter-segment difference feature corresponding to the sub-image feature X is determined through the inter-segment difference feature expression corresponding to the sub-image feature X; where t is an integer greater than or equal to 1 and less than T; based on the inter-segment difference feature corresponding to the sub-image feature, the channel offset coefficient corresponding to the sub-image feature is determined; The sub-image features are adjusted based on the channel offset coefficient to obtain reference sub-image features; Along the feature offset direction assigned to the sub-image features, feature offset processing is performed on the image features corresponding to each of the T video segments in the reference sub-image features on the M feature channels to obtain the reference sub-image features after feature offset processing. Based on the reference sub-image features after feature offset processing, the reference sub-image features, and the sub-image features, the target sub-image features are determined to perform feature exchange and feature fusion in the time dimension. The inter-segment difference features characterize the amount of motion information contained in the sub-image features.

2. The method according to claim 1, characterized in that, The step of determining the channel offset coefficient corresponding to the sub-image feature based on the inter-segment difference feature corresponding to the sub-image feature includes: The objective function determines the channel offset coefficient corresponding to the segment difference feature of the sub-image feature; the channel offset coefficient is proportional to the segment difference feature, and the channel offset coefficient is greater than or equal to 0 and less than or equal to 1.

3. The method according to claim 1, characterized in that, The feature extraction network includes N feature extraction layers, where N is an integer greater than 1; the method further includes: Through the i-th feature extraction layer, feature extraction processing is performed on the (i-1)-th target image features corresponding to each of the T video segments to obtain the i-th image features corresponding to the T video segments; where i is an integer greater than or equal to 1 and less than N, and when i equals 1, the (i-1)-th target image feature is the T video segments; the i-th image features corresponding to the T video segments include the C feature channels; The i-th image features corresponding to the T video segments are divided into X i-th sub-image features along the directions of the C feature channels; the i-th sub-image features include the M feature channels; For each i-th sub-image feature among the X i-th sub-image features, based on the difference between the i-th image features corresponding to the T video segments in the i-th sub-image features, feature shifting processing is performed on the i-th image features corresponding to the T video segments in the i-th sub-image features to obtain the i-th target sub-image features corresponding to the i-th sub-image. For each of the T video segments, the image features belonging to that video segment in the i-th target sub-image features corresponding to the i-th sub-image features of the X i-th sub-image features are stitched together to obtain the i-th target image feature corresponding to that video segment. The i-th target image features corresponding to each of the T video segments are input into the (i+1)-th feature extraction layer.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the target user's interest in action categories; If the action category of the target in the video sequence belongs to the action category of interest, the video sequence is recommended to the target user.

5. The method according to claim 1, characterized in that, The method further includes: Get the set of sensitive action categories; If the action category of the target running in the video sequence belongs to the set of sensitive action categories, the release of the video sequence is intercepted.

6. A motion recognition device, characterized in that, The device includes: The video acquisition module is used to acquire the video sequence to be identified; A video segmentation module is used to divide the video sequence into T video segments, where T is an integer greater than 1; The feature extraction module is used to perform feature extraction processing on the T video segments through a feature extraction network to obtain the image features corresponding to each of the T video segments. The feature fusion module is used to perform feature fusion processing on the image features corresponding to the T video segments based on the difference between the image features corresponding to each of the T video segments, so as to obtain the target video features; An action recognition module is used to determine the action category of a moving target in the video sequence based on the characteristics of the target video. The feature arrangement module is used to arrange the image features corresponding to each of the T video segments according to the time order of the T video segments, so as to obtain the image features corresponding to the T video segments; the image features corresponding to the T video segments include C feature channels, where C is an integer greater than 1; The feature segmentation module is used to divide the image features corresponding to the T video segments into X sub-image features in the directions of the C feature channels, where X is an integer greater than 1; the sub-image features include M feature channels, where M is a positive integer less than C; The feature fusion module includes: The feature offset unit is used to perform feature offset processing on the image features corresponding to the T video segments in the sub-image features for each of the X sub-image features, based on the difference between the image features corresponding to the T video segments in the sub-image features, to obtain the target sub-image feature corresponding to the sub-image feature; the feature offset processing method is adapted to the degree of attention of the feature channels in the sub-image features to motion information; The feature stitching unit is used to stitch together the target sub-image features corresponding to the X sub-image features to obtain the target video features; The feature offset unit is specifically used for: Based on the difference between the image features corresponding to the (t+1)th video segment and the image features corresponding to the tth video segment in the sub-image features, the inter-segment difference feature corresponding to the sub-image feature X is determined through the inter-segment difference feature expression corresponding to the sub-image feature X; where t is an integer greater than or equal to 1 and less than T; based on the inter-segment difference feature corresponding to the sub-image feature, the channel offset coefficient corresponding to the sub-image feature is determined; the sub-image feature is adjusted based on the channel offset coefficient to obtain a reference sub-image feature; along the feature offset direction allocated to the sub-image feature, feature offset processing is performed on the image features corresponding to the T video segments in the reference sub-image feature on the M feature channels to obtain the reference sub-image feature after feature offset processing; based on the reference sub-image feature after feature offset processing, the reference sub-image feature, and the sub-image feature, the target sub-image feature is determined to perform feature exchange and feature fusion in the time dimension, where the inter-segment difference feature characterizes the amount of motion information contained in the sub-image feature.

7. A device, characterized in that, The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the action recognition method according to any one of claims 1 to 5 according to the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the action recognition method according to any one of claims 1 to 5.

Images