Method executed by electronic equipment, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the fields of computer vision, artificial intelligence and the like, and the method comprises the steps: obtaining a behavior subject in a to-be-processed video and a related event thereof through employing an artificial intelligence AI network; providing a behavior subject selection interface based on the acquired behavior subject; receiving a behavior subject selected by a user through the selection interface; events related to the user-selected behavior subject are provided. Based on the method provided by the embodiment of the invention, the behavior subject and the related events in the video can be identified, and the actual application requirement can be better met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and computer vision technology. Specifically, the present application relates to a method executed by an electronic device, an electronic device, and a storage medium. Background Art

[0002] With the increasing popularity and replacement frequency of electronic devices such as mobile phones, users have higher and higher demands on the functions of electronic devices. Video processing capability is one of them. How to improve the video processing capability of electronic devices to better meet practical application needs is a continuous goal in this field. Summary of the invention

[0003] The purpose of the embodiments of the present application is to improve the recognition accuracy of event clips in a video.

[0004] According to one aspect of an embodiment of the present application, a method performed by an electronic device is provided, the method comprising:

[0005] Get the video to be processed;

[0006] Adopt AI (Artificial Intelligence) network to obtain the behavior subject and related events in the video to be processed;

[0007] Based on the acquired behavior subject, provide a behavior subject selection interface;

[0008] Receiving a behavior subject selected by a user through the selection interface;

[0009] Provides events related to the subject of the behavior selected by the user.

[0010] According to another aspect of an embodiment of the present application, a method performed by an electronic device is provided, the method comprising:

[0011] Using an AI network, extracting semantic features of each frame of the video to be processed in the video to be processed, wherein the semantic features include spatial semantic features and temporal semantic features;

[0012] Determining an event in the video to be processed based on the extracted semantic features of each video frame to be processed;

[0013] Provide events in the video to be processed.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the method provided by any optional embodiment of the present application.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method provided in any embodiment of the present application is implemented.

[0017] Compared with the existing related technologies, the solution provided by the embodiment of the present application can more accurately identify event clips in the video and can better meet the actual needs. The optional implementation methods of the present application and the corresponding effects will be described in detail below in conjunction with the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0019] Figure 1 A flowchart of a method performed by an electronic device provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the principle of a convolution operation provided in an embodiment of the present application;

[0021] Figure 3a A schematic diagram of an ADT (Adjacent Dazzle Transformer) network provided in an embodiment of the present application;

[0022] Figure 3b A schematic diagram of the implementation principle of a specific convolution operation provided in an embodiment of the present application;

[0023] Figure 4 A flowchart of a method performed by an electronic device provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a video processing method and an AI network provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of another video processing method provided in an embodiment of the present application;

[0028] Figure 9a A schematic diagram of generating an event fragment provided in an embodiment of the present application;

[0029] Figure 9b A schematic diagram of presenting event fragments to a user provided in an embodiment of the present application;

[0030] Fig.10 A schematic diagram of a video processing method and an AI network provided in an embodiment of the present application;

[0031] Fig.11 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0032] Fig.12 A schematic diagram of a visualization effect of a neural network provided in an embodiment of the present application;

[0033] Fig.13 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0034] Fig.14 A schematic diagram of the structure of a feature extraction module in a neural network provided in an embodiment of the present application;

[0035] Fig.15a A schematic diagram of a feature extraction principle of a neural network provided in an embodiment of the present application;

[0036] Fig.15b A schematic diagram of the effect of an attention mechanism provided in an embodiment of the present application;

[0037] Fig.15c A schematic diagram of a feature extraction process provided in an embodiment of the present application;

[0038] Fig.16a A schematic diagram of the principle of a convolution operation provided in an embodiment of the present application;

[0039] Fig.16b A schematic diagram of the effect of a convolution operation provided in an embodiment of the present application;

[0040] Fig.16c A schematic diagram showing the comparative effects of different convolution operations provided in the embodiments of the present application;

[0041] Fig.17a A schematic diagram of the principle and effect of offsetting a block in a feature graph provided in an embodiment of the present application;

[0042] Fig.17b A schematic diagram of a feature extraction process of a neural network provided in an embodiment of the present application;

[0043] Fig.18 A schematic diagram of the effect of a neural network provided in an embodiment of the present application;

[0044] Fig.19 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0045] Fig.20a A schematic diagram of a feature extraction process of a neural network provided in an embodiment of the present application;

[0046] Fig.20b A schematic diagram of a feature extraction principle of a neural network provided in an embodiment of the present application;

[0047] Fig.21 A schematic diagram of a feature processing process of a neural network provided in an embodiment of the present application;

[0048] Fig. 22 A schematic diagram for comparing a high-quality behavior subject and a low-quality behavior subject provided in an embodiment of the present application;

[0049] Fig.23a and Figure 23b A schematic diagram of the effect of a neural network provided in an embodiment of the present application;

[0050] Fig.24a and Figure 24b A schematic diagram of a feature extraction process of a neural network provided in an embodiment of the present application;

[0051] Fig.25 A schematic diagram of the aggregation results of the behavioral subject provided in the embodiment of the present application;

[0052] Fig.26 A schematic diagram of a process for generating an event fragment provided in an embodiment of the present application;

[0053] Fig. 27 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0054] Fig.28a A schematic diagram of a user interface provided in an embodiment of the present application;

[0055] Fig.28b A schematic diagram of another user interface provided in an embodiment of the present application;

[0056] Fig.29 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present application as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present application. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0058] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present application. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present application is provided for illustration purposes only and not to limit the purpose of the present application as defined in the appended claims and their equivalents.

[0059] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to a "component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may be connected through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0060] The term "include" or "may include" refers to the presence of corresponding disclosed functions, operations, or components that can be used in various embodiments of the present application, rather than limiting the presence of one or more additional functions, operations, or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0061] The term "or" used in various embodiments of the present application includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, and may also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0062] Unless defined differently, all terms (including technical terms or scientific terms) used in this application have the same meanings understood by those skilled in the art of this application. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally, unless clearly defined in this application.

[0063] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0064] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0065] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0066] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0067] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0068] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0069] Among them, the method performed by the electronic device provided in the embodiment of the present application can be implemented using an AI network, and the video to be processed (image sequence) can be used as the input data of the AI ​​network, and the behavior subject appearing in the video and the event fragments associated with the behavior subject can be identified through the AI ​​network. Among them, the AI ​​network can also be called an AI model, a neural network or a neural network model. The AI ​​network is obtained through training, and the network parameters or model parameters refer to the network parameters of the AI ​​network obtained through training and learning, such as the weights and biases of the neural network. "Obtained through training" here means that a basic artificial intelligence model is trained with multiple training data through a training algorithm to obtain a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose).

[0070] The method provided in the present application may relate to the field of visual understanding of artificial intelligence technology, which is a technology used to recognize and process things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / positioning or image enhancement.

[0071] In real life, as short videos become more and more popular on various social media platforms, people like to share events related to specific objects that users prefer, and identifying events in long videos (the long and short in long videos and short videos are relative) is one of the needs in people's lives. In the current related technologies, although some schemes can also identify events in videos, the recognition results are not ideal, especially for events involving fast action behaviors. The current related technologies are difficult to identify fast motion behaviors, and cannot identify events related to specific objects from videos. If a user wants to find an event related to a specific object in a long video, the current related technologies cannot achieve this. Therefore, there are at least the following problems in the existing related technologies that need to be improved:

[0072] Problem A: It is impossible to accurately identify events in the video, and for events that include fast-moving behaviors, it is even more impossible to accurately provide users with events that include complete behaviors.

[0073] Problem B: It is impossible to identify the behavior object / behavior subject in the video, and it is impossible to identify and provide users with related events of a specific behavior object.

[0074] The purpose of the solution provided in the embodiment of the present application is to improve or solve one of the problems existing in the existing related technologies. Optionally, the method provided in the embodiment of the present application can solve the problem that the existing technology cannot identify the subject of the behavior in the video and its associated event clips or the problem that the event clip recognition accuracy is not enough, especially for videos containing fast-moving behaviors.

[0075] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the implementation methods in the following optional embodiments can be implemented separately, and in the event of conflicts between the implementation methods, they can refer to, learn from or combine with each other, or one or some steps in different implementation methods can be replaced with each other, and the same terms, similar features and similar implementation steps in different implementation methods will not be described repeatedly.

[0076] In an embodiment of the present application, a method executed by an electronic device is provided. The electronic device may be any electronic device, such as a terminal device or a server. Figure 1 As shown, the method includes:

[0077] Step S110: Obtain the video to be processed;

[0078] Step S120: Use the AI ​​network to obtain the behavior subject in the video to be processed.

[0079] Optionally, the above step S120 can be implemented as: using an AI network to obtain the behavior subject and its related events in the video to be processed.

[0080] like Figure 1 As shown, the method may also include:

[0081] Step S130: Based on the acquired behavior subject, a behavior subject selection interface is provided;

[0082] Step S140: receiving the behavior subject selected by the user through the selection interface, and providing events related to the behavior subject selected by the user.

[0083] In the embodiment of the present application, the video to be processed is a video including multiple video frames to be processed. Among them, the source of the video to be processed is not limited in the embodiment of the present application, and the video to be processed can be an original video or a video obtained by preprocessing the original video. The original video can be any video, which can be a video collected by the video acquisition device of the terminal device itself (such as the camera of a smart phone), a video stored in an electronic device or a video to be transmitted, or a video downloaded from the Internet or a video sent to the electronic device by other electronic devices.

[0084] Optionally, the video to be processed may be an image sequence consisting of some video frames in the original video. For example, a video image sequence may be obtained by extracting frames from the original video at a set time interval, or an image sequence may be obtained by sampling the original video at a preset sampling rate. The video to be processed includes a plurality of continuous video frames. It is understandable that if the video to be processed is a video obtained by sampling / extracting frames, the continuity of video frames means that the video frames obtained by sampling / extracting frames are relatively continuous. For example, the image sequence in the original video has a total of 100 frames, namely the 1st frame, the 2nd frame, ... the 100th frame. If one frame is extracted every two frames, the video to be processed includes the 1st frame, the 4th frame, the 7th frame, etc. The 1st frame and the 4th frame of the video to be processed are continuous, the 4th frame and the 7th frame are continuous, and the 1st frame and the 7th frame are directly adjacent to the 4th frame.

[0085] In the embodiment of the present application, the subject may also be referred to as an object or an object, and the behavior subject refers to the subject that has performed a specific behavior or a preset behavior, that is, the behavior subject may include but is not limited to an object that has moved / moved in the video. For example, for a video containing a scene of athletes playing football, the athlete is a behavior subject in the video. For another example, a bright moon hanging in the sky in the video frame can also be a behavior subject even if it may not move.

[0086] An event clip in a video, also referred to as an event in a video, refers to a video clip in which an event occurs in the video. It can be understood as an event clip in the video that contains preset behaviors. Identifying an event in a video means identifying whether a certain preset behavior or certain preset behaviors have occurred in the video. A video clip consisting of video frames in which a preset behavior occurs is an event. Among them, the preset behavior can be pre-set according to actual application requirements or application scenarios. For example, in a sports-related scenario, the preset behaviors may include high jump, long jump, shooting, and football kicking. Correspondingly, an event related to a behavior subject can be understood as an event clip corresponding to a subject that has undergone a preset behavior.

[0087] The method provided in the embodiment of the present application can use the trained AI network to identify the subject of behavior in the video to be processed, and can find events related to the subject of behavior. Based on this, the identified subject of behavior can be provided to the user, and the user can select the subject of behavior that the user is interested in. Based on the user's selection, the electronic device can provide the user with the event of the subject of behavior selected by the user. Based on the solution provided in the embodiment of the present application, even if there is a fast-moving subject in the video, the solution can accurately identify the subject and its related behaviors / events.

[0088] Optionally, in step S120, obtaining the subject of behavior and related events in the video to be processed includes:

[0089] Extracting semantic features of the video to be processed, wherein the semantic features of the video to be processed include semantic features of the subject in each frame of the video to be processed, and the semantic features of each frame of the video to be processed include spatial semantic features and temporal semantic features;

[0090] Based on the extracted semantic features, the behavioral subjects and their related events in the video to be processed are determined.

[0091] Among them, the video frames to be processed are frames in the video frames to be processed, and the video frames to be processed can be all frames in the original video, or they can be some video frames obtained by frame extraction or sampling in the original video. Optionally, for each video frame to be processed in the video to be processed (hereinafter directly referred to as a video frame), the semantic features of the video frame can also be referred to as the semantic feature map, spatiotemporal features or spatiotemporal feature map of the video frame. Among them, the semantic features of any video frame to be processed can be obtained by extracting features from the video frame and adjacent frames using an AI network. Among them, the adjacent frames of any video frame to be processed may include the preceding frame of the video frame, and the preceding frame at least includes the previous video frame of the video frame, that is, the preceding frame can be one frame or multiple frames, such as the two frames before the video frame.

[0092] For any video frame to be processed, based on the video frame, the spatial features of the image content of the video frame itself can be extracted, and because the video frame and its preceding frame are multiple frames that are continuous in time, the temporal features of the video frame can be extracted based on the interaction between the video frame and its preceding frame. Therefore, based on the video frame and its preceding frame, the spatiotemporal features of the video frame can be learned through the AI ​​network, that is, the spatiotemporal features that contain both spatial feature information and temporal feature information. Therefore, based on the spatiotemporal features of each video frame in the video to be processed, the event segments in the video can be more accurately identified.

[0093] For a video, the more the scene information changes (image content changes) in each frame image in the video, the greater the probability that an event occurs in the image. That is to say, for an event segment, the change in image content between video frames in the segment is relatively large. For example, for an event segment containing the behavior of long jump, long jump is a fast motion behavior. In different video frames, the position of the long jumper is constantly changing. For adjacent frames in the event segment, due to the movement of the long jumper, the same position across frames is not within the same object range. Therefore, if only the image features of each frame in the video itself are obtained, or the time features at the same position of different video frames are obtained, it is impossible to accurately identify and locate the event in the video (such as the above-mentioned long jump behavior). The solution provided by the embodiment of the present application takes into account the above-mentioned various factors, adopts an AI network obtained through training, and identifies events in the video by obtaining the semantic features of each video frame with both temporal feature information and spatial feature information, thereby improving the accuracy of the recognition result. Even if the video contains fast-moving behavior, based on the interaction of the video frame and its preceding frame, the feature learning of the behavior can be learned from it.

[0094] The embodiments of the present application do not make any sole limitation on the network structure of the AI ​​network used in the method provided in the embodiments of the present application.

[0095] As an optional solution, the semantic features of the video frame to be processed can be extracted in the following way:

[0096] For each video frame to be processed in the video to be processed, extracting a first semantic feature of the video frame and a second semantic feature of an adjacent frame based on a convolution module;

[0097] Determine semantically related patches in the video frame and semantically related patches in adjacent frames based on a first semantic feature of the video frame and a second semantic feature of an adjacent frame;

[0098] Extracting spatial semantic features of the subject in the video frame from semantically related patches in the video frame, and extracting temporal semantic features of the subject in the video frame from semantically related patches in adjacent frames;

[0099] The extracted spatial semantic features and temporal semantic features are fused to obtain the semantic features of the video frame.

[0100] In the embodiment of the present application, the blocks in the video frame can also be called patches, and one patch corresponds to an image block of the video frame. For the video frame to be processed in the video, if the video frame contains a subject, then the patches occupied by the subject in the video frame should be semantically related. As for the behavioral subject that may exist in the video, that is, the behavioral subject to be identified, since the behavioral subject is a subject that has performed a specific behavior, if a video frame and its adjacent frames contain the same subject, then the patches where the subject is located in a video frame and the patches occupied by the subject in the adjacent frames of the video frame should also be semantically related. Therefore, based on the semantically related patches in the video frame, the spatial semantic features of the subject in the video frame can be extracted, and based on the semantically related patches in the adjacent frames, the temporal semantic features of the subject in the adjacent frames of the video frame can be extracted. Then, the semantic features of the video frame can be obtained by fusing the spatial semantic features extracted from the video frame and the temporal semantic features extracted from the adjacent frames of the video frame.

[0101] The specific neural network structure of the convolution module used to extract the first semantic feature of the video frame and the second semantic feature of the adjacent frame is not limited in the present embodiment. As an optional solution, for each video frame to be processed, the semantic feature of the video frame is extracted based on the convolution module, including:

[0102] Use the first convolutional layer to perform convolution on the video frame;

[0103] Spatially rearrange the features of each channel among the features extracted by the first convolutional layer;

[0104] The second convolutional layer is used to perform convolution processing on the rearranged features, and the features of each space in the features extracted by the second convolutional layer are rearranged through channels to obtain the semantic features of the video frame.

[0105] It should be noted that, in the embodiments of the present application, the use of a neural network to extract features from video frames or patches in video frames may be to directly extract features from video frames using a neural network, or to extract features from features / feature maps of video frames using a neural network structure. For example, in the above-mentioned use of the first convolutional layer to perform convolution processing on video frames, the input of the first convolutional layer may be a video frame or a feature of a video frame, such as a feature map of a video frame obtained by extracting features using a commonly used convolutional network, and the first convolutional layer performs further feature extraction based on the feature map.

[0106] In this optional scheme of the present application, a new convolutional feature extraction scheme is provided. For the sake of ease of description, this feature extraction scheme is referred to as a specific convolution operation below. By adopting this convolution operation, each feature point in the semantic features of the extracted video frame can obtain a global receptive field and learn the global information of the video frame. Therefore, based on the feature map extracted by the convolution operation, the semantic features of the video frame with better feature expression ability can be obtained, which provides a better basis for more accurately identifying the subject in the video and its related events.

[0107] As an example, Figure 3b A schematic diagram of an optional implementation of a specific convolution operation proposed in an embodiment of the present application is shown, such as Figure 3b As shown in , the input feature is a video frame or a feature of a video frame. Assume that the input is a feature map of H*W*C1, where H and W represent the height and width of the feature map, respectively, and C1 represents the number of channels of the feature map. The first convolution operation (first convolution layer) can adopt a traditional convolution operation, and the size of the convolution kernel can be greater than 1. For any feature point in the input feature map, the feature point can learn the local information of the feature map through the first convolution operation. Optionally, the size of the feature map obtained by the first convolution operation remains unchanged. The first convolution operation outputs a feature map of H*W*C2, and C1 and C2 can be equal or unequal. After that, the first feature rearrangement operation can be performed on the output feature map of the first convolution operation. Specifically, the channel features of the output feature map can be rearranged in space (spatial rearrangement) to obtain a feature map of 1*C2*H*W. The number of channels of the feature map after the reshoot is H*W, and 1*C2 is the size of the feature map of one channel. That is to say, the feature map of one channel after the rearrangement is a feature map composed of the feature values ​​of a feature point in the C2 channel before the rearrangement. Afterwards, a 1*1 convolution operation can be performed to allow each feature point to learn the global receptive field of the feature map, and a new feature map of 1*C2*H*W of the fused global information can be obtained. Then, a second feature rearrangement operation (channel rearrangement) is performed on the new feature map to obtain an output feature map of H*W*C2, in which each feature point has learned the local information and global information in the feature map.

[0108] For each video frame to be processed in the video to be processed, the above-mentioned specific convolution operation can be used to obtain the semantic features of each frame, such as the first semantic features of any video frame to be processed and the second semantic features of its adjacent frames. After obtaining the semantic features of each video frame to be processed, further feature extraction can be performed based on the first semantic features of the video frame and the second semantic features of its adjacent frames to obtain the semantic features of the video frame including spatial semantic features and temporal semantic features. Optionally, based on the first semantic features of the video frame and the second semantic features of its adjacent frames, semantically related block patches in the video frame and semantically related patches in adjacent frames can be determined, and then based on the determined semantically related patches, the corresponding spatial semantic features and temporal semantic features of the subject in the video frame can be extracted.

[0109] Optionally, determining semantically related patches in the video frame and semantically related patches in the adjacent frames based on the first semantic feature of the video frame and the second semantic feature of the adjacent frames includes:

[0110] fusing the first semantic feature with the second semantic feature;

[0111] Based on the fused features, in the video frame and adjacent frames, the spatial position offset information of other patches semantically related to each patch of the video frame relative to the patch is determined;

[0112] Based on the spatial position offset information, semantically related patches in a video frame and semantically related patches in adjacent frames are determined.

[0113] Among them, the specific method of fusing the first semantic feature and the second semantic feature is not limited in the embodiment of the present application. Optionally, the first semantic feature and the second semantic feature can be spliced ​​to obtain a fused feature. Since the fused feature contains the semantic features of at least two adjacent video frames (the current video frame and its adjacent frames), for the current video frame, based on the fused feature, the patch semantically related to each patch in the current video frame can be more accurately determined. The semantically related patch includes the semantically related patch in the current video frame and the semantically related patch in the adjacent frame. Specifically, based on the fused feature, the position offset information of the patch semantically related to each patch relative to the patch can be determined, and based on the position offset information, it can be known which patches are semantically related.

[0114] By adopting the above optional scheme, for each video frame to be processed, other patches related to each patch of the video frame can be accurately found on the video frame, so that more accurate subject contour information of the subject that may exist in the video frame can be obtained based on these patches, and by accurately finding other patches related to each patch on the adjacent frames of the video frame, more accurate relative position information of the subject in the video frame can be obtained based on these patches. Therefore, based on the semantically related patches in the current video frame to be processed and its adjacent frames, the spatial semantic features and temporal semantic features of the subject in the current video frame can be better learned, wherein the temporal semantic features integrate the relative position information of the position of the subject in the current video frame and the position of the subject in its adjacent frames, so the motion information of the subject can be better learned.

[0115] After obtaining the semantic features of each video frame to be processed that integrates the spatial semantic features and the temporal semantic features, the behavior subject appearing in the video frame can be identified based on the semantic features of each video frame, and then the related events of the behavior subject can be determined. Optionally, based on the extracted semantic features of the video to be processed, determining the behavior subject and its related events in the video to be processed includes:

[0116] Based on the semantic features of each video frame to be processed, an object mask module is used to determine the area where the subject is located in each video frame to be processed;

[0117] For each video frame to be processed, based on the semantic features of the video frame and the area where the subject is located in the video frame, determine the regional features of the area where the subject of the video frame is located;

[0118] Based on the regional features of the area where the subject is located in each video frame to be processed, the behavior subject and related events in the video to be processed are determined.

[0119] Among them, the object mask module can perform semantic segmentation on the video frame based on the semantic features of the video frame, and segment the object area (the area where the subject is located) and the non-object area in the video frame. The specific neural network structure of the object mask module is not limited in this embodiment of the application. The object mask module can theoretically be any image segmentation network. Based on the semantic features of the video frame, the image segmentation network can be used to obtain a mask feature map of the video frame. The mask feature map is a binary feature map. The feature values ​​of the object area in the mask feature map are different from the feature values ​​of the non-object area. For example, the feature values ​​of the object area are all 1, and the feature values ​​of the non-object area are all 0. Based on the mask feature map, the object area and the non-object area in the video frame can be known.

[0120] Optionally, in order to more accurately determine the object area in the video frame, the above-mentioned method of using the object mask module based on each video frame to be processed to determine the area where the subject is located in each video frame to be processed may include:

[0121] For each video frame to be processed, the semantic features of the video frame are fused with the semantic features of adjacent frames;

[0122] Use the object mask module to segment the fused features and obtain the area where the subject is located in the video frame.

[0123] Since the purpose of video processing is to identify the subject appearing in the video, and the subject is a moving subject, the position and / or form of the subject in different video frames are different. Therefore, in order to more accurately determine the position of the subject in the video frame, object segmentation can be performed based on the semantic features of the video frame and the semantic features of its adjacent frames, and the area where the subject is located in the video frame is obtained based on the segmentation result. Optionally, the semantic features of the video frame and the semantic features of its adjacent frames can be spliced, and the spliced ​​features are used as the input of the object mask module, and the object mask module outputs the object segmentation result of the video frame, that is, the mask feature map.

[0124] After the object area in the video frame is determined by the object mask module, the regional features of the object area can be obtained based on the semantic features of the video frame. For example, the semantic features of the video frame are used to fill the pixel values ​​of the object area in the mask feature map. The pixel values ​​of the object area in the filled mask feature map are the same as the pixel values ​​of the corresponding area in the semantic features of the video frame, and the pixel values ​​of the non-object area are 0.

[0125] After determining the regional features of the region where the subject is located in each frame of the video to be processed, the behavior subject and related events in the video to be processed can be determined based on the regional features of the region where the subject is located.

[0126] Optionally, based on the regional features of the region where the subject is located in each frame of the video to be processed, determining the subject of behavior and related events in the video to be processed includes:

[0127] For each video frame to be processed, based on the regional features of the area where the subject is located in the video frame, the subject features corresponding to the video frame are obtained;

[0128] Based on the subject features of the video frame, a subject recognition model is used to determine whether the video frame contains a behavior subject;

[0129] Based on the determined behavior subject features, the behavior subject and related events in the video to be processed are obtained, wherein the behavior subject features are subject features of the video frame containing the behavior subject.

[0130] Among them, the subject recognition model is used to determine whether there is a behavioral subject in the video frame. The subject feature corresponding to a video frame to be processed can be the regional feature of the area where the subject is located in the video frame, or it can be a feature obtained after further processing the regional feature. For example, the regional feature of the area where the subject is located can be adjusted to a feature of a corresponding size according to a set feature size, and the feature is used as the subject feature. Since the subject feature represents the relevant information of the subject in the video frame, and the subject feature is a fusion of the spatial semantic features of the subject in the video frame and its temporal semantic features in the adjacent frames, therefore, based on the subject feature corresponding to the video frame, the subject recognition model is used to determine whether there is a behavioral subject in the video frame.

[0131] The embodiments of the present application do not make any sole limitation on the specific structure of the subject recognition model. Optionally, the subject recognition model may be a classification model, and the presence of a behavioral subject in a video frame and the absence of a behavioral subject in a video frame are two different classification results.

[0132] Optionally, in order to improve the accuracy of the behavior subject recognition result, the subject features corresponding to the video frame are obtained based on the regional features of the region where the subject is located in the video frame, including:

[0133] The regional features of the area where the subject is located in the video frame are fused with the semantic features of the video frame to obtain the target features of the video frame;

[0134] The target features of the video frame and the regional features of the area where the subject is located in the video frame are fused to obtain the subject features of the video frame.

[0135] Optionally, the subject feature of the video frame may be obtained by fusing the semantic feature of the video frame with the regional feature of the region where the subject is located in the video frame.

[0136] The method of fusing the features of the video frame (semantic features or target features) with the regional features of the area where the subject of the video frame is located to obtain the subject features can make the obtained subject features contain both the local features of the object area in the video frame and the global features of the video frame. The subject features obtained based on this scheme can more accurately determine whether the video frame contains the behavioral subject.

[0137] The specific method of fusing the video frame features and the regional features is not limited in the present embodiment. Optionally, the video frame features and the regional features can be adjusted to features of preset sizes, and the adjusted video frame features and regional features can be spliced ​​to obtain the main features.

[0138] Optionally, the regional features of the region where the subject is located in the video frame are fused with the spatiotemporal features of the video frame to obtain the target features of the video frame, including:

[0139] The regional features of the area where the subject is located in the video frame and the spatiotemporal features of the adjacent frames are fused, and based on the fused features, the target regional features of the subject in the video frame are extracted from the area where the subject is located in the video frame;

[0140] The target region features and the spatiotemporal features of the video frame are fused to obtain the target features of the video frame.

[0141] In this optional solution, based on the semantic features of the video frame, the area where the subject is located in the video frame determined by the object mask module can be understood as a coarse-grained object area recognition result. In order to further improve the accuracy of the determined object area, the regional features of the object area determined by subject segmentation and the semantic features of the adjacent frames of the video frame can be fused. Based on the fused features, the object contour can be further calculated in the segmented object area to obtain the target area features of the object in the video frame. Compared with the regional features obtained by segmentation, the target area features are features with more fine-grained object contours and absolute positions. By fusing the obtained target area features with the semantic features of the video frame, the target features of the video frame can be obtained, and the target features can help better identify the subject in the video. Optionally, as described above, the features of the video frame and the regional features of the area where the subject is located in the video frame can be fused to obtain the subject features of the video frame.

[0142] After obtaining the subject features corresponding to each frame of the video to be processed, the subject recognition module can be used based on the subject features of the video frame to identify the video frames containing the subject, and then the subject in the video can be identified based on the identified subject features. Based on the identification result of the subject, events related to the subject can be obtained. Optionally, based on the determined subject features, the subject in the video to be processed and its related events are obtained, including:

[0143] Aggregating the determined behavior subject characteristics to obtain at least one aggregation result;

[0144] Based on each video frame corresponding to each aggregation result, the behavior subject and its related events corresponding to each aggregation result are obtained.

[0145] Since the subjects included in different video frames may be the same or different, in order to obtain the events related to each subject, after determining the subject features, all subject features of the same subject can be found through feature aggregation. The video frames corresponding to all subject features in a specific aggregation result can be considered to contain the same subject. Based on the video frames corresponding to all subject features in the aggregation result, the subject corresponding to the aggregation result and its related events can be obtained.

[0146] The specific manner of aggregating the determined behavior subject features is not limited in the embodiments of the present application. In theory, any feature clustering scheme can be adopted. For example, the behavior subject features can be clustered based on the similarity between the behavior subject features.

[0147] As an optional solution, the determined behavior subject characteristics can be aggregated in the following way:

[0148] For each behavioral subject feature, determining at least one similar subject feature of the behavioral subject feature from the determined behavioral subject features;

[0149] Based on the characteristics of the behavior subject and its similar subject characteristics, the fusion characteristics of the behavior subject are extracted;

[0150] Aggregate the fusion features corresponding to the determined behavior subject features.

[0151] In the optional scheme provided by the present application, similar subject features of each behavioral subject feature can be found based on the similarity between the features. Optionally, a similar subject feature of a behavioral subject feature may refer to a behavioral subject feature whose similarity with the behavioral subject feature is greater than a set threshold. The specific method for calculating the feature similarity can be selected according to actual needs, and the similarity between the features can be obtained by methods including but not limited to calculating the cosine similarity, Euclidean distance, etc. between the features.

[0152] Among them, for each subject feature, the subject feature and its similar subject features can be considered as the subject features of the same subject corresponding to different angles and different scenes. Therefore, the semantic information of the subject's multi-angle scenes can be learned from the subject feature and its similar subject features, and the fusion feature of the subject with better feature expression ability can be obtained. Aggregation based on fusion features can further improve the accuracy of the aggregation results.

[0153] This application does not make any specific limitation on the specific method of extracting the fusion features of the behavior subject from the behavior subject features and the features of similar subjects. Optionally, the behavior subject features and the features of similar subjects can be added, or the average is calculated after addition, or the weighted sum is performed to obtain the fusion features.

[0154] As an optional solution provided by the present application, based on the characteristics of the behavior subject and the characteristics of its similar subjects, the fusion characteristics of the behavior subject are extracted, including:

[0155] Fusing similar subject features of the behavior subject features, and extracting features from the fused features using at least two different feature extraction methods to obtain at least two fused subject features;

[0156] Based on the correlation between the behavior subject characteristics and the characteristics of each fusion subject, the weight corresponding to each fusion subject characteristic is obtained;

[0157] The weights corresponding to the features of each fusion subject are used to perform weighted fusion on the features of the fusion subject to obtain the fusion features of the behavior subject.

[0158] Among them, for each behavior subject feature, the behavior subject feature and its similar subject features can be spliced, and at least two different feature extraction methods can be used to extract features from the spliced ​​features to obtain at least two fused subject features. By using different feature extraction methods to extract features from the spliced ​​vector, multiple features (fused subject features) corresponding to different feature spaces and containing information of different dimensions can be obtained. Based on the correlation between the behavior subject feature and each of the multiple features, the multiple features can be weightedly fused to obtain a fused feature corresponding to the behavior subject feature that integrates multi-angle and multi-scene information of the behavior subject.

[0159] Optionally, an attention mechanism may be used to realize the fusion of the above-mentioned multiple features, wherein the behavior subject feature may be used as a query vector (query) of the attention mechanism, and the above-mentioned multiple features corresponding to the behavior subject feature may be concatenated as the key (key) of the attention mechanism, and the weight corresponding to each feature of the multiple features may be calculated, and based on the weights corresponding to the multiple features, the multiple features may be weighted summed to obtain a fused feature corresponding to the behavior subject feature, and the fused feature may be used as a target subject feature corresponding to the behavior subject feature, and feature aggregation may be performed through the target subject features corresponding to all the behavior subject features to obtain at least one aggregation result.

[0160] Optionally, for each aggregation result, the behavior subject in the video frame corresponding to any behavior subject feature in the aggregation result can be taken as the behavior subject corresponding to the aggregation result, that is, the representative behavior subject, and the video frames corresponding to all the behavior subjects in the aggregation result (or after some processing on these video frames) are sorted according to the sequence of the video frames in the video to obtain events related to the behavior subject corresponding to the aggregation result.

[0161] As an optional solution, based on each video frame corresponding to each aggregation result, the behavior subject and related events corresponding to each aggregation result are obtained, including:

[0162] For each aggregation result, based on each behavior subject feature in the aggregation result, determine the quality of the behavior subject in the video frame corresponding to each behavior subject feature;

[0163] Based on the quality of the behavior subject in the video frame corresponding to the aggregation result, the behavior subject in the video to be processed is determined, and based on each video frame corresponding to the aggregation result, the related event of the behavior subject is determined.

[0164] Optionally, for each aggregation result, after determining the subject quality corresponding to each subject in the aggregation result, the subject in the video frame corresponding to the subject feature with the highest quality can be used as the representative subject corresponding to the aggregation result, and the video frames corresponding to the aggregation result can be combined to obtain the relevant event of the subject. The representative subject can be used as the label of the event. For example, the video frame with the highest subject quality or the regional image of the subject in the video frame, i.e., the sub-image, can be used as the label of the corresponding event, and the label can be associated with the event segment to obtain the event segment associated with the label, and the labels of each subject in the video can be provided to the user. Optionally, object recognition can also be performed on the subject feature representing the subject to identify which subject the feature corresponds to, such as which person, and then the event segment of which object (subject label) can be provided to the user. Of course, it is also possible not to identify, and use the sub-image representing the subject as the subject label, and provide the subject label and its associated event segment to the user.

[0165] Optionally, the quality of a subject corresponding to a subject feature can be realized by using a classification network. The subject feature can be input into the classification network obtained through training to obtain the object quality score corresponding to the vector. The higher the score, the higher the subject quality. Among them, the evaluation criteria for quality are not limited in this embodiment of the application. It can be configured according to requirements. By training the subject quality identifier, the subject quality identifier can learn the corresponding quality evaluation criteria. For example, whether the object in the video frame is positive, whether the pixel is high, and whether the face is clear can be used as evaluation criteria.

[0166] For each aggregation result, based on each video frame corresponding to the aggregation result, the related events of the behavior subject are obtained, which may include at least one of the following:

[0167] Method 1: removing the background in each video frame corresponding to the aggregation result, and obtaining related events based on each video frame with the background removed;

[0168] Method 2: based on the main area in each video frame corresponding to the aggregation result, each video frame is cropped, and related events are obtained based on each cropped video frame.

[0169] Among them, for method one, the event segment is obtained by removing the background in the video frame, and the size of the video frame can remain unchanged. For method two, the event segment is obtained by cropping the non-subject area in the video frame, that is, the image in the event segment is a sub-image that only retains the subject area in the video frame.

[0170] After determining each subject in the video to be processed and the events related to each subject, the label of each subject (such as the above-mentioned highest quality video frame or sub-image) can be provided to the user, such as displayed to the user through a user terminal, and the user can select the subject of interest. Based on the user's selection, the events of the selected subject can be provided to the user. Optionally, the user can also watch, forward, save or edit the events of the subject of interest.

[0171] In some actual application scenarios, the user may only want to know the highlights in the video, and is not particularly concerned about which subject the highlights are. Considering this demand, another embodiment of the present application further provides a method performed by an electronic device, which may include:

[0172] Use the AI ​​network to obtain events in the video to be processed and provide events in the video to be processed.

[0173] In this solution, it is possible to ignore the subjects in the video and only identify the event segments contained in the video to be processed. Of course, in this solution, after obtaining the event in the video to be processed, the subject identification of the event can be performed to obtain the subject corresponding to the event.

[0174] In an embodiment of the present application, the above-mentioned use of the AI ​​network to obtain events in the video to be processed may include:

[0175] Using an AI network, semantic features of each video frame to be processed are extracted, and the semantic features include spatial semantic features and temporal semantic features. Based on the extracted semantic features of each video frame to be processed, events in the video to be processed are determined.

[0176] Among them, in this embodiment, the optional scheme of using an AI network to obtain the semantic features of each video frame to be processed in the video to be processed can adopt the scheme for obtaining the semantic features of the video frame to be processed provided in any of the previous embodiments, which will not be repeated here.

[0177] Optionally, the above-mentioned determining an event in the video to be processed based on the extracted semantic features of each video frame to be processed may include:

[0178] The semantic features of each video segment in the video to be processed are integrated to obtain the segment features corresponding to each video segment;

[0179] Based on the segment features of the video segments, events in the video to be processed are determined.

[0180] Among them, any video segment can be a set number of continuous video frames in the video to be processed, such as every 5 or every 10 continuous video frames as a video segment. Similarly, the video to be processed in this embodiment can be the original video, or it can be a video obtained by adopting the original video.

[0181] After obtaining the semantic features of each video frame to be processed in the video to be processed through the AI ​​network extraction, the semantic features of the multiple video frames contained in each video clip can be fused, and whether the video clip is an event clip can be determined based on the fused clip features. The specific method of fusing the semantic features of each video frame of the video clip to obtain the video clip is not limited in the embodiment of the present application. Optionally, the semantic features of each video frame can be spliced ​​to obtain the clip features, or, after obtaining the semantic features of each video frame in the video to be processed, further feature extraction can be performed to obtain new features of each video frame, and the new features of each video frame in the video clip can be fused to obtain the clip features. As an optional method, feature extraction can be performed based on the semantic features of each video frame to obtain the target features of each video frame. For example, the target features of the video frame can be obtained by using the scheme provided in the previous text, and the target features of each video frame in each video clip can be spliced ​​to obtain the clip features of the video frame, or feature extraction can be performed on the spliced ​​features to obtain the clip features.

[0182] After obtaining the segment features of each video segment, optionally, a scoring model (such as a classification model) can be used based on the segment features of the video segment to obtain a score for whether the video segment is an event segment. The higher the score, the higher the possibility that the video segment is an event segment. Video segments with scores not lower than a preset threshold can be determined as event segments and displayed to users, and segments below the threshold will not be displayed to users.

[0183] In the scheme provided in the embodiment of the present application, the events present in the video, or the subject of behavior and the associated events of the subject of behavior appearing in the video can be accurately identified based on the semantic features of the video frame including the temporal semantic features and the spatial semantic features. In practical applications, the scheme in the corresponding embodiment can be adopted according to the actual application requirements (for example, whether it is necessary to identify the subject of behavior or event fragments in the video, or to identify the subject of behavior and its associated event fragments).

[0184] Optionally, the semantic features of each to-be-processed video frame in the to-be-processed video may be obtained by:

[0185] Encode each to-be-processed video frame respectively to obtain a first feature map of each video frame;

[0186] For each video frame to be processed in the video to be processed, at least one first operation is performed on the video frame, and a semantic feature map, i.e., a semantic feature, of the video frame is obtained based on the second feature map of the video frame obtained by the last first operation; wherein the first operation includes:

[0187] Extracting a first contour feature map and a first position feature map of an object in the video frame based on the current feature map of the video frame and the current feature map of a preceding frame (such as a previous frame) of the video frame; obtaining a second feature map of the video frame based on the first contour feature map and the first position feature map corresponding to the video frame;

[0188] Among them, the current feature map of the video frame corresponding to the first first operation is the first feature map of the video frame, the current feature map of the video frame corresponding to the first operation other than the first one is the second feature map of the video frame obtained by the previous first operation, and the current feature map of the previous frame is the second feature map of the previous frame obtained by the current first operation.

[0189] Among them, the embodiment of the present application does not limit the method for obtaining the first feature map of each video frame in the video. In theory, any feature extraction network can be used for implementation, or the video frame can be directly used as the first feature map of the video frame. Optionally, the first feature map of each video frame can be implemented by feature extraction through a convolutional neural network, that is, the AI ​​network includes a convolutional neural network, the input of the convolutional neural network is the video frame, and the output is the first feature map of the video frame. Optionally, when performing feature extraction on each video frame separately through a convolutional neural network, it can be determined whether the size of the video frame is a pre-set fixed size. If not, each video frame can be processed into an image of a fixed size and then input into the convolutional neural network.

[0190] Optionally, the preceding frame of any video frame may use only the previous frame of the frame, or may use the previous frames of the frame. When the preceding frame of any current frame uses multiple frames, when processing is performed based on the feature map of the current frame and the feature map of its preceding frame, the feature map of the preceding frame may be a fused feature map of the feature maps of multiple frames. For example, if the preceding frames are the first two frames of the current frame, the feature values ​​of the corresponding positions in the feature maps of the first two frames may be averaged to obtain a fused feature map. For example, for the first first operation, the input may be the first feature map of the current frame and the fused feature map of the first two frames of the current frame, and the input of the second first operation may be the second feature map of the current video frame and the fused feature map of the new feature maps of the first two frames obtained by the second operation.

[0191] As an example, Figure 2 FIG. 1 shows a schematic diagram of a principle of encoding a video frame into a fixed number of patches provided by an embodiment of the present application, wherein: Figure 2 The image matrix in is the video frame, the values ​​in the matrix are the pixel values ​​at the corresponding positions in the video frame, the kernel matrix is ​​the model parameter (weight matrix) of the convolutional neural network, the size of the matrix is ​​the size of the convolution kernel, and the convolution kernel size in this example is 3*3. The output matrix is ​​the first feature map of the video frame. The eigenvalue of a position in the first feature map is obtained by multiplying the 9 pixel values ​​of the 3*3 image block in the video frame with the weight values ​​of the corresponding positions in the 3*3 convolution kernel and then adding them. For example, Figure 2 The eigenvalue 89 in the output matrix is ​​obtained by multiplying the kernel matrix and the pixel value of the image block in the upper left corner of the image matrix (with 9 pixel values ​​filled) and then adding them together.

[0192] A convolutional neural network is used to extract features from each video frame. Each video frame can be encoded into a fixed number of patches (also called blocks). One patch corresponds to an image block of the video frame, and the size of the image block is determined by the receptive field size of the convolutional neural network.

[0193] In order to enable the frame feature map of each video frame to learn the temporal and spatial features at the same time, after obtaining the first feature map of each video frame, for any video frame, the AI ​​network can be used to further extract features of the video frame based on the first feature map of the video frame and the first feature map of the preceding frame of the video frame to obtain the spatiotemporal feature map of the video frame. In the above optional scheme provided by the present application, for any video frame, the AI ​​network can be used to perform the first operation on the video frame for a set number of times, and the spatiotemporal feature map of the video frame can be obtained based on the second feature map obtained by the last first operation.

[0194] In an embodiment of the present application, a neural network that implements at least one of the first operations described above may be referred to as an ADT network, and the ADT network may include multiple cascaded ADT modules, each of which implements a first operation, and an input of an ADT module includes an output of a previous ADT module. A feature map containing object contours and relative position information, i.e., semantic features of a video frame, may be obtained through the ADT network.

[0195] For the convenience of description, in some of the following embodiments, the preceding frame of each to-be-processed video frame will be described by taking the previous video frame of the video frame as an example.

[0196] As an example, Figure 3a , a schematic diagram of the principle of feature extraction of video frames through an ADT network is shown. In this example, the ADT network includes two ADT modules, wherein the feature map T represents the first feature map of the current frame, and the feature map T-1 represents the first feature map of the frame before the current frame. It can be seen from the figure that for any current video frame, the input of the first first operation (the first ADT module) includes the first feature map T of the video frame, and the output feature map of the first first operation corresponding to the video frame before the video frame, and the input of the second first operation includes the feature map of the video frame output by the first first operation, and the output feature map of the second second operation corresponding to the video frame before the video frame.

[0197] For each video frame, through the above-mentioned first operation, the static posture information in space and the dynamic relative position information in time of the object (also referred to as object, target or subject) in the video frame can be learned based on the feature map of the current frame and the feature map of the previous frame of the current frame, so that a new feature map containing the object contour information and position information can be obtained, and then based on the new feature map of each video frame in the video, the behavior transformation of the moving object can be better identified, and the motion behavior can be accurately identified. Among them, the number of the first operation, that is, the number of ADT modules included in the ADT network, is not limited in the embodiment of the present application.

[0198] Optionally, for any video frame, extracting a first contour feature map and a first position feature map of an object in the video frame based on the first current feature map of the video frame and the first current feature map of a preceding frame of the video frame may include:

[0199] Based on the first current feature map of the video frame and the first current feature map of the preceding frame of the video frame, extracting a first weight feature map corresponding to the preceding frame of the video frame and a second weight feature map corresponding to the video frame;

[0200] Based on the first current feature map and the first weight feature map of the preceding frame of the video frame, a first position feature map corresponding to the video frame is obtained, and based on the first current feature map and the second offset feature map of the video frame, a first contour feature map corresponding to the video frame is obtained.

[0201] Since the event in the video is caused by a specific action of an object in the video, if there is an event in the video, the object in the video (the position, posture, etc. of the object) changes, and the positional relationship between the related image content in the video frame also changes. In order to more accurately identify the area related to the object in the video frame, in the optional scheme provided in the embodiment of the present application, for any video frame, an AI network can be used to learn the first weight feature map and the second weight feature map corresponding to the video frame based on the feature map of the video frame and the feature map of its previous frame, wherein the weight feature map can also be called an offset feature map, and the first weight feature map can be understood as the offset map of the image information in the previous frame relative to the object / subject of the previous frame, and the second weight feature map can be understood as the offset map of the image information in the current video frame relative to the object in the video frame. The first weight feature map and the second weight feature map can respectively characterize the spatial position offset of each patch in the previous frame that is semantically related to the patch of the current video frame, and the position offset of the patch in the current video frame that is semantically related to the patch in the video frame. By learning the weighted feature map, semantically related patches in the feature map of the video frame can be found more accurately, rather than simply taking adjacent patches in the feature map as related patches, thereby improving the accuracy of event recognition.

[0202] After obtaining the weight feature map corresponding to the current frame and the weight feature map corresponding to the previous frame of the current frame, the weight feature map can be used as the offset feature map in the deformable convolution network calculation, and the feature map of the corresponding frame can be convolved to obtain a new feature map. This optional scheme of the present application provides a new offset calculation scheme in the deformable convolution operation. For the previous frame of the current frame, when performing a convolution operation on the feature map of the previous frame (which can be the first current feature map of the previous frame, or a new feature map obtained by extracting features from the first current feature map again), the weight feature map of the previous frame obtained in the above manner is used to implement the deformable convolution of the frame to generate a relative position result. Similarly, a deformable convolution operation can be performed on the feature map of the current frame based on the weight feature map of the current frame to generate a contour result.

[0203] Optionally, for each video frame to be processed, the first current feature map of the video frame and the first current feature map of the preceding frame of the video frame can be fused to obtain a fused feature map, and the first weight feature map and the second weight feature map corresponding to the video frame can be extracted based on the fused feature map. The specific method of fusion is not limited in this embodiment of the application, and may include but is not limited to splicing.

[0204] As an optional solution, for each video frame to be processed, the first weight feature map and the second weight feature map corresponding to the video frame can be obtained by:

[0205] Performing the second operation on the third feature map of the video frame and the third feature map of its preceding frame respectively, to obtain fourth feature maps corresponding to the video frame and the preceding frame respectively, wherein the third feature map of any video frame is the first current feature map of the video frame, or is a feature map obtained by performing feature extraction on the first current feature map of the video frame;

[0206] The fourth feature map of the video frame and the fourth feature map of the preceding frame of the video frame are spliced, and based on the spliced ​​fusion feature map, a first weight feature map corresponding to the preceding frame of the video frame and a second weight feature map corresponding to the video frame are extracted; wherein the second operation includes:

[0207] Performing feature rearrangement on the third feature map to obtain a fifth feature map, wherein the number of channels of the fifth feature map is the number of feature points in a feature map of one channel in the third feature map, and the height and width of the fifth feature map are 1 and the number of channels of the third feature map, respectively; performing feature extraction on the fifth feature map to obtain a sixth feature map whose size information feature map remains unchanged; performing feature rearrangement on the sixth feature map to obtain a fourth feature map having the same size information as the third feature map.

[0208] Among them, the above-mentioned second operation is the specific convolution operation described in the previous embodiment. By adopting this convolution operation, each feature point / patch in the feature map can obtain a global receptive field and obtain the global information of the feature map. Therefore, based on the feature map obtained by the convolution operation, a more accurate weight feature map can be learned to obtain a more accurate object contour and relative position.

[0209] It can be understood that, in actual implementation, the above-mentioned second operation can be performed once or multiple times. If it is performed multiple times, the input of the current second operation is the output of the previous second operation. Among them, the process of performing the second operation on any current video frame and its preceding frame is the same. Taking the current video frame as an example, optionally, for any first operation, the input of the first second operation in the first operation, that is, the third feature map, can be the input of the first operation, that is, the first current feature map of the current video frame, or it can be a feature map obtained by extracting features from the feature map again. Optionally, a deformable convolution network can be used to perform a deformable convolution operation on the first current feature map of the video frame to obtain the third feature map of the video frame, and the third feature map is used as the input feature map of a specific convolution operation. An optional implementation of the feature convolution operation is as follows. Figure 3b shown.

[0210] For any video frame, after obtaining the fourth feature map of the video frame and the fourth feature map of the previous frame of the video frame through the above-mentioned specific convolution operation, the fourth feature map of the video frame and the fourth feature map of the previous frame can be fused (such as splicing), and the first weighted feature map and the second weighted feature map corresponding to the video frame can be extracted based on the fused feature map, and based on the weighted feature map, a feature map containing more accurate object contour and position information is obtained.

[0211] Optionally, for each video frame to be processed, obtaining a second feature map of the video frame based on a first contour feature map and a first position feature map corresponding to the video frame may include:

[0212] A second feature map of the video frame is obtained by fusing the first position feature map, the first contour feature map, and the first current feature map of the video frame corresponding to the video frame.

[0213] The embodiments of the present application do not make a single limitation on the fusion method. Optionally, the first position feature map, the first contour feature map and the first current feature map corresponding to each first operation may be feature maps of the same size. The output feature map of the current first operation, i.e., the second feature map, may be obtained by adding the first position feature map, the first contour feature map and the first current feature map corresponding to the current first operation (or the feature map obtained by extracting features from the first current feature map again, such as the third feature map described in the later embodiments).

[0214] Optionally, for each video frame to be processed, after obtaining the second feature map of the last first operation corresponding to the video frame, the second feature map of the video frame obtained by the last first operation can be used as the spatiotemporal feature map (semantic feature) of the video frame, or feature extraction can be performed again on the second feature map of the video frame obtained by the last first operation to obtain the spatiotemporal feature map of the video frame.

[0215] Optionally, in order to further improve the accuracy of the recognition results, for any video frame, when extracting the above-mentioned first weight feature map and second weight feature map corresponding to the video frame based on the first current feature map of the video frame and the first current feature map of the preceding frame of the video frame, a more fine-grained weight feature extraction method can be used. Specifically, the third feature map of any video frame includes multiple blocks, each block is an area where at least one feature point in the third feature map is located, wherein the third feature map of any video frame is the first current feature map of the video frame, or is a feature map obtained by extracting features from the first current feature map of the video frame. In other words, a block (a patch) of any feature map can be a pixel point in the feature map, or a region of multiple pixels.

[0216] The extracting of the first weight feature map corresponding to the preceding frame of the video frame and the second weight feature map corresponding to the video frame may include:

[0217] Each block in the third feature map of the video frame is used as a query block respectively. For any query block, the query block is spliced ​​with the fused feature map to obtain a spliced ​​feature map, and based on the spliced ​​feature map, a first weight feature map and a second weight feature map corresponding to the query block are extracted;

[0218] For each query block of the video frame, a first feature block corresponding to the query block is obtained based on the third feature map of the video frame and the second weight feature map corresponding to the query block, and a second feature block corresponding to the query block is obtained based on the third feature map of the preceding frame of the video frame and the first weight feature map corresponding to the query block; wherein the first contour feature map corresponding to the video frame includes the first feature maps corresponding to each query block of the video frame, and the corresponding first position feature map in the video frame includes the second features corresponding to each query block of the video frame.

[0219] In this optional solution, the first weight feature map corresponding to a video frame includes the first weight feature map of each query block of the video frame, and the second weight feature map corresponding to the video frame includes the second weight feature map of each query block of the video frame. Correspondingly, the first contour feature map of each video frame includes the first feature blocks corresponding to all query blocks of the video frame, and the first position feature map of the video frame includes the second feature blocks corresponding to all query blocks of the video frame. Among them, the first weight feature map corresponding to a query block of a video frame represents the spatial position offset information of each block in the previous video frame of the video frame that is semantically related to the query block, and the second weight feature map corresponding to the query block represents the spatial position offset information of each block in the video frame that is semantically related to the query block.

[0220] Through this optional scheme, for each query block of a video frame, based on the fused feature map of the video frame and its previous frame, several blocks related to the query block can be found from the feature map of the video frame, and several blocks related to the target block in the previous frame (that is, blocks related to the query block in the previous frame) can be found from the previous frame of the video frame. The target block refers to a block in the previous frame with the same position as the query block, so that a new query block can be obtained by learning the information of these related blocks, that is, a new feature block corresponding to the query block that has learned more semantic information.

[0221] In actual implementation, the number of blocks related to each block can be pre-configured, assuming it is M, where the related blocks of a block can include the block itself and the surrounding M-1 blocks. Assuming M=9, the surrounding blocks of a block include 8 blocks (8 neighborhoods) around the block. The second weighted feature map corresponding to any query block can be interpreted as an offset feature map of the M semantically related blocks of the query block. The "offset" here can be understood as the offset of the related blocks relative to the objects in the video frame. Based on the offset feature map, the related blocks of the query block can be more accurately found in the feature map of the current video frame. Similarly, based on the first weighted feature map, blocks related to the query block can be more accurately found in the previous frame.

[0222] The optional solution of the present application provides a new deformable convolution operation. The second weight feature map corresponding to a query block in the current video frame can be understood as: the offset corresponding to the M related blocks of the query block in the extracted input feature map (the third feature map of the current video frame) (the offset of the deformation of the deformable convolution). Based on the input feature map and the offset corresponding to the M blocks, the convolution operation can be performed on the M blocks after the offset, so that the shape of the convolution operation is closer to the shape of the object in the video frame. Therefore, based on this solution, more accurate object contour information and relative position information of the video frame can be obtained.

[0223] Optionally, obtaining the second feature map of the video frame based on the first contour feature map and the first position feature map corresponding to the video frame includes:

[0224] For each query block of the video frame, a new feature block corresponding to the query block is obtained by fusing the first feature block, the second feature block, and the query block corresponding to the query block;

[0225] Based on the new feature blocks corresponding to the query blocks of the video frame, a second feature map of the video frame is obtained.

[0226] Optionally, for each query block, a new feature block corresponding to the query block can be obtained by adding the first feature block, the second feature block, and the query block corresponding to the query block. The second feature map of any video frame includes new feature blocks corresponding to each query block of the video frame.

[0227] From the previous description, it can be seen that for any video frame, after one or more first operations are performed to obtain the second feature map of the video frame output by the last operation (the feature map of the video frame output by the last ADT module in the cascaded multiple ADT modules), the second feature map can be used as the spatiotemporal feature map (semantic feature) of the video frame, or further feature extraction can be performed on the second feature map to obtain a spatiotemporal feature map.

[0228] It can be understood that when implementing the solution provided in the embodiments of the present application, some feature extraction steps may or may not be executed. For example, the input features of the ADT network may be each video frame, or may be the features of each video frame obtained by image block encoding. When using the ADT network to further extract features from the input features, the first specific convolution operation may be performed on the basis of the input features of the ADT network, or may be to first perform feature extraction on the input features (such as convolution processing using deformable convolution), and then perform a specific convolution operation on the extracted features.

[0229] After obtaining the semantic features of each video frame, the subject of behavior and its related events in the video frame can be determined based on the semantic features of each video frame. In order to more accurately identify the subject of behavior and its related events, further feature extraction can be performed based on the semantic features of each video frame to be processed in the following way to obtain the target features of each video frame:

[0230] For each to-be-processed video frame in the video, perform the third operation on the video frame at least once, and obtain the target feature of the video frame based on the seventh feature map of the video frame obtained by the last third operation; wherein the third operation includes:

[0231] Based on the second current feature map of the video frame and the second current feature map of the previous frame of the video frame, identify the object in the video frame to obtain a first object feature map of the video frame (regional features of the region where the subject is located in the video frame), wherein the feature value of the target region where the object is located in the first object feature map is the pixel value of the corresponding region in the video frame, and the feature value of the non-target region in the object feature map is 0; based on the first object feature map of the video frame and the second current feature map of the video frame, obtain a seventh feature map of the video frame;

[0232] Among them, the second current feature map of the video frame corresponding to the first third operation is the second feature map of the video frame obtained by the last first operation; the second current feature map of the video frame corresponding to the third operation other than the first one is the seventh feature map of the video frame obtained by the last third operation, and the second current feature map of the preceding frame of the video frame is the seventh feature map of the preceding frame obtained by the current third operation.

[0233] In the embodiment of the present application, the neural network that implements the third operation at least once can be called an AVT (Adjacent Variation Transformer) network. The AVT network can include multiple cascaded AVT modules, each AVT module implements the third operation once, and the input of an AVT module includes the output of the previous AVT module. Through the AVT network, a feature map containing more accurate contour information and absolute position information of the object can be obtained. The feature extraction process of the AVT network is similar to Figure 3a The feature extraction diagram shown in the ADT network is similar to that shown in , except that the input and output feature maps are different.

[0234] For any video frame, the feature extraction principle corresponding to the first AVT module of the video frame is as follows: the input feature map includes the output feature map of the video frame obtained by the ADT network (that is, the feature map of the video frame output by the last ADT module, the input feature map of the AVT module of the video frame), and the output feature map of the first AVT module corresponding to the previous video frame of the video frame. Based on the feature maps of these two video frames, such as the spliced ​​feature maps of the two video frames, the AVT module can identify the object in the video frame and obtain a first object feature map. Based on the object feature map and the input feature map of the video frame, an output feature map containing the outline and position of the object in the video can be obtained, that is, the seventh feature map mentioned above.

[0235] Optionally, for any video frame, the first object feature map of the video frame may be obtained in the following manner:

[0236] The second current feature map of the video frame is spliced ​​with the second current feature map of the preceding frame of the video frame (i.e., the semantic features of the video frame and the semantic features of its adjacent frames are fused), and a mask feature map of the video frame is obtained based on the spliced ​​feature map, wherein the feature value of the area where the object is located in the mask feature map is 1, and the feature value of the area where the object is not located is 0;

[0237] For an area whose feature value is 1 in the mask feature map, the feature value of the corresponding area in the second current feature map of the video frame is used to fill the feature value of the area to obtain a first object feature map.

[0238] Optionally, the acquisition of the mask feature map can be implemented using an image segmentation network. The input of the image segmentation network includes the feature map of the current frame and the feature map of its previous frame. Based on the feature maps of multiple frames, the image segmentation network can identify the object area and non-object area in the feature map of the current frame, and output the mask feature map. The mask feature map is a binary feature map. After that, the pixel values ​​in the current feature map of the video frame can be used to fill the pixel values ​​of the object area in the binary feature map (that is, only the pixel values ​​of the object area in the input feature map are retained, and the pixel values ​​of other areas are set to 0) to obtain the first object feature map. Optionally, the first object feature map of the video frame can be used as the seventh feature map of the video frame, or the first object feature map can be extracted again to obtain the seventh feature map of the video frame. In order to further improve the accuracy of object recognition, as an optional method, the seventh feature map of the video frame can be obtained in the following way:

[0239] Using the first object feature map of the video frame as the first current feature map of the video frame, and the second current feature map of the preceding frame of the video frame as the first current feature map of the preceding frame, and obtaining the second object feature map of the video frame by performing the first operation;

[0240] The second current feature map and the second object feature map of the video frame are fused to obtain a seventh feature map of the video frame.

[0241] In this optional solution, in addition to a neural network (such as an image segmentation network) for identifying the area where the object is located in the image, the AVT module may also include an ADT module. The principle of the ADT module is the same as the principle of the ADT described above, except that the input feature map is different. The input feature map of the ADT module in any AVT module includes the first object feature map of the current video frame and the seventh feature map of the previous frame of the video frame (the output feature map of the previous frame obtained by the current AVT module), and the output feature map is a more fine-grained object feature map of the current video frame. Afterwards, the output feature map of the video frame corresponding to the AVT module can be obtained through the input feature map of the AVT module of the fused video frame and the fine-grained object feature map. After each video frame is processed separately through the AVT network, a new spatiotemporal feature map, i.e., the target feature, of each video frame can be obtained.

[0242] It should be noted that, no matter it is an AVT module or an ADT module, since the first video frame in the video does not have a preceding frame, the feature map of the preceding frame of the video frame may be a preset feature map.

[0243] After obtaining the target feature map of each video frame of the video to be processed through the solution provided by any embodiment of the present application, since the target feature map of each video frame contains temporal and spatial image features, it is possible to accurately identify the events existing in the video based on the target feature maps of these video frames. Optionally, the solution provided by the present application can also identify the behavioral subjects of the events in the video frames (that is, non-static objects in the video) and their associated events, thereby providing users with events related to specific objects. For example, the video is a video related to playing football. Based on the solution provided in the embodiment of the present application, a video clip of a specific football player in the video can be identified.

[0244] As an optional solution, the above-mentioned identifying event segments in the video based on the target features of each video frame may include:

[0245] For any video frame, based on the target feature of the video frame and the first object feature map, the object feature (subject feature) of the video frame is obtained, and based on the object feature, whether the video frame contains the action subject is determined;

[0246] Based on each first subject feature, an event segment in the video is identified, wherein the first subject refers to an object feature of a video frame containing the subject.

[0247] In the embodiment of the present application, the subject of behavior refers to an object that has performed a preset behavior or a preset action. In practical applications, some objects that appear in the video may not be the subject of behavior. For example, there is a lying cat in each frame of the video to be processed, and the position of the cat in each frame of the video is the same or basically unchanged. Then the cat can be considered as the background in the video frame and does not belong to the subject of behavior. In order to improve the accuracy of event recognition, after obtaining the target feature map of each video frame, for each frame, it can be determined whether the object in the video frame is the subject of behavior based on its target feature map, and then only the video frame corresponding to the subject of behavior can be used for event recognition.

[0248] Optionally, for any video frame, the object features corresponding to the video frame can be obtained based on the first object feature map and the target feature map of the video frame. Since the first object feature map identifies the area where the object in the video frame is located, the object feature map can be cropped to obtain a feature sub-map of the area where the object is located in the feature map, such as the minimum circumscribed rectangle surrounding the area where the object is located in the first object feature map can be used as the feature sub-map. Afterwards, the feature vector of the object in the video frame is obtained by fusing the target feature map of the video frame and the feature sub-map of the area where the object is located, and whether the object is a behavior subject is determined based on the object feature vector. Optionally, the target feature map and the feature sub-map of the video frame can be converted into feature vectors of fixed lengths respectively, and the two converted feature vectors are concatenated to obtain the object feature vector. Based on the vector, a classification result is obtained through a classification network, and the classification network can be a binary classification network. According to the output of the network, it can be determined whether the object is a behavior subject. For example, the output of the classification network can be a probability value (also called a score), and the probability value represents the possibility that the object is a behavior subject. If the probability value is less than a preset threshold, it is determined that the object is not a behavior subject. If the probability value is greater than the preset threshold, it is determined that the object is a behavior subject.

[0249] Optionally, after determining whether the object in each frame of the video frame is a behavior subject, the identifying event segments in the video based on each first behavior subject feature includes:

[0250] Aggregating each second behavior subject feature to obtain at least one aggregation result, wherein the second behavior subject feature is the first behavior subject feature, or is a target subject feature obtained by extracting the first behavior subject feature;

[0251] For each aggregation result, an event segment corresponding to the aggregation result is generated based on each video frame corresponding to the second behavior subject feature in the aggregation result.

[0252] Through vector aggregation, we can find the behavior subject features corresponding to the same behavior subject. The video frames corresponding to the behavior subject features in an aggregation result can be regarded as video frames containing the same object. Then, after obtaining the aggregation result corresponding to each behavior subject, we can generate the corresponding event fragments according to the video frames corresponding to the aggregation result.

[0253] Optionally, for each first behavior subject feature, the target subject vector corresponding to the first behavior subject feature is obtained in the following manner:

[0254] Using the first behavior subject feature as a query subject feature, determining at least one similar feature of the query subject feature from each first behavior subject feature except the query subject vector;

[0255] Based on the query subject feature and at least one similar feature of the query subject feature, at least one fourth operation is performed, and a new subject feature obtained by the last fourth operation is used as a target subject feature, wherein the fourth operation includes:

[0256] Based on the correlation between the current subject feature corresponding to the query subject feature and at least one similar feature, a weight corresponding to the at least one similar feature is obtained, and based on the weight corresponding to the at least one similar feature, at least one similar feature is fused to obtain a new subject feature corresponding to the query subject feature;

[0257] The current subject feature corresponding to the first fourth operation is the query subject feature, and the current subject vector corresponding to the fourth operation other than the first one is the new subject feature obtained by the last fourth operation.

[0258] According to the optional solution provided by the present application, a first subject feature and at least one similar feature thereof can be considered as features of the same subject from different angles or different scenes. By adopting this optional solution, each first subject feature can learn the semantic information of the subject corresponding to the feature from multiple angles and multiple scenes from itself and several other similar subject features. The learned target subject features have better feature expression capabilities and are more conducive to the accurate identification of the subject and the event fragments related to it.

[0259] In an embodiment of the present application, the neural network that implements the above-mentioned at least one fourth operation can be called a CCT (Context Contrast Transformer) network. The CCT network is a neural network based on an attention mechanism, wherein the query subject feature is the Q (query vector) in the attention mechanism, and the K vector (key vector) and the V vector (value vector) in the attention mechanism can be obtained based on at least one similar feature of the Q vector. Based on the Q vector and the K vector, a weight vector can be calculated, and the V vector is weighted using the weight vector to obtain a new feature corresponding to the Q vector.

[0260] As an optional solution, for each query subject feature, the weight of at least one similar feature corresponding to the query subject feature can be obtained in the following manner:

[0261] The similar features are spliced ​​together, and at least two different feature extraction methods are used to extract features from the spliced ​​features to obtain at least two new features; the at least two new features are spliced ​​together to obtain a spliced ​​feature; based on the splicing vector and the query subject feature, the weights corresponding to the query subject feature and the similar features are determined.

[0262] By using different feature extraction methods to extract features from the spliced ​​features, multiple features corresponding to different feature spaces and containing information of different dimensions can be obtained. These vectors are spliced ​​as the K vector of the attention mechanism, which can enable the query vector to better learn the semantic information of the same object from multiple angles and scenes. Optionally, for the V vector, a similar method as above can be used to splice the features extracted by using multiple different feature extraction methods. The K vector and the K vector can be the same or different.

[0263] By adopting the above method, each first behavior subject feature can be processed separately to obtain the corresponding target subject feature, and based on all the target subject features corresponding to the video, the subject in the video and its related events can be identified.

[0264] After all the behavior subject features are aggregated to obtain the aggregated results, for each aggregated result, the event fragment corresponding to the aggregated result can be obtained in the following way:

[0265] Perform at least one of the following operations on each video frame corresponding to the aggregation result, and generate an event segment corresponding to the aggregation result based on each video frame after the operation:

[0266] Operation 1: cropping the video frame based on the object area in the video frame;

[0267] Operation 2: Set the pixel value of the background area in the video frame to 0;

[0268] The object area is the area where the object is located in the video frame, and the background area is the area outside the object area.

[0269] By adopting the above operation 1, the video frame can be cropped based on the area where the object is located in the video frame, and the event segment can be obtained based on the cropped sub-image containing the object. For example, the cropped sub-image corresponding to each aggregation result can be first processed into a uniform size, and the sub-images after the uniform size are sorted according to the time sequence of the video frame where the sub-image is located, so as to obtain the event segment composed of these groups of images. Optionally, for each aggregation result, each video frame can also be filtered according to the time interval between each video frame corresponding to the collection result. For example, if the time interval between a video frame and its adjacent video frames before and after is too large, the video frame can be considered to be an isolated frame and can be deleted. An event segment can be generated based on the video frames before the isolated frame, and an event segment can be generated based on the video frames after the isolated frame. Of course, if the number of images contained in an event segment is too small, such as less than a set number, the event segment can also be deleted.

[0270] By adopting the above operation 2, the background of each video frame can be deleted to generate an event segment in which the pixel values ​​of the background area are all 0.

[0271] After identifying an event in the video to be processed, or a behavior subject and its related events, the event or subject can be provided to the user. The embodiment of the present application also provides a method executed by an electronic device, such as Figure 4 As shown, the method may include the following steps:

[0272] Step S410: In response to a target operation of a user on a first video, displaying segment information of at least one event segment in the first video;

[0273] Step S420: In response to a user's processing operation on the segment information of at least one segment in the at least one event segment, corresponding processing is performed on the at least one segment.

[0274] The at least one event segment may be part or all of the event segments in the first video. The first video may be any video, and the event segments in the video frame may be identified by using the method in any of the embodiments provided in the foregoing text of the present application. Optionally, the first video may be sampled to obtain a video to be processed, and any of the schemes provided in the embodiments of the present application may be used to identify the event segments in the video to be processed, or to identify the behavior subjects in the video to be processed and at least one event segment associated with each behavior subject.

[0275] In an embodiment of the present application, the above-mentioned target operation may be any operation in a preconfigured first operation set. The above-mentioned processing operation may be any operation in a preconfigured second operation set. For any target operation, the information displayed to the user in response to the target operation may be relevant information of one or more behavioral subjects in the video, such as the label of the behavioral subject, or relevant information of the segment, such as the image sequence of the segment, or the cover of the segment (such as any image in the segment), etc., or relevant information of the behavioral subject and relevant information of its related event segment. When displaying multiple information, multiple information may be displayed simultaneously, or part of the information may be displayed first, and then other information may be displayed after receiving the relevant operation of the user.

[0276] Optionally, the target operation may include but is not limited to at least one of the following:

[0277] Video playback operations; video information viewing operations; specified operations on any behavioral subject or specified area in the video frame.

[0278] Optionally, the above processing operation may include at least one of the following:

[0279] Clip saving operation; Clip sharing operation; Clip playback operation; Clip merging operation; Clip deletion operation; Operation to trigger display of image sequence in clip; Clip editing operation; Clip publishing operation.

[0280] As an optional manner, in response to the target operation, displaying the segment information of at least one event segment in the first video may include at least one of the following:

[0281] displaying relevant information of at least one event segment;

[0282] Displaying relevant information of the behavior subject associated with at least one event segment and relevant information of at least one event segment associated with each behavior subject;

[0283] Displaying relevant information of at least one event segment of the target behavior subject associated with the target operation; for example, a user can long press an object in a video frame during video playback. If the object is a behavior subject, information of the event segment associated with the behavior subject can be displayed to the user;

[0284] Displaying relevant information of a subject associated with at least one event segment, and in response to a user's triggering operation of the second prompt information on any subject, displaying relevant information of at least one event segment related to the any subject; for example, the tags of all subjects identified in the first video may be displayed to the user, and the user may select a subject of interest, and then the information of the event segment of the subject selected by the user may be presented to the user;

[0285] An event viewing control of the target behavior subject associated with the target operation is displayed, and in response to a trigger operation on the time viewing control, relevant information of at least one event segment of the target behavior subject is displayed; for example, a user can select the behavior subject of the video frame he is interested in, and the selection operation can be considered as a target operation. In response to the target operation, an operable control can be displayed, through which the user is prompted to view the highlight segments, i.e., event segments, of the behavior subject corresponding to the target operation. The user can click on the control, and then the information of the behavior segment of the behavior subject is displayed to the user.

[0286] Of course, in actual implementation, if the user selects a certain event segment or performs an operation on a certain event segment / behavior subject, the corresponding event segment can also be played directly to the user. The implementation form of the "operation" described in the above embodiments may include but is not limited to touch screen operation, or it may be voice operation or user input / operation obtained by other means.

[0287] In order to better understand and illustrate the method provided in the embodiments of the present application, the optional implementation methods of the method provided in the present application are further described below in combination with the principles of the scheme provided in the present application and several optional embodiments. The steps of different embodiments can be combined or replaced with each other without conflict.

[0288] Figure 5 The schematic diagram of the principle of a method performed by an electronic device provided by the present application is shown, wherein the electronic device can be a user terminal / terminal device, or a server, such as an application server. The following description takes the terminal device as the execution subject of the method. Figure 5 As shown, the method may include the following steps:

[0289] Step S100: obtaining an original video, performing frame extraction processing on the original video, and obtaining a video to be processed.

[0290] The step of extracting frames from the original video is an optional step, and the original video can also be directly used as the video to be processed. The specific method of extracting frames from the original video is not limited in the embodiment of the present application. Optionally, the original video can be extracted at a set frame number interval or a set time interval, such as extracting one frame every 2 frames, or extracting one frame every set milliseconds, and the extracted video frame sequence is used as the video to be processed.

[0291] Step S200: Input the video to be processed into the AI ​​network, and use the AI ​​network to process the video to be processed to obtain events in the video, or behavioral subjects in the video and related events.

[0292] Among them, the event is a video clip in which the event occurs, such as a video clip containing / appearing a set action, and the event clip is obtained based on at least part of the video frames in the video to be processed. Optionally, in addition to the event clip, the output by the AI ​​network may also include the behavior subject of the event (also called the event object), which refers to the target object associated with the event in the event clip, such as the execution subject of the set action that appears in the event clip. As an example, if a person appears in the video to be processed to perform a long jump, then this person is a behavior subject, and the event clip corresponding to this behavior subject is generated based on the frames of video of this person in the long jump, such as combining the frames of video of this person doing the long jump to obtain the event clip, or cropping the frames of video of this person doing the long jump according to the area where he appears in the video frame, and combining the cropped frames containing this person to obtain the event clip.

[0293] Step S300: Provide an event, or a behavior subject and its related events.

[0294] Optionally, the terminal device can directly display the event clips output by the AI ​​network to the user. The specific display form is not limited in the embodiments of the present application. For example, each event clip can be displayed to the user in the form of a list. Optionally, the terminal device can display each subject in the video to the user, and the user can select the subject of interest. The terminal device can display the event clips of the subject selected by the user to the user, for example, displaying the cover of the event clip of the subject (such as any image of the subject included in the event clip) to the user, or displaying each frame image of the event clip to the user in a list, thumbnail, list or other form. It can also be that after the user clicks on the subject of interest, the event clip of the subject is played to the user.

[0295] The following is based on Figure 5 The principles shown in the embodiment of the present invention are further described in combination with several optional embodiments of the present invention. The neural network involved in the embodiments of the present invention can be called a network, a module or a model. For example, an ADT network can be called an ADT module or an ADT model.

[0296] Optional embodiment 1

[0297] like Figure 6 As shown, the AI ​​network in this embodiment may include a cascaded image block encoding module, an ADT network capable of obtaining spatial contour information and temporal relative position information of an object (or object, subject), an AVT network capable of obtaining absolute position information and fine-grained contour information of the object, and a CCT network and a post-processing network capable of obtaining multi-angle and multi-scene information of the object, and the input of the latter network in each cascaded network includes the output of the previous network.

[0298] Among them, the input of the AI ​​network is multiple continuous video frames in the video to be processed. Image block coding can be used to encode each input video frame into a preset number of patches or blocks. The ADT network can further extract the semantic information of each video frame based on the output of the image block coding module to identify the fast-moving behavior in the image. Specifically, the ADT network can locate the relative position of the same behavior subject between adjacent frames and capture the coarse-grained contour of the behavior subject. Combining the above two (contour and position), the behavior transformation of the object can be accurately identified, so that the ADT network can accurately identify fast-moving behavior.

[0299] Based on the output of the AVT network, the AVT network can locate the absolute position of the subject, thereby obtaining the true position of the object in the frame. The AVT network can obtain a fine-grained outline containing semantic information such as the color and posture of the object. Combining the above two, the AVT network can accurately identify the subject. Based on the output of the AVT network, the subject module can determine whether the object in the video frame is a subject, and can also give a quality score to each subject. Based on all the subjects determined by the subject module, the CCT network can be used to learn the multi-angle and multi-scene semantic information of each subject. The post-processing network is used to aggregate the subjects and identify the related events of each subject in the video.

[0300] Figure 7 A schematic flow chart of a video processing method (i.e., a method executed by an electronic device) provided by this embodiment is shown below. Figure 6 The AI ​​network architecture shown and Figure 7 The flowchart shown illustrates the method, which may include the following steps:

[0301] Step S1: Sample the video frames and determine whether the number of sampled video frames reaches the set number. When the number of video frames is lower than the threshold, the process ends and no content is output. When the number of video frames is higher than or equal to the threshold, object and event recognition begins.

[0302] The input of this step is a video (the original video to be processed), the original video is sampled / framed, and the video to be processed whose number of sampled video frames is not less than the set threshold is processed. If the number of video frames of the video to be processed is small, no processing is performed. Optionally, this step can also be replaced by judging the number of video frames included in the original video. If the number of video frames of the original video is not less than the preset threshold, the original video is sampled, and the sampled video is used as the video to be processed to enter the next step of processing. If the number of video frames of the original video is small, video sampling and subsequent processing may not be performed.

[0303] Step S2: pre-process the original video frame.

[0304] This step is an optional step, and each video frame in the sampled video to be processed can be preprocessed. The preprocessing may include but is not limited to scaling the video frames. Because the resolution of the original video frames is usually relatively high, in order to speed up the calculation and improve the processing efficiency, the size of the video frames can be readjusted. For example, if the size of the video frame is larger than the set size, the video frame is reduced to the set size. If the size of the video frame is not larger than the set size, the size of the video frame may not be adjusted, or the video frame may be adjusted to the set size.

[0305] Step S3: Use the AI ​​network to perform object and event recognition on each video frame to be processed to obtain recognition results of event segments.

[0306] The input of this step is all the video frames after sampling and preprocessing. This step may include the following steps S31 to S36.

[0307] Step S31: Use Patch Embedded to encode each video frame into multiple patches / blocks of the same size.

[0308] The input of this step is all the video frames after sampling and preprocessing, and preliminary encoding is performed for each video frame. In this embodiment, each video frame can be encoded by using an image block encoding method to obtain the initial encoding result of each video frame, that is, the initial feature map. Optionally, the image block encoding of this step can be implemented by using a convolutional network, and a convolutional network is used to extract features of each video frame to obtain an initial feature map of each video frame, wherein each pixel point (i.e., feature point) in the initial feature map of each video frame corresponds to an image block on the video frame. Optionally, the kernel size (Kernel Size) and the convolution step size (stride) of the convolution kernel of the convolutional network can be equal, so that the image blocks corresponding to each pixel point on the encoded feature map on the video frame do not overlap each other. Of course, the convolution step size can also be smaller than the kernel size of the convolution kernel, so that there will be partial overlapping areas between the image blocks corresponding to adjacent feature points on the initial feature map. For each video frame, it is also possible to first divide the video frame into multiple image blocks according to a preset size, and then perform feature extraction on each image block to obtain the encoding results of each image block in the video frame. At this time, the encoding result of a video frame includes the encoding results corresponding to each image block in the video frame.

[0309] For the convenience of description, the initial feature map of each video frame obtained by image block encoding is referred to as feature map A, which is the first feature map in the previous text.

[0310] Step S32: Input each video frame into the ADT network composed of ADT modules to extract the coarse-grained contour and relative position information of the object, and output a frame feature map with the coarse-grained contour and relative position information of the object.

[0311] The input of this step is the output of the image block encoding, that is, the feature map A of each video frame. Based on the feature map A of each video frame, the trained ADT network is used to further extract the frame features and object features of each video frame to obtain a new feature map of each video frame. The feature map of each video frame extracted by the ADT network is referred to as feature map B, that is, the second feature map of each video frame output by the last ADT module of the ADT network in the previous text.

[0312] Step S33: Input the feature map of each video frame output in the previous step into the AVT network composed of AVT modules to extract the absolute position and fine-grained contour information of the object, and output the frame feature map with the above information and the segmented object feature map.

[0313] The input of this step is the feature map B of each video frame output by the AVT network. Based on the feature map B of each video frame, the AVT network is used to perform more fine-grained feature extraction to obtain the feature map C of each video frame. The feature map C of each video frame output by this step includes the feature map C1 of each video frame (the seventh feature map output by the last AVT module) and the feature map C2 of the object contained in each video frame (the object feature map output by the last AVT module).

[0314] in, Figure 6 The "M×" shown in indicates that the ADT network may include M (M≥1) superimposed ADT modules, and the AVT network may also include M superimposed AVT modules. The number of feature extraction modules corresponding to the ADT network and the AVT network may be the same or different. The ADT network and the AVT network may be referred to as feature extractors in the solution provided in the embodiment of the present application.

[0315] Step S34: The extracted object features are judged. If the extracted objects are behavior subjects and the number exceeds the set threshold, subsequent operations are performed. Otherwise, the process ends without outputting any content.

[0316] This step is based on the feature map of each video frame output in the previous step, and determines whether each video frame contains a behavior subject through a behavior subject identification module (behavior subject identifier).

[0317] Optionally, for each video frame, a rectangular feature map can be cropped from the object feature map C2 of the video frame to fit the object contour, and the frame feature map C1 of the video frame and the cropped object feature map are adjusted to vectors of the same size and spliced. After the splicing is completed, the vector (behavior subject feature) will be passed through a trained behavior subject identifier to generate a score (behavior subject discrimination score). Vectors exceeding a certain score threshold will be treated as behavior subject vectors (the object in the video frame is the behavior subject) and enter the next step of processing. If the behavior subject discrimination score corresponding to a video frame is less than the score threshold, it is considered that the video frame does not contain a behavior subject and the video frame can no longer be processed.

[0318] Optionally, for a video to be processed, if the number of video frames containing the behavioral subject is relatively small, the video may not be processed subsequently; if the number of video frames containing the behavioral subject exceeds a set threshold, subsequent processing may be performed based on the video frames containing the behavioral subject.

[0319] Optionally, the identified subject vectors can also be passed through a trained subject quality identifier to give each vector a quality score. The quality score corresponding to a video frame represents the quality of the subject / object contained in the video frame. The higher the score, the higher the quality.

[0320] Step S35: object information interaction. In this step, the trained information interaction network (CCT network) can be used to allow the behavior subject to perform information interaction to help better identify the behavior subject.

[0321] The input of this step is all the subject vectors identified in the previous step. The correlation between the subject vectors can be learned through a neural network, so that each subject vector can fuse the information of the subject vector associated with it to obtain a new subject vector corresponding to each video frame.

[0322] Optionally, for each subject vector, you can first find several subject vectors that are most similar to it from all subject vectors. For example, you can use cosine similarity to find K vectors that are similar to itself. Then use each subject vector as a query vector (query vector). For each query vector, input the query vector and the K vectors similar to it into a network composed of Context Contrast Transformer (CCT network). The query vector interacts with the K vectors, so that the query vector obtains its own multi-angle and multi-scene semantic information from the K vectors. The network outputs a new query subject vector with multi-angle and multi-scene semantic information. After each subject vector passes through the CCT network, it can learn to obtain its corresponding new subject vector.

[0323] Step S36: Post-process the behavior subject.

[0324] The input of this step is each new behavior subject vector obtained via the CCT network. This step may include a behavior subject aggregation stage S361 and an event confirmation stage S362.

[0325] In the first post-processing stage S361, all new subject vectors obtained in the previous step can be aggregated to obtain at least one aggregation result. The specific method of aggregation processing is not limited in the embodiment of the present application. Optionally, all subject vectors can be aggregated using graph propagation. After aggregation, all subject vectors are divided into multiple different aggregations. Among them, the subjects corresponding to these subject vectors in each aggregation obtained after aggregation are regarded as the same subject, such as the same person. The output of behavior aggregation is a subject vector with an aggregation label. The specific label of an aggregation represents the subject in the aggregation.

[0326] In the second stage of post-processing S362, the frame where the subject vector is located in each aggregation can be found, and each frame where the subject is located can be cropped, and rectangular frames can be cropped according to the shape of the subject. These cropped new rectangular frames are formed into continuous segments according to a certain frame spacing. The segment is the event. Optionally, for each aggregation, the frame corresponding to the subject vector with the highest quality score in the aggregation can also be found, and the subject can be segmented in the frame. The output of the post-processing step can include the segmented subject and the corresponding event in each aggregation.

[0327] Step S37: Show the entire video to the user and let the user select the subject of the behavior from the video.

[0328] Step S38: Output the behavior subject and related events selected by the user.

[0329] like Figure 6 and Figure 7 In the schematic diagram shown, the output after post-processing includes three behavior subjects and event segments corresponding to each behavior subject. All behavior subjects segmented in the post-processing stage S362 can be displayed to the user first, and the user can select a behavior subject of interest, and then output the behavior subject and related events. Of course, all behavior subjects and related events can also be displayed to the user.

[0330] Optional embodiment 2

[0331] The video processing principle of Example 2 can be further referred to Figure 6 , Figure 8 The flowchart of the video processing method of Embodiment 2 is shown. Embodiment 2 differs from Embodiment 1 in that the second stage S362 of post-processing and the implementation of Steps S37 and S38 are different. Embodiment 2 provides a user interaction method different from that of Embodiment 1.

[0332] The steps before the post-processing stage and the first stage of the post-processing stage of Example 2 may be the same as steps S31 to S35 and the first stage S361 of the post-processing in Example 1.

[0333] The steps after the first stage of post-treatment of Example 2 may be as follows:

[0334] In the second stage of post-processing S362, the frame where the subject vector is located in each aggregation can be found, and the background of each frame where the subject is located can be removed. For example, the pixel values ​​of the background area in the frame where the subject is located can be set to 0, the original pixel values ​​of the subject area can be retained, and the video frames with only the subject can be output. These new video frames can be formed into continuous segments according to a certain frame spacing, and the segment is the event. Then find the frames corresponding to all the subject vectors in the aggregation, and segment the subject in the frame to obtain the segmented subject and event in each aggregation.

[0335] By comparison, it can be seen that the difference between the second stage of post-processing in Example 1 and Example 2 is that: Example 1 crops the video frames according to the shape of the subject in each video frame, and obtains the event segment based on the cropped new video frames containing the subject, while Example 2 removes the background area in each video frame, and obtains the event segment based on each video frame with the background removed. Of course, in actual implementation, it is also possible not to crop the video frames or remove the background, and directly obtain the event segment corresponding to each aggregation based on each video frame corresponding to each aggregation.

[0336] As an example, Figure 9a A schematic diagram of the effect of Example 2 is shown. By removing the background, the background area is all white, and the subject area is an image processed with the original pixel values ​​in the video frame. For each aggregation, the event corresponding to the aggregation can be obtained based on all processed images corresponding to all subject vectors in the aggregation. The subject corresponding to the aggregation is the subject in the image with the highest quality score. The subject in this example is a person, and the event that occurred is a person doing long jump.

[0337] When the user is watching the video, the behavior subject obtained by the second stage 362 of post-processing can be displayed to the user. As an optional implementation method, if the user is interested in a certain behavior subject, he can long press the behavior subject in the video. After the long press, a related event button will be displayed on the page. When the user clicks the button, the behavior subject and its related events will be displayed to the user, such as Figure 8 and Figure 9b As shown in the schematic diagram, the user recognizes two events in the video he is watching, such as Figure 9bThe cover of the two event clips shown below. When the user watches the video, long press the subject of the behavior in the video frame displayed on the user interface, and the user interface will display Figure 9b The "hand" button shown in the figure can display Figure 9b The subject of the behavior and related events are shown on the right. Optionally, as shown in the figure, the user can also perform further operations on the related events, such as sharing the information of the event, or saving the video clip, and can also click on the event clip to play it, etc.

[0338] Optional embodiment 3

[0339] Example 3 can be regarded as a simplified version of Example 1. Fig.10 A schematic flow chart of the video processing method of Embodiment 3 is shown. Fig.11 The schematic diagram of the framework structure of the video processing system corresponding to the embodiment 3 is shown. The video frame sampling, video preprocessing and image block encoding steps of the embodiment 3 can be the same as these steps (steps S1, S2 and S31) of the scheme 1. Fig.10 and Fig.11 As shown, in Embodiment 3, after obtaining the feature map of each video frame through image block coding, the processing steps may be as follows:

[0340] Step 32: Each video frame is input into the trained MoViNet network (Mobile Video Networks), the semantic information of the video frame is extracted, and a frame feature map with the above information is output.

[0341] The input of this step is the output of the previous step, that is, the feature map of each video frame output by the image block encoding module. Based on the feature map of each video frame, a new feature map with semantic information corresponding to each video frame can be extracted using a mobile video network.

[0342] Step S33: Input the frame feature map output in the previous step into the ADT network to extract the coarse-grained contour and relative position information of the object, and output a frame feature map with the above information.

[0343] Step S34: Input the frame feature map output in the previous step into the segment confirmation module. The segment confirmation module can process the same number of consecutive frame feature maps and output multiple consecutive frame feature maps as vectors of fixed length (segment vectors) representing a video segment.

[0344] like Fig.10As shown, the input of this step is the output of the ADT network. The fragment confirmation module is also a trained neural network, which can process the features of a set number of video frames and output a feature vector of a set length corresponding to the set number of video frames. The feature maps of a set number of video frames in this step are processed by one or more processing methods such as feature fusion, feature extraction, and feature pooling. Through this step, feature dimension reduction processing can be performed on the feature maps of multiple video frames to obtain fragment vectors of multiple video frames.

[0345] For example, if the set number is 10 frames and the ADT network outputs feature maps of 27 frames, the feature maps of 27 frames can be supplemented to feature maps of 30 frames (an integer multiple of the set number). For example, the feature map of the last frame is copied to the previous multiple (a multiple of the ratio of the number of video frames to the set number is rounded up), and then every 10 frames are taken as a video segment, and the feature maps of every 10 frames are passed through the segment confirmation module to obtain a segment vector.

[0346] Step S35: Input each segment vector outputted in the previous step into the scoring model to score the segment vector.

[0347] The network structure of the scoring model is not limited in this embodiment of the application. Optionally, the scoring model is composed of multiple trained fully connected layers, and each segment vector is input into the model to output a score of the segment vector, such as Fig.10 The scores shown in 0.6, 0.3, ..., 0.7, the score of a segment vector represents the possibility that the video segment corresponding to the segment vector is an event segment, and the higher the score, the higher the possibility. Video segments with scores higher than the set score threshold can be displayed to the user as event segments, and segments with scores lower than the threshold are not displayed to the user. The specific display method can be configured according to needs, and can adopt but is not limited to the method of displaying events in Example 1 or Example 2, such as directly displaying each event segment to the user, or as Fig.11 As shown, when watching a video, the user can long press a video frame (such as long press an object in the video frame, such as a person), and the event segment to which the video frame belongs can be displayed to the user, or the user can click Fig.11 The following event snippets are shown below ( Fig.11 The terminal device displays any clip of the event clip selected by the user to the user, and the user can also save, share, play, etc. the event clip.

[0348] As another optional embodiment, the ADT network in Embodiment 3 may also be replaced by an AVT network.

[0349] The embodiments of the present application innovatively propose an ADT network, an AVT network, and a CCT network. The feature extraction steps in the various optional embodiments described above can be combined or replaced without conflict. In actual implementation, the AI ​​network in each optional embodiment can include one or more of the ADT network, the AVT network, and the CCT network. Some structures in the AI ​​network are also optional structures, and some structures can be replaced by other networks. For example, Fig.10 The image block encoding module or mobile video network can be removed, or other feature extraction networks can be used to implement it.

[0350] The following describes the steps that may be involved in the optional embodiments of the present application and the optional implementation schemes of the neural network structure (such as image block coding, ADT network, AVT network, CCT network, etc.) that the AI ​​network may include.

[0351] ① Video preprocessing

[0352] For a video obtained on a terminal device (such as a mobile device), all video frames can be sampled at a fixed frame rate, and then it can be determined whether the image format of the video frame is the set image format. If not, the sampled video frame can be converted to the set image format, such as RGB (Red, Green, Blue) format. Optionally, for each video frame of the set image format, it can be converted to a preset size. For example, the short side of each video frame can be reduced or enlarged to a fixed size, and the long side changes size according to the short side scaling ratio. After that, a new video frame with a fixed length and width size can be cropped according to the center point of each frame and input into the AI ​​network for subsequent calculations.

[0353] ②Patch Embedded

[0354] Patch Embedded can use a two-dimensional convolution kernel to sample image blocks for each input video frame (such as the video frames after the above preprocessing). The calculation formula of the convolution processing principle can be expressed as follows:

[0355]

[0356] Among them, x and y represent the x-coordinate (horizontal coordinate) and y-coordinate (vertical coordinate) of a pixel point in a video frame, p*q represents the size of the convolution kernel, p and q can be the same, in which case the convolution kernel is a p*q square, w represents the weight of the convolution kernel (network parameter of the convolution network), and v represents the pixel value of the coordinate (x, y). A convolution calculation is to multiply each pixel value in the image block of size p*q in the video frame with the corresponding weight in the weight matrix of the convolution kernel of size p*q, and then add the p*q product values ​​to obtain a feature value in the feature map. The convolution kernel is used to continuously slide and convolve on the video frame to obtain the initial feature map corresponding to the video frame.

[0357] like Figure 2 The schematic diagram of the convolution calculation principle shown in FIG. 1 shows that the kernel matrix represents the weight matrix of the convolution kernel in formula (1). The matrix size is p*q. The values ​​in the kernel matrix represent the weight w of the convolution kernel. Figure 2 In the diagram shown, the size of the convolution kernel is 3*3, the weights are {0,-1,0,-1,5,-1,0,-1,0}, and the weight values ​​in the weight matrix are obtained by training the neural network based on the training samples. The step size of the convolution can be 3.

[0358] Figure 2 The output matrix in represents the feature map of the video frame obtained after the convolution operation. Figure 2 The pixel values ​​of the image block shown in the figure are {105, 102, 100, 103, 99, 103, 101, 98, 104}. The eigenvalue after convolution calculation is 89. The specific calculation process is as follows:

[0359] 0*105+(-1)*102+0*100+(-1)*103+5*99+(-1)*103+0*101+(-1)*98+0*104=89.

[0360] After the same video frame is convolved with C different convolution kernels, it will output a feature map with C channels. Each video frame is downsampled and encoded using the convolution kernel, and the output feature map with C channels is input into the subsequent AI network to facilitate feature extraction of the video frame.

[0361] ③ADT Network

[0362] The ADT network proposed in the embodiment of the present application is a network composed of Adjacent Dazzle Transformer (ADT), which can extract coarse-grained contours and relative position information of objects in a frame, thereby improving the recognition rate of fast-moving behaviors.

[0363] Fig.12 A schematic diagram showing the visualization effect of the ADT network is shown in Fig.12 As shown, when the ADT network processes the feature map of each video frame, it needs the feature map of the current video frame and at least one preceding frame of the current frame, and the at least one preceding frame includes at least the previous frame of the current frame. In the following embodiments, the preceding frame takes the previous frame of the current frame as an example.

[0364] Each video frame of the video to be processed is divided into multiple patches of the same size after image block encoding (it can be understood here that each pixel in the feature map obtained after image block encoding corresponds to an image block / patch on the video frame), such as Fig.12 A small block in a video frame as shown in . Each patch in each frame needs to exchange information with other patches in the current frame and the previous frame. Fig.12 As shown in the schematic diagram on the left side of FIG, a patch in the current frame T can obtain the object contour information in space by exchanging information with other patches related to the patch in frame T. Fig.12 As shown in the schematic diagram on the right side of the figure, a patch in frame T can interact with other patches related to the patch in the previous frame (frame T-1) of frame T in time to obtain the relative position information of the object. Through the ADT network, information can be exchanged between video frames in space and time at the same time to obtain the coarse-grained object contours and relative positions in the video frames, so that fast-moving behaviors in the video can be accurately identified.

[0365] for Fig.12 In the example shown, the video contains an event of a long jump by a long jumper. The ADT network provided by the embodiment of the present application can generate a coarse-grained contour based on the shape and relative position of the object in space by tracking the same object (athlete) in the previous frame in time. Each video frame is regarded as a sequence of patches of the same size ( Fig.12 For each query block in each video frame, for example Fig.12In the spatial interaction or temporal interaction shown in , a small rectangular box in frame T is located at the thigh of the athlete in frame T. In spatial interaction, multiple rectangular blocks on the left side of frame T are related blocks of the query block (i.e., patches that interact with the query block). In temporal interaction, multiple rectangular blocks on the left side of frame T-1 are related blocks corresponding to the query block in the feature map of the previous frame. The related blocks of the query block in the current frame and its previous frame can locate the entire body of the athlete within the coarse-grained range of frame T. Specifically, the query block interacts with the related blocks on frame T to obtain the contour information of the person. Therefore, the ADT network can obtain the coarse-grained contour of the fast-moving object. The query block interacts with the color-related blocks on frame T-1 to obtain the relative position of the person. Therefore, the ADT network can obtain the relative position of the fast-moving object. Then, by combining the two, the behavior transformation of the fast-moving object can be obtained, and the fast-moving behavior can be accurately identified.

[0366] Fig.13 FIG. 4 shows an optional network structure diagram of the ADT network provided in the embodiment of the present application, such as Fig.13 As shown, the ADT network is mainly composed of multiple superimposed ADT modules. Fig.13 In the network structure shown, the ADT network includes M layers of ADT modules. The input of the ADT network is the feature map of the continuous frames after Patch Embedding. When processing the feature map of each frame, the feature map of the current frame and the processing result of the previous frame of the current frame are required as input. When processing the feature map of the first frame, the feature map of the previous frame can be a preset feature map of the same size ( Fig.13 The feature map of each video frame is extracted by the feature extraction module of the M-layer ADT module, and then a new frame feature map is output. For any video frame, the input of the ADT module of any layer includes the feature map of the video frame output by the ADT module of the previous layer of the layer, and the feature map of the previous frame of the video frame output by the ADT module. By using multiple levels of ADT modules, the semantic information of the video frame can be learned from shallow to deep.

[0367] The process of feature extraction using the ADT network can include the following steps:

[0368] (1) Use the ADT module of the first layer to extract the spatiotemporal information of the feature map of the current video frame. Optionally, the size of the output feature map remains unchanged. The calculation principle of the ADT module can be expressed by the following formula (2):

[0369] K = Concat(Conv(Movement(X t-1 )), Conv(Movement(X t )))

[0370] W t =Conv(Concat(K,Q t ))[:,:,1:2N]

[0371] W t-1 =Conv(Concat(K,Q t ))[:,:,2N+1:4N]

[0372] X output =W t-1 *Vt-1+W t *V t +Q t

[0373] Q t =X t

[0374] V t-1 =X t-1 W v

[0375] V t =X t W v (2)

[0376] Among them, X t represents the feature map at the current time T, X t-1 It represents the output result feature map of the previous frame at time T-1, that is, the feature map of the previous frame is extracted by the ADT module of the first layer, X output It represents the feature map at the current time T output after an ADT operation, that is, the output of the ADT module of the first layer corresponding to the current frame.

[0377] In the above formula, Movement(a) means using a motion module to further extract features from feature map a, where the motion module can be implemented based on a deformable convolutional network (DCN). Conv(b) means performing a convolution operation on feature map b, and Concat(c, d) means concatenating feature map c and feature map d. v Indicates that the feature map V is feature mapped. The feature mapping can be realized by the trained mapping matrix or feature extraction network. Q can use the original feature Figure X , or it can be a new feature map obtained by extracting features from X. In the above formula, through Conv(Concat(K, Q t))The number of channels of the feature map output after feature extraction is 4N, [:,:,1:2N] means taking the feature maps of the first 2N channels among the 4 channels, and [:,:,2N+1:4N] means taking the feature maps of channels 2N+1 to 4N (the last 2N channels).

[0378] As shown in formula (2), to process a feature map of the current frame at time T, it is necessary to input the feature maps of the current frame and the previous frame at the same time. For an object at time T in the current frame, the spatial contour information of the object can be obtained from the current frame, and the temporal relative position information of the object can be obtained from the previous frame to realize the extraction of spatiotemporal information, such as Fig.12 Schematic diagram of the effect shown.

[0379] (2) Continue to use the ADT module of the second layer to extract spatiotemporal information from the feature map of the current frame. The size of the output feature map remains unchanged. The calculation principle is shown in the above formula (2). The input of the ADT module of the second layer is the output of the ADT module of the first layer, including the output of the current frame and the output of the previous frame of the current frame.

[0380] (3) Similarly, after M layers of ADT operations, a new feature map of each frame is output, and the output size remains unchanged.

[0381] The following is a detailed description of the optional implementation method of feature extraction using the ADT module.

[0382] The ADT module can be mainly composed of two parts. As shown in the above formula (2), the first part can use the Movement module to obtain the rough contour and relative position information of the object, and the second part can further perform feature extraction through the Transformer. A new Transformer is proposed in an embodiment of the present application. The attention mechanism (which can be called AD attention) in the Transformer, Q (query vector), K (key vector), and V (value vector) can obtain more accurate coarse-grained contour and relative position information of the object. The ADT module proposed in the embodiment of the present application can more accurately obtain the spatiotemporal posture changes of the object, and therefore can better identify fast-moving behaviors.

[0383] The specific neural network structure of the ADT module is not limited in the present embodiment. The ADT module includes an ADT attention layer as an optional solution. Fig.14 A schematic diagram showing an optional structure of the ADT module provided in an embodiment of the present application is shown in FIG. Fig.14As shown in FIG, the ADT module may include a cascaded ADA (Adjacent Dazzle Attention) layer, a residual connection and normalization (Add&Norm) layer, a feedforward network layer, and a residual connection and normalization (Add&Norm) layer. Among them, the structure except the ADA layer is an optional structure (it may be present or absent, and may also be replaced by other feature extraction networks). Fig.14 In the network structure shown, the input feature map of the ADT module is subjected to feature extraction by the ADA layer, and the obtained new feature map and the input feature map of the ADT module are used as the input of the Add&Norm layer. After feature map addition and layer normalization, the output feature map of the Add&Norm layer is obtained. The output feature map and the feature map after processing by the feedforward network are used as the input of a new Add&Norm layer, and the output of the Add&Norm is used as the output of the ADT module. Fig.13 Each ADT in represents the ADT operation of the current frame. The ADT module in layer 1 extracts shallow semantic information of the video frame, and the ADT module in layer M extracts deep semantic information of the video frame.

[0384] The ADT network provided in an embodiment of the present application is described below. The ADT network includes one or more cascaded ADT modules.

[0385] Fig.15a A schematic diagram of the network structure and feature extraction principle of an ADA mechanism in an ADT module provided in an embodiment of the present application is shown. The feature extraction process of the ADT module is described below in conjunction with the network structure.

[0386] For each video frame, due to the Patch Embedded module, the frame feature map is composed of multiple patches of the same size. Each patch can be understood as a feature point / pixel point in the feature map, and a feature point corresponds to an image block in the video frame. To extract information from the feature map of each frame, it is necessary to complete the calculation of all patches on the feature map. Fig.15a The schematic diagram shows the ADT single frame feature map at time T ( Fig.15a In frame T, it can be understood that Fig.15a Although the input of the ADT module and the output of the mobile module are expressed in the form of video frames, in the actual processing process, both the input and output are the calculation process of one patch of the feature map, and all other patches of the video frame can be calculated synchronously in the same way.

[0387] For each patch of frame T, when calculating the patch, the patch is used as the query patch (Querypatch), such as Fig.15aAs shown, the calculation process of the ADT module for a patch on a single-frame feature map at time T can include the following steps:

[0388] (1) To calculate a query patch on the feature map at time T, the frame feature maps at time T and time T-1 need to be input.

[0389] Among them, for the ADT module of layer 1, that is, the first ADT module, Fig.15a The frame T-1 and frame T in the above respectively represent the feature map of frame T-1 and the feature map of frame T obtained by image block coding. Here, frame T corresponds to the current frame, and frame T-1 corresponds to the previous frame of the current frame. For the ADT modules except layer 1, Fig.15a Frame T in the figure is the output feature map of the previous ADT module of the current frame, and frame T-1 is the output feature map of the ADT module of the current layer of the previous frame. That is, during implementation, it is necessary to perform the ADT operation of the current layer on the previous frame, and then perform the ADT operation of the current layer on the current frame, because the ADT operation of the current layer of the current frame needs to apply the output feature map of the ADT operation of the current layer of the previous frame.

[0390] In the following description, frame T may also be described as a feature map at time T or a feature map of frame T, and frame T-1 may be described as a feature map at time T-1 or a feature map of frame T-1.

[0391] For the query patch at time T, N surrounding patches for generating offsets can be defined in advance, such as Fig.15a As shown in the figure, there are multiple small blocks (small rectangular boxes) in frame T. For the patch at the same position as the query patch on the feature map at time T-1, N patches at the same position for generating offsets are also defined in advance, such as Fig.15a As shown in frame T-1 in .

[0392] (2) The Movement module is composed of trained Deformable Convolutional Networks. After passing through the Movement module, the N patches on the feature map at time T and time T-1 will be offset, and these patches will be offset to the object area related to the location of the query patch.

[0393] like Fig.15aAs shown in the schematic diagram, the object in the video frame is a person doing long jump. Before being processed by the mobile module, the associated patch of the query patch is a number of patches adjacent to it. After being processed by the mobile module, the position of the associated patch of the query patch in the feature map is offset to the area where the person in the video frame is located. It can be understood that the associated patch of the query patch can be found in the area close to the object in the video frame through the mobile module, so that the contour and position information of the object in the image can be more accurately identified.

[0394] (3) The feature graphs at time T and time T-1 output in the previous step undergo a specific convolution operation ( Fig.15a After the Conv connected to the mobile module in the network, two new feature maps will be output, and each patch on the new feature map can obtain global information. This specific convolution operation is a convolution processing method proposed in an embodiment of the present application, which will be introduced in detail later.

[0395] (4) The two feature maps outputted in step (3) are concatenated. The concatenated feature maps can be called Keys, which are feature maps that combine the spatial information at time T and the temporal information at time T-1.

[0396] (5) A query patch on the key and feature graph at time T ( Figure 1 The query in the query is spliced ​​to obtain a new feature map. This operation combines the query patch with the spatiotemporal information and strengthens the dominant position of the query patch.

[0397] (6) The new feature map outputted in step (5) is fed into a 1*1 convolution, and two Offshoots (offset feature map / weight feature map) are outputted, namely the Offshoots at time T and the Offshoots at time T-1, which are used for the offset of N patches on the feature map at time T and the offset of N patches on the feature map at time T-1, respectively, to help more accurately find other patches related to the Query patch. The detailed description of obtaining Offshoots will be introduced later.

[0398] (7) In the current step, the two feature maps at time T and time T-1 after the Movement module are defined as Values ​​at time T and Values ​​at time T-1.

[0399] Among them, the Offshoots at time T is used as the offset of the deformable convolution operation, and the convolution operation is performed based on the offset and the feature map Values ​​at time T, and a new feature map Outline result (outline feature map) is output. The method of calculating Offshoots proposed in the embodiment of the present application is a new method, which is different from the current offset calculation method of the deformable convolution operation. The Offshoots provided in the embodiment of the present application is used as the offset of the deformable convolution calculation to implement the deformable convolution calculation, which can more accurately find other patches related to the Query patch on the feature map at time T, and obtain more accurate object contour information.

[0400] Similarly, the Offshoots at time T-1 is used as the offset corresponding to the feature map at time T-1, and a variable convolution operation is performed with the feature map Values ​​at time T-1 to output a new feature map Relative Position result. By using the Offshoots calculation method provided in the embodiment of the present application, other patches related to the Query patch can be more accurately found on the feature map at time T-1, thereby obtaining more accurate relative position information.

[0401] (8) The feature Outline result (second feature block) at time T, the feature RelativePosition result (first feature block) at time T-1 and the Query patch are fused (e.g., added) and output to obtain a new Query result, that is, the new feature obtained after the Query patch undergoes an ADT operation.

[0402] After performing the above processing on each patch corresponding to each video frame using the same process, a new Query result corresponding to each patch is obtained. The new feature map of a video frame after an ADT operation is the new Query result of all patches corresponding to the video frame.

[0403] It can be understood that when the first ADT operation is performed on each video frame, the input feature map is the feature map obtained by encoding the video frame through image blocks. For ADT operations other than the first one, the input feature map is the feature map output by the previous ADT operation.

[0404] For the embodiments of the present application Fig.15aIn the neural network structure of the ADT module, the AD attention mechanism is the core component of the ADT module. For each video frame, when the ADT module is used to further learn the semantic information of the image, first, the input feature map of the video frame can be convolved (deformable convolution) by the mobile module of the mobile network including multiple convolution modules. Each convolution operation can predict the moving direction and distance of each patch, and then the patch can be moved to the rough area of ​​the object. Therefore, the moving patch can form a coarse-grained outline of the object in frame T and a relative position in frame T-1. Furthermore, the Query, Keys and Values ​​in the novel attention mechanism proposed in the embodiment of the present application fuse the information of frames T and T-1 at an early stage, so the outline and relative position information of the object can be well fused. The behavior transformation / change of the object can be well identified.

[0405] Fig.15b A schematic diagram showing the visualization effect of the AD attention mechanism in the ADT module provided in an embodiment of the present application is shown as follows: Fig.15b As shown in , Query is the Q vector in the attention mechanism, which is a block in the video frame (such as the eigenvalue of a feature point). The K vector (Keys in the figure, corresponding to the Keys in the above step (4)) and V vector (Values ​​in the figure, corresponding to the Values ​​in the above step (7)) of the attention mechanism are obtained based on frame T and frame T-1. By interacting the K vector and the Q vector, the corresponding weight feature map can be learned. Based on the weight feature map and the V vector, a new feature with the contour information and relative position information of the object in the image after interactive learning can be obtained, that is, the new Query corresponding to the Q vector. The new feature obtained by this scheme obtains the coarse-grained contour of the object from the relevant blocks of frame T and the relative position of the problem from the relevant blocks of frame T-1.

[0406] Fig.15cThe effect diagram of the calculation process of a new Query for a patch (query block) in frame T is shown in FIG. Query in the figure represents the query block, and the output represents the new Query. Based on frame T, the convolution calculation in the moving module is adopted, and the preset number of related blocks of frame T and Query in frame T are re-moved to the new positions in frame T and T-1, and the moved blocks form the coarse-grained contour of the object in frame T and the relative position in frame T-1. The next step is the operation of Q, K, and V in the novel attention mechanism, where Q comes from the query block, and the query block can be directly used as Q, or feature extraction or feature conversion can be performed on the query block to generate Q. K is obtained based on the feature convolution operation, and V comes from the feature map output by the moving module. After Q and K are connected, the weights of frames T and T-1 are generated by convolution operation. Then, the relative position result is generated by multiplying the weight and value of frame T-1, and the contour result is generated by multiplying the weight and value of frame T. The output can be generated by adding the query block, the relative position result, and the contour result.

[0407] New convolution operation

[0408] Fig.16a FIG. 4 is a schematic diagram showing a new specific convolution operation in the ADT proposed in an embodiment of the present application. Fig.16b A diagram showing the visualization effect of the convolution operation, Fig.16c The convolution result of this convolution operation is compared with the traditional convolution result. Figures 16a to 16c The new convolution operation provided in the embodiment of the present application is introduced. The following is an example of a video frame. Fig.16a As shown, the process of the convolution operation may include the following steps:

[0409] (1) The input is the frame feature map output by the Movement module. Each feature map consists of M patches of the same size (the feature map output by image block encoding includes M patches).

[0410] in, Fig.16a The (M, H, W, C) shown in the figure represents the parameters of the feature map, M represents the number of patches, H and W represent the height and width of the feature map, and C represents the number of channels of the feature map. It can be seen that Fig.16a The number of channels and the size of the feature maps output by each layer in the convolutional network shown are unchanged.

[0411] (2) Use the traditional 3*3 convolution kernel to extract local information of the feature map.

[0412] (3) The Transpose operation is used to rearrange the channel features in space to obtain the rearranged feature map (M, 1, C, H*W), as shown in Fig.16a and Fig.16b As shown in the figure, the size of the feature map after rearrangement through a conversion module is 1*C, and the number of channels of the feature map is H*W. That is to say, the conversion operation converts the C feature values ​​of each feature point in the feature map of C channels obtained by feature extraction through traditional convolution into a feature map of one channel.

[0413] (3) After the Transpose operation, 1*1Conv can be used to allow each patch to obtain information about the global receptive field.

[0414] like Fig.16a and Fig.16b As shown in the figure, since the number of channels of the feature map after the above conversion is H*W, and the feature map of one channel is a feature map composed of the eigenvalues ​​of a feature point on all channels, when the 1*1 convolution is used to perform a convolution operation on the converted feature map, the eigenvalues ​​of the H*W channels at each position in the converted feature map will participate in the convolution calculation. Since the eigenvalues ​​of these H*W channels contain the eigenvalues ​​of all feature points in the feature map before conversion, the eigenvalues ​​obtained at each position after the 1*1 convolution operation integrate the global information of the video frame, rather than the local information.

[0415] (5) Use the Transpose operation to restore the feature map output in step (4) to the input size, that is, rearrange the feature map (M, 1, C, H*W) into a feature map of (M, H, W, C), in which each patch obtains global information.

[0416] It can be seen from the above convolution operation process that the above new convolution operation provided by the embodiment of the present application is different from the traditional convolution method. Fig.16c The comparison of the results of the convolution operation provided in the embodiment of the present application and the traditional convolution operation is shown. It can be seen from the figure that the receptive field of the traditional convolution operation is limited, and each patch can only obtain local information. For example, for a convolution operation with a convolution kernel size of 3*3, the result of the convolution operation corresponding to each patch can only obtain information of 9 patches. The above-mentioned new convolution operation provided in the embodiment of the present application allows each patch on the output feature map to obtain global information, which can help to generate more accurate offsets, i.e., Offshoots, and obtain more accurate object contours and relative positions.

[0417] Calculation of Offshoots

[0418] Fig.17aA diagram showing a visualization of the meaning of Offshoots, Fig.17b The schematic diagram of the principle of a new deformable convolution operation in the ADT proposed in the embodiment of the present application is shown. In the embodiment of the present application, Offshoots refers to allowing the patch on the feature map to obtain the offset distance on the X-axis and Y-axis to find a new patch, such as Fig.17a The left side of the diagram shows a schematic diagram of the offset effect of a patch. The pink block (the block not filled with black) is offset to the blue block (the block filled with black) based on its offset distance △x and △y. When N patches are offset to the area where the object is located, the contour of the object can be obtained based on these offset patches, as shown in Fig.17a The right side of the figure shows the visualization effect of Offshoots. The blocks filled with diagonal lines in the figure represent a query vector. The N pink blocks not filled with black are the associated patches before the offset corresponding to the query vector. The blue blocks filled with black are the N associated patches after the offset corresponding to the query vector. It can be seen that the associated patches after the offset are closer to the subject / object in the video frame (the person doing the long jump in the figure). The meaning of Offshoots can be expressed as follows:

[0419] Offshoots = (△x, △y) (3)

[0420] In formula (3), △x and △y are represented as two feature maps of the same size, which are used to predict the offset of the patch in the X-axis and Y-axis respectively, that is, each patch corresponds to the offset in the horizontal and vertical directions. Offshoots are mainly generated by the second part of the ADT structure, Q, K, V operations. In the ADT module, the feature maps of two adjacent frames are operated at the same time to generate Offshoots. Assuming that the set number of associated patches of each query patch is N, corresponding to the ADT structure shown in Figure 15, the output of the 1*1 convolution is a feature map including 4N channels, that is, the number of input channels of the 1*1 convolution operation is N, among which the feature maps of the first 2N channels (the feature maps of the X-axis and Y-axis corresponding to each patch in the N patches) are the offset feature maps corresponding to the current frame in the two adjacent video frames, and the feature maps of the last 2N channels are the offset feature maps corresponding to the previous frame.

[0421] Fig.17b The specific process of generating the Offshoots of a feature graph at time T is shown. The four steps in the larger dotted box in the figure lead to the generation of the final Offshoots. The generation process can include the following steps:

[0422] (1) The input is the feature map at time T after the mobile module. Assume that the size of the feature map is 32*32*C, where C is the number of output channels of the mobile module. For each patch of the video frame, the N patches associated with the patch on the feature map output by the mobile module are also offset, and the rough outline of the object is obtained based on the area formed by these N patches.

[0423] (2) This step corresponds to Fig.17b In step 1, the feature map output by the mobile module at time T is subjected to a specific convolution operation to output a new feature map Key with a size of 32*32*C. Each patch on the feature map Key obtains the global information in the input feature map. This operation allows the N patches to have a clearer judgment of their own positions, helping the subsequent Offshoots to generate more accurate offsets and move to an area that better fits the shape of the object.

[0424] (3) This step corresponds to Fig.17b In step 2, the feature map Key at time T and the feature map Key at time T-1 are concatenated. The concatenated feature map is called feature map Keys, with a size of 32*32*2C. This operation combines the spatial information at time T and the temporal information at time T-1. The process of obtaining the feature map Key at time T-1 is the same as the process of obtaining the feature map Key at time T.

[0425] (4) This step corresponds to Fig.17b In step 3, the feature map Keys and a query patch on the feature map at time T are concatenated to obtain a new feature map (Keys & Query in the figure) with a size of 32*32*3C. When concatenating features, if the sizes of the features to be concatenated are inconsistent, the features need to be converted into feature maps of the same size before concatenation.

[0426] The operation in step 3 merges the query patch and the spatiotemporal information, strengthening the dominance of the query patch and allowing subsequent offsets to revolve around the query, looking for other patches related to the query patch.

[0427] (5) This step corresponds to Fig.17bIn step 4, the feature map Keys & Query passes through a 1*1 convolution, and outputs Offshoots with a size of 32*32*2N, which is used to predict the offset of N patches corresponding to a Query patch on the feature map at time T. The Offshoots corresponding to a video frame consists of a feature map of size 32*32*2N, with 2N channels. The first N channel feature maps are called △x, with a size of 32*32*N, which are used to predict the offset of N patches on the X axis. The last N channel feature maps are called △y, with a size of 32*32*N, which are used to predict the offset of N patches on the Y axis.

[0428] Combining the spatial global information obtained in step (2) and the spatiotemporal information obtained in step (3), based on the query information (feature map Keys & Query) obtained in step (4), a new offset Offshoots is generated after information fusion in step (5). Since the offset obtains a larger receptive field and better information fusion, the Offshoots can allow the patch to be offset to a more accurate position.

[0429] According to the Offshoots generation process of the feature map at time T, Offshoots can make the patch generate a new offset, which helps to more accurately find other patches related to the query patch and obtain a more accurate object contour. The Offshoots generation process of the feature map at time T-1 that can be performed simultaneously in the ADT result is similar to that at time T, so a more accurate relative position can be obtained. Based on the spatial contour information and the time relative position information, the behavior changes of the object can be better captured, and the events in the video can be more accurately identified, even fast-moving behaviors can have a good recognition effect.

[0430] The new feature map of each video frame extracted by the ADT network can be used as the input feature map of the AVT network, and a more fine-grained feature extraction can be performed through the AVT network.

[0431] ④AVT network

[0432] The AVT network proposed in the embodiment of the present application is a network composed of AVT modules, which can extract the fine-grained contour and absolute position information of objects in the frame, so as to more accurately identify objects. For object recognition, the more accurate the posture and position of the object are, the higher the accuracy of object recognition. The AVT network can help obtain the absolute position and fine-grained contour of the object in the frame.

[0433] Fig.18The diagram shows a visualization effect diagram of an AVT network proposed in an embodiment of the present application. The input of the AVT network can be the feature map of each video frame input by the ADT network. Of course, if the AI ​​network does not use the ADT network, the input of the AVT network can also be the feature map of each video frame output by the image block encoding module. When the AVT network is used for AVT operation, the processing of each video frame needs to be based on the feature map of the video frame and the previous frame of the video frame. For example, the current video frame is the video frame at time T. Fig.18 As shown in the effect diagram, the AVT network can first segment the approximate outline and absolute position of the object in the video frame through the object mask module ( Fig.18 Then perform ADT operation within the rough contour range to obtain further detailed contours and absolute positions ( Fig.18 The fine-grained graph T in the image helps improve the accuracy of object recognition.

[0434] Fig.19 An optional network structure diagram of the AVT network provided in the embodiment of the present application is shown below. Fig.19 The network structure shown and Fig.18 The visualization diagram shown in the figure introduces the feature processing principle in the AVT network. Fig.19 As shown, the AVT network may include multiple superimposed AVT modules. Fig.19 In the schematic diagram, the AVT network includes M layers of AVT modules.

[0435] The input of the AVT network is the feature map of consecutive frames after passing through the ADT network. When processing each frame feature map, the current frame feature map and the previous frame result are required as input. When processing the first frame feature map, the previous frame feature map can be a feature map with the same size value of 0. The feature map of each video frame is subjected to M layers of AVT operations to output a new frame feature map. The processing flow of the AVT network may include the following steps:

[0436] (1) The AVT module of the first layer is used to further extract the spatiotemporal information of the feature map of the current video frame. The size of the output feature map can remain unchanged. The calculation principle of the AVT module can be expressed as follows:

[0437] W t =Warp(Mask(Concat(X t , X t-1 )))

[0438] X output =ADT(W t , X t-1 )+X t (4)

[0439] Among them, X t represents the feature map at the current time T, X t-1 It represents the output result feature map of the previous frame at time T-1, that is, the feature map of the previous frame is extracted by the ADT module of the first layer, X output It represents the feature map at the current time T after an AVT operation, that is, the output of the ADT module of the first layer corresponding to the current frame. Wrap(d) means filling the feature map with feature values, Mask(e) means using the object mask module to perform an object mask operation (Mask operation) on the feature map e, and the object mask module may include an image segmentation network and an activation function layer. ADT(f, m) means performing an ADT operation on the feature map f and feature m.

[0440] As shown in formula (4), when the AVT network is used to process the feature map, a Mask operation is added. For an object at time T in the current frame, the absolute position information of the object can be obtained through this operation. Optionally, the ADT operation can still be retained in the AVT network to further complete the spatiotemporal information extraction and obtain more fine-grained contour and position information of the object. (2) Continue to use the second-layer AVT module to further extract the spatiotemporal information of the feature map of the current frame. The size of the output feature map remains unchanged. The calculation method is shown in the above formula (4).

[0441] (3) Similarly, after M layers of AVT operations, a new feature map of each frame is output, and the output size remains unchanged.

[0442] Fig.20a FIG. 4 shows a schematic diagram of a network structure and feature extraction principle of an AVT module provided in an embodiment of the present application. Fig.20a As shown, an AVT module may include an object mask module, a deformation module and an ADT module. Fig.20a Taking the example of using the AVT module to perform an AVT operation on the feature map of a single video frame at time T, the principle of the AVT module is described in detail. The calculation process of the AVT operation may include the following steps:

[0443] (1) Taking the feature map of the video frame at time T as an example, the input of the first AVT operation of the video frame is the feature map T of the video frame output by the ADT network and the output feature map T-1 of the first AVT operation of the previous video frame of the video frame. The input of each AVT operation except the first AVT operation is the output feature map T of the previous AVT operation of the video frame and the output feature map T-1 of the current AVT operation of the previous video frame of the video frame.

[0444] (2) The feature map T and the feature map T-1 are concatenated to obtain a new feature map, which is used to represent the features of the current video frame.

[0445] like Fig.20a As shown in the figure, it is assumed that the feature map of each video frame output by the ADT network is a W*H*C feature map, where W and H are the width and height of the feature map, and C is the number of output channels of the feature map output by the ADT network. The feature map T of the current frame at time T and the feature map T-1 of the previous frame of the current frame are concatenated to obtain a W*H*2C feature map.

[0446] (3) The spliced ​​feature map is input into the object mask module, and the mask feature map T of the current frame is obtained through the output of the module.

[0447] By using the mask module, the approximate contour and absolute position information of the object in the video frame can be obtained. The specific neural network structure of the mask network is not limited in the present embodiment. Optionally, the object mask module may include trained multi-layer convolution and excitation functions (such as Sigmoid function). The object mask network is a segmentation network with F in represents the input of the object mask module, and the output of the object mask module is F out It can be expressed as follows:

[0448] F out =Sigmoid(H conv (F in ))

[0449] Among them, F in It is the feature map after the concatenation of feature map T and feature map T-1, such as Fig.20a As shown, the input feature map of the object mask module is W*H*2C, H conv (.) indicates multi-layer convolution operation. Sigmoid (.) indicates sigmoid operation. After the object mask operation, a mask feature map T (or mask feature map, mask map, mask map) with the approximate outline and absolute position of the object is generated. The mask feature map is a binary feature map, such as Fig.20a The size shown in the figure is W*H, and the number of channels is 1. The feature map has only values ​​0 and 1. The area with a value of 1 in the mask feature map indicates the area where the object is located, and the area with a value of 0 indicates that there is no object. After the segmentation network, the absolute position and approximate outline of the object in the frame at time T can be obtained.

[0450] (4) This step is used to fill the feature value of the area with a value of 1 in the mask feature map T. The filling value can be the feature map T from step (1). After the feature value filling, a deformed feature map T corresponding to the video frame at time T containing only the segmented object area is obtained, which can also be called a visible feature map, visible map or deformed map.

[0451] like Fig.20a As shown, this step can be implemented by using a deformation module, which can process the feature map of each channel of the feature map T of the W*H*C feature map separately based on the mask map T. The processing result is that the feature values ​​of each feature point corresponding to each position with a value of 1 in the mask feature map T in each channel are retained, and the feature values ​​of each feature point corresponding to each position with a value of 1 in the mask feature map T are removed, and finally the display feature map T of W*H*C is obtained, in which only the feature values ​​of the area containing the object in the feature map T of the current frame are retained, and the feature values ​​of other areas are 0. The approximate outline of the object in the video frame can be seen in the deformed feature map T.

[0452] Through the deformation operation, the feature values ​​are filled into the approximate contour area of ​​the mask map T, thereby obtaining the deformed feature map T containing semantic information such as color and posture.

[0453] (5) The visible feature map T obtained in step (4) and the input feature map T-1 in step (1) are input into the ADT module, and a fine-grained feature map T is output. The operation of the ADT module here is consistent with the principle of the ADT module in the ADT network, except that the input feature map at time T is different. The ADT module further calculates the object contour in the segmented object area to obtain a finer-grained object contour and absolute position.

[0454] The input of the ADT operation in the AVT module has both temporal and spatial interactions in the ADT operation. The spatial interaction is for the deformation graph T, and the temporal interaction is for the deformation graph T and the feature graph T-1. As an example, Fig.20b A schematic diagram showing the effect of ADT operation in an AVT module on spatial interaction is shown in FIG. Fig.20b As shown in , the spatial interaction is performed in the rough contour area of ​​the deformation map T. To generate a fine-grained feature map T containing the fine-grained contour and absolute position information of the object, as shown in Fig.20b The fine-grained graph T shown in .

[0455] (6) Add the fine-grained feature map T obtained in step (5) and the input feature map T in step (1) to obtain the final output of the current AVT operation.

[0456] The above steps show an AVT operation for a single frame. All other video frames in the video need to go through the above operation. AVT operation can help obtain the fine-grained outline and absolute position of the object, helping to better identify the object.

[0457] ⑤Behavior subject judgment module

[0458] The behavior subject judgment module is mainly used to screen behavior subjects and score their quality. Fig.21 FIG. 1 shows a schematic diagram of the principle of a behavior subject judgment module provided in an embodiment of the present application, such as Fig.21 As shown, the input of this module includes the feature map of each video frame output by the AVT network and the object feature map output by the deformation module of the AVT module of the last layer in the AVT network. For each video frame, the processing flow of this module may include the following steps:

[0459] (1) A rectangular feature map of the object is cut out from the deformation feature map. The rectangle must contain the feature value region where the entire object is located and fit the feature value region. Specifically, the rectangular feature map can be the minimum circumscribed rectangle of all non-zero pixel values ​​in the visible feature map.

[0460] (2) The feature map of the video frame is fused with the rectangular feature map to obtain an object vector. Optionally, the rectangular feature map can be resized and stretched into a vector of a fixed length, and the frame feature map can also be resized and stretched into a vector of the same fixed length. Then, the two feature vectors of the same length are concatenated to form an object vector.

[0461] (3) The object vectors obtained in step (2) are passed through the behavior subject recognition module, which consists of a trained classification network and gives each object vector a score ( Fig.21 The behavior score in the image is used to determine whether the object vector is a behavior subject vector, that is, to determine whether the object has a preset behavior. These preset behaviors can be defined in advance, for example, preset behaviors can include shooting, blinking, jumping, etc. Only object vectors that exceed a certain score threshold will be determined as behavior subject vectors and enter the next calculation.

[0462] (4) Object vectors that are judged to be action subject vectors will pass through the object / subject quality recognition module, which consists of a trained classification network and gives each action subject vector a quality score ( Fig.21 For example, the quality of the subject vector can be judged based on whether the object is frontal, whether the pixels are high, whether the face is clear, etc. If the object has a frontal, high-pixel, and clear surface, the quality score is high. If the object is non-frontal, low-pixel, and the face is not clear, the quality score is low, such as Fig. 22The object vectors shown on the left side of are the subject vectors with higher quality scores. Fig. 22 The object vector on the right is the subject vector with a lower quality score.

[0463] like Fig.21 As shown, the behavior subject judgment module can output a behavior subject vector with a quality score corresponding to each video frame in the video to be processed.

[0464] ⑥CCT Network

[0465] The CCT network proposed in the embodiment of the present application is a network composed of Context Contrast Transformer (CCT). For object recognition, the richer the object can understand itself in multiple angles and multiple scenes, the higher the object recognition accuracy. The CCT network allows the subject to obtain semantic information about itself from multiple angles and multiple scenes, which can help better identify objects.

[0466] Through the efficient information exchange function of the CCT network, each subject (query subject) can exchange information with similar objects in the video (similar subjects), helping the subject to obtain multi-angle and multi-scene information about itself from similar objects, thereby improving the accuracy of object recognition. Fig.23a As shown in the schematic diagram, by allowing the man in a shirt to perform CCT operations with the same man in a shirt from different angles, and allowing the man to perform CCT operations with the same man in a suit in different scenes, the man can obtain multi-angle and multi-scene information about himself, thereby more accurately identifying the subject of the behavior.

[0467] In the CCT network, the same behavior subjects can be aggregated together to achieve information interaction of the same subject in multiple scenarios and multiple angles, such as Figure 23b As shown in the figure, the man on the left (the subject vector corresponding to the video frame where the man is located) is used as the query subject. Since the query subject has learned the scene information when he is wearing a suit, when he encounters an object wearing the same suit and with a similar front view, the two objects will be clustered together and considered to be the same person. Since the query subject has learned different angles of itself when it is wearing a shirt, when it encounters an object wearing the same shirt and with a similar lower face, the two objects will be clustered and considered to be the same person.

[0468] Fig.24a FIG. 1 shows a schematic diagram of a CCT network provided in an embodiment of the present application. Fig.24a As shown in step 2, the CCT network is mainly composed of multiple superimposed CCT modules (multi-layer CCT shown in the figure). Fig.24aL CCT modules are shown in FIG. 5 . Through multiple levels of CCT modules, the interaction of semantic information between the behavior subject vectors of the same subject can be realized from shallow to deep. Fig.24a Step 1 shown in the figure is a pre-step before CCT network operation. Fig.24a As shown, the specific implementation process of step 1 and step 2 may include the following steps:

[0469] (1) Among all the subject vectors in a video output by the subject judgment module, one subject vector is selected as the query vector. K vectors that are most similar to the query vector can be found from other subject vectors in the video. For example, the cosine similarity between the vectors can be calculated to find the K vectors that are most similar to the query vector. Fig.24a Similar vectors in .

[0470] Among them, the cosine similarity can be set to an extremely high similarity threshold, so the objects in each video frame corresponding to the query vector and the K similar vectors found for the vector can be regarded as the same object.

[0471] (2) Input the query vector and K similarity vectors into the CCT network. Use the CCT module of layer 1 (the first layer) to allow the query vector and K similarity vectors to interact with each other and output a new query vector. The size of the output new query vector remains unchanged. As an optional solution, the specific calculation principle of the CCT module can be shown as follows.

[0472] X a =Pool 1 (Concat(X 1 , X 2 , ..., X k ))

[0473] X b =Pool 2 (Concat(X 1 , X 2 , ..., X k ))

[0474] X c =Pool 3 (Concat(X 1 , X 2 , ..., X k ))

[0475] X d =Pool 4 (Concat(X 1 , X 2, ..., X k ))

[0476] K = Concat(X a , X b , X c , X d )

[0477] V = Concat(X a , x b , X c , X d )

[0478] X output =Softmax(X q ×K)×V (5)

[0479] Among them, X q represents the input Query vector, X 1 , X 2 , ..., X k are K vectors similar to the input Query vector, Concat represents the feature concatenation operation, Pool 1 , Pool 2 , Pool 3 , Pool 4 Indicates that 4 different convolution kernels are used to perform convolution operations to output 4 vectors, such as vectors of 4 different lengths. Softmax represents the activation function, X output The new Query vector is output.

[0480] As shown in formula (5), for the processing of a query vector, the query vector and its K similarity vectors can be input into CCT, so that the query vector and the K similarity vectors can exchange information. Since the objects in the video frames corresponding to the query vector and the K similarity vectors can be regarded as the same object, the query vector can learn multi-scene and multi-angle semantic information about itself from the K similarity vectors.

[0481] (3) The new query vector and K similarity vectors output by the CCT module of layer 1 are used as the input of the CCT module of layer 2. The CCT module of layer 2 outputs a new query vector. The size of the output vector remains unchanged. The calculation principle is shown in the above formula (5).

[0482] (4) Similarly, after L layers of CCT operations, a new query vector with the same size is output. The query vector output by the last layer of CCT module is the output vector obtained after the input query vector is processed by the CCT network.

[0483] Each behavior subject vector determined by the behavior subject determination module must be processed as a query vector through the above steps 1-4 until the CCT operation is completed for each behavior subject vector to obtain a new feature vector corresponding to each behavior subject vector.

[0484] The CCT module proposed in the embodiment of the present application is a new Transformer structure. The CCT module adds concatenation and pooling operations of multiple input vectors (pool operation in formula (5)) on the basis of the traditional Transformer structure. While saving the amount of calculation, it enables the Query vector to learn multi-angle and multi-scenario semantic information from other input vectors.

[0485] Figure 24b A schematic diagram showing a principle of a calculation process of performing a CCT operation on a single Query vector provided by an embodiment of the present application is shown, such as Figure 24b As shown, the calculation process can be as follows:

[0486] (1) Input the query vector and the K similar vectors found by cosine similarity.

[0487] (2) Concatenate the K vectors, use four convolution kernels of different lengths for pooling, and output four vectors of different lengths.

[0488] (3) Concatenate the four vectors of different lengths output from step (2) to obtain a new vector, called Key (query vector).

[0489] (4) The process of obtaining the Value (value vector) is the same as that of obtaining the Key. You can repeat steps 2 and 3 to obtain the Value. It should be noted that the network parameters of the convolution kernel used to obtain the Value and the network parameters of the convolution kernel used to obtain the Key can be the same or different. In other words, the Value and Key corresponding to a Query vector can be the same vector or different vectors.

[0490] (5) Multiply the Key and Query vectors, pass them through the Softmax layer, and then multiply them with the Value to get a new Query vector.

[0491] ⑦Post-processing

[0492] All the behavior subject vectors output after passing through the CCT network can be post-processed by two modules, namely the behavior subject aggregation module and the event confirmation module. The final behavior subjects and events are generated through post-processing and displayed to the user.

[0493] Among them, the behavior subject aggregation module is mainly used to aggregate all behavior subject vectors, such as using the graph propagation algorithm to aggregate vectors, and aggregate similar objects as a cluster. The objects in each cluster will be treated as the same object. Multiple different objects can form multiple clusters, such as Fig.25 The visualization diagram shown in the figure shows that by aggregating all the subject vectors, three aggregation results are obtained. Each subject output by the subject aggregation module can have its own aggregation label.

[0494] The function of the event confirmation module is to allow all subject vectors in the same aggregation to find corresponding video frames, and to generate event segments of the subject corresponding to the aggregation based on these frames. Optionally, all video frames corresponding to the same aggregation can be cropped out of a rectangular frame area that fits the subject in the video frame as a new video frame based on the size of the subject in the video frame, and two or more continuous rectangular video frames form an event segment. If the spacing between video frames exceeds a certain spacing, the video frames are considered discontinuous, and a single discontinuous video frame is discarded. Through the above processing, the events in the same aggregation can be obtained. The subject in the video frame corresponding to the subject vector with the highest quality score in an aggregation and the event under the aggregation to which the vector belongs can be used as an event related to a subject and the subject.

[0495] like Fig.26 In the schematic diagram shown, the video frames in the video to be processed corresponding to all the subject vectors in an aggregation include frames 3 to 6 shown in the figure. The subject area in each frame can be cropped according to the shape of the subject in each of the four frames (such as the subject shape in the visible feature map of each frame output by the last AVT layer of the AVT network), and the event fragments of the subject corresponding to the aggregation are obtained based on the continuous frames in the cropped frames.

[0496] Find the subject with the highest quality score in each aggregation and the events under the aggregation, so that the subject and related events to be recommended to the user are determined. Optionally, the subject corresponding to each aggregation can be displayed to the user, and the user can select a subject he is interested in, and the selected subject and related events will be displayed to the user.

[0497] Of course, the post-processing method in the optional embodiment 2 provided above may also be adopted. After obtaining each aggregation through the behavior subject aggregation module and finding the video frames corresponding to all behavior subject vectors in each aggregation, the pixel values ​​of the areas other than the area where the behavior subject is located in each video frame may be removed, and based on each video frame corresponding to each aggregation and with the pixel values ​​of the non-subject area removed, an event segment of the behavior subject corresponding to each aggregation is generated.

[0498] This application does not limit the specific method of displaying at least one of the behavior subject or event clip in the video to the user, and can be configured according to actual application requirements and application scenarios. As two examples, Fig.28a and Fig.28b Schematic diagrams of two user interfaces are shown, such as Fig.28a As shown, the video in this example contains content related to cats. When a user watches the video on his mobile phone, he can long press any area of ​​the video frame with a cat or long press the cat in the video. At this time, in response to the user's operation, the tag of the cat and related event clips can be displayed to the user, such as Fig.28a In the user interface on the right, the subject of the behavior can be the cat in the frame with the highest object quality in the event segment, and the related events are the two event segments of the cat in the video. Fig.28b The video shown is a video about a person. While watching the video, the user can long press, slide in a specified direction in the video, or perform other specified operations. In response to the user's operation, the labels of multiple subjects in the video can be displayed to the user, and related events of at least one subject can also be displayed, for example, relevant information of the event segment of the subject in the video frame where the user's above operation is located can be displayed. Optionally, the user can also select the subject of interest in the subject's label list. Based on the user's selection, the related events of the subject selected by the user can be displayed to the user. That is, the solution provided in the embodiment of the present application can also support the switching of subjects.

[0499] This application proposes an event recognition method based on deep learning. Given a video, this method can be used to identify event segments in the video, or to identify and segment the subject of the video, and record event segments related to the subject of the video. The solution provided by the embodiment of this application can have at least the following beneficial effects:

[0500] (1) The solution provided in the embodiment of the present application can accurately identify events in the video and can also accurately identify fast-moving behaviors. For example, this can be achieved by using the ADT network proposed in the embodiment of the present application.

[0501] (2) The solution provided by the embodiment of the present application can accurately identify the subject of behavior and events related to the subject of behavior in the video. For example, the AVT network and CCT network proposed by the embodiment of the present application can accurately identify the subject of behavior and its related events. This can meet the needs of users to find related events of user-specified objects in the video. Fig. 27As shown, the user inputs a video, and the AI ​​network provided by the embodiment of the present application can generate the subject of behavior in the video and its related events. The schematic diagram identifies three subjects in the video, some of which have one event segment and some have two. Optionally, each identified subject (such as the sub-graph of the subject in the video frame corresponding to the vector with the highest score in the aggregation result corresponding to the subject, or the video frame with the background removed) can be displayed to the user first, and the user can select a subject of interest, and then the specific subject selected by the user and its related events can be provided to the user.

[0502] Optionally, the feature extraction network composed of ADT and AVT provided by the embodiment of the present application can improve the recognition rate of behaviors in the video, especially the recognition effect of fast-moving behaviors is also very good. The neural network structure in the AI ​​network provided by the embodiment of the present application can be obtained based on a public training set (such as ImageNet-21K data set) or other training sets. It has been tested and proved that compared with the relevant technology, the recognition accuracy of the AI ​​network provided by the present application is significantly improved, and the model parameter amount of the AI ​​network can be smaller, the amount of calculation can also be effectively reduced, and it is easier to deploy in mobile terminals (such as mobile phones), reducing problems such as mobile phone heating and jamming, so it can better meet the actual application needs. In addition, the CCT network provided by the embodiment of the present application can select objects (similar vectors) from the dimension of the entire video for information interaction, improve the accuracy of object recognition, and can effectively reduce the amount of calculation and improve processing efficiency compared to using the traditional Transformer mechanism to complete information interaction.

[0503] An embodiment of the present application also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present application.

[0504] Fig.29 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Fig.29 As shown, the electronic device 4000 may include at least one processor 4001, and may also include a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0505] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0506] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.29 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0507] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0508] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0509] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0510] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0511] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0512] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0513] The above text and drawings are provided only as examples to help readers understand the present application. They are not intended to and should not be interpreted as limiting the scope of the present application in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present application, and other similar implementation means based on the technical ideas of the present application are adopted, which also fall within the protection scope of the embodiments of the present application.

Claims

1. A method performed by an electronic device, characterized in that: include: Use artificial intelligence (AI) networks to obtain the behavioral subjects and related events in the video to be processed; Based on the acquired behavior subject, provide a behavior subject selection interface; Receiving a behavior subject selected by a user through the selection interface; Provides events related to the subject of the behavior selected by the user.

2. The method according to claim 1, characterized in that The step of obtaining the subject of behavior and related events in the video to be processed includes: Extracting semantic features of the video to be processed, wherein the semantic features include semantic features of the subject in each frame of the video to be processed, and the semantic features of each frame of the video to be processed include spatial semantic features and temporal semantic features; Based on the extracted semantic features, the behavioral subjects and their related events in the video to be processed are determined.

3. The method according to claim 2, characterized in that Extract semantic features of the video to be processed, including: For each to-be-processed video frame in the to-be-processed video, extracting a first semantic feature of the video frame and a second semantic feature of an adjacent frame based on a convolution module; Determining semantically related block patches in the video frame and semantically related patches in the adjacent frames based on the first semantic feature and the second semantic feature; Extracting spatial semantic features of the subject in the video frame from semantically related patches in the video frame, and extracting temporal semantic features of the subject in the video frame from semantically related patches in the adjacent frames; The spatial semantic feature and the temporal semantic feature are fused to obtain the semantic feature of the video frame.

4. The method according to claim 3, characterized in that For each video frame to be processed, the semantic features of the video frame are extracted based on the convolution module, including: Using a first convolutional layer, performing convolution processing on the video frame; Spatially rearrange the features of each channel among the features extracted by the first convolutional layer; The second convolutional layer is used to perform convolution processing on the rearranged features, and the features of each space in the features extracted by the second convolutional layer are rearranged through channels to obtain the semantic features of the video frame.

5. The method according to claim 3, characterized in that: Determining semantically related patches in the video frame and semantically related patches in the adjacent frames based on the first semantic feature and the second semantic feature includes: fusing the first semantic feature with the second semantic feature; Based on the fused features, in the video frame and the adjacent frames, respectively determine spatial position offset information of other patches semantically related to each patch of the video frame relative to the patch; Based on the spatial position offset information, semantically related patches in the video frame and semantically related patches in the adjacent frames are determined.

6. The method according to any one of claims 2 to 5, characterized in that Based on the extracted semantic features, determine the behavior subject and its related events in the video to be processed, including: Based on the semantic features of each video frame to be processed, an object mask module is used to determine the area where the subject is located in each video frame to be processed; Determine the regional features of the region where the subject of the video frame is located based on the semantic features of the video frame and the region where the subject of the video frame is located; Based on the regional features of the area where the subject is located in each video frame to be processed, the behavior subject and related events in the video to be processed are determined.

7. The method according to claim 6, characterized in that Based on each video frame to be processed, an object mask module is used to determine the area where the subject is located in each video frame to be processed, including: For each video frame to be processed, fusing the semantic features of the video frame with the semantic features of the adjacent frames; The object mask module is used to perform subject segmentation on the fused features to obtain the area where the subject is located in the video frame.

8. The method according to claim 6 or 7, characterized in that: Based on the regional features of the region where the subject is located in each video frame to be processed, the behavior subject and related events in the video to be processed are determined, including: For each video frame to be processed, based on the regional features of the area where the subject is located in the video frame, obtaining the subject features corresponding to the video frame; Based on the subject features of the video frame, a subject recognition model is used to determine whether the video frame contains a behavior subject; Based on the determined behavior subject features, the behavior subject and related events in the video to be processed are obtained, wherein the behavior subject features are subject features of the video frame containing the behavior subject.

9. The method according to claim 8, characterized in that Based on the regional features of the region where the subject is located in the video frame, the subject features corresponding to the video frame are obtained, including: Fusing the regional features of the region where the subject is located in the video frame with the semantic features of the video frame to obtain the target features of the video frame; The target features of the video frame and the regional features of the region where the subject is located in the video frame are fused to obtain the subject features of the video frame.

10. The method according to claim 9, characterized in that The regional features of the region where the subject is located in the video frame and the fusion features of the video frame are fused to obtain the target features of the video frame, including: Fusing the regional features of the region where the subject is located in the video frame with the semantic features of the adjacent frames, and extracting the target regional features of the subject in the video frame from the region where the subject is located in the video frame based on the fused features; The target region feature and the semantic feature of the video frame are fused to obtain the target feature of the video frame.

11. The method according to claims 8 to 10, characterized in that Based on the determined behavior subject features, the behavior subject and its related events in the video to be processed are obtained, including: Aggregating the determined behavior subject characteristics to obtain at least one aggregation result; Based on each video frame corresponding to each aggregation result, the behavior subject and its related events corresponding to each aggregation result are obtained.

12. The method according to claim 11, characterized in that Aggregate the identified behavioral subject characteristics, including: For each behavioral subject feature, determining at least one similar subject feature of the behavioral subject feature from the determined behavioral subject features; Based on the characteristics of the behavior subject and the characteristics of similar subjects, a fusion feature of the behavior subject is extracted; Aggregate the fusion features corresponding to the determined behavior subject features.

13. The method according to claim 12, characterized in that Based on the characteristics of the behavior subject and its similar subject characteristics, the fusion characteristics of the behavior subject are extracted, including: Fusing similar subject features of the behavior subject features, and extracting features from the fused features using at least two different feature extraction methods to obtain at least two fused subject features; Based on the correlation between the behavior subject feature and each fusion subject feature, a weight corresponding to each fusion subject feature is obtained; The weights corresponding to the features of each fusion subject are used to perform weighted fusion on the features of the fusion subject to obtain the fusion features of the behavior subject.

14. The method according to any one of claims 11 to 13, characterized in that Based on each video frame corresponding to each aggregation result, the behavior subject and its related events corresponding to each aggregation result are obtained, including: For each aggregation result, based on each behavior subject feature in the aggregation result, determining the quality of the behavior subject in the video frame corresponding to each behavior subject feature; Based on the quality of the behavior subject in the video frame corresponding to the aggregation result, the behavior subject in the video to be processed is determined, and based on each video frame corresponding to the aggregation result, the related event of the behavior subject is determined.

15. The method according to claim 14, characterized in that The obtaining of the relevant events of the behavior subject based on each video frame corresponding to the aggregation result includes at least one of the following: removing the background in each video frame corresponding to the aggregation result, and obtaining related events based on each video frame without the background; Based on the main area in each video frame corresponding to the aggregation result, each video frame is cropped, and related events are obtained based on each cropped video frame.

16. A method performed by an electronic device, characterized in that: include: Using an AI network, extracting semantic features of each frame of the video to be processed in the video to be processed, wherein the semantic features include spatial semantic features and temporal semantic features; Determining an event in the video to be processed based on the extracted semantic features of each video frame to be processed; Provide events in the video to be processed.

17. The method according to claim 16, characterized in that Determining an event in the video to be processed based on the extracted semantic features of each video frame to be processed, including: The semantic features of each video segment in the video to be processed are integrated to obtain segment features corresponding to each video segment; Based on the segment features of the video segment, an event in the video to be processed is determined.

18. An electronic device, characterized in that: The electronic device comprises at least one processor, and the at least one processor is configured to execute the method according to any one of claims 1 to 17.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images