A video processing method, apparatus and device

Abstract

The present application provides a video processing method, apparatus and device, which relate to the field of computer vision technology and are used to improve the accuracy of video action detection. The video processing method includes: extracting global time context information and local time context information features from the first fusion feature according to the first fusion feature of the video to be detected and a preset feature enhancement network, to obtain the global feature and the local feature of the video to be detected, where the first fusion feature includes the spatial flow feature and the temporal flow feature of the video to be detected; according to the global feature and the local feature, and in combination with a preset boundary prediction network, performing boundary localization on each action in the video to be detected, and determining the video segment corresponding to each action.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular, to a video processing method, apparatus, and device. Background Art

[0002] In the field of video processing, video action detection plays an important role in multiple technical fields. For example, in the sports event scenario, it can help analyze the technical movements of athletes and provide improvement suggestions; in the industrial production field, it can detect in real time whether the operations of workers comply with safety regulations, detect potential hazards in a timely manner, and ensure production safety; in the intelligent security field, it can identify abnormal behaviors and give alarms in a timely manner. In addition, video action detection is also widely used in technical fields such as the education field and the medical field.

[0003] With the rapid development of artificial intelligence, a temporal action detection algorithm has been proposed to achieve automatic detection of video actions. The temporal action detection algorithm is based on the understanding of video content and automatically locates and identifies each action in the video, thereby completing the action detection task. However, the current temporal action localization detection algorithm cannot correctly understand the content meaning in the video, resulting in low accuracy of video action detection. Summary of the Invention

[0004] This application provides a video processing method, apparatus, and device for improving the accuracy of video action detection.

[0005] In a first aspect, an embodiment of this application provides a video processing method, including: extracting global time context information and local time context information features from the first fusion feature according to the first fusion feature of the video to be detected and a preset feature enhancement network, to obtain the global feature and local feature of the video to be detected, where the first fusion feature includes the spatial flow feature and temporal flow feature of the video to be detected; and performing boundary localization on each action in the video to be detected according to the global feature and the local feature, in combination with a preset boundary prediction network, to determine the video segment corresponding to each action.

[0006] In the embodiments of the present application, since the first fusion feature includes the spatial stream feature and the temporal stream feature of the video to be detected, the global feature and the local feature obtained based on the first fusion feature consider both the spatial feature and the temporal feature of the video to be detected, which is beneficial to improving the understanding of the video content. Moreover, in the embodiments of the present application, the global feature and the local feature are obtained through global temporal context information extraction and local temporal context information extraction, and the global feature and the local feature are used for action boundary localization. When the boundary prediction network performs boundary localization on each action in the video to be detected, it considers both the dynamic development and change of the video content in the long time period of the video to be detected and the dynamic change in the short time period of the video to be detected, which helps to understand the detailed changes in the short time of the video to be detected. In other words, with the combination of the global feature and the local feature, the boundary prediction network can more comprehensively understand the semantic information and content meaning of the video, thereby improving the accuracy of each action boundary localization, that is, improving the accuracy of video action detection.

[0007] In a possible implementation manner, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; according to the first fusion feature of the video to be detected and a preset feature enhancement network, global temporal context information and local temporal context information features of the first fusion feature are extracted to obtain the global feature and the local feature of the video to be detected, including: using the attention enhancement network to extract the dependency relationship between non-adjacent video frames in the first fusion feature to obtain the global feature; using the adaptive graph convolutional network to extract the dependency relationship between adjacent video frames in the first fusion feature to obtain the local feature.

[0008] In this implementation manner, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network, and the global feature and the local feature are respectively extracted from the first fusion feature, which is beneficial to more comprehensively understanding the content meaning of the video to be detected, thereby improving the accuracy of video action detection.

[0009] In a possible implementation manner, the attention enhancement network is composed of a temporal expansion front convolutional block and an attention network; the temporal expansion front convolutional block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract the feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

[0010] In this implementation manner, by setting a temporal expansion front convolutional block in the attention enhancement network to expand the temporal receptive field and enhance the perception range of the attention network for the first fusion feature in the time dimension, the context understanding ability of the video to be detected is improved.

[0011] In a possible implementation manner, according to the global feature and the local feature, in combination with a preset boundary prediction network, perform boundary localization on each action in the video to be detected, and determine the video segment corresponding to each action, including: performing feature fusion on the global feature and the local feature to obtain a second fused feature; inputting the second fused feature into the boundary prediction network, and using the boundary prediction network to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action, where each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, where a candidate video segment of each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; calculating the confidence scores of the at least one candidate video segment according to the starting boundary probability sequence and the two-dimensional confidence map of each action; and determining the candidate video segment with the highest confidence score among the at least one candidate video segment as the video segment corresponding to each action.

[0012] In a possible implementation manner, the boundary prediction network includes a first branch network and a second branch network; wherein, the first branch network is used to perform temporal convolution processing on the second fused feature to obtain the starting boundary probability sequence of each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fused feature through a boundary matching mechanism, and form each candidate starting boundary in the candidate starting boundary set with each candidate ending boundary in the candidate ending boundary set into a matching pair, determine the confidence of each matching pair, and generate the two-dimensional confidence map of each action, where a matching pair is used to indicate a candidate video segment.

[0013] In this implementation manner, the multi-branch structure design of the boundary prediction network enables the boundary prediction network to predict action boundaries from both the temporal and spatial dimensions simultaneously, improving the robustness and accuracy of the prediction. Among them, temporal convolution is good at processing temporal data and is suitable for distinguishing background frames and action frames, while the two-dimensional confidence map can more intuitively display the starting position and confidence of the action, which is beneficial to more accurately locate the action boundary.

[0014] In a possible implementation manner, before obtaining the global feature and local feature of the video to be detected, the method further includes: extracting the spatial feature of each video frame in the video to be detected by using a convolutional neural network to obtain the spatial flow feature of the video to be detected; extracting the temporal flow feature of the video to be detected according to the optical flow between adjacent video frames in the video to be detected; performing feature fusion on the spatial flow feature and the temporal flow feature to obtain the first fusion feature.

[0015] In a second aspect, an embodiment of the present application provides a video processing device, including: a feature extraction module, configured to extract global time context information and local time context information features from the first fusion feature according to the first fusion feature of the video to be detected and a preset feature enhancement network, to obtain the global feature and local feature of the video to be detected, where the first fusion feature includes the spatial flow feature and temporal flow feature of the video to be detected; a boundary localization module, configured to perform boundary localization on each action in the video to be detected according to the global feature and the local feature, in combination with a preset boundary prediction network, to determine the video segment corresponding to each action.

[0016] In a possible implementation manner, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; the feature extraction module is specifically configured to: extract the dependency relationship between non-adjacent video frames in the first fusion feature by using the attention enhancement network to obtain the global feature; extract the dependency relationship between adjacent video frames in the first fusion feature by using the adaptive graph convolutional network to obtain the local feature.

[0017] In a possible implementation manner, the attention enhancement network is composed of a temporal expansion pre-convolutional block and an attention network; the temporal expansion pre-convolutional block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract the feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

[0018] In a possible implementation manner, the boundary positioning module is specifically configured to: perform feature fusion on the global feature and the local feature to obtain a second fused feature; input the second fused feature into the boundary prediction network, and use the boundary prediction network to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action, where each value in the starting boundary probability sequence is used to indicate the probability that a time point is the action starting boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, where one candidate video segment of each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; calculate the confidence scores of the at least one candidate video segment according to the starting boundary probability sequence and the two-dimensional confidence map of each action; and determine the candidate video segment with the highest confidence score among the at least one candidate video segment as the video segment corresponding to each action.

[0019] In a possible implementation manner, the boundary prediction network includes a first branch network and a second branch network; where the first branch network is used to perform temporal convolution processing on the second fused feature to obtain the starting boundary probability sequence of each action; the second branch network is used to determine a set of candidate starting boundaries and a set of candidate ending boundaries corresponding to each action from the second fused feature through a boundary matching mechanism, and form each candidate starting boundary in the set of candidate starting boundaries with each candidate ending boundary in the set of candidate ending boundaries into a matching pair, determine the confidence of each matching pair, and generate the two-dimensional confidence map of each action, where a matching pair is used to indicate a candidate video segment.

[0020] In a possible implementation manner, the feature extraction module is further configured to, before obtaining the global feature and the local feature of the video to be detected, use a convolutional neural network to extract the spatial feature of each video frame in the video to be detected to obtain the spatial flow feature of the video to be detected; extract the temporal flow feature of the video to be detected according to the optical flow between adjacent video frames in the video to be detected; and perform feature fusion on the spatial flow feature and the temporal flow feature to obtain the first fused feature.

[0021] In a third aspect, an embodiment of the present application provides a video processing device, including: at least one processor, and a memory communicatively connected to the at least one processor; where the memory stores instructions executable by the at least one processor, and the at least one processor implements the method as described in the first aspect and any possible implementation manner by executing the instructions stored in the memory.

[0022] Fourthly, an embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when run on a computer, cause the computer to execute the method as described in the first aspect and any possible implementation manners thereof.

[0023] Fifthly, an embodiment of the present application provides a computer program product containing computer instructions, which, when run on a computer, cause the method as described in the first aspect and any possible implementation manners thereof to be implemented.

[0024] For the beneficial effects of the second to fifth aspects, reference may be made to the content described in the first aspect above, and details are not repeated here. Description of the Drawings

[0025] Figure 1 It is a schematic diagram of an application scenario of a video processing method provided by an embodiment of the present application;

[0026] Figure 2 It is a schematic diagram of the structure of a temporal action localization network provided by an embodiment of the present application;

[0027] Figure 3 It is a schematic flowchart of a video processing method provided by an embodiment of the present application;

[0028] Figure 4 It is a schematic diagram of the structure of an attention enhancement network provided by an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of the structure of a temporal expansion front convolutional block provided by an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the structure of a multi-head attention mechanism provided by an embodiment of the present application;

[0031] Figure 7 It is a schematic diagram of the structure of a video processing device provided by an embodiment of the present application;

[0032] Figure 8 It is a schematic diagram of the structure of a video processing device provided by an embodiment of the present application. Detailed Embodiments

[0033] To make the objectives, technical solutions, and advantages of this application more clear and understandable, the following will describe the technical solutions in the embodiments of this application clearly and completely in combination with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts fall within the scope of protection of this application. Without conflict, the embodiments in this application and the features in the embodiments can be combined arbitrarily with each other. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0034] In the description and claims of this application and the above accompanying drawings, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices. "Multiple" in this application can mean at least two, for example, it can be two, three, or more, and the embodiments of this application do not make limitations.

[0035] The following will explain the exemplary embodiments of this application in combination with the accompanying drawings, including various details of the embodiments of this application to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described here without departing from the scope of the disclosure of this application. Similarly, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures. It should be noted that in the embodiments of this application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned, and they should be considered exemplary. The purpose is only to illustrate the feasibility in the implementation of the technical solutions of this application, but it does not mean that the applicant has already or necessarily used this solution.

[0036] In the technical solutions of this application, the acquisition, transmission, storage, use, etc. of data all comply with the requirements of relevant national laws and regulations.

[0037] The following will explain some terms related to the embodiments of this application.

[0038] 1. Multi-head Self-Attention Mechanism (MSA): A technology developed based on the self-attention mechanism, aiming to enhance the expression ability and generalization ability of the model by processing multiple "attention heads" in parallel.

[0039] 2. Adaptive Graph Convolution (AGC): A variant of the graph convolutional network for processing graph data, aiming to automatically learn the connection relationships and weights between different nodes in the graph structure. In this network, to capture local context, a new graph convolutional layer is designed to construct local branches. First, a video graph is constructed, represented as:

[0040]

[0041] where and E represent the sets of nodes and edges respectively. Each node represents a segment, and each edge represents the dependency relationship between two segments.

[0042] Define the adjacency matrix related to graph G . The weight of an edge, that is, , represents the relationship connection degree between two connected nodes i and j. The larger the weight, the stronger the relationship between the two line segments and the more they are associated with each other. For local context modeling, the edges between two nodes are constructed based on their temporal distance. The edge set is defined as:

[0043]

[0044] where T is the number of segments, and δ is defined as the maximum connection distance. δ is a parameter used to limit the connection range. In adaptive graph convolution, the binary mask matrix M is usually used to control the connections between nodes. It determines which other nodes each node can be connected to. Specifically, δ indicates how many other nodes can be connected at most within the range before and after the current node. For example, if δ = 1, then each node will only be connected to its previous node, the current node, and the next node (i.e., considering directly adjacent segments). If δ increases, the range of connected nodes will also expand accordingly, enabling the capture of more extensive local context information.

[0045] Please refer to Figure 1 , which is a schematic diagram of the application scenario of a video processing method provided in an embodiment of this application. As Figure 1 shown, this schematic diagram includes device 110 and video processing device 120. Among them, device 110 and video processing device 120 can communicate wired or wirelessly.

[0046] ​Among them, the device 110 may refer to a collection device with video collection function, such as a camera, a video camera, a terminal device with video collection function, etc. The terminal device is specifically, for example, a mobile phone, a personal computer (PC), a tablet computer, a laptop computer, a personal digital assistant, a mobile internet device (MID), etc. Alternatively, the device 110 refers to a video storage device with video data storage function, such as a video recorder, a terminal device or a server with video storage function, a mobile storage device, etc. The content of the terminal device can be referred to the content described above, which will not be elaborated here. The video processing device 120 refers to a device or a device with data processing ability. When the video processing device 120 is a device with data processing ability, it can specifically be a functional module in a device, such as a functional module in the device 110. The embodiments of the present application do not limit this.

[0047] Exemplarily, a temporal action detection network runs in the video processing device 120, and the temporal action detection network is used to perform action detection tasks. When the video processing device 120 receives a video to be detected from the device 110, the video to be detected can be input into the temporal action detection network, so that the temporal action detection network performs action detection on the video to be detected, thereby obtaining video segments corresponding to each action in the video to be detected.

[0048] Next, in combination with Figure 2 the structural schematic diagram of a temporal action detection network shown below, the process of the video processing device 120 in the embodiments of the present application using the temporal action detection network to process the video to be detected will be described.

[0049] As Figure 2 shown, the temporal action detection network 200 includes a feature extraction network 201, a feature enhancement network 202, and a boundary prediction network 203.

[0050] The video processing device 120 can input the video to be detected into the feature extraction network 201 to obtain a first fused feature, and then input the first fused feature into the feature enhancement network 202, so that the feature enhancement network extracts global temporal context information and local temporal context information from the first fused feature, obtains global features and local features, and further uses the boundary prediction network 203 to perform boundary localization on each action in the video to be detected according to the global features and local features, thereby determining the video segments corresponding to each action. Among them, the detailed processing process of the video processing device for the video to be detected will be described in detail below.

[0051] It should be noted that the temporal action detection network used in the embodiments of this application is a temporal action detection network trained based on sample data, and the sample data can be selected according to the application scenario of the temporal action detection network. For example, in the scenario of sports events, the sample data can be event videos of various sports, such as football event videos.

[0052] Exemplarily, taking a football event video as an example, the training process of the temporal action detection network will be illustrated. The video processing device can segment the football event video, extract the video segments corresponding to each action, and use a video annotation tool to annotate the start time and label category of the video segments, generating a JSON file. Among them, the video annotation tool is, for example, the BILS Video Annotation Tool. The label category is used to indicate the action category of each action. For example, shooting, scoring, corner kick, penalty kick, kick-off, offside flagging, replay of aerial confrontation, and replay of goal. Convert the video segments into a format that the model can process. Specifically, sample the video at intervals of to divide the video into segments. The length of the sliding window is 128, and the maximum duration is set to 64, which can cover 98% of the action instances in the video. The spatial size of each frame is set to 96×96 to generate sample data. Divide the sample data into a training set and a test set according to a preset ratio. Input the training set into the initial temporal action detection network, and continuously adjust the model parameters based on the training results and the loss function until the training conditions are met, obtaining the trained temporal action detection network. The training conditions are, for example, that the loss value reaches a preset threshold, and the preset threshold can be specifically set according to actual needs. The embodiments of this application do not limit this. Further, test the trained temporal action detection network based on the test set, and determine whether the trained temporal action detection network meets the evaluation requirements based on the model evaluation parameters. If so, end the training process. If not, repeat the above steps until the trained temporal action detection network meets the evaluation requirements.

[0053] Optionally, during the training process, data augmentation techniques can be used to increase the diversity of the training data. Data augmentation techniques are, for example, random cropping, horizontal flipping, etc.

[0054] Please refer to Figure 3 for the flowchart of a video processing method provided by the embodiments of this application. Among them, the following description is based on the video processing device executing Figure 3 the steps shown. The video processing device is, for example, Figure 1 the video processing device 120 shown in Figure 3 The feature enhancement network involved is, for example, Figure 2 the feature enhancement network 202 shown in Figure 3 The boundary prediction network involved is, for example, Figure 2The boundary prediction network 203 shown.

[0055] S301. According to the first fusion feature of the video to be detected and a preset feature enhancement network, perform global temporal context information and local temporal context information feature extraction on the first fusion feature to obtain the global feature and local feature of the video to be detected. The first fusion feature includes the spatial flow feature and temporal flow feature of the video to be detected.

[0056] Among them, the feature enhancement network can be composed of an attention enhancement network and an adaptive graph convolutional network. The attention enhancement network is used to extract the global feature of the video to be detected, and the adaptive graph convolutional network is used to extract the local feature of the video to be detected.

[0057] Specifically, the video processing device can use the attention enhancement network to extract the dependence relationship between non-adjacent video frames in the first fusion feature to obtain the global feature. Use the adaptive graph convolutional network to extract the dependence relationship between adjacent video frames in the first fusion feature to obtain the local feature. Among them, the non-adjacent video frames described in the embodiments of the present application may refer to video frames that are far apart in the video to be detected. The specific implementation manners of obtaining the global feature by using the attention enhancement network and obtaining the local feature by using the adaptive graph convolutional network will be described separately below.

[0058] 1. Obtain the global feature by using the attention enhancement network.

[0059] Please refer to Figure 4 , which is a schematic structural diagram of an attention enhancement network provided by an embodiment of the present application. As Figure 4 shown, the attention enhancement network is composed of a temporal expansion pre-convolution block and an attention network. Among them, the attention network is shown as a multi-head attention mechanism layer and a feed-forward neural network layer in Figure 4 . The temporal expansion pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension. In other words, the temporal expansion pre-convolution block is a lightweight network component used to expand the temporal receptive field and enhance the multi-head attention mechanism.

[0060] After the video processing device inputs the first fusion feature into the attention enhancement network, it first passes through the temporal expansion pre-convolution block to process the first fusion feature to obtain the processed first fusion feature. Among them, the structural diagram of the temporal expansion pre-convolution block is as Figure 5 shown. Please refer to Figure 5, the temporal expansion pre-convolution block consists of a gated linear unit, a parallel 1×1 convolutional layer, a 3×1 average pooling layer with a stride of 1, and a 7×1 convolutional layer with a large kernel size. Among them, the parallel 1×1 convolutional layer and the 3×1 average pooling layer with a stride of 1 can expand the receptive field, and the 3×1 average pooling layer can also smoothly filter out tiny noise frames. To avoid overfitting of large-sized kernels, the 7×1 convolutional layer can be implemented using depthwise separable convolution. Moreover, in the temporal expansion pre-convolution block, residual connections are used for each of its components, and normalization is used after each component.

[0061] After passing through the temporal expansion pre-convolution block, the processed first fusion feature is input into the multi-head attention mechanism layer. After multiple attention heads perform parallel processing on the processed first fusion feature, a multi-head attention result is obtained, and then the multi-head attention result is processed to obtain an enhanced feature.

[0062] Specifically, please refer to Figure 6 , which is a schematic structural diagram of a multi-head attention mechanism provided by an embodiment of this application. As Figure 6 shown, the multi-head attention mechanism includes an input sequence, a linear transformation, a scaled dot-product attention, and h parallel self-attention heads. Among them, Figure 6 the Q in

[0063] represents the query (Query, Q) matrix, the K represents the key (Key, K) matrix, and the V represents the value (Value, V) matrix. Specifically, it can be expressed as:

[0064] K = g(Fg): The key matrix, which is also obtained from the input feature Fg through the linear transformation g.

[0065] V = h(Fg): The value matrix, which is obtained from the input feature Fg through the linear transformation h.

[0066] The input sequence is usually represented as a word embedding matrix (X). Each attention head first converts the input sequence into Q, K, and V matrices through a linear transformation. Specifically, the input sequence is represented as , and first is projected onto three matrices: the query , the key , and the value , and the formula is as follows:

[0067]

[0068]

[0069]

[0070] where is the weight matrix.

[0071] Among them, a single self-attention head can be expressed as:

[0072]

[0073] where softmax represents row-wise normalization, represents the dimension of.

[0074] After passing through h parallel self-attention heads, the multi-head attention result can be expressed as:

[0075]

[0076] where is the weight matrix. The value of h can be set according to actual needs, and this application embodiment does not limit it. For example, in this application embodiment, h = 8, that is, 8-head attention.

[0077] For each attention head, the attention map is calculated by matching the transformed input feature with the input feature where f and g are learnable linear transformations. In other words, the generated attention map A is expressed as:

[0078]

[0079] where is the generated attention map, is the dimension of.

[0080] The attention map A is obtained by calculating the dot product of the query matrix Q and the key matrix K and normalizing it through the softmax function. After obtaining the attention map A, the elements (or vectors) in the value matrix V can be weighted by using the attention map A, so as to obtain the weighted sum (i.e., enhanced feature). Among them, the weighted value matrix V (i.e., V = h(Fg)) represents the input feature after linear transformation. Specifically, the vectors in V contain the information for subsequent aggregation and weighting. The weighted sum is obtained by multiplying the attention map A by the value matrix V, which contains the global context information of the input feature.

[0081] Finally, after the multi-head attention mechanism layer, the enhanced features are processed through a feedforward neural network layer to obtain global features. The feedforward neural network layer consists of two linear layers with ReLU activations and residual connections after each layer. The final output is the global feature.

[0082] 2. Use adaptive graph convolutional network to extract local features.

[0083] The first fused feature is input into the adaptive graph convolutional network to construct a video graph. The relevant content of constructing the video graph can be referred to the content in the glossary above. Based on the difference calculation formula, the difference between two nodes m and n in the video graph is calculated to obtain the difference calculation result. The difference calculation formula is as follows:

[0084]

[0085] in, is the activation function, is a trainable parameter vector used to transform The dimension is reduced to 1. and Represents the feature vectors of two different nodes m and n in the graph. This difference calculation It is a commonly used method in graph convolutional networks to measure the similarity or difference between nodes. By calculating the difference, a scalar value can be obtained, which reflects the distance or similarity between two nodes in the feature space.

[0086] Further, based on the difference calculation results, update The elements in , that is, the weight of the edge between two nodes, use the updated adjacency matrix The input features are aggregated using a fixed binary mask matrix M (used to limit the connection range). The aggregation process is implemented through a dot product operation, and the calculation formula is as follows:

[0087]

[0088] in Including trainable weights, Cin is the number of channels (or feature dimensions) of the input features, and Cout is the number of channels (or target feature dimensions) of the output features. This matrix usually contains trainable weights used to linearly transform the input features during the adaptive graph convolution process. is fixed The binary mask matrix, according to Limit the connection range and control the connection influence between the current stage and the surrounding nodes. ⊙ is the dot product. Fully parameterized and optimized during the training phase. The elements in it can be any values without any constraints, which means that the edges of the graph are completely learned based on the training data and the positions of the nodes in the video. is a data-dependent graph that adaptively learns the unique graph of each video, where the unique graph is constructed based on the video content or video features. The calculation of can be correspondingly referred to the content described above.

[0089] After the aggregation operation, a new node feature matrix is obtained , which contains richer local temporal context information. In other words, represents the local feature.

[0090] In a possible implementation manner, before obtaining the global feature and the local feature, it is also necessary to extract the spatial flow feature and the temporal flow feature from the video to be detected according to the feature extraction network, so as to fuse them to obtain the first fusion feature.

[0091] Among them, the feature extraction network in the embodiments of the present application can be implemented by a two-stream network. The two-stream network consists of two parallel networks that respectively process spatial information and temporal information to obtain richer and more accurate feature representations. The two-stream network is such as the I3D two-stream network.

[0092] In the two-stream network, for the spatial flow, a convolutional neural network (CNN) is used to extract the spatial features of each video frame in the video to be detected, and the spatial flow feature of the video to be detected is obtained. Among them, the spatial features may include texture features, color features, shape features, etc. of the video frame, which will not be listed one by one here. The spatial flow feature can help capture the static elements and details in the video and is crucial for identifying objects in the scene and understanding the composition of the picture. For the temporal flow, the optical flow between adjacent video frames in the video to be detected is used to estimate the pixel-level motion information between adjacent video frames, and the temporal flow feature of the video to be detected is extracted. Among them, the optical flow can capture the position movement and shape change of the objects in the video, thereby providing detailed information about the object motion and actions. It should be understood that the objects described in the embodiments of the present application may refer to people or objects in the video. After the spatial flow feature and the temporal flow feature are extracted, a feature fusion operation is performed on the spatial flow feature and the temporal flow feature to obtain the first fusion feature. Exemplarily, the spatial flow feature and the temporal flow feature can be concatenated to obtain the first fusion feature.

[0093] Exemplarily, the video processing device preprocesses the video to be detected, taking the video to be detected as = where is the video length, is the nth-frame RGB image. To reduce the computational cost, samples are regularly taken at a frame interval of σ, resulting in consecutive and non-overlapping segments of length σ, where . After inputting it into the two-stream network, a visual feature sequence of length is extracted through the two-stream network:

[0094]

[0095] F represents the visual feature sequence extracted from the video to be detected, that is, the first fusion feature. This sequence is obtained by processing each frame (or sampled frames) in the video to be detected and passing through the two-stream network, represents the feature vector extracted at time step , represents the index of the nth time step or segment in the feature sequence F. F is the set of . R represents the set of real numbers here, that is, each element of the feature vector is a real number. Therefore, indicates that F is a matrix with C rows columns, where C is the dimension of the feature, is the number of feature vectors (corresponding to segments)

[0096] Since the segments are consecutive and non-overlapping, the starting frame of each segment will be at every σth frame position in the video frame sequence. That is, the first segment starts from the first frame, and the second segment starts from the 1 + σ = 6th frame. Therefore, F is equivalent to a set of features of the starting frames of segments of length σ frames, and the starting frame is used to represent the segment where the starting frame is located.

[0097] S302. According to the global features and local features, combined with a preset boundary prediction network, perform boundary localization on each action in the video to be detected, and determine the video segment corresponding to each action.

[0098] Among them, the actions in the video to be detected are related to the scene where the video to be detected is located. For example, taking the video to be detected as a sports event video, each action can refer to shooting, scoring, corner kick, penalty kick, kick-off, etc., and the video segment corresponding to each action can refer to the video segment of this action from the starting moment to the ending moment in the video to be detected.

[0099] Specifically, the video processing device can perform feature fusion on global features and local features to obtain a second fused feature, and then input the second fused feature into the boundary prediction network. The boundary prediction network is used to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action. Each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action. The two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action. A candidate video segment of each action consists of video frames between a candidate starting boundary and a candidate ending boundary of each action. Further, according to the starting boundary probability sequence and the two-dimensional confidence map of each action, the confidence score of at least one candidate video segment is calculated, and the candidate video segment with the highest confidence score is selected from at least one candidate video segment as the video segment corresponding to each action.

[0100] The following describes the specific method for using the boundary prediction network and the second fused feature to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action.

[0101] The boundary prediction network includes a first branch network and a second branch network. The video processing device inputs the second fused feature into the first branch network. The first branch network performs temporal convolution processing on the second fused feature to distinguish the background frames and action frames of the video to be detected. Through further convolution and pooling operations, a sequence representing the probability of the start of an action at each time point, that is, the starting boundary probability sequence, is obtained. Exemplarily, the starting probability sequence can be expressed as {p1, p2, …, p i}, where p i represents the probability that the i-th frame is the starting boundary of the action. Among them, the higher the probability value, the greater the possibility that the action starts at this time point, or the greater the possibility that this time point is the starting boundary of the action.

[0102] The video processing device inputs the second fused feature into the second branch network. The second branch network uses the boundary matching mechanism to determine the candidate starting boundary set and the candidate ending boundary set corresponding to each action from the second fused feature, and combines each candidate starting boundary in the candidate starting boundary set with each candidate ending boundary in the candidate ending boundary set to form a matching pair, determines the confidence of each matching pair, and generates the two-dimensional confidence map of each action. A matching pair is used to indicate a candidate video segment.

[0103] Among them, the second branch network includes a boundary matching module (BPM) and a proposal evaluation module (PEM). Exemplarily, the network structures of a boundary matching module and a proposal evaluation module provided in an embodiment of the present application are shown in Table 1.

[0104] Table 1

[0105]

[0106] Specifically, the video processing device may first input the second fusion feature into the segment generation module. The segment generation module is used to determine a candidate start boundary set and a candidate end boundary set corresponding to each action from the second fusion feature, and form a matching pair (a candidate video segment) by combining each candidate start boundary in the candidate start boundary set with each candidate end boundary in the candidate end boundary set. Further, the segment generation module scores each matching pair and retains the matching pairs (candidate video segments) with scores greater than a preset score value. In other words, the segment generation module is used to generate at least one candidate video segment. For ease of description, the matching pairs with scores greater than the preset score value are hereinafter referred to as at least one candidate video segment.

[0107] Input at least one candidate video segment into the boundary matching module. The boundary matching module optimizes the start boundary and end boundary of at least one candidate video segment and generates a two-dimensional confidence map.

[0108] Optionally, the boundary matching module further includes a temporal evaluation module (TEM). The temporal evaluation module includes two 3×3 convolutional layers, a max pooling layer, and normalization with a depth of 2. In this implementation, the temporal evaluation module can be used to evaluate the temporal accuracy of at least one candidate video segment. Exemplarily, the temporal evaluation module can evaluate the quality of the video segment by calculating the temporal consistency, duration, and context information of the video segment to improve the accuracy of the two-dimensional confidence map.

[0109] Each row in the two-dimensional confidence map represents the confidence of candidate video segments with the same specific start boundary but different durations, and each column represents the confidence of candidate video segments with the same specific duration but different start boundaries. It should be understood that different durations represent different end boundaries.

[0110] Further, the video processing device can calculate the confidence score of at least one candidate video segment according to the start boundary probability sequence of each action and the two-dimensional confidence map. Exemplarily, the video processing device can multiply and fuse the start boundary probability sequence of each action and the two-dimensional confidence map to determine the confidence score of each candidate video segment. The calculation formula can be as follows:

[0111]

[0112] Where, represents the start boundary of the candidate video segment, Indicates the end boundary of the candidate video segment, is the probability of the candidate video segment at the start boundary ; is the probability of the candidate video segment at the end boundary ; is the classification confidence, and

[0113] It should be noted that the classification confidence is usually the probability value obtained when classifying action categories through a classifier (such as a softmax layer). The regression confidence is obtained in the regression task, which represents the accuracy of the model's prediction of the start and end times of the action. and can be directly output by the trained model and do not need to be manually set or calculated.

[0114] Based on the above calculation method, the video processing device can calculate the confidence scores of at least one candidate video segment, and then can determine the candidate video segment with the highest confidence score among at least one candidate video segment as the video segment corresponding to each action. This video segment indicates the start boundary, end boundary, and confidence score of each action.

[0115] Optionally, when determining the video segment corresponding to each action according to the confidence scores of at least one candidate video segment, a non-maximum suppression method can be used to remove redundancy and obtain the final video segment corresponding to each action.

[0116] The confidence score is calculated by comprehensively considering the probability of the start boundary of each action, the probability of the end boundary (duration), and the matching degree between the start boundary and the end boundary. Since the confidence score comprehensively considers multiple factors, the accuracy of determining the video segment corresponding to each action based on the confidence score is also higher.

[0117] It should be understood that in the embodiments of the present application, the start boundary indicates the start moment of the action in the video, and the end boundary indicates the end moment of the action in the video.

[0118] Based on the same inventive concept, the embodiments of the present application provide a video processing device, which is used to implement any of the above video processing methods, such as Figure 3 the video processing method shown, and, this device can also implement the functions of the video processing device in the foregoing text.

[0119] Please refer to Figure 7 which is a schematic structural diagram of a video processing device provided by the embodiments of the present application. As Figure 7 shown, the video processing device 700 includes a feature extraction module 701 and a boundary localization module 702.

[0120] Exemplarily, the feature extraction module 701 is configured to perform global temporal context information and local temporal context information feature extraction on the first fusion feature according to the first fusion feature of the video to be detected and a preset feature enhancement network, so as to obtain the global feature and the local feature of the video to be detected. The first fusion feature includes the spatial flow feature and the temporal flow feature of the video to be detected; the boundary localization module 702 is configured to perform boundary localization on each action in the video to be detected according to the global feature and the local feature, in combination with a preset boundary prediction network, so as to determine the video segment corresponding to each action.

[0121] In a possible implementation manner, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; the feature extraction module 701 is specifically configured to: use the attention enhancement network to extract the dependency relationship between non-adjacent video frames in the first fusion feature to obtain the global feature; use the adaptive graph convolutional network to extract the dependency relationship between adjacent video frames in the first fusion feature to obtain the local feature.

[0122] In a possible implementation manner, the attention enhancement network is composed of a temporal expansion pre-convolution block and an attention network; the temporal expansion pre-convolution block is used to enhance the perception range of the first fusion feature in the temporal dimension, and the attention network is used to extract the feature information of the first fusion feature at different temporal scales according to a preset multi-head attention mechanism.

[0123] In a possible implementation manner, the boundary localization module 702 is specifically configured to: perform feature fusion on the global feature and the local feature to obtain a second fusion feature; input the second fusion feature into the boundary prediction network, and use the boundary prediction network to determine the start boundary probability sequence of each action and the two-dimensional confidence map of each action, where each value in the start boundary probability sequence is used to indicate the probability that a time point is the action start boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, where one candidate video segment of each action is composed of the video frames between one candidate start boundary and one candidate end boundary of each action; calculate the confidence score of at least one candidate video segment according to the start boundary probability sequence and the two-dimensional confidence map of each action; determine the candidate video segment with the highest confidence score among at least one candidate video segment as the video segment corresponding to each action.

[0124] In a possible implementation manner, the boundary prediction network includes a first branch network and a second branch network; wherein, the first branch network is used to perform temporal convolutional processing on the second fusion feature to obtain a sequence of starting boundary probabilities for each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fusion feature through a boundary matching mechanism, and form a matching pair by combining each candidate starting boundary in the candidate starting boundary set with each candidate ending boundary in the candidate ending boundary set, determine the confidence of each matching pair, and generate a two-dimensional confidence map for each action, where a matching pair is used to indicate a candidate video segment.

[0125] In a possible implementation manner, the feature extraction module 701 is further configured to, before obtaining the global feature and the local feature of the video to be detected, extract the spatial feature of each video frame in the video to be detected by using a convolutional neural network to obtain the spatial flow feature of the video to be detected; extract the temporal flow feature of the video to be detected according to the optical flow between adjacent video frames in the video to be detected; perform feature fusion on the spatial flow feature and the temporal flow feature to obtain a first fusion feature.

[0126] Based on the same inventive concept, an embodiment of the present application provides a video processing device, which is used to implement any one of the above video processing methods, for example Figure 3 the video processing method shown, and, this device can also implement the functions of the video processing apparatus in the foregoing text.

[0127] Please refer to Figure 8 , which is a schematic structural diagram of a video processing device provided by an embodiment of the present application. As Figure 8 shown, the video processing device 800 includes at least one processor 801 and a memory 802 communicatively connected to at least one processor 801.

[0128] Among them, the processor 801 can be a general-purpose processor or a dedicated processor, etc. The processor 801, for example, includes: a baseband processor or a central processing unit, etc. The baseband processor can be used to process communication protocols and communication data. The central processing unit can be used to control the video processing device 800, execute software programs and / or process data. Different processors can be independent devices or can be provided in one or more processing circuits, for example, integrated on one or more application-specific integrated circuits.

[0129] In one embodiment, the memory 802 stores instructions executable by at least one processor 801, and at least one processor 801 realizes the functions of the foregoing video processing apparatus by executing the instructions stored in the memory 802, and correspondingly, can also realize the steps executed by the foregoing video processing apparatus.

[0130] In this embodiment, the video processing device 800 can also implement the functions of the foregoing video processing device 700, and at least one processor 801 in the video processing device 800 can also implement the functions of the foregoing feature extraction module 701 and boundary localization module 702.

[0131] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing computer instructions that, when run on a computer, cause the computer to execute any of the foregoing video processing methods, for example Figure 3 the video processing method shown.

[0132] Based on the same inventive concept, an embodiment of the present application provides a computer program product containing computer instructions that, when run on a computer, cause any of the foregoing video processing methods, for example Figure 3 the video processing method shown to be implemented.

[0133] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0135] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured product including instruction means that implement the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, causing a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 in one block or a plurality of blocks. Figure 1

[0137] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.​

Claims

1. A video processing method, characterized in that: include: Extracting global temporal context information and local temporal context information features from a first fused feature of a video to be detected and a preset feature enhancement network to obtain global features and local features of the video to be detected, wherein the first fused feature includes spatial stream features and temporal stream features of the video to be detected; Based on the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined; The feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network, and the attention enhancement network and the adaptive graph convolutional network are respectively used to extract global features and local features of the video to be detected based on the first fusion features; The attention enhancement network includes a temporal expansion pre-convolution block and an attention network, wherein the temporal expansion pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension.

2. The method according to claim 1, characterized in that According to the first fusion feature of the video to be detected and a preset feature enhancement network, global time context information and local time context information feature extraction are performed on the first fusion feature to obtain the global feature and local feature of the video to be detected, including: Extracting dependencies between non-adjacent video frames in the first fusion feature using the attention enhancement network to obtain the global feature; The adaptive graph convolutional network is used to extract the dependency between adjacent video frames in the first fusion feature to obtain the local feature.

3. The method according to claim 2, characterized in that The attention network is used to extract feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

4. The method according to claim 1, wherein Based on the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined, including: Performing feature fusion on the global feature and the local feature to obtain a second fused feature; Inputting the second fused feature into the boundary prediction network, and using the boundary prediction network to determine a starting boundary probability sequence for each action and a two-dimensional confidence map for each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; and the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, wherein a candidate video segment for each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; Calculating a confidence score of the at least one candidate video segment according to the starting boundary probability sequence and the two-dimensional confidence map of each action; A candidate video segment with the highest confidence score among the at least one candidate video segment is determined as the video segment corresponding to each action.

5. The method according to claim 4, characterized in that: The boundary prediction network includes a first branch network and a second branch network; wherein, the first branch network is used to perform temporal convolution processing on the second fusion feature to obtain a starting boundary probability sequence for each action; the second branch network is used to determine the candidate starting boundary set and the candidate ending boundary set corresponding to each action from the second fusion feature through a boundary matching mechanism, and each candidate starting boundary in the candidate starting boundary set and each candidate ending boundary in the candidate ending boundary set are combined into a matching pair, the confidence of each matching pair is determined, and a two-dimensional confidence map for each action is generated, wherein a matching pair is used to indicate a candidate video segment.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the global features and local features of the video to be detected, the method further includes: Extracting spatial features of each video frame in the video to be detected using a convolutional neural network to obtain spatial flow features of the video to be detected; Extracting a temporal flow feature of the video to be detected based on the optical flow between adjacent video frames in the video to be detected; The spatial stream features and the temporal stream features are fused to obtain the first fused features.

7. A video processing device, characterized in that: include: a feature extraction module, configured to extract global temporal context information and local temporal context information features from a first fused feature of a video to be detected and a preset feature enhancement network, thereby obtaining global features and local features of the video to be detected, wherein the first fused feature includes spatial stream features and temporal stream features of the video to be detected; A boundary positioning module is used to locate the boundary of each action in the video to be detected based on the global features and the local features in combination with a preset boundary prediction network, and determine the video segment corresponding to each action; The feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network, and the attention enhancement network and the adaptive graph convolutional network are respectively used to extract global features and local features of the video to be detected based on the first fusion features; The attention enhancement network includes a temporal expansion pre-convolution block and an attention network, wherein the temporal expansion pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension.

8. A video processing device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 6 by executing the instructions stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises computer instructions, which, when executed on a computer, enable the method according to any one of claims 1 to 6 to be implemented.

Images