Video processing method, device and equipment

Abstract

The invention provides a video processing method, device and equipment, relates to the technical field of computer vision, and is used for improving the accuracy of video action detection. The video processing method comprises the following steps: performing global time context information and local time context information feature extraction on a first fusion feature according to the first fusion feature of a to-be-detected video and a preset feature enhancement network to obtain a global feature and a local feature of the to-be-detected video, the first fusion feature comprises a spatial stream feature and a time stream feature of the to-be-detected video; and according to the global features and the local features, in combination with a preset boundary prediction network, performing boundary positioning on each action in the to-be-detected video, and determining a video clip corresponding to each action.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video processing method, device and equipment. Background Art

[0002] In the field of video processing, video motion detection plays an important role in many technical fields. For example, in sports events, it can help analyze the technical movements of athletes and provide suggestions for improvement; in industrial production, it can detect in real time whether workers' operations comply with safety regulations, detect potential hazards in a timely manner, and ensure production safety; in the field of intelligent security, it can identify abnormal behaviors and issue alarms in a timely manner. In addition, video motion detection is also widely used in technical fields such as education and medical fields.

[0003] With the rapid development of artificial intelligence, a temporal action detection algorithm has been proposed to realize the automatic detection of video actions. The temporal action detection algorithm automatically locates and identifies each action in the video based on the understanding of the video content, thereby completing the action detection task. However, the current temporal action positioning detection algorithm cannot correctly understand the content and meaning of the video, resulting in low accuracy of video action detection. Summary of the invention

[0004] The present application provides a video processing method, apparatus and device for improving the accuracy of video action detection.

[0005] In a first aspect, an embodiment of the present application provides a video processing method, comprising: based on a first fusion feature of a video to be detected and a preset feature enhancement network, extracting global temporal context information and local temporal context information features of the first fusion feature to obtain global features and local features of the video to be detected, wherein the first fusion feature includes spatial stream features and temporal stream features of the video to be detected; based on the global features and the local features, combined with a preset boundary prediction network, locating the boundaries of each action in the video to be detected, and determining the video segment corresponding to each action.

[0006] In the embodiment of the present application, since the first fusion feature includes the spatial flow feature and the temporal flow feature of the video to be detected, the global feature and the local feature obtained based on the first fusion feature not only consider the spatial feature of the video to be detected, but also consider the temporal feature of the video to be detected, which is conducive to improving the understanding of the video content. In addition, the embodiment of the present application obtains global features and local features by extracting global temporal context information and local temporal context information, and uses global features and local features to locate the action boundary, so that the boundary prediction network considers the dynamic development and changes of the video content of the video to be detected in a long period of time when locating the boundary of each work in the video to be detected, and also considers the dynamic changes of the video to be detected in a short period of time, which is helpful to understand the changes in details of the video to be detected in a short period of time. In other words, under the combination of global features and local features, the boundary prediction network can more comprehensively understand the semantic information and content meaning of the video, thereby improving the accuracy of locating the boundary of each action, that is, improving the accuracy of video action detection.

[0007] In a possible implementation, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; based on a first fusion feature of the video to be detected and a preset feature enhancement network, global time context information and local time context information feature extraction are performed on the first fusion feature to obtain the global and local features of the video to be detected, including: using the attention enhancement network to extract the dependency between non-adjacent video frames in the first fusion feature to obtain the global feature; using the adaptive graph convolutional network to extract the dependency between adjacent video frames in the first fusion feature to obtain the local feature.

[0008] In this embodiment, a feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network, and global features and local features are respectively extracted from the first fusion features, which is conducive to a more comprehensive understanding of the content meaning of the video to be detected, thereby improving the accuracy of video action detection.

[0009] In one possible implementation, the attention enhancement network consists of a time-extended pre-convolution block and an attention network; the time-extended pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

[0010] In this embodiment, by setting a temporal expansion pre-convolution block in the attention enhancement network, the temporal receptive field is expanded, the perception range of the attention network to the first fusion feature in the time dimension is enhanced, and the context understanding ability of the video to be detected is improved.

[0011] In a possible implementation, based on the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined, including: performing feature fusion on the global features and the local features to obtain a second fused feature; inputting the second fused feature into the boundary prediction network, and using the boundary prediction network to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, wherein a candidate video segment for each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; according to the starting boundary probability sequence and the two-dimensional confidence map of each action, the confidence score of the at least one candidate video segment is calculated; and the candidate video segment with the highest confidence score in the at least one candidate video segment is determined as the video segment corresponding to each action.

[0012] In a possible implementation, the boundary prediction network includes a first branch network and a second branch network; wherein the first branch network is used to perform temporal convolution processing on the second fusion feature to obtain a starting boundary probability sequence for each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fusion feature through a boundary matching mechanism, and each candidate starting boundary in the candidate starting boundary set and each candidate ending boundary in the candidate ending boundary set are combined into a matching pair, the confidence of each matching pair is determined, and a two-dimensional confidence map for each action is generated, wherein a matching pair is used to indicate a candidate video segment.

[0013] In this implementation, the multi-branch structure design of the boundary prediction network enables the boundary prediction network to predict the action boundary from both the temporal and spatial dimensions, thereby improving the robustness and accuracy of the prediction. Among them, the temporal convolution is good at processing temporal data and is suitable for distinguishing background frames from action frames, while the two-dimensional confidence map can more intuitively display the action starting position and confidence, which is conducive to more accurate positioning of the action boundary.

[0014] In a possible implementation, before obtaining the global features and local features of the video to be detected, the method further includes: using a convolutional neural network to extract the spatial features of each video frame in the video to be detected to obtain the spatial flow features of the video to be detected; extracting the temporal flow features of the video to be detected based on the optical flow between adjacent video frames in the video to be detected; and performing feature fusion on the spatial flow features and the temporal flow features to obtain the first fusion features.

[0015] In a second aspect, an embodiment of the present application provides a video processing device, comprising: a feature extraction module, used to extract global temporal context information and local temporal context information features of the first fusion feature of the video to be detected and a preset feature enhancement network, so as to obtain the global features and local features of the video to be detected, wherein the first fusion feature includes the spatial flow features and temporal flow features of the video to be detected; a boundary positioning module, used to perform boundary positioning for each action in the video to be detected based on the global features and the local features in combination with a preset boundary prediction network, and determine the video segment corresponding to each action.

[0016] In a possible implementation, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; the feature extraction module is specifically used to: use the attention enhancement network to extract the dependency between non-adjacent video frames in the first fusion feature to obtain the global feature; use the adaptive graph convolutional network to extract the dependency between adjacent video frames in the first fusion feature to obtain the local feature.

[0017] In one possible implementation, the attention enhancement network consists of a time-extended pre-convolution block and an attention network; the time-extended pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

[0018] In a possible implementation, the boundary positioning module is specifically used to: perform feature fusion on the global feature and the local feature to obtain a second fused feature; input the second fused feature into the boundary prediction network, and use the boundary prediction network to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, wherein a candidate video segment for each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; calculate the confidence score of the at least one candidate video segment based on the starting boundary probability sequence and the two-dimensional confidence map of each action; and determine the candidate video segment with the highest confidence score in the at least one candidate video segment as the video segment corresponding to each action.

[0019] In a possible implementation, the boundary prediction network includes a first branch network and a second branch network; wherein the first branch network is used to perform temporal convolution processing on the second fusion feature to obtain a starting boundary probability sequence for each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fusion feature through a boundary matching mechanism, and each candidate starting boundary in the candidate starting boundary set and each candidate ending boundary in the candidate ending boundary set are combined into a matching pair, the confidence of each matching pair is determined, and a two-dimensional confidence map for each action is generated, wherein a matching pair is used to indicate a candidate video segment.

[0020] In a possible implementation, the feature extraction module is also used to, before obtaining the global features and local features of the video to be detected, use a convolutional neural network to extract the spatial features of each video frame in the video to be detected to obtain the spatial flow features of the video to be detected; extract the temporal flow features of the video to be detected based on the optical flow between adjacent video frames in the video to be detected; and perform feature fusion on the spatial flow features and the temporal flow features to obtain the first fusion features.

[0021] In a third aspect, an embodiment of the present application provides a video processing device, comprising: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the method described in the first aspect and any possible implementation manner by executing the instructions stored in the memory.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect and any possible implementation manner.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables the method described in the first aspect and any possible implementation method to be implemented.

[0024] The beneficial effects of the second to fifth aspects can be referred to the contents described in the first aspect above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of an application scenario of a video processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a sequential action positioning network provided in an embodiment of the present application; Figure 3 A schematic diagram of a video processing method provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an attention enhancement network provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a timing expansion pre-convolution block provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a multi-head attention mechanism provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0027] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0028] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0029] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0030] Some terms involved in the embodiments of the present application are explained below.

[0031] 1. Multi-head Self-Attention Mechanism (MSA): It is a technology developed based on the self-attention mechanism, which aims to enhance the expressiveness and generalization ability of the model by processing multiple "attention heads" in parallel.

[0032] 2. Adaptive Graph Convolution (AGC): It is a variant of graph convolution network for processing graph data, which aims to automatically learn the connection relationship and weights between different nodes in the graph structure. In this network, in order to capture local context, a new graph convolution layer is designed to construct local branches. First, a video graph is constructed, which is represented as:

[0033] in and E represent the set of nodes and edges respectively. Each node represents a fragment, and each edge represents the dependency between two fragments.

[0034] Define the adjacency matrix associated with the graph G The weight of an edge is , represents the degree of connection between two connected nodes i and j. The larger the weight, the stronger the relationship between the two segments and the more they are related to each other. For local context modeling, the edge between two nodes is constructed based on their temporal distance. The edge set is defined as:

[0035] Where T is the number of fragments and δ is defined as the maximum connection distance. δ is a parameter used to limit the range of connections. In adaptive graph convolution, a binary mask matrix M is usually used to control the connections between nodes. It determines which other nodes each node can be connected to. Specifically, δ indicates how many other nodes can be connected at most within the range before and after the current node. For example, if δ=1, then each node will only be connected to its previous node, current node, and next node (i.e., considering directly adjacent fragments). If δ increases, the range of connected nodes will also expand accordingly, thereby being able to capture a wider range of local context information.

[0036] Please refer to Figure 1 , is a schematic diagram of an application scenario of a video processing method provided in an embodiment of the present application. Figure 1 As shown, the schematic diagram includes a device 110 and a video processing apparatus 120. The device 110 and the video processing apparatus 120 can perform wired or wireless communication.

[0037] Among them, the device 110 may refer to a collection device with a video collection function, such as a camera, a video camera, a terminal device with a video collection function, etc., and the terminal device may specifically be a mobile phone, a personal computer (PC), a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), etc. Alternatively, the device 110 refers to a video storage device with a video data storage function, such as a video recorder, a terminal device or server with a video storage function, a mobile storage device, etc. The content of the terminal device may correspond to the content described above and will not be repeated here. The video processing device 120 refers to a device or apparatus with data processing capabilities. When the video processing device 120 is a device with data processing capabilities, it may specifically be a functional module in a device, such as a functional module in the device 110, and the embodiments of the present application do not limit this.

[0038] Exemplarily, a sequential action detection network is run in the video processing device 120, and the sequential action detection network is used to perform action detection tasks. When the video processing device 120 receives the video to be detected from the device 110, the video to be detected can be input into the sequential action detection network, so that the sequential action detection network performs action detection on the video to be detected, thereby obtaining a video clip corresponding to each action in the video to be detected.

[0039] Combine the following Figure 2 The structural diagram of a sequential motion detection network shown in the figure illustrates the process of the video processing device 120 in the embodiment of the present application processing the video to be detected by using the sequential motion detection network.

[0040] like Figure 2 As shown, the temporal action detection network 200 includes a feature extraction network 201, a feature enhancement network 202 and a boundary prediction network 203.

[0041] The video processing device 120 can input the video to be detected into the feature extraction network 201 to obtain the first fused feature, and then input the first fused feature into the feature enhancement network 202, so that the feature enhancement network extracts the global temporal context information and the local temporal context information of the first fused feature to obtain the global feature and the local feature, and then locates the boundary of each action in the video to be detected by using the boundary prediction network 203 according to the global feature and the local feature, so as to determine the video segment corresponding to each action. The detailed processing process of the video to be detected by the video processing device will be described in detail below.

[0042] It should be noted that the temporal motion detection network used in the embodiment of the present application is a temporal motion detection network trained based on sample data. The sample data can be selected according to the application scenario of the temporal motion detection network. For example, in a sports event scenario, the sample data can be event videos of various sports, such as football event videos.

[0043] Exemplarily, taking a football match video as an example, the training process of the temporal action detection network is illustrated. The video processing device can segment the football match video, extract the video clips corresponding to each action, and use the video annotation tool to annotate the start time and label category of the video clips to generate a JSON file. Among them, the video annotation tool is, for example, the BILS Video Annotation Tool. The label category is used to indicate the action category of each action, such as shooting, scoring, corner kick, penalty kick, kick-off, offside flag waving, replay of aerial confrontation and replay of goal. Convert the video clips into a format that the model can process, specifically, In order to sample the video at intervals, the video is divided into segments, the length of the sliding window is 128, and the maximum duration is set to 64, which can cover 98% of the action instances in the video. The space size of each frame is set to 96×96 to generate sample data. The sample data is divided into a training set and a test set according to a preset ratio, and the training set is input into the initial temporal action detection network. The model parameters are continuously adjusted based on the training results and the loss function until the training conditions are met to obtain the trained temporal action detection network. The training conditions are, for example, that the loss value reaches a preset threshold. The preset threshold can be set according to actual needs, and the embodiments of the present application do not limit this. Furthermore, the trained temporal action detection network is tested based on the test set, and it is determined based on the model evaluation parameters whether the trained temporal action detection network meets the evaluation requirements. If so, the training process is terminated. If not, the above steps are repeated until the trained temporal action detection network meets the evaluation requirements.

[0044] Optionally, during the training process, data augmentation techniques may be used to increase the diversity of training data, such as random cropping and horizontal flipping.

[0045] Please refer to Figure 3 , is a flow chart of a video processing method provided in an embodiment of the present application. The following is a video processing device executing Figure 3 The video processing device is, for example, Figure 1 The video processing device 120 shown, Figure 3 The feature enhancement network involved is, for example, Figure 2 The feature enhancement network 202 shown, Figure 3 The boundary prediction network involved is, for example, Figure 2 Boundary prediction network 203 is shown.

[0046] S301, based on the first fusion feature of the video to be detected and a preset feature enhancement network, extract the global time context information and the local time context information features of the first fusion feature to obtain the global features and local features of the video to be detected, wherein the first fusion feature includes the spatial stream features and the temporal stream features of the video to be detected.

[0047] The feature enhancement network can be composed of an attention enhancement network and an adaptive graph convolutional network. The attention enhancement network is used to extract global features of the video to be detected, and the adaptive graph convolutional network is used to extract local features of the video to be detected.

[0048] Specifically, the video processing device may use an attention enhancement network to extract dependencies between non-adjacent video frames in the first fusion feature to obtain global features. An adaptive graph convolutional network may be used to extract dependencies between adjacent video frames in the first fusion feature to obtain local features. Among them, the non-adjacent video frames described in the embodiment of the present application may refer to video frames that are far apart in the video to be detected. The specific implementation methods of extracting global features using an attention enhancement network and extracting local features using an adaptive graph convolutional network are described below.

[0049] 1. Use the attention-enhanced network to extract global features.

[0050] Please refer to Figure 4 , is a schematic diagram of the structure of an attention enhancement network provided in an embodiment of the present application. Figure 4 As shown in Figure 2, the attention enhancement network consists of a temporal expansion pre-convolution block and an attention network, where the attention network is Figure 4 The diagram shows a multi-head attention mechanism layer and a feedforward neural network layer. The temporal expansion pre-convolution block is used to enhance the perception range of the first fusion feature in the temporal dimension. In other words, the temporal expansion pre-convolution block is a lightweight network component used to expand the temporal receptive field and enhance the multi-head attention mechanism.

[0051] After the video processing device inputs the first fusion feature into the attention enhancement network, it first passes through the temporal expansion pre-convolution block to process the first fusion feature and obtain the processed first fusion feature. The structure diagram of the temporal expansion pre-convolution block is shown in FIG. Figure 5 Please refer to Figure 5 The temporal expansion pre-convolution block consists of a gated linear unit (Gated LinearUnit), a parallel 1×1 convolution layer, a 3×1 average pooling layer with a stride of 1, and a 7×1 convolution layer with a large kernel size. The parallel 1×1 convolution layer and the 3×1 average pooling layer with a stride of 1 can expand the receptive field, and the 3×1 average pooling layer can also smoothly filter out tiny noise frames. In order to avoid overfitting of large-size kernels, the 7×1 convolution layer can be implemented using deep separable convolution. In addition, in the temporal expansion pre-convolution block, residual connections are used for each of its components, and normalization is used after each component.

[0052] After passing through the time-expanded pre-convolution block, the processed first fusion feature is input into the multi-head attention mechanism layer. After multiple attention heads process the processed first fusion feature in parallel, the multi-head attention result is obtained, and then the multi-head attention result is processed to obtain the enhanced feature.

[0053] For details, please refer to Figure 6, is a schematic diagram of the structure of a multi-head attention mechanism provided in an embodiment of the present application. Figure 6 As shown in Figure 1, the multi-head attention mechanism includes input sequence, linear transformation, scaled dot product attention, and h parallel self-attention heads. Figure 6 The Q in the table represents the query matrix, K represents the key matrix, and V represents the value matrix. Specifically, it can be illustrated as follows: Q=f(Fg): Query matrix, which is obtained from the input features Fg through linear transformation f.

[0054] K=g(Fg): key matrix, also obtained from the input feature Fg through linear transformation g.

[0055] V=h(Fg): Value matrix, which is obtained from the input features Fg through linear transformation h.

[0056] The input sequence is usually represented as a word embedding matrix (X). Each attention head first converts the input sequence into Q, K, V matrices through linear transformation. Specifically, the input sequence is represented as , first Projection into three matrices: Query ,key Sum , the formula is as follows:

[0057]

[0058]

[0059] in is the weight matrix.

[0060] Among them, a single self-attention head can be expressed as:

[0061] Among them, softmax means row-by-row normalization. express Dimension.

[0062] After h parallel self-attention heads, the multi-head attention result can be expressed as:

[0063] in, is the weight matrix. The value of h can be set according to actual needs, and the embodiment of the present application does not limit this. For example, in the embodiment of the present application, h=8, that is, 8-head attention.

[0064] For each attention head, the attention map is obtained by transforming the input features With input features The calculation is done by matching, where f and g are learnable linear transformations. In other words, the generated attention map A is expressed as:

[0065] in is the generated attention map, yes Dimension.

[0066] The attention map A is obtained by calculating the dot product of the query matrix Q and the key matrix K and normalizing it with the softmax function. After obtaining the attention map A, the attention map A can be used to weight the elements (or vectors) in the value matrix V to obtain the weighted sum (i.e., enhanced features), where the weighted value matrix V (i.e., V=h(Fg )) represents the input features after linear transformation. Specifically, the vectors in V contain information for subsequent aggregation and weighting. The weighted sum is obtained by multiplying the attention map A with the value matrix V, which contains the global context information of the input features.

[0067] Finally, after the multi-head attention mechanism layer, the enhanced features are processed through the feedforward neural network layer to obtain the global features. The feedforward neural network layer contains two linear layers with ReLU activation and residual connections after each layer. The final output is the global features.

[0068] 2. Use adaptive graph convolutional network to extract local features.

[0069] The first fusion feature is input into the adaptive graph convolutional network to construct a video graph. The relevant content of constructing the video graph can correspond to the content of the glossary in the previous text. Based on the difference calculation formula, the difference between the two nodes m and n in the video graph is calculated to obtain the difference calculation result. The difference calculation formula is as follows:

[0070] in, is the activation function, is a trainable parameter vector used to transform The dimension is reduced to 1. and Represents the feature vectors of two different nodes m and n in the graph. This difference calculation It is a commonly used method in graph convolutional networks to measure the similarity or difference between nodes. By calculating the difference, a scalar value can be obtained, which reflects the distance or similarity between two nodes in the feature space.

[0071] Further, based on the difference calculation results, update The elements in , that is, the weights of the edges between two nodes, use the updated adjacency matrix The input features are aggregated using a fixed binary mask matrix M (used to limit the range of connections). The aggregation process is implemented through a dot product operation, and the calculation formula is as follows:

[0072] in Including trainable weights, Cin is the number of channels (or feature dimensions) of the input features, and Cout is the number of channels (or target feature dimensions) of the output features. This matrix usually contains trainable weights for linear transformation of input features during adaptive graph convolution. is fixed The binary mask matrix, according to Limit the connection range and control the connection influence between the current stage and the surrounding nodes. ⊙ is the dot product. Fully parameterized and optimized during the training phase. The elements in can be arbitrary values ​​without any constraints, which means that the edges of the graph are learned entirely based on the training data and the positions of the nodes in the video. It is a data-dependent graph that adaptively learns a unique graph for each video, where the unique graph is constructed based on the video content or video features. The calculation of can refer to the content described above.

[0073] After aggregation operation, the new node feature matrix is ​​obtained , which contains richer local temporal context information. In other words, That is, it represents local features.

[0074] In a possible implementation, before obtaining global features and local features, it is necessary to extract spatial stream features and temporal stream features from the video to be detected according to a feature extraction network to fuse them to obtain a first fused feature.

[0075] Among them, the feature extraction network in the embodiment of the present application can be implemented by a dual-stream network, which is composed of two parallel networks, processing spatial information and temporal information respectively to obtain richer and more accurate feature representation. The dual-stream network is such as an I3D dual-stream network.

[0076] In the dual-stream network, for the spatial stream, a convolutional neural network (CNN) is used to extract the spatial features of each video frame in the video to be detected, and the spatial stream features of the video to be detected are obtained, wherein the spatial features may include texture features, color features, shape features, etc. of the video frame, which are not listed here one by one. The spatial stream features can help capture static elements and details in the video, which is essential for identifying objects in the scene and understanding the composition of the picture. For the temporal stream, the optical flow between adjacent video frames in the video to be detected is used to estimate the pixel-level motion information between adjacent video frames through the optical flow, and the temporal stream features of the video to be detected are extracted. Among them, the optical flow can capture the position movement and shape change of the object in the video, thereby providing detailed information about the object movement and action. It should be understood that the object described in the embodiment of the present application may refer to a person or object in the video. After extracting the spatial stream features and the temporal stream features, the spatial stream features and the temporal stream features are subjected to feature fusion operations to obtain a first fusion feature. Exemplarily, the spatial stream features and the temporal stream features can be spliced ​​together to obtain the first fusion feature.

[0077] Exemplarily, the video processing device pre-processes the video to be detected, so that the video to be detected is = ,in is the video length, is the nth frame RGB image. In order to reduce the computational cost, we regularly sample at frame interval σ, and get contiguous and non-overlapping segments of length σ, where After it is input into the two-stream network, it is extracted into a length of The visual feature sequence is:

[0078] F represents the visual feature sequence extracted from the video to be detected, that is, the first fusion feature. This sequence is obtained by processing each frame (or sampled frame) in the video to be detected and passing it through the two-stream network. Indicates that at time step The feature vector extracted at represents the index of the nth time step or segment in the feature sequence F. F is Here R represents the set of real numbers, that is, the eigenvector Each element of is a real number. Therefore, Indicates that F is a C row A matrix of columns, where C is the dimension of the features, is the number of eigenvectors (corresponding to fragments)

[0079] Since the segments are continuous and non-overlapping, the starting frame of each segment will be the position of every σ frames in the video frame sequence. That is, the first segment starts from frame 1, and the second segment starts from frame 1+σ=6, so F is equivalent to The feature set of the starting frame of a segment of length σ frames is used to characterize the segment where the starting frame is located.

[0080] S302, based on the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined.

[0081] Among them, the actions in the video to be detected are related to the scene in which the video to be detected is located. For example, taking the video to be detected as a sports event video, each action can refer to a shot, a goal, a corner kick, a penalty kick, a kick-off, etc., and the video clip corresponding to each action can refer to the video clip of this action from the start time to the end time in the video to be detected.

[0082] Specifically, the video processing device can perform feature fusion on the global features and the local features to obtain a second fused feature, and then input the second fused feature into the boundary prediction network, and use the boundary prediction network to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action, and the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, and a candidate video segment for each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action. Further, according to the starting boundary probability sequence and the two-dimensional confidence map of each action, the confidence score of at least one candidate video segment is calculated, and from the at least one candidate video segment, the candidate video segment with the highest confidence score is selected as the video segment corresponding to each action.

[0083] The following describes a specific method of using the boundary prediction network and the second fusion feature to determine the starting boundary probability sequence of each action and the two-dimensional confidence map of each action.

[0084] The boundary prediction network includes a first branch network and a second branch network. The video processing device inputs the second fusion feature into the first branch network, performs temporal convolution processing on the second fusion feature through the first branch network, distinguishes the background frame and the action frame of the video to be detected, and obtains a sequence representing the probability of the action starting at each time point through further convolution and pooling operations, that is, the starting boundary probability sequence. Exemplarily, the starting probability sequence can be expressed as {p1, p2, ..., p i}, where p iIndicates the probability that the i-th frame is the starting boundary of the action. The higher the probability value, the greater the possibility that the action starts at this time point, or the greater the possibility that this time point is the starting boundary of the action.

[0085] The video processing device inputs the second fused feature into the second branch network, uses the second branch network to determine the candidate start boundary set and the candidate end boundary set corresponding to each action from the second fused feature through a boundary matching mechanism, and combines each candidate start boundary in the candidate start boundary set with each candidate end boundary in the candidate end boundary set into a matching pair, determines the confidence of each matching pair, and generates a two-dimensional confidence map for each action, where a matching pair is used to indicate a candidate video segment.

[0086] The second branch network includes a boundary matching module (BPM) and a proposal evaluation module (PEM). Exemplarily, a network structure of a boundary matching module and a proposal evaluation module provided in an embodiment of the present application is shown in Table 1.

[0087] Table 1

[0088] Specifically, the video processing device may first input the second fusion feature into the segment generation module, use the segment generation module to determine the candidate start boundary set and the candidate end boundary set corresponding to each action from the second fusion feature, and combine each candidate start boundary in the candidate start boundary set and each candidate end boundary in the candidate end boundary set into a matching pair (a candidate video segment). Further, the segment generation module scores each matching pair and retains the matching pairs (candidate video segments) with a score greater than a preset score value. In other words, the segment generation module is used to generate at least one candidate video segment. For ease of description, the matching pairs with a score greater than the preset score value are referred to as at least one candidate video segment below.

[0089] At least one candidate video segment is input into a boundary matching module, a start boundary and an end boundary of the at least one candidate video segment are optimized by the boundary matching module, and a two-dimensional confidence map is generated.

[0090] Optionally, the boundary matching module also includes a temporal evaluation module (TEM), which includes two 3×3 convolutional layers and a maximum pooling layer, and a normalization with a depth of 2. In this embodiment, the temporal accuracy of at least one candidate video segment can be evaluated by the temporal evaluation module. Exemplarily, the temporal evaluation module can evaluate the quality of the video segment by calculating the temporal consistency, duration, and contextual information of the video segment to improve the accuracy of the two-dimensional confidence map.

[0091] Each row in the two-dimensional confidence map represents the confidence of candidate video segments with the same specific start boundary but different durations, and each column represents the confidence of candidate video segments with the same specific duration but different start boundaries. It should be understood that different durations represent different end boundaries.

[0092] Furthermore, the video processing device may calculate the confidence score of at least one candidate video segment according to the starting boundary probability sequence and the two-dimensional confidence map of each action. Exemplarily, the video processing device may multiply and fuse the starting boundary probability sequence and the two-dimensional confidence map of each action to determine the confidence score of each candidate video segment. The calculation formula may be as follows:

[0093] in, represents the starting boundary of the candidate video segment, Indicates the end boundary of the candidate video segment, The candidate video segment is at the starting boundary The probability of For candidate video segments at the end boundary The probability of is the classification confidence, is the regression confidence.

[0094] It should be noted that classification confidence is usually the probability value obtained when the action category is classified by a classifier (such as a softmax layer). Regression confidence is obtained in the regression task, which indicates the accuracy of the model's prediction of the start and end time of the action. and It can be directly output from the trained model without manual setting or calculation.

[0095] Based on the above calculation method, the video processing device can calculate the confidence score of at least one candidate video segment, and then determine the candidate video segment with the highest confidence score in at least one candidate video segment as the video segment corresponding to each action, and the video segment indicates the starting boundary, ending boundary and confidence score of each action.

[0096] Optionally, when determining the video segment corresponding to each action according to the confidence score of at least one candidate video segment, a non-maximum suppression method may be used to remove redundancy to obtain the final video segment corresponding to each action.

[0097] The confidence score is calculated by comprehensively considering the probability of the starting boundary of each action, the probability of the ending boundary (duration), and the degree of match between the starting boundary and the ending boundary. Since the confidence score comprehensively considers many factors, the accuracy of determining the video segment corresponding to each action based on the confidence score is also higher.

[0098] It should be understood that the starting boundary described in the embodiments of the present application indicates the starting moment of the action in the video, and the ending boundary indicates the ending moment of the action in the video.

[0099] Based on the same inventive concept, an embodiment of the present application provides a video processing device, which is used to implement any of the above-mentioned video processing methods, for example Figure 3 The video processing method shown in the figure can also realize the functions of the video processing device mentioned above.

[0100] Please refer to Figure 7 , is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 7 As shown, the video processing device 700 includes a feature extraction module 701 and a boundary positioning module 702 .

[0101] Exemplarily, the feature extraction module 701 is used to extract global temporal context information and local temporal context information features of the first fusion feature of the video to be detected based on the first fusion feature of the video to be detected and a preset feature enhancement network, so as to obtain the global features and local features of the video to be detected, wherein the first fusion feature includes the spatial flow features and temporal flow features of the video to be detected; the boundary positioning module 702 is used to locate the boundary of each action in the video to be detected based on the global features and local features in combination with a preset boundary prediction network, and determine the video segment corresponding to each action.

[0102] In one possible implementation, the feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; the feature extraction module 701 is specifically used to: use the attention enhancement network to extract the dependency between non-adjacent video frames in the first fusion feature to obtain the global feature; use the adaptive graph convolutional network to extract the dependency between adjacent video frames in the first fusion feature to obtain the local feature.

[0103] In one possible implementation, the attention enhancement network consists of a time-extended pre-convolution block and an attention network; the time-extended pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

[0104] In a possible implementation, the boundary positioning module 702 is specifically used to: perform feature fusion on global features and local features to obtain a second fused feature; input the second fused feature into a boundary prediction network, and use the boundary prediction network to determine a starting boundary probability sequence for each action and a two-dimensional confidence map for each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, wherein a candidate video segment for each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; calculate the confidence score of at least one candidate video segment based on the starting boundary probability sequence and the two-dimensional confidence map of each action; and determine the candidate video segment with the highest confidence score in the at least one candidate video segment as the video segment corresponding to each action.

[0105] In one possible implementation, the boundary prediction network includes a first branch network and a second branch network; wherein the first branch network is used to perform temporal convolution processing on the second fused feature to obtain a starting boundary probability sequence for each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fused feature through a boundary matching mechanism, and each candidate starting boundary in the candidate starting boundary set and each candidate ending boundary in the candidate ending boundary set are combined into a matching pair, the confidence of each matching pair is determined, and a two-dimensional confidence map for each action is generated, wherein a matching pair is used to indicate a candidate video segment.

[0106] In a possible implementation, the feature extraction module 701 is also used to extract the spatial features of each video frame in the video to be detected using a convolutional neural network before obtaining the global features and local features of the video to be detected, so as to obtain the spatial flow features of the video to be detected; extract the temporal flow features of the video to be detected based on the optical flow between adjacent video frames in the video to be detected; and perform feature fusion on the spatial flow features and the temporal flow features to obtain a first fusion feature.

[0107] Based on the same inventive concept, an embodiment of the present application provides a video processing device, which is used to implement any of the above-mentioned video processing methods, for example Figure 3 The video processing method shown, and the device can also realize the functions of the video processing device mentioned above.

[0108] Please refer to Figure 8 , is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 8 As shown, the video processing device 800 includes at least one processor 801 and a memory 802 communicatively connected to the at least one processor 801 .

[0109] The processor 801 may be a general-purpose processor or a dedicated processor. The processor 801 may include, for example, a baseband processor or a central processing unit. The baseband processor may be used to process communication protocols and communication data. The central processing unit may be used to control the video processing device 800, execute software programs and / or process data. Different processors may be independent devices or may be arranged in one or more processing circuits, for example, integrated in one or more dedicated integrated circuits.

[0110] In one embodiment, the memory 802 stores instructions that can be executed by at least one processor 801. The at least one processor 801 implements the functions of the video processing device described above by executing the instructions stored in the memory 802. Accordingly, the steps performed by the video processing device described above can also be implemented.

[0111] In this embodiment, the video processing device 800 can also implement the functions of the previous video processing apparatus 700 , and at least one processor 801 in the video processing device 800 can also implement the functions of the previous feature extraction module 701 and the boundary positioning module 702 .

[0112] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes any of the video processing methods described above, for example, Figure 3 The video processing method shown.

[0113] Based on the same inventive concept, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed on a computer, enables any of the above-mentioned video processing methods, for example, Figure 3 The video processing method shown is implemented.

[0114] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video processing method, characterized in that: include: According to a first fusion feature of the video to be detected and a preset feature enhancement network, extracting global temporal context information and local temporal context information features from the first fusion feature to obtain global features and local features of the video to be detected, wherein the first fusion feature includes spatial stream features and temporal stream features of the video to be detected; According to the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined.

2. The method according to claim 1, characterized in that The feature enhancement network is composed of an attention enhancement network and an adaptive graph convolutional network; According to the first fusion feature of the video to be detected and a preset feature enhancement network, extracting global time context information and local time context information features from the first fusion feature to obtain the global features and local features of the video to be detected, including: Extracting the dependency between non-adjacent video frames in the first fusion feature using the attention enhancement network to obtain the global feature; The adaptive graph convolutional network is used to extract the dependency between adjacent video frames in the first fusion feature to obtain the local feature.

3. The method according to claim 2, characterized in that The attention enhancement network consists of a time-extended pre-convolution block and an attention network; the time-extended pre-convolution block is used to enhance the perception range of the first fusion feature in the time dimension, and the attention network is used to extract the feature information of the first fusion feature at different time scales according to a preset multi-head attention mechanism.

4. The method according to claim 1, characterized in that According to the global features and the local features, combined with a preset boundary prediction network, the boundary of each action in the video to be detected is located, and the video segment corresponding to each action is determined, including: Performing feature fusion on the global feature and the local feature to obtain a second fused feature; Inputting the second fused feature into the boundary prediction network, and using the boundary prediction network to determine a starting boundary probability sequence of each action and a two-dimensional confidence map of each action, wherein each value in the starting boundary probability sequence is used to indicate the probability that a time point is the starting boundary of each action; and the two-dimensional confidence map is used to indicate the confidence of at least one candidate video segment corresponding to each action, wherein a candidate video segment of each action is composed of video frames between a candidate starting boundary and a candidate ending boundary of each action; Calculating a confidence score of the at least one candidate video segment according to the starting boundary probability sequence and the two-dimensional confidence map of each action; A candidate video segment with a highest confidence score among the at least one candidate video segment is determined as the video segment corresponding to each action.

5. The method according to claim 4, characterized in that: The boundary prediction network includes a first branch network and a second branch network; wherein the first branch network is used to perform temporal convolution processing on the second fused feature to obtain a starting boundary probability sequence for each action; the second branch network is used to determine a candidate starting boundary set and a candidate ending boundary set corresponding to each action from the second fused feature through a boundary matching mechanism, and each candidate starting boundary in the candidate starting boundary set and each candidate ending boundary in the candidate ending boundary set are combined into a matching pair, the confidence of each matching pair is determined, and a two-dimensional confidence map for each action is generated, wherein a matching pair is used to indicate a candidate video segment.

6. The method according to any one of claims 1 to 5, characterized in that: Before obtaining the global features and local features of the video to be detected, the method further includes: Using a convolutional neural network to extract the spatial features of each video frame in the video to be detected, to obtain the spatial flow features of the video to be detected; Extracting a temporal flow feature of the video to be detected based on the optical flow between adjacent video frames in the video to be detected; The spatial stream feature and the temporal stream feature are fused to obtain the first fused feature.

7. A video processing device, characterized in that: include: A feature extraction module is used to extract global time context information and local time context information features of the first fused features of the video to be detected according to a first fused feature and a preset feature enhancement network, so as to obtain global features and local features of the video to be detected, wherein the first fused feature includes spatial stream features and temporal stream features of the video to be detected; The boundary positioning module is used to perform boundary positioning on each action in the video to be detected based on the global features and the local features in combination with a preset boundary prediction network, and determine the video segment corresponding to each action.

8. A video processing device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method according to any one of claims 1 to 6 by executing the instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed on a computer, enable the method according to any one of claims 1 to 6 to be implemented.

Images