A frame offset artificial intelligence prediction method for primary and backup dual paths of audio and video streams

Abstract

The application provides a kind of main and standby double audio and video stream frame offset artificial intelligence prediction method, it is related to video processing technical field.The method comprises: the first video stream and second video stream are preprocessed, obtain first video frame sequence and second video frame sequence;First video frame sequence and second video frame sequence are respectively processed by feature extraction, obtain first space-time feature map and second space-time feature map;First partition feature vector and second partition feature vector are obtained, and input time sequence relation model, obtain time sequence relation feature information, and input multilayer perception network model, obtain frame offset.According to the application, the same preprocessing can be carried out on two video streams, the influence caused by different encoding methods is reduced, and the frame offset can be determined by understanding the content and time sequence of the video frame, further reducing the influence of the video frame difference caused by different video encoding methods, and improving the alignment accuracy.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an artificial intelligence prediction method for frame offset of primary and backup dual-channel audio and video streams. Background Technology

[0002] In related technologies, methods for aligning two broadcast television video streams often employ simple image feature extraction methods to extract image features from both streams separately, calculate image similarity, find multiple consecutive pairs of identical frames, and determine whether the content broadcast by the two video streams at the current moment is the same. However, due to differences in encoding methods and other reasons, each frame in the two video streams may have differences, making it difficult to accurately find the exact time point when completely identical images appear using a fixed similarity threshold, resulting in insufficient alignment accuracy.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides an artificial intelligence prediction method for frame offset of primary and backup dual-channel audio and video streams, which can solve the technical problem of low alignment accuracy when there are differences in encoding methods and other aspects between the two video streams.

[0005] According to a first aspect of the present invention, an artificial intelligence prediction method for frame offset of primary and backup dual-channel audio and video streams is provided, comprising:

[0006] Preprocess the first and second video streams to be aligned to obtain the first video frame sequence of the first video stream and the second video frame sequence of the second video stream.

[0007] Feature extraction processing is performed on the first video frame sequence and the second video frame sequence respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence.

[0008] Obtain the first partition feature vector of multiple partitions of the first spatiotemporal feature map, and obtain the second partition feature vector of multiple partitions of the second spatiotemporal feature map;

[0009] The feature vectors of the first and second partitions are input into the time series relationship model for processing to obtain time series relationship feature information.

[0010] The temporal relationship feature information is processed by a multilayer perceptron model to obtain the frame offsets of the first video stream and the second video stream.

[0011] According to the present invention, preprocessing is performed on a first video stream and a second video stream to be aligned to obtain a first video frame sequence of the first video stream and a second video frame sequence of the second video stream, including:

[0012] The first video stream and the second video stream are subjected to unified decoding, frame sampling and normalization to obtain the first video frame sequence and the second video frame sequence, respectively.

[0013] According to the present invention, feature extraction processing is performed on a first video frame sequence and a second video frame sequence respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence, including:

[0014] The first video frame sequence is processed by a first 3D convolutional neural network model to obtain a first spatiotemporal feature map.

[0015] By using a second 3D convolutional neural network model with at least some parameters identical to the first 3D convolutional neural network model, feature extraction processing is performed on the second video frame sequence to obtain a second spatiotemporal feature map.

[0016] According to the present invention, obtaining the first partition feature vector of multiple partitions of a first spatiotemporal feature map includes:

[0017] The first spatiotemporal feature map is divided along the temporal direction, width direction, and height direction to obtain multiple first partition feature maps;

[0018] Flatten the feature map of the first partition into the feature vector of the first partition;

[0019] Based on the temporal direction index of the first partition feature map in the first spatiotemporal feature map, and the indexes in the width and height directions, determine the first spatiotemporal position code corresponding to the first partition feature map;

[0020] The first spatiotemporal location code is fused with the first partition feature vector to obtain the first partition spatiotemporal feature vector.

[0021] According to the present invention, obtaining the second partition feature vector of multiple partitions of the second spatiotemporal feature map includes:

[0022] The second spatiotemporal feature map is divided along the temporal direction, width direction, and height direction to obtain multiple second partition feature maps;

[0023] Flatten the feature map of the second partition into the feature vector of the second partition;

[0024] Based on the temporal direction index of the second partition feature map in the second spatiotemporal feature map, and the indexes in the width and height directions, determine the second spatiotemporal position code corresponding to the second partition feature map;

[0025] The second spatiotemporal location code is fused with the second partition feature vector to obtain the second partition spatiotemporal feature vector.

[0026] According to the present invention, the first partition feature vector and the second partition feature vector are input into a time-series relation model for processing to obtain time-series relation feature information, including:

[0027] At each level of the temporal relationship model, a cross-attention mechanism is used to process the first input feature vector and the second partition feature vector corresponding to the first partition feature vector to obtain the first output feature vector corresponding to the first partition feature vector, which is used as the first input feature vector of the next level. In the first level of the temporal relationship model, the first input feature vector is the first partition feature vector.

[0028] After processing through multiple levels of the temporal relationship model, the first output feature vector is fused to obtain temporal relationship feature information.

[0029] According to the present invention, at each level of the temporal relationship model, a cross-attention mechanism is used to process the first input feature vector and the second partition feature vector corresponding to the first partition feature vector to obtain the first output feature vector corresponding to the first partition feature vector, which serves as the first input feature vector for the next level, including:

[0030] The query matrix is ​​obtained by processing the first input feature vector corresponding to the first partition feature vector with the query weight matrix.

[0031] The key-value matrix is ​​obtained by processing the feature vector of the second partition with the key-value weight matrix;

[0032] The numerical matrix is ​​obtained by processing the feature vector of the second partition with the numerical weight matrix;

[0033] Based on the query matrix, the key-value matrix, and the numerical matrix, a fusion feature matrix is ​​obtained;

[0034] The fused feature matrix is ​​processed through a multi-layer perceptron to obtain the first output feature vector corresponding to the first partition feature vector.

[0035] According to the present invention, processing the temporal relationship feature information through a multilayer perceptron model to obtain the frame offsets of the first video stream and the second video stream includes:

[0036] The temporal relationship feature information is processed through the first mapping layer of the multilayer perceptron model to obtain the first mapping feature information;

[0037] The first mapping feature information is obtained by processing the activation layers of the multilayer perceptron model;

[0038] The frame offset is obtained by processing the first activation feature information through the second mapping layer of the multilayer perceptron model.

[0039] According to a second aspect of the present invention, a frame offset artificial intelligence prediction system for primary and backup dual-channel audio and video streams is provided, comprising:

[0040] The preprocessing module is used to preprocess the first video stream and the second video stream to be aligned, to obtain the first video frame sequence of the first video stream and the second video frame sequence of the second video stream.

[0041] The feature extraction module is used to perform feature extraction processing on the first video frame sequence and the second video frame sequence respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence.

[0042] The partitioning module is used to obtain the first partition feature vector of multiple partitions of the first spatiotemporal feature map and the second partition feature vector of multiple partitions of the second spatiotemporal feature map.

[0043] The temporal relationship module is used to input the feature vectors of the first partition and the feature vectors of the second partition into the temporal relationship model for processing to obtain temporal relationship feature information;

[0044] The frame offset module is used to process the temporal relationship feature information through a multilayer perceptron model to obtain the frame offsets of the first video stream and the second video stream.

[0045] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the frame offset artificial intelligence prediction method for the primary and backup dual-channel audio and video streams.

[0046] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0047] According to the present invention, the impact of different encoding methods can be reduced by performing the same preprocessing on two video streams. Furthermore, the frame offset can be determined by understanding the content and temporal relationship of video frames, further reducing the impact of video frame differences caused by different video encoding methods and improving alignment accuracy. Moreover, the first and second video streams can be preprocessed in a unified manner to reduce the impact of video frame differences caused by different video encoding methods, obtaining a first and second video frame sequence with consistent data format, frame rate, resolution, etc., providing an accurate data foundation for understanding video meaning and aligning videos. 3D convolution can also be used to obtain spatiotemporal feature maps of continuous video frame sequences. The temporal relationship between consecutive video frame sequences can be fused in the spatiotemporal feature map, facilitating the understanding of the content of the continuous video frame sequence and helping to improve the prediction accuracy of frame offset. Furthermore, the spatiotemporal feature map can be partitioned to obtain partition feature vectors suitable for the Transformer model. Encoding can be performed based on the temporal and spatial positions of the partition feature vectors, facilitating the model to determine the position of each partition, thereby more accurately understanding the video content and the relationship between the first and second video streams, thus improving the accuracy of frame offset. Furthermore, a cross-attention mechanism can be used to cross-reference and fuse the feature information of the first and second video frame sequences to obtain the spatial and temporal dependencies between the first input feature vector and the second partition feature vector. This achieves spatiotemporal information fusion, enabling alignment and comparison of dual video streams and providing a sufficient and accurate data foundation for frame offset prediction. Moreover, a multilayer perceptron model can be used to analyze temporal relationship feature information, and frame offsets can be determined through classification or regression, thereby achieving the goal of rectifying video temporal data.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0050] Figure 1 An exemplary flowchart of an artificial intelligence prediction method for frame offset of primary and backup dual-channel audio and video streams according to an embodiment of the present invention is shown.

[0051] Figure 2 An exemplary schematic diagram illustrates the application of the AI-based frame offset prediction method for primary and backup dual-channel audio and video streams according to an embodiment of the present invention.

[0052] Figure 3 A block diagram of an artificial intelligence prediction system for frame offset of primary and backup dual-channel audio and video streams according to an embodiment of the present invention is shown as an example. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Figure 1 An exemplary flowchart illustrates an artificial intelligence prediction method for frame offset of primary and backup dual-channel audio and video streams according to an embodiment of the present invention. The method includes:

[0056] Step S1: Preprocess the first video stream and the second video stream to be aligned to obtain the first video frame sequence of the first video stream and the second video frame sequence of the second video stream.

[0057] Step S2: Perform feature extraction processing on the first video frame sequence and the second video frame sequence respectively to obtain the first spatiotemporal feature map of the first video frame sequence and the second spatiotemporal feature map of the second video frame sequence.

[0058] Step S3: Obtain the first partition feature vector of multiple partitions of the first spatiotemporal feature map, and obtain the second partition feature vector of multiple partitions of the second spatiotemporal feature map.

[0059] Step S4: Input the feature vectors of the first partition and the feature vectors of the second partition into the time series relationship model for processing to obtain the time series relationship feature information;

[0060] Step S5: Process the temporal relationship feature information through a multilayer perceptron model to obtain the frame offset of the first video stream and the second video stream.

[0061] The AI-based frame offset prediction method for primary and backup dual-channel audio and video streams according to embodiments of the present invention can reduce the impact of different encoding methods by performing the same preprocessing on the two video streams. Furthermore, it can determine the frame offset by understanding the content and timing relationship of the video frames, further reducing the impact of video frame differences caused by different video encoding methods and improving alignment accuracy.

[0062] Example 1:

[0063] According to one embodiment of the present invention, in step S1, the two video streams to be aligned can be preprocessed uniformly to reduce the differences in video frames caused by different video encoding methods.

[0064] According to one embodiment of the present invention, preprocessing a first video stream and a second video stream to be aligned to obtain a first video frame sequence of the first video stream and a second video frame sequence of the second video stream includes: performing unified decoding, frame sampling and normalization processing on the first video stream and the second video stream to obtain the first video frame sequence and the second video frame sequence respectively.

[0065] According to one embodiment of the present invention, by unifying decoding, frame sampling and normalization processing, the data format, frame rate, resolution and other conditions of the obtained first video frame sequence and second video frame sequence are kept consistent, reducing the impact of inconsistent video encoding and helping to improve the accuracy of alignment.

[0066] According to an embodiment of the present invention, normalization can be performed using the following formulas (1) and (2):

[0067]

[0068] Among them, X p For the p-th video stream, when p=1, X p For the first video stream, when p=2, X p This is the second video stream. D(X) p ) indicates that for X p Decoding is performed to obtain multiple video frames. The dimensions are T×H×W×C, where T is the number of frames, H is the height of each video frame, W is the width of each video frame, and C is the number of channels per pixel. Before processing, each pixel has three channels of data: R, G, and B. Therefore, C = 3. μ represents the pixel value of the c-th channel of the h-th pixel in the h-th height direction and the w-th pixel in the w-th width direction in the t-th video frame. c σ represents the mean pixel value of the c-th channel of each pixel in each video frame. cX′ represents the standard deviation of the pixel value of the c-th channel of each pixel in each video frame. p [t,h,w,c] represents the pixel value of the c-th channel of the h-th pixel in the height direction and the w-th pixel in the width direction in the t-th video frame after normalization. When p=1, it can represent the pixel value of the c-th channel of the h-th pixel in the height direction and the w-th pixel in the width direction in the t-th video frame of the first video frame sequence. When p=2, it can represent the pixel value of the c-th channel of the h-th pixel in the height direction and the w-th pixel in the width direction in the t-th video frame of the second video frame sequence. That is, normalization can be performed in the above way to determine the pixel value of each channel of each pixel in each video frame of the first video frame sequence and the second video frame sequence.

[0069] In this way, the first and second video streams can be preprocessed in a unified manner to reduce the impact of differences in video frames caused by different video encoding methods. This results in a first and second video frame sequence with consistent data format, frame rate, resolution, and other conditions, providing an accurate data foundation for understanding the meaning of the video and aligning the video.

[0070] Example 2:

[0071] According to one embodiment of the present invention, in step S2, after obtaining the first video frame sequence and the second video frame sequence, feature information of the first video frame sequence and the second video frame sequence can be obtained respectively to facilitate understanding of the video content and thus more accurately determine the frame offset of the first video stream and the second video stream. In the example, a 3D convolutional neural network model can be used for feature extraction processing.

[0072] According to one embodiment of the present invention, feature extraction processing is performed on a first video frame sequence and a second video frame sequence to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence, including: performing feature extraction processing on the first video frame sequence using a first 3D convolutional neural network model to obtain a first spatiotemporal feature map; and performing feature extraction processing on the second video frame sequence using a second 3D convolutional neural network model with at least some parameters identical to the first 3D convolutional neural network model to obtain a second spatiotemporal feature map.

[0073] According to one embodiment of the present invention, the first 3D convolutional neural network model and the second 3D convolutional neural network model can be two branches of a 3D convolutional neural network model. The parameters of the two branches can be completely identical or at least partially identical (e.g., the parameters of the convolution kernels are the same), which makes the feature extraction process of the two branches consistent and facilitates the comparison of features of video frames in two video frame sequences.

[0074] According to one embodiment of the present invention, both the first 3D convolutional neural network model and the second 3D convolutional neural network model may include multiple 3D convolutional layers. Each 3D convolutional layer may have multiple 3D convolutional kernels, which can process the input tensor to obtain an output tensor, and the output tensor can be used as the input tensor of the next 3D convolutional layer. The first video frame sequence and the second video frame sequence are respectively used as the input tensors of the first 3D convolutional layer of the first 3D convolutional neural network model and the second 3D convolutional neural network model, and the dimensions of the first video frame sequence and the second video frame sequence are both T×H×W×C. After processing by the 3D convolutional layers, the dimensions of the output tensor and the input tensor may be different.

[0075] According to one embodiment of the present invention, each 3D convolutional layer may have multiple 3D convolutional kernels, and the size of each 3D convolutional kernel is K. t ×K h ×K w K t =2,K h =3,K w =3. The parameters of the convolution kernel can be obtained through training.

[0076] According to an embodiment of the present invention, the 3D convolutional layer can process the input tensor of the layer using the following formula (3) to obtain the output tensor, wherein the input tensor is the output tensor of the previous 3D convolutional layer.

[0077]

[0078] in, To perform 3D convolution processing on the feature map of the m-th channel in the input tensor of the l-th 3D convolutional layer, the c-th convolutional kernel is located at the δ-th channel. t Frame, δ h row, δ w Column weight parameters, For the m-th channel of the input tensor of the l-th 3D convolutional layer, at the t·s-th position... t +δ t Frame i·s h +δ h row, j·s w +δ w The pixel values ​​of the columns of pixels, where s t Let s be the time step. h Let s be the step size in the height direction. w Let be the step size in the width direction, t be the number of steps in the time direction, i be the number of steps in the height direction, j be the number of steps in the width direction, and C be the step size in the width direction. l-1 The number of channels in the input tensor. Let be the bias of the c-th convolutional kernel in the l-th 3D convolutional layer. For activation function, Let m be the pixel value of the c-th channel in the output tensor of the l-th 3D convolutional layer, located at the i-th row and j-th column of the t-th frame, where m ≤ C. l-1 -1, δ t ≤K t -1, δ h ≤K h -1, δ w ≤K w -1, and i, j, t, s t s h s w m, C l-1 K t K h K h δ t δ h δ w c and l are all integers.

[0079] According to an embodiment of the present invention, the pixel values ​​of each pixel at each position of the 3D feature map of each channel output by each 3D convolutional layer can be determined by the above formula (3), thereby obtaining the output tensor. Multiple 3D convolutional layers in the first 3D convolutional neural network model can perform the above feature extraction processing on the first video frame sequence to obtain a first spatiotemporal feature map. Similarly, multiple 3D convolutional layers in the second 3D convolutional neural network model can perform the above feature extraction processing on the second video frame sequence to obtain a second spatiotemporal feature map.

[0080] In this way, spatiotemporal feature maps of continuous video frame sequences can be obtained using 3D convolution. The temporal relationship between consecutive video frame sequences can be fused into the spatiotemporal feature maps, which facilitates the understanding of the content of the continuous video frame sequences and helps improve the prediction accuracy of frame offsets.

[0081] Example 3:

[0082] According to an embodiment of the present invention, in step S3, when performing the understanding and processing of two video contents and determining the frame offset, a Transformer-based model can be used for processing. Before using the model, the first spatiotemporal feature map and the second spatiotemporal feature map can be partitioned, and the information obtained based on the partitioning can be used as the input information of the Transformer-based model, i.e., multiple tokens. The attention mechanism of the Transformer model is then used to fuse the information of each partition to further understand the video content and determine the relationship between the first video stream and the second video stream, thereby improving the accuracy of the frame offset.

[0083] According to an embodiment of the present invention, obtaining a first partition feature vector of a plurality of partitions of a first spatiotemporal feature map includes: dividing the first spatiotemporal feature map along a temporal direction, a width direction, and a height direction to obtain a plurality of first partition feature maps; flattening the first partition feature maps into first partition feature vectors; determining a first spatiotemporal position code corresponding to the first partition feature map based on the sequence number of the first partition feature map in the first spatiotemporal feature map in the temporal direction, and the sequence numbers in the width direction and the height direction; and fusing the first spatiotemporal position code with the first partition feature vector to obtain a first partition spatiotemporal feature vector.

[0084] According to an embodiment of the present invention, obtaining a second partition feature vector of multiple partitions of a second spatiotemporal feature map includes: dividing the second spatiotemporal feature map along the temporal direction, width direction, and height direction to obtain multiple second partition feature maps; flattening the second partition feature maps into second partition feature vectors; determining a second spatiotemporal position code corresponding to the second partition feature map based on the sequence number of the second partition feature map in the temporal direction and the sequence numbers in the width direction and height direction of the second partition feature map in the second spatiotemporal feature map; and fusing the second spatiotemporal position code with the second partition feature vector to obtain a second partition spatiotemporal feature vector.

[0085] According to an embodiment of the present invention, as described above, the method of partitioning the first spatiotemporal feature map and obtaining the first partition spatiotemporal feature vector is consistent with the method of partitioning the second spatiotemporal feature map and obtaining the second partition spatiotemporal feature vector. Taking the acquisition of the first partition spatiotemporal feature vector as an example, the size of each first partition feature map can be set, for example, the size of each first partition feature map is T. p ×P h ×P w In the example, T p =2, P h =P w =16, that is, the temporal direction of each first spatiotemporal feature map can be divided into n t =T T / T p The height direction can be divided into n parts. h =H T / P h The width direction can be divided into n parts. w =W T / P w T T H represents the number of frames in the first spatiotemporal feature map. T W represents the height of the first spatiotemporal feature map in each frame. T This represents the width of the first spatiotemporal feature map in each frame. The set of partition indices is I = {(x,y,z)|0≤x}. <n t ,0≤y <nh ,0≤z <n w}, where x, y, and z are all integers. For the (x, y, z)th partition, xT represents the temporal direction in the first spatiotemporal feature map. p up to (x+1)T p The range of -1, yP in the height direction h to (y+1)P h The range of -1, zP in the width direction w To (z+1)P w The area enclosed by the range of -1.

[0086] According to one embodiment of the present invention, the first partition feature map of each partition can be flattened into a first partition feature vector. That is, the pixel values ​​of each pixel in the first partition feature map are sequentially used as data in the first partition feature vector to obtain the first partition feature vector corresponding to each partition. The length of the first partition feature vector is T. p ×P h ×P w Furthermore, the above processing can be performed on the first spatiotemporal feature maps of multiple feature channels, and multiple first partition feature vectors can be obtained for each feature channel.

[0087] According to one embodiment of the present invention, a spatiotemporal position code corresponding to the first partition feature map can be determined based on the temporal direction index (e.g., the xth index) and the indexes in the height and width directions (e.g., the (y, z)th index) of the first partition feature map in the first spatiotemporal feature map. For example, the temporal direction index is encoded to obtain a temporal position code, for example, by processing x through a fully connected layer to obtain a vector as the temporal position code, and the spatial index is encoded to obtain a spatial position code, for example, by processing (y, z) through a fully connected layer to obtain a vector as the spatial position code. The temporal position code and the spatial position code are summed or concatenated to obtain the first spatiotemporal position code. Further, the first spatiotemporal position code can be concatenated or summed with the first partition feature vector at the corresponding position to obtain the first partition spatiotemporal feature vector.

[0088] In this way, the spatiotemporal feature map can be partitioned to obtain partition feature vectors suitable for the Transformer model. The partition feature vectors can be encoded based on their temporal and spatial positions, making it easier for the model to determine the position of each partition. This allows for a more accurate understanding of the video content and the relationship between the first and second video streams, thereby improving the accuracy of frame offsets.

[0089] Example 4:

[0090] According to an embodiment of the present invention, in step S4, the timing relationship model is the Transformer-based model described above, which can fuse the first partition feature vector and the second partition feature vector of the above multiple channels to determine the timing relationship between the first video stream and the second video stream, and then determine the frame offset between them.

[0091] According to an embodiment of the present invention, the first partition feature vector and the second partition feature vector are input into a temporal relationship model for processing to obtain temporal relationship feature information. The process includes: at each level of the temporal relationship model, using a cross-attention mechanism to process the first input feature vector and the second partition feature vector corresponding to the first partition feature vector to obtain a first output feature vector corresponding to the first partition feature vector, which is used as the first input feature vector of the next level. In the first level of the temporal relationship model, the first input feature vector is the first partition feature vector. After processing through multiple levels of the temporal relationship model, the first output feature vector is fused to obtain temporal relationship feature information.

[0092] According to an embodiment of the present invention, a cross-attention mechanism can be used to fuse the feature vectors of the first partition and the feature vectors of the second partition to determine the temporal relationship between them. At each level of the temporal relationship model, the cross-attention mechanism is used to process the first input feature vector corresponding to the first partition feature vector and the second partition feature vector to obtain a first output feature vector corresponding to the first partition feature vector, which serves as the first input feature vector for the next level. This includes: processing the first input feature vector corresponding to the first partition feature vector with a query weight matrix to obtain a query matrix; processing the second partition feature vector with a key-value weight matrix to obtain a key-value matrix; processing the second partition feature vector with a numerical weight matrix to obtain a numerical matrix; obtaining a fused feature matrix based on the query matrix, the key-value matrix, and the numerical matrix; and processing the fused feature matrix through a multilayer perceptron layer to obtain the first output feature vector corresponding to the first partition feature vector.

[0093] According to an embodiment of the present invention, each level of the temporal relationship model is a Transformer-based model. In each level, a cross-attention mechanism can be used to process the first input feature vector and the second partition feature vector corresponding to the first partition feature vector to obtain the spatial and temporal dependencies between the first input feature vector and the second partition feature vector, thereby realizing the comprehensive fusion of spatiotemporal information to determine the temporal relationship between the first video stream and the second video stream, and then determine the frame offset between them.

[0094] According to one embodiment of the present invention, a query weight matrix W can be generated. QKey-value weight matrix W K , and numerical weight matrix W V The first input feature vectors of the level are combined into a matrix. For example, in the first level, multiple first partition feature vectors are used as the first input feature vectors and combined into a matrix. If the number of levels is not 1, the first output feature vectors of the previous level are combined into a matrix, which is used as the first input matrix. The first input matrix is ​​then used to perform matrix multiplication with the query weight matrix to obtain the query matrix.

[0095] According to one embodiment of the present invention, the query matrix Q can be determined by the following formula (4):

[0096]

[0097] in, It is the first input matrix of the l-th level, and also the matrix composed of the first output feature vectors of the (l-1)-th level (i.e., the first output matrix).

[0098] According to one embodiment of the present invention, in each level, the second partition feature vector can be formed into a matrix, namely, the second input matrix, and matrix multiplication operation can be performed with the key value weight matrix and the numerical weight matrix to obtain the key value matrix and the numerical matrix.

[0099] According to an embodiment of the present invention, the key matrix K and the numerical matrix V can be obtained by the following formulas (5) and (6):

[0100] K = H2W K (5)

[0101] V = H2W V (6)

[0102] H2 is the second input matrix.

[0103] According to an embodiment of the present invention, the query matrix, the key-value matrix, and the numerical matrix can be fused using a cross-attention mechanism to obtain a fused feature matrix. The fused feature matrix can be obtained using the following formula (7):

[0104]

[0105] Among them, K T Let d be the transpose of K. k Let K be the dimension of each vector in K, Softmax be the activation function, Cross_Attention be the cross-attention mechanism, and Cross_Attention(Q,K,V) be the fused feature matrix.

[0106] According to one embodiment of the present invention, the above-mentioned fused feature matrix is ​​normalized and processed by a multilayer perceptron layer (e.g., a linear operation layer and an activation layer) to obtain a first output matrix, that is, a matrix composed of the first output feature vectors corresponding to the first partition feature vectors, and is used as the first input matrix of the next layer.

[0107] According to one embodiment of the present invention, after multiple levels of processing, the first output feature vector is fused to obtain temporal relationship feature information. In other words, the first output matrix of the last level is used as the temporal relationship feature information.

[0108] In this way, the feature information of the first video frame sequence and the feature information of the second video frame sequence can be cross-linked and fused through the cross-attention mechanism to obtain the spatial and temporal dependencies between the first input feature vector and the second partition feature vector, thereby realizing the fusion of spatiotemporal information. This enables the alignment and comparison of the two video streams, providing a sufficient and accurate data foundation for the prediction of frame offset.

[0109] Example 5:

[0110] According to one embodiment of the present invention, in step S5, the temporal relationship feature information can be processed by a multilayer perceptron model to obtain the frame offset of the first video stream and the second video stream. That is, the frame offset of the two video streams can be determined by analyzing the feature information (i.e., temporal relationship feature information) that enables comparison of the two video streams through a multilayer perceptron model.

[0111] According to an embodiment of the present invention, processing the temporal relationship feature information through a multilayer perceptron model to obtain the frame offset of a first video stream and a second video stream includes: processing the temporal relationship feature information through a first mapping layer of the multilayer perceptron model to obtain first mapping feature information; processing the first mapping feature information through an activation layer of the multilayer perceptron model to obtain first activation feature information; and processing the first activation feature information through a second mapping layer of the multilayer perceptron model to obtain the frame offset.

[0112] According to one embodiment of the present invention, a multilayer perceptron model can be composed of multiple fully connected layers and activation layers, which can perform mapping, activation, and other processing on temporal relationship feature information, and finally output frame offset. In the example, prediction can be performed by regression or classification. For example, when predicting by regression, the specific data of the frame offset can be directly output. When predicting by classification, the probability of the frame offset being multiple values ​​can be output, and the value corresponding to the maximum probability is taken as the frame offset. The present invention does not limit this.

[0113] According to an embodiment of the present invention, when using regression for prediction, the frame offset e can be obtained according to the following formula (8):

[0114] e=W2σ(W1H TR +b1)+b2(8)

[0115] Where W1 and W2 are weight matrices, b1 and b2 are bias vectors, σ is the activation function, and H... TR This refers to temporal relationship characteristic information.

[0116] According to one embodiment of the present invention, the first mapping layer can process the temporal relationship feature information through W1 and b1 to obtain the first mapping feature information. After processing by the activation function, the first activation feature information is obtained. Then, the second mapping layer can continue to process the first activation feature information through W2 and b2 to obtain the frame offset e.

[0117] According to one embodiment of the present invention, when making predictions using a classification method, the dimensions of the first mapping level and the second mapping level can be adjusted accordingly so that the obtained e is a vector with multiple dimensions. Then, the vector is activated by the softmax function to obtain the probability that the frame offset is a multiple value, and the value corresponding to the maximum probability is used as the frame offset.

[0118] In this way, temporal relationship features can be analyzed through a multilayer perceptron model, and frame offsets can be determined by classification or regression, thereby achieving the purpose of rectifying video temporal data.

[0119] Example 6:

[0120] Figure 2 An exemplary schematic diagram illustrates the application of the AI-based frame offset prediction method for primary and backup dual-channel audio and video streams according to an embodiment of the present invention.

[0121] According to one embodiment of the present invention, a first video frame sequence and a second video frame sequence can be processed by a first 3D convolutional neural network model and a second 3D convolutional neural network model with the same parameters to obtain a first spatiotemporal feature map and a second spatiotemporal feature map, respectively. The first spatiotemporal feature map and the second spatiotemporal feature map are then partitioned and flattened to obtain a first partition feature vector and a second partition feature vector. Further, a first spatiotemporal position code can be added to the first partition feature vector to obtain a first partition spatiotemporal feature vector, and a second spatiotemporal position code can be added to the second partition feature vector to obtain a second partition spatiotemporal feature vector.

[0122] According to one embodiment of the present invention, the first and second partition spatiotemporal feature vectors can be used as input tokens for a temporal relationship model. After layer normalization, they can be fused through a cross-attention mechanism, and after layer normalization and processing by a multilayer perceptron, a first output feature vector is obtained. This process can be iterated multiple times to obtain temporal relationship feature information, which is then used by the multilayer perceptron model to determine the frame offset via regression.

[0123] The AI-powered frame offset prediction method for primary and backup dual-channel audio and video streams according to embodiments of the present invention can reduce the impact of different encoding methods by performing the same preprocessing on both video streams. Furthermore, it can determine the frame offset by understanding the content and temporal relationship of video frames, further reducing the impact of differences in video frames caused by different video encoding methods and improving alignment accuracy. Moreover, by performing preprocessing on the first and second video streams in a unified manner, the impact of differences in video frames caused by different video encoding methods can be reduced, resulting in a first and second video frame sequence with consistent data format, frame rate, resolution, etc., providing an accurate data foundation for understanding the meaning of the video and aligning the video. Additionally, 3D convolution can be used to obtain spatiotemporal feature maps of continuous video frame sequences. The temporal relationship between consecutive video frame sequences can be fused in the spatiotemporal feature maps, facilitating the understanding of the content of the continuous video frame sequences and helping to improve the prediction accuracy of frame offsets. Furthermore, the spatiotemporal feature map can be partitioned to obtain partition feature vectors suitable for the Transformer model. These partition feature vectors can be encoded based on their temporal and spatial positions, facilitating the model's determination of each partition's location. This leads to a more accurate understanding of the video content and the relationship between the first and second video streams, thereby improving the accuracy of frame offsets. A cross-attention mechanism can also be used to cross-fertilize the feature information of the first and second video frame sequences to obtain the spatial and temporal dependencies between the first input feature vector and the second partition feature vector. This achieves spatiotemporal information fusion, enabling alignment and comparison of the two video streams and providing a sufficient and accurate data foundation for frame offset prediction. Moreover, a multilayer perceptron model can be used to analyze temporal relationship feature information, and frame offsets can be determined through classification or regression, thus achieving the goal of rectifying video temporal data.

[0124] Figure 3 An exemplary block diagram of a primary / backup dual-channel audio / video stream frame offset artificial intelligence prediction system according to an embodiment of the present invention is shown, the system comprising:

[0125] The preprocessing module is used to preprocess the first video stream and the second video stream to be aligned, to obtain the first video frame sequence of the first video stream and the second video frame sequence of the second video stream.

[0126] The feature extraction module is used to perform feature extraction processing on the first video frame sequence and the second video frame sequence respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence.

[0127] The partitioning module is used to obtain the first partition feature vector of multiple partitions of the first spatiotemporal feature map and the second partition feature vector of multiple partitions of the second spatiotemporal feature map.

[0128] The temporal relationship module is used to input the feature vectors of the first partition and the feature vectors of the second partition into the temporal relationship model for processing to obtain temporal relationship feature information;

[0129] The frame offset module is used to process the temporal relationship feature information through a multilayer perceptron model to obtain the frame offsets of the first video stream and the second video stream.

[0130] According to an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein when the computer program instructions are executed by a processor, the frame offset artificial intelligence prediction method for the primary and backup dual-channel audio and video streams is implemented.

[0131] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0132] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting frame offset of primary and backup audio and video streams, characterized in that, The method comprises the following steps: preprocessing first and second video streams to be aligned to obtain a first video frame sequence of the first video stream and a second video frame sequence of the second video stream; extracting features from the first and second video frame sequences respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence; obtaining first partition feature vectors of multiple partitions of the first spatiotemporal feature map and second partition feature vectors of multiple partitions of the second spatiotemporal feature map; inputting the first and second partition feature vectors into a time sequence relationship model for processing to obtain time sequence relationship feature information; processing the time sequence relationship feature information through a multilayer perception network model to obtain a frame offset of the first and second video streams; obtaining first partition feature vectors of multiple partitions of the first spatiotemporal feature map comprises the following steps: dividing the first spatiotemporal feature map along a time sequence direction, a width direction and a height direction to obtain multiple first partition feature maps; flattening the first partition feature maps into first partition feature vectors; determining a first spatiotemporal position code corresponding to the first partition feature map according to a sequence number of the first partition feature map in the time sequence direction and sequence numbers of the first partition feature map in the width direction and the height direction in the first spatiotemporal feature map; fusing the first spatiotemporal position code and the first partition feature vector to obtain a first partition spatiotemporal feature vector; the manner of partitioning the first spatiotemporal feature map and obtaining the first partition spatiotemporal feature vector is consistent with the manner of partitioning the second spatiotemporal feature map and obtaining the second partition spatiotemporal feature vector; the time sequence relationship model is a model based on a Transformer, and the first and second partition feature vectors are input into the time sequence relationship model for processing to obtain the time sequence relationship feature information, which comprises the following steps: at each level of the time sequence relationship model, a cross-attention mechanism is used to process a first input feature vector corresponding to the first partition feature vector and a second input feature vector corresponding to the second partition feature vector to obtain a first output feature vector corresponding to the first partition feature vector as a first input feature vector of a next level, wherein at a first level of the time sequence relationship model, the first input feature vector is the first partition feature vector; after processing through multiple levels of the time sequence relationship model, the first output feature vectors are fused to obtain the time sequence relationship feature information.

2. The method of claim 1, wherein, The method comprises the following steps: uniformly decoding, frame sampling and normalizing the first and second video streams to obtain the first and second video frame sequences respectively.

3. The method of claim 1, wherein, The method comprises the following steps: extracting features from the first and second video frame sequences respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence, which comprises the following steps: extracting features from the first video frame sequence through a first 3D convolutional neural network model to obtain the first spatiotemporal feature map; The second video frame sequence is subjected to feature extraction processing by a second 3D convolutional neural network model which is identical to the first 3D convolutional neural network model at least in part, to obtain a second spatiotemporal feature map.

4. The method of claim 1, wherein, At each level of the time sequence relationship model, the first input feature vector corresponding to the first partition feature vector and the second partition feature vector are processed using a cross-attention mechanism to obtain a first output feature vector corresponding to the first partition feature vector as the first input feature vector of the next level, including: The first input feature vector corresponding to the first partition feature vector is processed with a query weight matrix to obtain a query matrix; The second partition feature vector is processed with a key-value weight matrix to obtain a key-value matrix; The second partition feature vector is processed with a numerical weight matrix to obtain a numerical matrix; A fusion feature matrix is obtained according to the query matrix, the key-value matrix, and the numerical matrix; The fusion feature matrix is processed by a multi-layer perception network level to obtain the first output feature vector corresponding to the first partition feature vector.

5. The method of claim 1, wherein, The frame offset between the first video stream and the second video stream is obtained by processing the time sequence relationship feature information by a multi-layer perception network model, including: The time sequence relationship feature information is processed by a first mapping level of the multi-layer perception network model to obtain first mapping feature information; The first mapping feature information is processed by an activation level of the multi-layer perception network model to obtain first activation feature information; The first activation feature information is processed by a second mapping level of the multi-layer perception network model to obtain the frame offset.

6. A frame offset artificial intelligence prediction system for primary and backup dual paths of audio and video streams, characterized in that, Including: A preprocessing module is configured to pre-process the first video stream and the second video stream to be aligned to obtain a first video frame sequence of the first video stream and a second video frame sequence of the second video stream; A feature extraction module is configured to perform feature extraction processing on the first video frame sequence and the second video frame sequence respectively to obtain a first spatiotemporal feature map of the first video frame sequence and a second spatiotemporal feature map of the second video frame sequence; A partition module is configured to obtain first partition feature vectors of multiple partitions of the first spatiotemporal feature map and obtain second partition feature vectors of multiple partitions of the second spatiotemporal feature map; A time sequence relationship module is configured to input the first partition feature vectors and the second partition feature vectors into a time sequence relationship model for processing to obtain time sequence relationship feature information; A frame offset module is configured to process the time sequence relationship feature information by a multi-layer perception network model to obtain a frame offset between the first video stream and the second video stream. The first partition feature vectors of multiple partitions of the first spatiotemporal feature map are obtained, including: The first spatiotemporal feature map is divided along the time sequence direction, the width direction, and the height direction to obtain multiple first partition feature maps; The first partition feature map is flattened into a first partition feature vector; A first spatiotemporal position code corresponding to the first partition feature map is determined according to the sequence number of the first partition feature map in the time sequence direction and the sequence numbers in the width direction and the height direction in the first spatiotemporal feature map; The first spatiotemporal position code is fused with the first partition feature vector to obtain a first partition spatiotemporal feature vector; The manner of partitioning the first spatio-temporal feature map and obtaining the first partitioned spatio-temporal feature vector is consistent with the manner of partitioning the second spatio-temporal feature map and obtaining the second partitioned spatio-temporal feature vector; The time sequence relationship model is a model based on a Transformer, and the first partitioned feature vector and the second partitioned feature vector are input into the time sequence relationship model for processing to obtain time sequence relationship feature information, including: At each level of the time sequence relationship model, a cross-attention mechanism is used to process the first input feature vector corresponding to the first partitioned feature vector and the second partitioned feature vector to obtain a first output feature vector corresponding to the first partitioned feature vector as a first input feature vector of a next level, wherein at a first level of the time sequence relationship model, the first input feature vector is the first partitioned feature vector; After processing through multiple levels of the time sequence relationship model, the first output feature vector is fused to obtain the time sequence relationship feature information.

7. A computer-readable storage medium, characterized in that, A computer program product has computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the method of any one of claims 1-5.

Images