Video data processing method and apparatus, electronic device, medium, and product

Abstract

The present disclosure provides a video data processing method, relates to the technical field of artificial intelligence, and particularly relates to the technical fields of video processing, chip design, deep learning and the like. The method comprises the following steps: decoding a target video code stream to obtain a plurality of video frames, and generating a token sequence of an image block in the video frames; based on a self-attention mechanism in a visual processing model, calculating attention weights corresponding to the token sequence according to a weight bias matrix; wherein the weight bias matrix is pre-constructed according to a space-time correlation between coding blocks in the target video code stream; processing the token sequence based on the attention weights to obtain a feature sequence, and generating a video processing result based on the feature sequence. The present disclosure effectively maps the compression domain coding information to the self-attention mechanism, realizes efficient guidance of the attention weights under the premise of not changing the model structure, and thus improves the accuracy and robustness of the video processing result.

Description

Technical Field

[0001] This relates to the field of artificial intelligence technology, particularly to video processing, chip design, and deep learning. Specifically, it relates to a video data processing method, apparatus, electronic device, medium, and product. Background Technology

[0002] When using video processing models to process video data, related techniques typically acquire video frames and generate word sequences of image blocks. After adding positional encoding to the words in the word sequences, the data is input into the video processing model, and spatiotemporal dependencies are modeled through a self-attention mechanism.

[0003] In this process, the self-attention mechanism's characterization of spatiotemporal relationships is highly dependent on the guidance of positional encoding. However, positional encoding can only provide relative or absolute positional information between words and lacks explicit prior constraints on the underlying spatiotemporal structure of the video. This makes it difficult for the video processing model to accurately capture the dynamic changes between frames, thereby affecting the accuracy and robustness of the video processing results. Summary of the Invention

[0004] This disclosure provides a video data processing method, apparatus, electronic device, medium, and product.

[0005] According to one aspect of this disclosure, a video data processing method is provided, the method comprising: The target video stream is decoded to obtain multiple video frames, and a word sequence of image blocks in the video frames is generated. Based on the self-attention mechanism in the visual processing model, the attention weights corresponding to the word sequence are calculated according to the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coding blocks in the target video bitstream; The word sequence is processed based on the attention weights to obtain a feature sequence, and a video processing result is generated based on the feature sequence.

[0006] According to another aspect of this disclosure, a video data processing apparatus is provided, the apparatus comprising: The bitstream decoding module is used to decode the target video bitstream to obtain multiple video frames and generate a word sequence of image blocks in the video frames; The weight calculation module is used to calculate the attention weights corresponding to the word sequence based on the self-attention mechanism in the visual processing model and the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coding blocks in the target video bitstream; The result generation module is used to process the word sequence based on the attention weight to obtain a feature sequence, and generate a video processing result based on the feature sequence.

[0007] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform the video data processing method according to any embodiment of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the video data processing method described in any embodiment of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video data processing method described in any embodiment of this disclosure.

[0010] This disclosure effectively maps compressed domain coding information to a self-attention mechanism, achieving efficient guidance of attention weights without changing the model structure, thereby improving the accuracy and robustness of video processing results.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of a video data processing method provided according to an embodiment of the present disclosure; Figure 2 This is a flowchart of another video data processing method provided according to an embodiment of the present disclosure; Figure 3 This is a flowchart of yet another video data processing method provided according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of a video data processing apparatus according to an embodiment of the present disclosure; Figure 5 A block diagram of an electronic device used to implement a video data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Figure 1 This is a flowchart illustrating a video data processing method according to an embodiment of this disclosure. This embodiment is applicable to video understanding tasks based on visual processing models, such as video classification, object detection, semantic segmentation, or action recognition, and is particularly suitable for video processing scenarios requiring the capture of object motion trends, inter-frame continuity, and high dynamic range scenes. The method can be executed by a video data processing apparatus, which can be implemented in hardware and / or software and can be configured in an electronic device. (Reference) Figure 1 The method specifically includes the following: S101. Decode the target video stream to obtain multiple video frames, and generate a word sequence of image blocks in the video frames.

[0015] S102. Based on the self-attention mechanism in the visual processing model, calculate the attention weights corresponding to the word sequence according to the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed according to the spatiotemporal correlation between coding blocks in the target video bitstream.

[0016] S103. The word sequence is processed based on the attention weights to obtain a feature sequence, and a video processing result is generated based on the feature sequence.

[0017] The target video stream is a binary data sequence formed by video encoding based on a video coding standard. A coded block is the basic unit constituting the target video stream; each coded block corresponds to a pixel region in a video frame, has a unique coded block index, and carries the coded block type and corresponding reference relationship information. By parsing the compressed domain of the target video stream, each coded block can be extracted sequentially.

[0018] Typically, video coding standards utilize historical coding blocks to predict the current coding block during the encoding process to eliminate spatiotemporal redundancy. This process naturally characterizes the inherent predictive dependencies between coding blocks, constituting the spatiotemporal correlation between them. The specific choice of video coding standard is not limited here; it is determined based on actual business needs. For example, the video coding standard could be H.264 or H.265, etc.

[0019] A video frame is reconstructed after decoding the target video stream. An image patch, generally called a patch, is obtained by dividing a video frame according to a preset size. An image patch is the basic unit for word generation. A word, generally called a token, is a high-dimensional vector obtained by linearly mapping an image patch. All words corresponding to image patches are arranged in spatial order to form a word sequence.

[0020] The visual processing model is a neural network model built on the Transformer architecture, with self-attention mechanism at its core. The weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coded blocks in the target video bitstream. Spatiotemporal correlation describes the structured dependencies between different coded blocks in the spatial or temporal dimensions. This correlation is formed during the encoding process and reflects the inter-block prediction and reference relationships determined at the encoding stage.

[0021] In the self-attention mechanism, attention weights are used to measure the strength of the association between any two words in the word sequence. The word sequence is the input of the self-attention mechanism, and the feature sequence is the output of the self-attention mechanism. Each word in the feature sequence, while retaining its own local visual information, further incorporates spatiotemporal association information with other words in the sequence, and this spatiotemporal association information is guided by the spatiotemporal correlation between coding blocks carried by the weight bias matrix.

[0022] Video processing results refer to the prediction results of tasks such as classification, detection, or segmentation output after the feature sequence is input into the downstream task head.

[0023] The video frame sequence obtained by decoding the target video stream includes multiple video frames. Each frame is divided into image blocks and then linearly mapped into a word sequence. Each word in the sequence carries the local visual information of its corresponding image block, providing a basic input unit for subsequent self-attention processing.

[0024] By leveraging the self-attention mechanism in visual processing models, a weight bias matrix is ​​loaded when calculating the attention weights of a word sequence. Although the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coded blocks in the target video stream, since each word is generated from image blocks in a video frame through linear mapping, and image blocks and coded blocks share the same pixel coordinate space in the video frame, a positional mapping correspondence between words and coded blocks can be established. Based on this positional mapping, the spatiotemporal correlation between coded blocks can be mapped to the spatiotemporal correlation between words. Specifically, when the image blocks corresponding to two words respectively cover two coded blocks with spatiotemporal correlation, this correlation can be passed to the pair of words through the corresponding element values ​​in the weight bias matrix. Therefore, the weight bias matrix can be effectively used to calculate the attention weights of a word sequence. Its essence is to transform the structured prior at the coded block level in the compressed domain into word-level attention guidance through spatial mapping, enabling the self-attention mechanism to explicitly incorporate the inter-block prediction and reference relationships determined in the encoding stage when calculating the correlation strength between words, rather than relying solely on the pixel content similarity between words.

[0025] The self-attention mechanism first generates a query matrix, a key matrix, and a value matrix by linearly mapping the word sequence. The query and key matrices are used to calculate attention weights, while the value matrix carries the original features of the word sequence. After the attention weights corresponding to the word sequence are determined, the attention weight matrix is ​​multiplied by the value matrix. This involves weighting and summing the feature vectors of each word in the value matrix according to the weight coefficients of the corresponding rows in the attention weight matrix. This ensures that the features of each output word incorporate information from other words in the entire sequence, and this fusion process is guided by the spatiotemporal correlation between coding blocks carried by the weight bias matrix. Compared to the input word sequence, the resulting feature sequence, while retaining local visual information, further incorporates information from other words in the sequence. Furthermore, the fusion process, due to the intervention of the weight bias matrix, prioritizes and strengthens the information transfer between word pairs that already have strong correlations in the compressed domain. Finally, this feature sequence is input into the downstream task head to generate the video processing result.

[0026] This disclosed technical solution pre-constructs a weight bias matrix based on the spatiotemporal correlation between coded blocks in the target video bitstream, and directly uses this matrix to calculate the attention weights of the word sequence in the self-attention mechanism. This achieves efficient mapping of compressed domain coded information to the self-attention mechanism, enabling efficient guidance of attention weights without altering the visual processing model structure. Since the weight bias matrix is ​​pre-constructed during the compressed domain parsing stage and directly loaded during inference, the additional overhead of real-time motion information calculation is avoided, significantly improving the computational efficiency of video processing. Furthermore, this disclosed technical solution enables the self-attention mechanism to prioritize information transmission between spatiotemporally correlated word pairs when fusing global context, compensating for the inability of positional encoding alone to characterize inter-frame dynamic changes, thereby improving the accuracy and robustness of the video processing results.

[0027] Figure 2 This is a flowchart of another video data processing method provided according to an embodiment of the present disclosure; this embodiment is an optional solution proposed based on the above embodiments, specifically, it provides a scheme for constructing a weight bias matrix.

[0028] See Figure 2 The video data processing method provided in this embodiment includes: S201. Perform compression domain parsing on the target video stream to determine the coding block index and coding block type of the current coding block, as well as the video frame type and video frame index of the video frame where the current coding block is located.

[0029] The current coding block refers to the basic unit being processed, serving as a reference for determining its spatiotemporal correlation with other coding blocks. The current coding block has a unique coding block index, which is a unique identifier of the spatial location of each coding block within a video frame, typically assigned according to raster scan order. The current coding block also carries a coding block type, indicating the specific prediction mode employed. Optionally, the coding block type includes inter-frame coding block, intra-frame coding block, or skipped coding block. The video frame type refers to the prediction coding type of the frame containing the current coding block. Optionally, the video frame type includes I-frame, P-frame, or B-frame. The video frame index identifies the temporal order of the video frame within the video frame sequence.

[0030] Compression domain parsing of the target video stream involves directly reading the coding block index and coding block type of the current coding block, as well as the video frame type and video frame index of the video frame in which it is located, at the bitstream level. This process does not require complete decoding and reconstruction of pixels, thus obtaining the block-level information determined in the encoding stage with extremely low overhead.

[0031] S202. Based on the coding block index, the coding block type, the video frame type, and the video frame index, determine the spatiotemporal correlation between the current coding block and other coding blocks in the target video stream.

[0032] During video encoding, various structured dependencies exist between different coding blocks. These structured dependencies, after quantization, become the spatiotemporal correlations between coding blocks. Among them, structured dependencies include inter-frame reference dependencies explicitly recorded in the form of syntax elements during encoding, as well as temporal continuity associations implied in the physical continuity of video content.

[0033] Inter-frame reference dependency can be determined based on video frame type, coding block type, and coding block index. The video frame type can determine the list of reference frames allowed for the current coding block. The coding block type is used to determine whether the block carries reference relationship information such as motion vectors and reference frame indexes. The coding block index can accurately locate the target coding block it depends on in the reference video frame, thereby obtaining the spatiotemporal correlation between the current coding block and other coding blocks.

[0034] Temporal continuity can be determined using video frame indices and coded block indices. Video frame indices can calculate the time interval between different video frames, while coded block indices are used to align the same spatial location across different time frames. Although the reference relationships between these video frame pairs are not explicitly recorded in the bitstream, based on the temporal and physical continuity of the video content, a reasonable spatiotemporal correlation can be inferred between coded blocks at the same spatial location in different time frames, thus obtaining the spatiotemporal correlation between the current coded block and other coded blocks.

[0035] S203. Based on the spatiotemporal correlation and its corresponding coding block index, construct the weight bias matrix.

[0036] Organized into a matrix according to the correspondence between the coding block indices, the row and column indices of the matrix correspond to the coding block indices respectively, and the element values ​​in the matrix are the spatiotemporal correlations between the corresponding two coding blocks. The resulting weight bias matrix is ​​essentially a numerical expression of the spatiotemporal dependencies at the coding block level.

[0037] The coded block index uniquely identifies the position of each coded block in the video frame space grid. Using the coded block index as the row and column index of the matrix, the spatiotemporal correlation between any two coded blocks can be accurately located to the corresponding element position in the matrix.

[0038] Because there is a spatially mapped relationship between the tokens and encoding blocks processed by the subsequent self-attention mechanism, and the matrix form can be structurally aligned with the original association matrix corresponding to the token sequence in the self-attention mechanism, the pre-constructed weight bias matrix can be directly superimposed on the self-attention calculation, achieving efficient guidance of attention weights. The original association matrix is ​​obtained by calculating the dot product similarity between the query matrix and the key matrix.

[0039] The weight bias matrix can be pre-built and stored offline, and then directly loaded and used during self-attention computation, avoiding the additional overhead of real-time computation of motion information or optical flow, and significantly reducing inference costs.

[0040] This disclosed technical solution directly extracts the coded block index, coded block type, video frame type, and video frame index by performing compressed domain parsing on the target video bitstream, obtaining structured information at the coded block level without fully decoding pixels. Based on this, and using the spatiotemporal correlations between coded blocks determined by the aforementioned parameters, it can simultaneously capture explicit inter-frame reference dependencies and implicit temporal continuity associations, thus more comprehensively characterizing the inherent motion patterns of the video content. Furthermore, the spatiotemporal correlations are constructed into a weight bias matrix according to the coded block index, achieving an efficient conversion from compressed domain structured dependencies to a numerical matrix. Since the weight bias matrix explicitly introduces prior knowledge from the encoding process, the self-attention mechanism gains stronger guidance capabilities for dynamic scenes without changing the model structure, effectively compensating for the deficiency that positional encoding alone cannot fully express inter-frame motion trends, thereby improving the accuracy and robustness of video processing tasks in high-dynamic scenes.

[0041] In an optional embodiment, determining the spatiotemporal correlation between the current coding block and other coding blocks in the target video stream based on the coding block index, the coding block type, the video frame type, and the video frame index includes: determining a reference frame list for the current coding block based on the video frame type; determining a first video frame in the reference frame list and a first coding block in the first video frame based on the reference frame list, the coding block index, and reference relationship information corresponding to the coding block type; determining a second video frame in the target video stream based on the video frame index and a preset frame span, and determining a second coding block in the second video frame based on the coding block index; and determining the spatiotemporal correlation between the current coding block and the first coding block and the second coding block, respectively.

[0042] The video frame type determines which other video frames the current coding block can reference during inter-frame prediction. The reference frame list is a set of indices of other video frames that the current frame is allowed to reference, determined by the video frame type. For example, a P-frame corresponds to the forward reference frame list L0, and a B-frame corresponds to both the forward list L0 and the backward list L1. The reference relationship information corresponds to the coding block type and indicates the syntax elements its prediction depends on, including reference frame indices, motion vectors, and skip flags. The skip flag indicates whether the coding block uses the implicit prediction rules defined in the coding standard.

[0043] The first video frame refers to the video frame that the current coding block actually references, determined from the reference frame index in the reference relationship information within the reference frame list. The first coding block refers to the coding block located within the first video frame, based on the coding block index and motion vector of the current coding block, and which has an explicit predictive dependency on the current coding block.

[0044] First, based on the video frame type of the video frame containing the current coding block, a list of allowed reference frames is constructed according to the prediction reference rules specified in the video coding standard. Then, based on this list of reference frames, combined with the coding block index of the current coding block and the reference relationship information carried by the coding block type, the first video frame can be uniquely identified in the list of reference frames. Furthermore, within the first video frame, the first coding block that has an explicit prediction dependency with the current coding block is located using the coding block index and motion vector.

[0045] Meanwhile, in order to capture the temporal continuity that is not explicitly recorded during the encoding process but exists objectively, a second video frame separated from the current frame by a preset frame span is determined in the target video bitstream based on the video frame index and the preset frame span. The second coding block is then determined in the second video frame using the same coding block index, i.e. the same spatial location.

[0046] The preset frame span is a pre-defined time interval value, such as 1, 2, or 3 frames, used to limit the search for temporally continuous coded blocks within a range of frames that are adjacent or spaced a fixed number of times. The second video frame refers to another video frame, determined by the video frame index of the current coded block and the preset frame span, that is separated from the current frame by the preset frame span. This video frame is not limited to the video frames in the reference frame list. The second coded block refers to the coded block in the second video frame that has the same coded block index as the current coded block, i.e., the same spatial location.

[0047] Given a first coding block and a second coding block, the spatiotemporal correlation between the current coding block and the first coding block, as well as the spatiotemporal correlation between the current coding block and the second coding block, can be defined as preset correlations. The specific values ​​of these preset correlations can be determined based on actual business needs and are not limited here.

[0048] The aforementioned technical solution, based on video frame type, reference relationship information, and coded block index, can accurately extract the inter-frame predictive dependencies recorded during the encoding process, ensuring the accuracy and reliability of spatiotemporal correlation. Simultaneously, based on the video frame index and preset frame span, it actively supplements the temporal continuity associations of the same spatial location across different time frames without relying on inter-frame reference dependencies, compensating for potentially missed prior information. These two aspects complement each other, making the constructed weight bias matrix denser and conforming to the natural temporal structure of the video content. The entire determination process is completed during the compressed domain parsing stage, without needing to decode pixel content, acquiring rich spatiotemporal prior information with extremely low computational overhead, providing a high-quality data foundation for the efficient guidance of subsequent self-attention mechanisms.

[0049] In an optional embodiment, determining a first video frame in the reference frame list and a first coding block in the first video frame based on the reference frame list, the coding block index, and the reference relationship information corresponding to the coding block type includes: if the coding block type is an inter-frame coding block, determining whether the reference relationship information includes a skip identifier; if the skip identifier is included, determining the first coding block based on the motion information derivation rules of the skip mode in the coding standard adopted by the target video stream; if the skip identifier is not included, determining the first video frame in the reference frame list based on the reference frame index in the reference relationship information; and determining the first coding block for the current coding block in the first video frame based on the coding block index and the motion vector in the reference relationship information.

[0050] Because video coding standards have clear syntactic definitions and derivation rules for the reference relationship between inter-frame coding blocks, the existence of the skip flag directly indicates the way motion information is obtained, while the reference frame index and motion vector provide complete addressing information for explicit reference in non-skip mode.

[0051] When the coding block type is an inter-frame coding block, the reference relationship information is first checked to see if a skip flag is included. If a skip flag is included, it indicates that the current coding block adopts skip mode. In this mode, the bitstream does not explicitly provide motion vectors and reference frame indices. Instead, it implicitly determines the referenced coding block based on the motion information derivation rules predefined in the coding standard used by the target video bitstream. Based on this, the first coding block can be directly determined without parsing additional motion parameters. Inter-frame coding blocks are encoded using a cross-frame prediction method. The skip flag is a flag indicating whether the current inter-frame coding block adopts skip mode. Skip mode is a special inter-frame prediction mode in which the coding block does not explicitly encode motion vectors and residual data; its motion information is implicitly determined by the motion information derivation rules predefined in the coding standard.

[0052] If the reference relationship information does not contain a skip flag, the reference frame index in the reference relationship information must be used to locate the first video frame in the reference frame list determined according to the video frame type. Then, based on the coding block index and motion vector of the current coding block, the specific position pointed to by the motion vector in the first video frame is calculated, thereby accurately determining the first coding block. The motion vector refers to the spatial offset from the current coding block to the first coding block in the first video frame, used to describe the displacement of the current coding block relative to the first coding block in the horizontal and vertical directions.

[0053] The above technical solution determines the first coding block by distinguishing whether or not a skip flag is present. When a skip flag is present, it is implicitly determined according to the motion information derivation rules in the coding standard. When no skip flag is present, it is explicitly located based on the reference frame index and motion vector. This allows for the accurate extraction of inter-frame reference dependencies between coding blocks with minimal compression domain parsing overhead, providing an efficient and reliable structured prior for constructing the weight bias matrix, and thus effectively guiding the self-attention mechanism.

[0054] In an optional embodiment, determining the spatiotemporal correlation between the current coding block and the first coding block and the second coding block respectively includes: if the reference relationship information includes a skip identifier, then determining the spatiotemporal correlation corresponding to the first coding block as a first correlation; if the reference relationship information does not include a skip identifier, then determining the spatiotemporal correlation corresponding to the first coding block as a second correlation; determining the spatiotemporal correlation between the current coding block and the second coding block as a third correlation; wherein the third correlation is the smallest and greater than zero, and the second correlation is between the first correlation and the third correlation.

[0055] The first, second, and third correlations are used to quantify the strength of spatiotemporal dependencies from different sources. The first correlation corresponds to the case where the reference relationship information contains a skip flag. Since the skip mode is the optimal prediction method selected by the encoder after motion estimation, it has the highest reliability, hence the first correlation value is the largest. The second correlation corresponds to the case where the reference relationship information does not contain a skip flag. Although inter-frame reference dependencies are also used in the encoding process, its reliability is slightly lower than that of the skip mode, hence the second correlation value is between the first and third correlations. The third correlation corresponds to the temporally continuous association between the current coding block and the second coding block. Its corresponding spatiotemporal dependency is not the inter-frame reference dependency actually used in the encoding process, and its reliability is the lowest, hence the third correlation value is the smallest and greater than zero.

[0056] If the reference relationship information includes a skip flag, it indicates that there is an inter-frame reference dependency between the first coding block and the current coding block that was actually used in the coding process and optimized by motion estimation. Its reliability is the highest, so it is assigned the first correlation. If the reference relationship information does not include the skip flag, then although there is still an inter-frame reference dependency actually used in the encoding process between the first coding block and the current coding block, this dependency may contain residual data or non-optimal prediction mode, and its reliability is slightly lower than that of the skip mode. Therefore, a second correlation is assigned between the first correlation and the third correlation. The second coding block is associated with the current coding block by a temporal continuity inferred from the video frame index and the preset frame span, rather than by the inter-frame reference dependency actually used in the encoding process. Its reliability depends entirely on the assumption of physical continuity of the video content, so it has the lowest reliability and is assigned a minimum and greater than zero third correlation.

[0057] Among them, the first correlation, the second correlation, and the third correlation are all positive real numbers. They can be set in advance as fixed hyperparameters based on coding experience, or they can be adaptively optimized as learnable parameters during model training to flexibly adapt to different video content and task requirements.

[0058] The aforementioned technical solution differentiates between the presence or absence of skip markers and temporal continuity by setting first, second, and third correlations. This allows the weight bias matrix to accurately reflect the reliability of dependencies between coding blocks. Inter-frame reference dependencies corresponding to skip markers have the highest reliability, followed by those in non-skipped modes, while temporal continuity inferred from spatial alignment and time intervals has the lowest reliability. Therefore, the self-attention mechanism adaptively allocates attention weights based on the reliability of dependency sources, avoiding indiscriminate treatment of all spatiotemporal correlations and thus improving the accuracy and robustness of video processing results in high-dynamic scenes or complex motion trajectories.

[0059] Figure 3 This is a flowchart of another video data processing method provided according to an embodiment of the present disclosure; this embodiment is an optional solution proposed based on the above embodiments.

[0060] See Figure 3 The video data processing method provided in this embodiment includes: S301. Decode the target video stream to obtain multiple video frames, and generate a word sequence of image blocks in the video frames.

[0061] S302. Determine the pixel coordinate range of the image block and the coded block in the video frame.

[0062] The pixel coordinate range refers to the pixel area covered by the image block or coded block in the video frame, which is generally determined by the coordinates of the upper left and lower right corners.

[0063] S303. Based on the pixel coordinate range, establish the positional mapping relationship between the image block and the coding block.

[0064] The positional mapping relationship is calculated based on the fact that image blocks and coded blocks share the same pixel coordinate space in a video frame, and is obtained through spatial positional overlap calculation. Specifically, spatial positional overlap calculation refers to determining the spatial correspondence between the two by judging whether there is an intersection between the pixel coordinate ranges of the image block and the pixel coordinate range of the coded block.

[0065] Based on the positional mapping relationship, one or more coding blocks covered by each image block can be determined. Since the image block is the basic unit of word generation, there is a one-to-one correspondence between words and image blocks, and thus the spatial correspondence between words and coding blocks can be determined.

[0066] S304. Based on the self-attention mechanism in the visual processing model, the attention weights corresponding to the word sequence are calculated according to the weight bias matrix and the position mapping relationship.

[0067] The weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coded blocks in the target video stream.

[0068] The weight bias matrix is ​​block-dimensional. When calculating attention weights based on the self-attention mechanism, it is necessary to use position mapping to spatially align the tokens with the code blocks, so that the weight bias matrix is ​​transformed from the code block dimension to the token dimension.

[0069] Position mapping serves as a bridge connecting compressed domain coding blocks and pixel domain lexical units. It can transform the weight bias matrix from the coding block dimension to the lexical dimension, enabling the self-attention mechanism to directly guide attention weights to lexical pairs with strong spatiotemporal correlation based on the weight bias matrix without changing the model structure. This accurately transfers the spatiotemporal correlation prior in the compressed domain to the self-attention calculation.

[0070] S305. The word sequence is processed based on the attention weights to obtain a feature sequence, and a video processing result is generated based on the feature sequence.

[0071] This disclosed technical solution achieves spatial alignment between compressed domain coding blocks and pixel domain words by determining the pixel coordinate range of image blocks and coding blocks in video frames and establishing positional mapping relationships. This enables the weight bias matrix to be accurately converted from the coding block dimension to the word dimension. The self-attention mechanism can directly utilize the spatiotemporal correlation between coding blocks to guide the calculation of attention weights without changing the model structure. This efficiently transfers the existing structured dependencies in the compressed domain to the global interaction process of word sequences, effectively improving the self-attention mechanism's ability to capture spatiotemporal correlations in high-dynamic scenes, thereby improving the accuracy and robustness of video processing results.

[0072] In an optional embodiment, establishing the positional mapping relationship between the image block and the coding block based on the pixel coordinate range includes: traversing each image block, calculating a first overlap area between the current image block and each coding block based on its pixel coordinate range for the current image block; and establishing the positional mapping relationship between the image block and each coding block based on the first overlap area and the coding grouping parameters corresponding to the target video bitstream.

[0073] For the current image frame, traverse all the image blocks obtained within the video frame. For multiple video frames in the entire video sequence, repeat the above traversal process frame by frame to establish the positional mapping relationship between the image blocks in each video frame and the coded blocks within that video frame.

[0074] Here, the current image block refers to the target image block being traversed and processed, and is the object of the operation for establishing positional mapping relationships. For the current image block, a first overlap area is calculated based on its pixel coordinate range and the pixel coordinate range of each coded block. The first overlap area refers to the intersection area of ​​the current image block and a single coded block in pixel coordinate space, calculated by comparing their pixel coordinate ranges, and is used to quantify the degree of correspondence between the image block and the coded block in pixel coordinate space. The first overlap area provides an objective quantitative basis for determining the coded blocks covered by the image block.

[0075] The first overlap area and the coding block parameters corresponding to the target video bitstream are used together to establish the positional mapping relationship between the image block and each coding block. The coding block parameters are information parsed from the target video bitstream used to divide the coding block into several independent regions, and are used to constrain the legality of the positional mapping relationship.

[0076] The coding group parameter is introduced because referencing between coding blocks in different groups is restricted or prohibited in video coding standards. If the coding blocks covered by an image patch belong to different groups, they cannot be simply mapped to a single coding block; otherwise, an illegal correspondence that violates coding constraints would be introduced. Therefore, when establishing positional mapping relationships, it is necessary to simultaneously constrain the correspondence based on the first overlap area and the coding group parameter to ensure that the established positional mapping accurately reflects spatial positional relationships and conforms to the constraints of cross-group references in the coding standard. This provides a reliable foundation for subsequently converting the weight bias matrix from the coding block dimension to the word dimension.

[0077] The above technical solution quantifies the spatial coverage of image blocks and coding blocks by calculating the first overlap area, and establishes a positional mapping relationship between image blocks and coding blocks based on coding grouping parameters. This mapping objectively reflects the actual coverage of coding blocks by image blocks, while adhering to the constraint of limited reference between coding blocks in different groups in the coding standard. This provides an accurate spatial alignment basis for the subsequent conversion of the weight bias matrix from the coding block dimension to the word dimension, ensuring that the spatiotemporal correlation prior in the compression domain can be reliably transferred to the self-attention calculation, thereby improving the self-attention mechanism's ability to capture spatiotemporal correlations.

[0078] In an optional embodiment, establishing the positional mapping relationship between the image block and each coding block based on the first overlap area and the coding grouping parameters corresponding to the target video stream includes: determining candidate coding blocks that overlap with the current image block based on the first overlap area, and the total number of candidate coding blocks; if the total number of coding blocks is greater than 1 and the candidate coding blocks belong to the same coding block group, then the candidate coding block with the largest first overlap area is taken as the target coding block corresponding to the position of the current image block; if the total number of coding blocks is greater than 1 and the candidate coding blocks belong to different coding block groups, then the current image block is subdivided into sub-regions based on the coding grouping parameters and the pixel coordinate range of the sub-regions is determined; different sub-regions do not overlap and the corresponding coding block groups are different; based on the pixel coordinate range of the sub-regions, the target coding block corresponding to the position of the current image block is determined and the area ratio of each target coding block in the current image block is recorded; based on the image block index of the current image block and the coding block index of the target coding block, the positional mapping relationship between the image block and each coding block is established.

[0079] Among them, candidate coding blocks and the current image block share a common spatial coverage area, which can be determined by analyzing whether the pixel coordinate ranges of the current image block and each coding block intersect in the horizontal and vertical directions. Candidate coding blocks constitute a selection set for establishing positional mapping relationships.

[0080] The total number of candidate coding blocks is used to determine the spatial coverage complexity between the current image block and the coding blocks.

[0081] If the total number of coding blocks is 1, it indicates that the current image block only has spatial overlap with a single coding block. In this case, the candidate coding block is directly determined as the target coding block, forming a one-to-one mapping.

[0082] If the total number of coding blocks is greater than 1, it is necessary to further determine whether the candidate coding blocks belong to the same coding block group based on the coding grouping parameters. If they belong to the same coding block group, it means that all overlapping coding blocks are located in the same independent region where mutual reference is allowed. In this case, the candidate coding block with the largest first overlapping area is selected as the target coding block to maximize the preservation of the dominant relationship of spatial coverage and maintain a one-to-one mapping. If they belong to different coding block groups, it means that the current image block covers coding blocks belonging to different reference constraint regions. In this case, the entire image block cannot be simply mapped to a single coding block. Therefore, the grouping boundary is determined based on the coding grouping parameters, and the current image block is divided into several sub-regions. Each sub-region only overlaps with coding blocks within the same coding block group. Then, the corresponding target coding block is determined for each sub-region based on the pixel coordinate range of the sub-region, and the area ratio of each target coding block in the current image block is recorded, thus forming a one-to-many mapping.

[0083] Finally, based on the image block index of the current image block and the coding block index of the determined target coding block, a positional mapping relationship between the image block and the coding block is established.

[0084] The above technical solution quantifies the spatial coverage relationship between image blocks and coding blocks through the first overlap area and distinguishes whether candidate coding blocks belong to the same coding block group based on coding grouping parameters. This allows the positional mapping relationship to flexibly adopt a one-to-one or one-to-many mapping form with area proportions according to the degree of spatial overlap and coding constraints. For cross-group image blocks, segmentation processing ensures that each sub-region corresponds only to the coding blocks in the same group, strictly adhering to the constraint of cross-group reference restriction in the coding standard while achieving spatial alignment. The positional mapping relationship established in this way provides an accurate and reliable spatial alignment basis for the subsequent conversion of the weight bias matrix from the coding block dimension to the word dimension, ensuring that the spatiotemporal correlation prior in the compression domain can be efficiently transferred to the self-attention calculation, thereby improving the accuracy and robustness of the video processing results.

[0085] In an optional embodiment, determining the target coding block corresponding to the current image block position based on the pixel coordinate range of the sub-region and recording the area ratio of each target coding block in the current image block includes: traversing each sub-region; for the current sub-region, calculating the second overlap area between the current sub-region and the candidate coding block based on its pixel coordinate range; taking the candidate coding block with the largest second overlap area as the target coding block corresponding to the current sub-region position, and determining the area ratio of the target coding block in the current image block based on the second overlap area; determining the target coding block corresponding to the current image block position based on the target coding block corresponding to the current sub-region position, and recording the area ratio of each target coding block in the current image block.

[0086] In this context, a sub-region refers to a non-overlapping spatial unit obtained by segmenting the image block along the group boundary when the candidate coding blocks covered by the image block belong to different coding block groups. Each sub-region only overlaps with coding blocks within the same coding block group. The current sub-region refers to the sub-region being traversed and processed, and is the object of operation for determining the target coding block. The second overlap area refers to the spatial intersection area between the current sub-region and the candidate coding block, calculated by comparing the pixel coordinate ranges of the two, and is used to quantify the degree of coverage of the coding block by the sub-region.

[0087] Since sub-regions are obtained by segmenting image blocks along group boundaries when they cross groups, each sub-region only spatially overlaps with coded blocks within the same coded block group. Based on the pixel coordinate range of each sub-region, a second overlap area is calculated between it and candidate coded blocks. The candidate coded block with the largest second overlap area is selected as the target coded block corresponding to that sub-region. The target coded block refers to the coded block that is ultimately determined to correspond to the current sub-region. Typically, the candidate coded block with the largest second overlap area is selected to ensure the closest spatial correspondence between the sub-region and the coded block, avoiding correspondence ambiguity caused by covering multiple coded blocks.

[0088] The area ratio of the target coding block in the current image block is determined based on the second overlap area, which quantifies the coverage weight of the coding block in the entire image block and provides an objective basis for weight allocation in subsequent one-to-many mappings.

[0089] Through the above operations, each sub-region can obtain a unique corresponding target coding block and its area ratio, and then the multiple target coding blocks corresponding to the current image block and their respective area ratios are obtained, forming a one-to-many positional mapping relationship between the image block and the coding block.

[0090] The above technical solution ensures that each sub-region obtains a unique and spatially closest coding block by calculating the second overlap area for each sub-region and selecting the candidate coding block with the largest area as the target coding block. Simultaneously, it determines the area proportion of each target coding block in the current image block based on the second overlap area, quantifying the coverage weight of the coding block in the entire image block. This enables the positional mapping relationship between image blocks and coding blocks to be accurately established in a weighted many-to-one form in cross-group scenarios. It preserves the quantization information of spatial coverage while strictly adhering to the cross-group reference constraints defined by the coding grouping parameters. This provides a precise weight allocation basis for subsequently converting the weight bias matrix from the coding block dimension to the word dimension, thereby improving the accuracy and effectiveness of transferring the compression domain prior to the self-attention mechanism.

[0091] In an optional embodiment, the self-attention mechanism based on the visual processing model calculates the attention weights corresponding to the lexical sequence according to the weight bias matrix and the position mapping relationship, including: converting the weight bias matrix into a new weight bias matrix based on the list of coding blocks corresponding to the lexical indexes in the position mapping relationship and the area ratio of each coding block in the corresponding image block; using the new weight bias matrix to correct the original association matrix corresponding to the lexical sequence to obtain an enhanced association matrix, and determining the attention weights corresponding to the lexical sequence based on the enhanced association matrix; wherein, the original association matrix is ​​obtained by calculating the dot product similarity between the query matrix and the key matrix; the query matrix, the key matrix, and the value matrix are obtained by linearly mapping the lexical sequence.

[0092] The lexical index uniquely identifies the position of each lexical in the lexical sequence and corresponds one-to-one with the image patch index. The coded block list refers to the set of indices of one or more coded blocks that spatially overlap with the corresponding image patch, determined according to the positional mapping relationship for each lexical. The area percentage refers to the proportion of the spatial coverage area of ​​each coded block in its corresponding image patch to the total area of ​​that image patch, used to quantify the weight of that coded block within the image patch. The row and column indices of the weight bias matrix correspond to the coded block indices, where the matrix elements represent the spatiotemporal correlation between coded block pairs. The new weight bias matrix is ​​obtained by converting the weight bias matrix from the coded block dimension to the lexical dimension through the positional mapping relationship, where the matrix elements represent the spatiotemporal correlation between lexical pairs.

[0093] First, based on the positional mapping relationship, the list of coding blocks corresponding to each word and the area ratio of each coding block in the corresponding image block are obtained. Using the area ratio as a weight coefficient, the corresponding element values ​​in the weight bias matrix of the coding block dimension are weighted and summed to calculate a new bias value for each pair of words, thus forming a new weight bias matrix in the word dimension.

[0094] Subsequently, the new weight bias matrix is ​​superimposed on the original association matrix to obtain the enhanced association matrix. This is done to combine the pixel content similarity between tokens with the prior spatiotemporal correlation between coding blocks to jointly determine the association strength between token pairs. The original association matrix is ​​the matrix calculated using the dot product similarity between the query matrix and the key matrix, reflecting the pixel content-based correlation between token pairs. The enhanced association matrix is ​​the matrix obtained by superimposing the new weight bias matrix on the original association matrix, so that the association strength between tokens simultaneously reflects pixel content similarity and the prior spatiotemporal correlation in the compressed domain.

[0095] Next, the enhanced association matrix is ​​normalized to obtain the attention weights corresponding to the word sequence. Optionally, after obtaining the attention weights corresponding to the word sequence, the attention weights are used to perform weighted aggregation on the logarithmic matrix to obtain the feature sequence. This ensures that when the output features of each word are fused with other word information in the sequence, they preferentially absorb word information that has strong spatiotemporal correlation in the compressed domain. Optional, adopt Determine the feature sequence. Among them, This refers to attention weight; Q , K and V These are the query matrix, key matrix, and value matrix, respectively. B For the new weight bias matrix; d k is the dimension of the key vector, used to scale the dot product result to stabilize the gradient; the softmax function normalizes along the row direction of the matrix so that the sum of the attention weights of each word to all words is 1.

[0096] The above technical solution obtains the list of coding blocks corresponding to each word and the area ratio of each coding block in the corresponding image block through position mapping relationship. It transforms the weight bias matrix of coding block dimension into a new weight bias matrix of word dimension, realizing the alignment of the spatiotemporal correlation prior in the compressed domain to the self-attention input space. The new weight bias matrix is ​​superimposed with the original correlation matrix obtained by the dot product of the query matrix and the key matrix, so that the enhanced correlation matrix reflects both the pixel content similarity between words and the structured dependency between coding blocks. The attention weight is determined based on the enhanced correlation matrix, so that the self-attention mechanism can efficiently integrate the existing spatiotemporal correlation prior in the compressed domain into the attention calculation without changing the model structure, thereby improving the accuracy and robustness of the correlation strength judgment between words, and ultimately improving the video processing results.

[0097] In an optional embodiment, converting the weight bias matrix into a new weight bias matrix based on the list of coding blocks corresponding to the lexical indexes in the positional mapping relationship and the area ratio of each coding block in the corresponding image block includes: for any two lexical pairs in the lexical sequence, determining the list of coding blocks corresponding to each lexical pair and the area ratio of each coding block; if both lexical pairs correspond to only one coding block, then directly using the element values ​​corresponding to these two coding blocks in the weight bias matrix as the spatiotemporal correlation between the lexical pairs; if at least one lexical pair corresponds to multiple coding blocks, then taking one coding block from each of the coding block lists corresponding to the two lexical pairs to form a coding block pair; weighting the element values ​​corresponding to the coding block pair based on the area ratio of the coding blocks in the coding block pair, and determining the spatiotemporal correlation between the lexical pairs based on the obtained weighting result; and determining the new weight bias matrix based on the spatiotemporal correlation between the lexical pairs.

[0098] Among them, a word pair refers to a combination of two words arbitrarily selected from a word sequence, which is used to measure the spatiotemporal correlation between the two.

[0099] First, for each word pair in the word sequence, determine the list of coding blocks corresponding to each word and the area ratio of each coding block. If two words each correspond to only one coding block, it indicates that the spatial correspondence of the word pair is clear and there is no ambiguity in coverage. In this case, the element values ​​corresponding to these two coding blocks in the weight bias matrix are directly used as the spatiotemporal correlation between the word pairs, realizing the one-to-one accurate transmission of the dependency relationship between coding blocks to word pairs.

[0100] If at least one term corresponds to multiple coding blocks, it indicates that the image block corresponding to that term covers multiple coding blocks, and the spatiotemporal correlation between coding blocks needs to be fused according to the coverage weight. In this case, one coding block is taken from each of the coding block lists corresponding to the two terms to form a coding block pair. The area ratios of the two coding blocks in their respective corresponding image blocks are multiplied, and then multiplied by the corresponding element value in the weight bias matrix to obtain the contribution value of the coding block pair to the spatiotemporal correlation of the term pair. The sum of the contribution values ​​of all coding block pairs yields the spatiotemporal correlation between the term pairs. This is done because the area ratio quantifies the weight of each coding block in its corresponding term. By using a weighted summation, the correlation between coding blocks is mapped to the term pairs according to the actual spatial coverage ratio, preserving the intensity differences in the original bias matrix while accurately reflecting the true spatial correspondence when an image block covers multiple coding blocks.

[0101] Based on the spatiotemporal correlation between all token pairs, a new weight bias matrix is ​​formed by organizing them according to the token index, thus completing the transformation from the coding block dimension to the token dimension.

[0102] The above technical solution achieves precise transfer of spatiotemporal correlation between coding blocks to the lexical dimension by distinguishing the number of coding blocks corresponding to lexical units in a lexical pair: when both lexical units in a lexical pair correspond to only one coding block, the corresponding element value in the weight bias matrix is ​​directly taken as the spatiotemporal correlation between the lexical pair; when at least one lexical unit corresponds to multiple coding blocks, the corresponding element values ​​of the coding block pair are weighted and summed based on the area ratio to obtain the spatiotemporal correlation between the lexical pair. The newly determined weight bias matrix is ​​directly aligned with the lexical sequence, providing data support for incorporating compression domain priors into self-attention computation.

[0103] In an optional embodiment, determining the pixel coordinate range of the image block and the coded block in the video frame includes: parsing the sequence parameter set or image parameter set in the target video bitstream to obtain the video frame size and the coded block size; determining the pixel coordinate range of the coded block based on the video frame size, the coded block size, and the coded block index; and determining the pixel coordinate range of the image block based on the image block size, the video frame size, the image block partitioning method, and the image block index.

[0104] The sequence parameter set and image parameter set are key syntactic structures in the target video bitstream. The sequence parameter set contains global parameters applicable to the entire video sequence, while the image parameter set contains local parameters applicable to a single frame or a group of frames. Together, they determine information such as the video frame size and coding block size. The video frame size refers to the number of pixels in the width and height directions of the reconstructed video frame after decoding, used to define the overall boundary of the pixel coordinate range. The coding block size refers to the pixel size of the coding block in the width and height directions, and is the basic unit for motion estimation and inter-frame prediction during video encoding. Both the video frame size and the coding block size are determined according to actual business requirements and are not limited here.

[0105] To determine the pixel coordinate range of a coded block, the sequence parameter set or image parameter set in the target video stream is first parsed to obtain the video frame size and the coded block size. Based on the video frame size and the coded block size, the number of coded blocks in the horizontal direction is calculated as the video frame width divided by the coded block width, and the number of coded blocks in the vertical direction is calculated as the video frame height divided by the coded block height. Since the coded block index is assigned according to the raster scan order, the row and column numbers of the coded block are obtained by performing integer division and modulo operations on the coded block index and the number of coded blocks in the horizontal direction. Combined with the coded block size, the pixel coordinate range of the coded block can then be calculated.

[0106] When determining the pixel coordinate range of an image block, the layout rules of the image blocks within the frame are first determined based on the image block division method. Based on this layout rule, the image block size, and the video frame size, the number of image blocks in the horizontal direction is calculated. Then, by performing integer and modulo operations on the image block index and the number of image blocks in the horizontal direction, the row and column numbers of the image blocks are obtained. Finally, the pixel coordinate range of the image block is calculated in conjunction with the image block size. The image block size refers to the pixel dimensions of the image block in the width and height directions, and is the basic unit for dividing a video frame. The image block division method refers to the specific strategy for dividing a video frame into image blocks, which determines the position and size of each image block. The image block index is a unique identifier for the spatial position of each image block in the video frame, and is assigned according to the division order.

[0107] The pixel coordinate range of a coded block is the pixel area covered by the coded block in a video frame. The pixel coordinate range of an image block is the pixel area covered by the image block in a video frame, and together with the pixel coordinate range of the coded block, they are used for subsequent spatial alignment determination.

[0108] The above technical solution obtains the video frame size and coding block size by parsing the sequence parameter set or image parameter set in the target video bitstream. Based on the video frame size, coding block size, and coding block index, it determines the pixel coordinate range of the coding block. Simultaneously, based on the image block size, video frame size, image block partitioning method, and image block index, it determines the pixel coordinate range of the image block. This provides a unified coordinate basis for subsequently establishing the positional mapping relationship between image blocks and coding blocks, ensuring that coding blocks and image blocks can achieve spatial alignment within the same coordinate system.

[0109] Figure 4 This is a schematic diagram of a video data processing apparatus according to an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to video understanding tasks based on visual processing models, such as video classification, object detection, semantic segmentation, or action recognition, and are particularly suitable for video processing scenarios that require capturing object motion trends, inter-frame continuity, and high dynamic range scenes. The apparatus can be implemented using software and / or hardware, and can implement the video data processing methods described in any embodiment of the present disclosure.

[0110] like Figure 4 As shown, the video data processing device 400 includes: The stream decoding module 401 is used to decode the target video stream to obtain multiple video frames and generate a word sequence of image blocks in the video frames; The weight calculation module 402 is used to calculate the attention weights corresponding to the word sequence based on the self-attention mechanism in the visual processing model and the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coding blocks in the target video bitstream; The result generation module 403 is used to process the word sequence based on the attention weight to obtain a feature sequence, and generate a video processing result based on the feature sequence.

[0111] This disclosed technical solution pre-constructs a weight bias matrix based on the spatiotemporal correlation between coded blocks in the target video bitstream, and directly uses this matrix to calculate the attention weights of the word sequence in the self-attention mechanism. This achieves efficient mapping of compressed domain coded information to the self-attention mechanism, enabling efficient guidance of attention weights without altering the visual processing model structure. Since the weight bias matrix is ​​pre-constructed during the compressed domain parsing stage and directly loaded during inference, the additional overhead of real-time motion information calculation is avoided, significantly improving the computational efficiency of video processing. Furthermore, this disclosed technical solution enables the self-attention mechanism to prioritize information transmission between spatiotemporally correlated word pairs when fusing global context, compensating for the inability of positional encoding alone to characterize inter-frame dynamic changes, thereby improving the accuracy and robustness of the video processing results.

[0112] Optionally, the device further includes a matrix construction module for constructing the weight bias matrix; the matrix construction module includes: a bitstream parsing submodule for performing compression domain parsing on the target video bitstream to determine the coding block index and coding block type of the current coding block, as well as the video frame type and video frame index of the video frame where the current coding block is located; a correlation determination submodule for determining the spatiotemporal correlation between the current coding block and other coding blocks in the target video bitstream based on the coding block index, the coding block type, the video frame type, and the video frame index; and the matrix construction submodule for constructing the weight bias matrix based on the spatiotemporal correlation and its corresponding coding block index.

[0113] Optionally, the correlation determination submodule includes: a list determination unit, configured to determine a reference frame list for the current coding block based on the video frame type; a first coding block determination unit, configured to determine a first video frame in the reference frame list and a first coding block in the first video frame based on the reference frame list, the coding block index, and reference relationship information corresponding to the coding block type; a second coding block determination unit, configured to determine a second video frame in the target video stream based on the video frame index and a preset frame span, and a second coding block in the second video frame based on the coding block index; and a correlation determination unit, configured to determine the spatiotemporal correlation between the current coding block and the first coding block and the second coding block, respectively.

[0114] Optionally, the first coding block determining unit is specifically configured to: if the coding block type is an inter-frame coding block, determine whether the reference relationship information includes a skip identifier; if the skip identifier is included, determine the first coding block based on the motion information derivation rules of the skip mode in the coding standard adopted by the target video bitstream; if the skip identifier is not included, determine the first video frame in the reference frame list based on the reference frame index in the reference relationship information; and determine the first coding block for the current coding block in the first video frame based on the coding block index and the motion vector in the reference relationship information.

[0115] Optionally, the correlation determination unit is specifically used to: if the reference relationship information includes a skip identifier, determine the spatiotemporal correlation corresponding to the first coding block as a first correlation; if the reference relationship information does not include a skip identifier, determine the spatiotemporal correlation corresponding to the first coding block as a second correlation; determine the spatiotemporal correlation between the current coding block and the second coding block as a third correlation; wherein the third correlation is the smallest and greater than zero, and the second correlation is between the first correlation and the third correlation.

[0116] Optionally, the weight calculation module 402 includes: a range determination submodule, used to determine the pixel coordinate range of the image block and the coding block in the video frame; a relationship determination submodule, used to establish a positional mapping relationship between the image block and the coding block based on the pixel coordinate range; and a weight calculation submodule, used to calculate the attention weight corresponding to the lexical sequence based on the weight bias matrix and the positional mapping relationship using the self-attention mechanism in the visual processing model.

[0117] Optionally, the relationship determination submodule includes: an area calculation unit, used to traverse each image block and, for the current image block, calculate the first overlap area between the current image block and each coding block based on its pixel coordinate range; and a relationship establishment unit, used to establish the positional mapping relationship between the image block and each coding block based on the first overlap area and the coding grouping parameters corresponding to the target video bitstream.

[0118] Optionally, the relationship establishment unit includes: a block total number determination subunit, used to determine candidate coding blocks that overlap with the current image block based on the first overlap area, and the total number of candidate coding blocks; a coding block determination subunit, used to select the candidate coding block with the largest first overlap area as the target coding block corresponding to the position of the current image block if the total number of coding blocks is greater than 1 and the candidate coding blocks belong to the same coding block group; an image block segmentation subunit, used to segment the current image block into sub-regions and determine the pixel coordinate range of the sub-regions based on the coding grouping parameters if the total number of coding blocks is greater than 1 and the candidate coding blocks belong to different coding block groups; different sub-regions do not overlap and the corresponding coding block groups are different; an area proportion determination subunit, used to determine the target coding block corresponding to the position of the current image block based on the pixel coordinate range of the sub-regions and record the area proportion of each target coding block in the current image block; and a relationship establishment subunit, used to establish a positional mapping relationship between the image block and each coding block based on the image block index of the current image block and the coding block index of the target coding block.

[0119] Optionally, the area proportion determination subunit is specifically used for: traversing each sub-region, calculating the second overlap area between the current sub-region and the candidate coding block based on its pixel coordinate range for the current sub-region; taking the candidate coding block with the largest second overlap area as the target coding block corresponding to the position of the current sub-region, and determining the area proportion of the target coding block in the current image block based on the second overlap area; determining the target coding block corresponding to the position of the current image block based on the target coding block corresponding to the position of the current sub-region, and recording the area proportion of each target coding block in the current image block.

[0120] Optionally, the weight calculation submodule includes: a matrix transformation unit, used to convert the weight bias matrix into a new weight bias matrix based on the list of coding blocks corresponding to the lexical indexes in the positional mapping relationship and the area ratio of each coding block in the corresponding image block; and a weight determination unit, used to modify the original association matrix corresponding to the lexical sequence using the new weight bias matrix to obtain an enhanced association matrix, and to determine the attention weight corresponding to the lexical sequence based on the enhanced association matrix; wherein the original association matrix is ​​obtained by calculating the dot product similarity between the query matrix and the key matrix; and the query matrix, the key matrix, and the value matrix are obtained by linearly mapping the lexical sequence.

[0121] Optionally, the matrix transformation unit is specifically used for: for any two words in the word sequence forming a word pair, determining the list of coding blocks corresponding to each word in the word pair and the area ratio of each coding block; if both words in the word pair correspond to only one coding block, then directly using the element values ​​corresponding to these two coding blocks in the weight bias matrix as the spatiotemporal correlation between the word pair; if at least one word in the word pair corresponds to multiple coding blocks, then taking one coding block from each of the coding block lists corresponding to the two words to form a coding block pair; based on the area ratio of the coding blocks in the coding block pair, performing weighted processing on the element values ​​corresponding to the coding block pair, and determining the spatiotemporal correlation between the word pair based on the obtained weighted processing result; The new weight bias matrix is ​​determined based on the spatiotemporal correlation between the word pairs.

[0122] Optionally, the range determination submodule includes: a size determination unit, used to parse the video frame size and coding block size obtained from the sequence parameter set or image parameter set in the target video bitstream; a first range determination unit, used to determine the pixel coordinate range of the coding block based on the video frame size, the coding block size and the coding block index; and a second range determination unit, used to determine the pixel coordinate range of the image block based on the image block size, the video frame size, the image block division method and the image block index.

[0123] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0125] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0126] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0127] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as video data processing methods. For example, in some embodiments, the video data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the video data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform video data processing methods by any other suitable means (e.g., by means of firmware).

[0129] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0134] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and weak business scalability inherent in traditional physical hosting and VPS services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0135] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0136] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video data processing method, the method comprising: The target video stream is decoded to obtain multiple video frames, and a word sequence of image blocks in the video frames is generated. Based on the self-attention mechanism in the visual processing model, the attention weights corresponding to the word sequence are calculated according to the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coding blocks in the target video bitstream; The word sequence is processed based on the attention weights to obtain a feature sequence, and a video processing result is generated based on the feature sequence.

2. The method according to claim 1, further comprising constructing the weight bias matrix as follows: The target video stream is parsed in the compression domain to determine the coding block index and coding block type of the current coding block, as well as the video frame type and video frame index of the video frame in which the current coding block is located; Based on the coded block index, the coded block type, the video frame type, and the video frame index, the spatiotemporal correlation between the current coded block and other coded blocks in the target video stream is determined; The weight bias matrix is ​​constructed based on the spatiotemporal correlation and its corresponding coding block index.

3. The method according to claim 2, wherein, The step of determining the spatiotemporal correlation between the current coding block and other coding blocks in the target video stream based on the coding block index, the coding block type, the video frame type, and the video frame index includes: Based on the video frame type, a reference frame list is determined for the current coding block; Based on the reference frame list, the coding block index, and the reference relationship information corresponding to the coding block type, a first video frame is determined in the reference frame list and a first coding block is determined in the first video frame. Based on the video frame index and the preset frame span, a second video frame is determined in the target video bitstream, and a second coding block is determined in the second video frame based on the coding block index; The spatiotemporal correlation between the current coding block and the first coding block and the second coding block is determined respectively.

4. The method according to claim 3, wherein, The step of determining a first video frame and a first coding block within the first video frame based on the reference frame list, the coding block index, and the reference relationship information corresponding to the coding block type includes: If the coding block type is an inter-frame coded block, then determine whether the reference relationship information includes a skip identifier; If the skip identifier is included, the first coding block is determined based on the motion information derivation rules of the skip mode in the coding standard adopted by the target video stream; If the skip identifier is not included, the first video frame is determined in the reference frame list based on the reference frame index in the reference relationship information; Based on the motion vector in the coded block index and reference relationship information, the first coded block is determined for the current coded block in the first video frame.

5. The method according to claim 3, wherein, Determining the spatiotemporal correlation between the current coding block and the first coding block and the second coding block respectively includes: If the reference relationship information includes a skip identifier, then the spatiotemporal correlation corresponding to the first coding block is determined to be the first correlation; If the reference relationship information does not include a skip identifier, then the spatiotemporal correlation corresponding to the first coding block is determined to be the second correlation; The spatiotemporal correlation between the current coding block and the second coding block is determined as the third correlation; The third correlation is the smallest and greater than zero, and the second correlation is between the first correlation and the third correlation.

6. The method according to claim 1, wherein, The self-attention mechanism based on the visual processing model calculates the attention weights corresponding to the word sequence according to the weight bias matrix, including: Determine the range of pixel coordinates of the image block and the coded block in the video frame; Based on the pixel coordinate range, establish the positional mapping relationship between the image block and the coded block; Based on the self-attention mechanism in the visual processing model, the attention weights corresponding to the word sequence are calculated according to the weight bias matrix and the position mapping relationship.

7. The method according to claim 6, wherein, The step of establishing the positional mapping relationship between the image block and the coded block based on the pixel coordinate range includes: Traverse each image block, and for the current image block, calculate the first overlap area between the current image block and each coded block based on its pixel coordinate range; Based on the first overlap area and the coding grouping parameters corresponding to the target video stream, a positional mapping relationship is established between the image block and each coding block.

8. The method according to claim 7, wherein, The step of establishing the positional mapping relationship between the image block and each coding block based on the first overlap area and the coding block parameters corresponding to the target video bitstream includes: Based on the first overlap area, candidate coding blocks that overlap with the current image block are determined, as well as the total number of candidate coding blocks; If the total number of coding blocks is greater than 1 and the candidate coding blocks belong to the same coding block group, then the candidate coding block with the largest first overlapping area is taken as the target coding block corresponding to the current image block position; If the total number of coding blocks is greater than 1 and the candidate coding blocks belong to different coding block groups, then the current image block is segmented into sub-regions based on the coding grouping parameters and the pixel coordinate range of the sub-regions is determined; different sub-regions do not overlap and the corresponding coding block groups are different; Based on the pixel coordinate range of the sub-region, determine the target coding block corresponding to the current image block position and record the area ratio of each target coding block in the current image block; Based on the image block index of the current image block and the coding block index of the target coding block, a positional mapping relationship between the image block and each coding block is established.

9. The method according to claim 8, wherein, The step of determining the target coding block corresponding to the current image block position based on the pixel coordinate range of the sub-region and recording the area ratio of each target coding block in the current image block includes: Traverse each sub-region, and for the current sub-region, calculate the second overlap area between the current sub-region and the candidate coding block based on its pixel coordinate range; The candidate coding block with the largest second overlap area is taken as the target coding block corresponding to the current sub-region position, and the area ratio of the target coding block in the current image block is determined based on the second overlap area. Based on the target coding block corresponding to the current sub-region position, determine the target coding block corresponding to the current image block position, and record the area ratio of each target coding block in the current image block.

10. The method according to claim 6, wherein, The self-attention mechanism in the visual processing model calculates the attention weights corresponding to the word sequence based on the weight bias matrix and the position mapping relationship, including: Based on the list of coding blocks corresponding to the term index in the position mapping relationship and the area ratio of each coding block in the corresponding image block, the weight bias matrix is ​​converted into a new weight bias matrix. The original association matrix corresponding to the lexical sequence is modified using the new weight bias matrix to obtain an enhanced association matrix, and the attention weights corresponding to the lexical sequence are determined based on the enhanced association matrix. The original association matrix is ​​obtained by calculating the dot product similarity between the query matrix and the key matrix; the query matrix, the key matrix, and the value matrix are obtained by linearly mapping the word sequence.

11. The method according to claim 10, wherein, The weight bias matrix is ​​converted into a new weight bias matrix based on the list of coding blocks corresponding to the lexical indices in the position mapping relationship and the area ratio of each coding block in the corresponding image block. For any two words in the word sequence that form a word pair, determine the list of coding blocks corresponding to each word in the word pair and the area ratio of each coding block; If both terms in a term pair correspond to only one coding block, then the element values ​​corresponding to these two coding blocks in the weight bias matrix are directly used as the spatiotemporal correlation between the term pairs. If at least one term in the term pair corresponds to multiple coding blocks, then one coding block is taken from each of the coding block lists corresponding to the two terms to form a coding block pair; Based on the area ratio of the corresponding coding blocks in the coding block pair, the element values ​​corresponding to the coding block pair are weighted, and the spatiotemporal correlation between the word pairs is determined based on the weighted processing result. The new weight bias matrix is ​​determined based on the spatiotemporal correlation between the word pairs.

12. The method according to claim 6, wherein, Determining the pixel coordinate range of the image block and the coded block in the video frame includes: The video frame size and coding block size are obtained by parsing the sequence parameter set or image parameter set in the target video bitstream; Based on the video frame size, the coding block size, and the coding block index, the pixel coordinate range of the coding block is determined; The pixel coordinate range of the image block is determined based on the image block size, the video frame size, the image block division method, and the image block index.

13. A video data processing apparatus, the apparatus comprising: The bitstream decoding module is used to decode the target video bitstream to obtain multiple video frames and generate a word sequence of image blocks in the video frames; The weight calculation module is used to calculate the attention weights corresponding to the word sequence based on the self-attention mechanism in the visual processing model and the weight bias matrix; wherein, the weight bias matrix is ​​pre-constructed based on the spatiotemporal correlation between coding blocks in the target video bitstream; The result generation module is used to process the word sequence based on the attention weight to obtain a feature sequence, and generate a video processing result based on the feature sequence.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the video data processing method according to any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the video data processing method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the video data processing method according to any one of claims 1-12.

Images