Video processing method, apparatus, device, medium, and product

Abstract

The application discloses a video processing method, device, equipment, medium and product. The processing method comprises the following steps: obtaining a to-be-encoded video, performing semantic feature extraction on the to-be-encoded video by using an encoder to obtain a semantic feature sequence; performing sliding common semantic extraction and quantization processing on the semantic feature sequence based on at least two different size time windows from large to small in stages to obtain a time quantization result and a time residual error; for each video frame in the same time window, performing sliding common semantic extraction and quantization processing based on at least two different size space windows from large to small in stages to obtain a space quantization result; and performing entropy coding on the time quantization result and the space quantization result respectively to obtain a compressed code stream. The above method can improve the compression ratio of the video, reduce the memory occupation of the compressed video, and improve the quality of the reconstructed video.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, and in particular relates to a video processing method, apparatus, equipment, medium and product. Background Technology

[0002] With the rapid development of video communication, traditional video encoding and decoding schemes can no longer meet the requirements of high efficiency and low memory usage in current technologies.

[0003] Therefore, semantic encoding and decoding techniques based on vector quantization have emerged. This technique maps the input video to a low-dimensional semantic feature space through an encoder, then performs quantization compression, and finally reconstructs the video through a decoder. However, existing semantic coding schemes typically use a single fixed-scale vector quantizer during the quantization process, which cannot fully adapt to the semantic information of the video content. Furthermore, the single-scale quantization strategy struggles to balance global consistency with local details, resulting in low compression efficiency, high memory usage of the compressed video, and low quality of the reconstructed video. Summary of the Invention

[0004] This application provides a video processing method, apparatus, device, medium, and product that can improve video compression efficiency.

[0005] In a first aspect, embodiments of this application provide a video processing method, including:

[0006] The video to be encoded is acquired, and semantic features are extracted from the video using an encoder to obtain a semantic feature sequence.

[0007] Based on at least two time windows of different sizes from large to small, the semantic feature sequence is progressively extracted and quantized using a sliding method to obtain the time quantization result and the time residual.

[0008] Based on a sliding motion using at least two spatial windows of varying sizes from large to small, common semantics are extracted and quantized for each video frame within the same time window to obtain spatial quantization results.

[0009] The temporal quantization results and spatial quantization results are entropy encoded separately to obtain a compressed bitstream.

[0010] Secondly, embodiments of this application provide a video processing apparatus, comprising:

[0011] The acquisition module is used to acquire the video to be encoded and to extract semantic features from the video to be encoded by the encoder to obtain a semantic feature sequence.

[0012] The processing module is used to perform sliding common semantic extraction and quantization processing on the semantic feature sequence step by step based on at least two time windows of different sizes from large to small, so as to obtain the time quantization result and time residual;

[0013] The processing module is also used to perform a sliding operation based on at least two spatial windows of different sizes from large to small, to extract and quantize the common semantics of each video frame in the same time window, and obtain the spatial quantization result.

[0014] The encoding module is used to entropy encode the temporal quantization results and spatial quantization results respectively to obtain a compressed bitstream.

[0015] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0016] Processor and memory storing computer program instructions;

[0017] The processing method for the video described in the first aspect above is used by the processor when executing computer program instructions.

[0018] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the video processing method described in the first aspect.

[0019] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when processed by a processor, implements the video processing method described in the first aspect.

[0020] The video processing methods, apparatus, devices, media, and products provided in this application eliminate spatiotemporal redundancy in videos by extracting and quantizing common semantics from semantic feature sequences through different time windows and different spatial windows. This reduces the memory footprint of the generated quantization results and improves compression efficiency. Simultaneously, entropy coding is used to compress the temporal and spatial quantization results, increasing the overall video compression ratio and reducing the video's memory footprint. Furthermore, by extracting common semantics through time windows and extracting spatial windows from the temporal residuals after quantization, residual propagation is achieved, preserving both commonalities and unique characteristics of the video, which is beneficial for video reconstruction and improves the reconstruction quality. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a video processing method provided in some embodiments of this application.

[0023] Figure 2 This is a flowchart illustrating another video processing method provided in some embodiments of this application.

[0024] Figure 3 This is a flowchart illustrating yet another video processing method provided in some embodiments of this application.

[0025] Figure 4 This is a flowchart illustrating yet another video processing method provided in some embodiments of this application.

[0026] Figure 5 This is a flowchart illustrating yet another video processing method provided in some embodiments of this application.

[0027] Figure 6 This is a schematic diagram of a video processing apparatus provided for some embodiments of this application.

[0028] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0031] Addressing the problems of low video compression efficiency, high memory usage of compressed videos, and low quality of reconstructed videos in existing technologies, this application provides a video processing method, apparatus, device, medium, and product that can solve these problems. The following is a detailed description of a video processing method provided by an embodiment of this application.

[0032] In some embodiments, such as Figure 1 As shown in the figure, this application embodiment provides a video processing method, which may include:

[0033] S110: Obtain the video to be encoded, and extract semantic features from the video to be encoded using an encoder to obtain a semantic feature sequence.

[0034] Here, the video to be encoded can be a video of a user making a video call using a user device, or a video uploaded by a user to the server through a user device.

[0035] The encoder described above can encode the input video into a semantic feature sequence rich in semantic information. This semantic feature sequence can include semantic features of the video, such as information about objects, scenes, actions, and activities. A pre-trained deep neural network encoder can be used, such as a temporal convolutional network, residual network, attention mechanism, downsampling network, or generative adversarial network (GAN). This encoder receives the video to be encoded, analyzes it frame-by-frame or segment-by-segment, and outputs a corresponding semantic feature sequence. This semantic feature sequence contains high-level semantic information (such as object, scene, and action information) of its corresponding video frame.

[0036] In some examples, the video undergoes semantic feature extraction by an encoder, resulting in a semantic feature sequence. ,in For the length of time, To achieve a feature space resolution, each video frame in the semantic feature sequence is divided into [a specific segment]. One grid, The number of channels represents the number of channels targeted at... For each grid cell, C feature descriptions are extracted.

[0037] S120: Based on at least two time windows of different sizes from large to small, the semantic feature sequence is extracted and quantized in a sliding manner to obtain the time quantization result and time residual.

[0038] Here, the extraction and quantification of common semantics of the semantic feature sequence through at least two time windows of different sizes from large to small can be specifically described as the output of the previous time window being used as the input of the next time window for step-by-step processing.

[0039] The aforementioned at least two time windows of different sizes refer to time windows of different durations. Semantic feature extraction and quantization are performed first using a larger time window, followed by a smaller time window. Furthermore, each time window slides along the time axis of the data for which common semantics need to be extracted with a fixed step size. After each slide, the window covers a new time period, and the common semantics extraction and quantization operation is performed on the data within that time period.

[0040] Common semantic features can be extracted from the semantic feature sequence step-by-step using at least two different time windows, from largest to smallest. Specifically, the residual from the common semantic extraction and quantization process of the previous time window can be used as the input for the next time window. Simultaneously, the common semantics output from each time window are quantized to obtain a time-condensed vector corresponding to each time window. The set of condensed vectors for each time window is taken as the time quantization result, and the residual from the output of the last time window is taken as the time residual.

[0041] In some examples, taking two time windows as an example, after extracting at least one common semantic element Yt1 from the semantic feature sequence using a sliding common semantic extraction method in the first time window, the common semantic element Yt1 can be quantized to obtain a time-condensed vector Qt1. By inverse quantizing and semantically reconstructing the time-condensed vector Qt1, the reconstructed feature sequence Yt1' is obtained, which can be used as a basis for semantic feature sequences. The residual Rt1 is determined by the difference between the reconstructed feature sequence Yt1' and the residual Rt1. Similarly, in the second time window, a sliding common semantic extraction is performed on the residual Rt1 to obtain at least two common semantics Yt2. Each common semantic Yt2 is quantized to obtain the corresponding time-condensed vector Qt2. Each time-condensed vector Qt2 is dequantized and semantically reconstructed to obtain the reconstructed feature sequence Yt2'. The aforementioned time residual can be determined based on the difference between the residual Rt1 and the at least two reconstructed feature sequences Yt2'.

[0042] It is conceivable that when there are multiple time windows, these multiple time windows can be iteratively processed in the two ways described above, which will not be elaborated here.

[0043] S130: Based on at least two spatial windows of different sizes from large to small, a sliding process is performed to extract and quantize the common semantics of each video frame in the same time window to obtain the spatial quantization result.

[0044] Here, the aforementioned spatial windows of different sizes refer to spatial windows of different sizes H'×W', where H'≤H and W'≤H. Semantic feature extraction and quantization can be performed first using a larger spatial window, followed by a smaller spatial window. Simultaneously, at least two spatial windows, from largest to smallest, can be used to progressively extract and quantize common semantics for each video frame. That is, the residual from the common semantic extraction and quantization of the previous spatial window can be used as the input for the common semantic extraction and quantization of the next spatial window. For each spatial window, during common semantic extraction and quantization of video frames, the window slides across the H×W video frames with a fixed step size, traversing the entire video frame. Common semantics are extracted and quantized for the local video frames covered by each spatial window, resulting in a spatially compressed vector corresponding to each spatial window. The set of compressed vectors corresponding to each spatial window is then used as the spatial quantization result.

[0045] In some examples, a two-space window example is used for detailed explanation:

[0046] The first, larger spatial window extracts at least one common semantic element Ys1 from the frame. This common semantic element Ys1 is quantized to obtain at least one corresponding spatial condensation vector Qs1. Each spatial condensation vector Qs1 is then dequantized and semantically reconstructed to obtain the corresponding reconstructed semantic element Ys1'. The residual Rs1 is determined based on the difference between the frame's semantic feature S and at least one reconstructed semantic element Ys1'. The second spatial window slides across the residual Rs1 to obtain at least two common semantic elements Ys2. Each common semantic element Ys2 is quantized to obtain the corresponding spatial condensation vector Qs2. Each spatial condensation vector Qs2 is then dequantized and semantically reconstructed to obtain the corresponding reconstructed semantic element Ys2'. The residual Rs2 is determined based on the difference between the residual Rs1 and the at least two reconstructed semantic elements Ys2'. The set of condensation vectors Qs1 and Qs2 is used as the spatial quantization result.

[0047] It is conceivable that when there are multiple spatial windows, these multiple spatial windows can be iterated in the two ways described above, which will not be elaborated here.

[0048] S140: Entropy encoding is performed on the time quantization result and the spatial quantization result respectively to obtain the compressed bitstream.

[0049] Entropy encoding can be performed on the time quantization results and the spatial quantization results respectively to obtain a compressed bitstream.

[0050] Here, during compression, the temporal quantization result can be entropy encoded to obtain the first sub-compressed bitstream, and the spatial quantization result can be entropy encoded to obtain the second sub-compressed bitstream. The first and second compressed bitstreams together form the compressed bitstream described above.

[0051] It can be conceivable that the temporal quantization result includes multiple temporal compression vectors, and the spatial quantization result includes multiple spatial compression vectors. Each temporal compression vector is generated by vector quantization of the common semantics within a time window, and each spatial compression vector is generated by vector quantization of the common semantics of a video frame within a spatial window. Entropy encoding can be performed on the temporal compression vectors in the temporal quantization result to obtain the first sub-compressed bitstream, and entropy encoding can be performed on the spatial compression vectors in the spatial quantization result to obtain the second sub-compressed bitstream.

[0052] In some examples, entropy encoding can be performed based on the hierarchical relationship between time windows and the hierarchical relationship between spatial windows. For multiple time-condensed vectors in the time quantization result, the corresponding time-condensed vectors can be entropy encoded sequentially according to the hierarchical order of the time windows from largest to smallest; similarly, for multiple spatial-condensed vectors in the spatial quantization result, the corresponding spatial-condensed vectors can be entropy encoded sequentially according to the hierarchical order of the spatial windows from largest to smallest.

[0053] In some examples, to further improve the efficiency of entropy coding, when entropy coding multiple time-condensed vectors sequentially, the time-condensed vectors corresponding to the same time window can be entropy-coded to obtain the first sub-compressed bitstream, and the spatial-condensed vectors corresponding to the same spatial window can be entropy-coded to obtain the second sub-compressed bitstream.

[0054] In some examples, when entropy encoding the temporal and spatial quantization results, the probability of each character appearing in the temporal quantization result can be counted to obtain the probability distribution of all characters in the temporal quantization result. Similarly, the probability of each character appearing in the spatial quantization result can be counted to obtain the probability distribution of all characters in the spatial quantization result. A binary Huffman tree can be constructed based on the probability distribution. Symbols with high probability are closer to the root of the tree, meaning that characters with high probability of occurrence have short codewords, while symbols with low probability are farther from the root of the tree, meaning that characters with low probability of occurrence have short codewords and long codewords.

[0055] This application's embodiments, by extracting and quantizing common semantics from semantic feature sequences using different temporal and spatial windows, can eliminate spatiotemporal redundancy in videos, accurately separate and compress global and local detail information, reduce the memory footprint of the generated quantization results, and improve compression efficiency. Simultaneously, entropy coding is used to compress the temporal and spatial quantization results, increasing the overall video compression ratio. Furthermore, by extracting common semantics through temporal windows and extracting spatial windows from the quantized temporal residuals, residual propagation is achieved, preserving both commonalities and unique characteristics of the video, which is beneficial for video reconstruction and improves reconstruction quality.

[0056] In some embodiments, such as Figure 2 As shown, at least two time windows of different sizes, including n time windows (n≥2), are used to progressively extract and quantize the common semantics of the semantic feature sequence based on at least two time windows of different sizes, resulting in time quantization results and time residuals, which may include:

[0057] S210: For the first time window, perform sliding common semantic extraction on the semantic feature sequence based on the first time window to obtain the first time common semantic.

[0058] The first time window can be used to extract common semantics in a sliding manner to obtain the first time common semantics Yt1.

[0059] In some examples, T=16 in the semantic feature sequence. , The length of the first time window is L1 = T / mt1 = 16. Through this first time window, a common semantic element Yt1 can be extracted from the 16 frames of the entire semantic feature sequence, with dimensions (1×64×32×128). The length of the first time window described above is merely an example; the actual length can be set manually. For example, setting the length of the first time window to 8 allows for sliding extraction of the common semantic element from the 16 frames of the semantic feature sequence, yielding two first-time common semantic elements Yt1.

[0060] S220: Based on a preset codebook, the common semantics of the first time are quantized to obtain the first time condensed vector.

[0061] The first-time common semantic Yt1 is obtained by calculating the similarity between it and all vectors in the preset codebook, and is used as the first-time condensed vector Qt1. Alternatively, the first-time condensed vector Qt1 corresponding to the first-time common semantic Yt1 is obtained based on the correspondence between the common semantics and vectors in the codebook.

[0062] In some examples, the first-time condensed vector Qt1 corresponding to the first-time common semantic Yt1 can be obtained based on the correspondence between the common semantics and vectors in the binary code table.

[0063] Taking the common semantic Yt1, with a dimension of (1×64×32×128), as an example, the first time condensation vector Qt1 is obtained, with a dimension of (1×64×32×128).

[0064] S230: Perform inverse quantization and semantic reconstruction on the first time-condensed vector to obtain the first reconstructed feature sequence.

[0065] The first time-condensed vector can be dequantized and restored to the original time window size through a time upsampling network to obtain the first reconstructed feature sequence Yt1'.

[0066] Taking the first time-condensed vector Qt1, with dimensions of (1×64×32×128), as an example, it is dequantized and extended to 16 frames through a temporal sampling network to obtain the first reconstructed feature sequence Yt1', with dimensions of (16×64×32×128).

[0067] S240: Based on the semantic feature sequence and the first reconstructed feature sequence, determine the first residual corresponding to the first time window.

[0068] The difference between the semantic feature sequence S and the first reconstructed feature sequence Yt1 can be calculated to obtain the first residual Rt1, which has a dimension of (16×64×32×128).

[0069] S250: For the i-th time window, perform sliding common semantic extraction on the i-1 residual corresponding to the (i-1)-th time window to obtain the i-th time common semantic, where i≥2.

[0070] For the second time window, continue using semantic feature sequences T=16, , For example, the length of the second time window is L2 = T / mt2 = 8. The common semantics in the 16-frame residual Rt1 are extracted using a sliding method through the second time window. Specifically, common semantics Yt2 are extracted once for the first 8 frames of residual Rt1, and once for the last 8 frames. The two common semantics Yt2 are then quantized to obtain two corresponding time-condensed vectors Qt2, each with dimensions of (1×64×32×128). Following the same approach, progressively decreasing time windows are used to perform sliding common semantic extraction and quantization on the residuals output from the previous time window, ultimately yielding the time quantization result and the time residuals. Here, the length of each time window can be set manually.

[0071] Similarly, for the i-th time window, we obtain the i-th time common semantic Yti corresponding to the i-th time window.

[0072] S260: Based on a preset codebook, the common semantics of time i are quantized to obtain the condensed vector of time i.

[0073] The common semantic term Yti at time i can be quantized to obtain the compressed vector Qti at time i. For example, the common semantic term Yti at time i can be quantized using a binary codebook to obtain the compressed vector Qti at time i.

[0074] S270: Perform inverse quantization and semantic reconstruction on the i-th time-condensed vector to obtain the i-th reconstructed feature sequence.

[0075] For Qti, determine the corresponding i-th reconstructed feature sequence Yti'.

[0076] Taking the second time window as an example, the two time-condensed vectors Qt2 can be dequantized and restored to the length of the second time window through a time upsampling network. That is, for each time-condensed vector Qt2, whose dimension is (1×64×32×128), the second reconstructed feature sequence Yt2' with a dimension of (8×64×32×128) is restored through dequantization and time upsampling network.

[0077] For the i-th time window, perform inverse quantization and semantic reconstruction on all the i-th time-condensed vectors corresponding to the i-th time window to obtain the i-th reconstructed feature sequence.

[0078] S280: Based on the (i-1)th residual and the i-th reconstructed feature sequence, determine the i-th residual corresponding to the i-th time window.

[0079] Taking the second time window of length 8 as an example, the second residual Rt2 can be determined based on the difference between the first residual Rt1, whose dimension is (16×64×32×128), and the two second reconstructed feature sequences Yt2'. The second residual Rt2 has a dimension of (16×64×32×128), where the dimension of each second reconstructed feature sequence Yt2' is (16×64×32×128).

[0080] For the i-th application window, the residual corresponding to the (i-1)-th time window can be dequantized and semantically reconstructed with all the i-th time-condensed vectors corresponding to the i-th time window to obtain the difference of the i-th reconstructed feature sequence, and the i-th residual Rti can be determined.

[0081] S290: Set i=i+1 and repeat the following steps: Extract the common semantics of the (i-1)th residual corresponding to the (i-1)th time window using a sliding method to obtain the common semantics of the i-th time; quantize the common semantics of the i-th time based on the preset codebook to obtain the condensed vector of the i-th time; perform inverse quantization and semantic reconstruction on the condensed vector of the i-th time to obtain the reconstructed feature sequence of the i-th time; determine the i-th residual corresponding to the i-th time window based on the (i-1)th residual and the reconstructed feature sequence of the i-th time window until i=n.

[0082] Repeat the process from S250 to S280 until the nth time window.

[0083] S300: The time condensation vector set corresponding to each time window is used as the time quantization result, and the residual corresponding to the nth time window is used as the time residual.

[0084] For n time windows, we obtain the time quantization results {Qt1, Qt2, ..., Qtn} and the time residuals Rtn.

[0085] This application employs a residual progressive layered video encoding method. It first extracts and retains the most macroscopic and important semantic information (such as scene theme and main actions) from the video using a larger time window, ensuring that the semantic content of the reconstructed video is correct and complete even at the lowest bitrate. This avoids the severe blurring or block artifacts that occur in traditional encoding at low bitrates. Subsequent extraction of more detailed semantic information through progressively smaller time windows further improves the quality of the reconstructed video.

[0086] In some embodiments, such as Figure 3 The at least two spatial windows of different sizes include m spatial windows, where m ≥ 2. For each video frame in the temporal residual, common semantics are extracted and quantized stepwise based on at least two spatial windows of different sizes to obtain spatial quantization results, which may include:

[0087] S310: For the first spatial window, based on the first temporal window, for each video frame in the temporal residual, perform sliding common semantic extraction on the semantic features corresponding to the video frame based on the first spatial window to obtain the first spatial common semantics.

[0088] Taking the aforementioned temporal residual Rtn, with dimensions of (16×64×32×128), as an example, for the temporal residual Rtn, a sliding common semantic extraction is performed on the semantic features S corresponding to the video frames based on the first spatial window to obtain the first spatial common semantics. The dimension of each video frame is (64×32×128). The size of the first spatial window can be H1×W1 = (64 / 1)×(32 / 1) = 64×32. The first spatial common semantic Ys1 of each frame in the temporal residual is extracted through the first spatial window, with a dimension of (64×32×128).

[0089] S320: Based on a preset codebook, the common semantics of the first space are quantized to obtain the first space condensed vector.

[0090] The common semantics of the first space can be quantized based on a preset codebook, such as a binary codebook, to obtain a first space condensed vector. For example, the common semantics of the first space Ys1, whose dimension is (64×32×128), can be quantized to obtain the first space condensed vector Qs1, whose dimension is (64×32×128).

[0091] S330: Perform inverse quantization and semantic reconstruction on the first spatial condensation vector to obtain the first reconstructed semantics.

[0092] The first spatial condensed vector can be dequantized and a spatial upsampling network can be used to perform semantic reconstruction to the first reconstructed semantics of the size of the first spatial window. For example, the first spatial condensed vector Qs1, with a dimension of (64×32×128), can be dequantized and semantically reconstructed to obtain the first reconstructed semantics Ys1', with a dimension of (64×32×128).

[0093] S340: Determine the first spatial residual based on the semantic features corresponding to the video frame and the first reconstructed semantics.

[0094] The difference between the semantic feature S corresponding to the video frame and the first reconstructed semantic Ys1' can be calculated to obtain the first spatial residual Rs1 corresponding to the video frame, which has a dimension of (64×32×128).

[0095] S350: For the j-th spatial window, perform sliding common semantic extraction on the (j-1)-th spatial residual corresponding to the (j-1)-th spatial window to obtain the common semantics of the j-th space, where j≥2.

[0096] For the second spatial window, taking the size of the second spatial window as H2×W2= (64 / 2)×(32 / 2) =32×16 as an example, the second spatial window performs sliding common semantic extraction on the first empty residual Rs1 to obtain 4 second common semantics Ys2, where the dimension of each second common semantic is (32×16×128).

[0097] For the j-th spatial window, a sliding common semantic extraction can be performed on the (j-1)-th spatial residual corresponding to the (j-1)-th spatial window to obtain the corresponding j-th spatial common semantic.

[0098] S360: Based on a preset codebook, the common semantics of the j-th space are quantized to obtain the condensed vector of the j-th space.

[0099] Taking the above four second common semantic Ys2, each with a dimension of (32×16×128), as an example, each second common semantic Ys2 can be quantized based on a preset codebook to obtain the second space condensation vector Qs2.

[0100] The common semantics of the j-th space can also be quantized to obtain the condensed vector of the j-th space.

[0101] S370: Perform inverse quantization and semantic reconstruction on the j-th spatial condensation vector to obtain the j-th reconstructed semantics.

[0102] Taking the above four second spatial condensation vectors Qs2 as an example, each second spatial condensation vector Qs2 has a dimension of (1×1×128). For each second spatial condensation vector Qs2, it can be dequantized and a spatial sampling network can be used to perform semantic reconstruction to the second reconstructed semantic Ys2' of the size of the second spatial window, resulting in four second reconstructed semantic Ys2', each with a dimension of (32×16×128).

[0103] Similarly, the j-th spatial condensation vector is subjected to inverse quantization and semantic reconstruction using a spatial sampling network to obtain the j-th reconstructed semantics.

[0104] S380: Based on the (j-1)th spatial residual and the jth reconstructed semantics, determine the jth spatial residual corresponding to the jth spatial window.

[0105] Taking the four second reconstructed semantics Ys2' as an example, the second spatial residual Rs2, with a dimension of (64×32×128), can be obtained based on the difference between the first spatial residual Rs1, whose dimension is (64×32×128), and the four second reconstructed semantics Ys2', with the dimension of each second reconstructed semantic Ys2' being (32×16×128).

[0106] Similarly, the j-th spatial residual can be obtained by taking the difference between the (j-1)-th spatial residual and the j-th reconstructed semantics.

[0107] S390: Set j to j+1 and repeat the following steps: extract common semantics from the (j-1)th spatial residual corresponding to the (j-1)th spatial window to obtain the common semantics of the jth space; quantize the common semantics of the jth space based on the preset codebook to obtain the jth space condensed vector; perform inverse quantization and semantic reconstruction on the jth space condensed vector to obtain the jth reconstructed semantics; determine the jth spatial residual corresponding to the jth spatial window based on the (j-1)th residual and the jth reconstructed semantics; until j=m is satisfied.

[0108] Repeat steps S350 to S380 until the value of j equals m.

[0109] S400: The set of spatially condensed vectors corresponding to each spatial window is used as the spatial quantization result.

[0110] For m time windows, the spatial quantization results {Qs1, Qs2, ..., Qsn} are obtained.

[0111] It should be noted that the sizes of the various spatial windows in this application embodiment are merely exemplary.

[0112] This application embodiment extracts common semantics by sliding the spatial window from large to small step by step and performs two-stage processing, which can effectively capture the common information of each video frame, eliminate spatial information redundancy to the maximum extent, effectively reduce the amount of compressed video data, and reduce the final memory usage of the video.

[0113] Furthermore, by using residual recursion, it is ensured that the details lost by quantization in one spatial window can be captured and supplemented by the next smaller spatial window. This enables the original video's semantic features to be restored with high quality during video reconstruction, effectively maintaining the semantic integrity and visual quality of the decoded video.

[0114] In some embodiments, the temporal quantization result includes temporal compression vectors corresponding to at least two time windows, and the spatial quantization result includes spatial compression vectors corresponding to at least two spatial windows. Entropy encoding is performed on the temporal quantization result and the spatial quantization result respectively to obtain a compressed bitstream, including:

[0115] The time-condensed vectors corresponding to at least two time windows are entropy encoded to obtain the first sub-compressed bitstream, and the spatial-condensed vectors corresponding to at least two spatial windows are entropy encoded to obtain the second sub-compressed bitstream. The first sub-compressed bitstream and the second sub-compressed bitstream constitute the compressed bitstream.

[0116] Here, entropy encoding can be performed on the time compression vector corresponding to each time window to obtain the first sub-compressed bitstream, and entropy encoding can be performed on the spatial compression vector corresponding to each spatial window to obtain the second sub-compressed bitstream.

[0117] Furthermore, the condensed vectors of all time windows can be grouped according to their corresponding time windows, resulting in multiple groups. That is, the condensed vectors corresponding to the same time window are grouped together, resulting in multiple groups. For example, all the first condensed vectors Qt1 generated by the first time window are grouped together, all the second condensed vectors Qt2 generated by the second time window are grouped together, and so on. Spatial condensed vectors are also grouped using the same strategy, that is, all the first spatial condensed vectors Qs1 are grouped together, and all the second spatial condensed vectors Qs2 are grouped together.

[0118] Similarly, spatial condensation vectors corresponding to the same spatial window can be grouped together to obtain multiple groups.

[0119] The time-compressed vectors belonging to the same group are grouped together and entropy-encoded separately to generate the first sub-compressed bitstream. For example, all first time-compressed vectors Qt1 and all second time-compressed vectors Qt2 are entropy-encoded separately to obtain the first sub-compressed bitstream. Similarly, all spatial-compressed vectors belonging to the same group are grouped together and entropy-encoded separately. For example, all first spatial-compressed vectors Qs1 and all second spatial-compressed vectors Qs2 are entropy-encoded separately to obtain the second sub-compressed bitstream. The first and second sub-compressed bitstreams constitute the compressed bitstream.

[0120] In some examples, when performing entropy encoding, a natural regression model can be used to statistically analyze the probability distribution of all characters appearing in all time condensed vectors corresponding to the same time window and the probability distribution of all characters appearing in all spatial condensed vectors corresponding to the same spatial window, and entropy encoding can be performed based on the above probability distributions.

[0121] In some examples, the entropy coding described above may include arithmetic coding.

[0122] This application embodiment obtains a first sub-compressed bitstream by entropy encoding the time-condensed vectors corresponding to different time windows, and a second sub-compressed bitstream by entropy encoding the spatial-condensed vectors corresponding to different spatial windows. This results in a final compressed bitstream with a clear structure, distinct layers, and ease of transmission and processing. Furthermore, during decoding, the user can manually configure which sub-compressed bitstream corresponding to which time window or spatial window to decode, improving decoding flexibility.

[0123] In some embodiments, the time compression vectors corresponding to at least two time windows are entropy encoded to obtain a first sub-compressed bitstream, including:

[0124] According to the time sequence, the time compression vector corresponding to each time window is entropy encoded sequentially to obtain the first sub-compressed bitstream; wherein, when encoding the current time compression vector, the time compression vectors corresponding to all previously encoded time windows are used as conditions;

[0125] Entropy encoding is performed on the spatial compression vectors corresponding to at least two spatial windows to obtain the second sub-compressed bitstream, including:

[0126] According to spatial order, the spatial compression vector corresponding to each spatial window is entropy encoded sequentially to obtain the second sub-compressed bitstream; wherein, when encoding the current spatial compression vector, the spatial compression vectors corresponding to all encoded spatial windows are used as conditions.

[0127] The time-condensed vector corresponding to each time window can be entropy encoded sequentially to obtain the first sub-compressed bitstream. Furthermore, when entropy encoding is performed on each time-condensed vector, the time-condensed vector corresponding to the already encoded time window can be used as a condition for conditional entropy encoding. At the same time, the spatial-condensed vector corresponding to each spatial window can be entropy encoded sequentially to obtain the second sub-compressed bitstream. Similarly, for the spatial-condensed vector corresponding to each spatial window, the spatial-condensed vector corresponding to all already encoded spatial windows can be used as a condition for conditional entropy encoding.

[0128] In some examples, for the first time-condensed vector and spatial-condensed vector, a preset baseline probability model can be used to entropy encode them to generate the corresponding compressed bitstream.

[0129] In some examples, for each time-condensed vector, entropy coding can be performed using the time-condensed vectors corresponding to all encoded time windows and the previous time window as joint conditions; and for each spatial-condensed vector, entropy coding can be performed using the spatial-condensed vectors corresponding to all encoded spatial windows and the previous spatial window as joint conditions.

[0130] This application embodiment sets up entropy encoding based on the time-condensed vector of the encoded time window when performing entropy encoding for the time-condensed vector corresponding to each time window. Similarly, when performing entropy encoding based on the spatial-condensed vector of the encoded spatial window, the entropy encoding is based on the spatial-condensed vector of the encoded spatial window. This enables the available historical information to become richer and the conditions to become more accurate as encoding progresses, allowing for a more thorough mining of the correlation between video data in the time and spatial dimensions, thereby significantly improving compression efficiency.

[0131] In some embodiments, such as Figure 4 As shown, after entropy encoding of the temporal quantization and spatial quantization results to obtain the compressed bitstream, the processing methods may further include:

[0132] S410: Perform entropy decoding on the compressed bitstream to obtain the reconstructed time quantization result and the reconstructed spatial quantization result.

[0133] Entropy decoding can be performed on the compressed bitstream. For example, the compressed bitstream includes a first sub-compressed bitstream with time information and a second sub-compressed bitstream with spatial information. Entropy decoding can be performed on the first and second sub-compressed bitstreams to obtain the reconstructed time quantization result and the reconstructed spatial quantization result.

[0134] S420: Perform dequantization and semantic reconstruction on the reconstructed temporal quantization results and the reconstructed spatial quantization results respectively to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics.

[0135] The reconstructed temporal quantization result is dequantized based on a preset codebook, and then semantic reconstruction is performed through a temporal semantic reconstruction network such as a transposer or a Long Short-Term Memory (LSTM) network to generate temporal reconstructed semantics. The reconstructed spatial quantization result is dequantized, and then semantic reconstruction is performed through a spatial semantic reconstruction network such as a transposed convolution or an upsampling network to generate spatial reconstructed semantics.

[0136] S430: Integrate temporal reconstruction semantics and spatial reconstruction semantics to obtain the reconstructed semantic feature sequence.

[0137] A reconstructed semantic feature sequence can be obtained by fusing temporal and spatial semantic reconstructions through a pre-defined neural network.

[0138] S440: Input the reconstructed semantic feature sequence into the preset decoder to obtain the reconstructed video.

[0139] The reconstructed semantic feature sequence can be input into a preset decoder, which can map the semantic features back to the original pixel space and finally output the reconstructed video.

[0140] The decoder described above is a deep learning model pre-trained on a training video dataset. This deep learning module can convert semantic feature sequences into image frames, thereby obtaining the reconstructed video.

[0141] This application embodiment obtains the corresponding reconstructed temporal quantization result and reconstructed spatial quantization result by entropy decoding of the compressed bitstream. The reconstructed temporal quantization result and reconstructed spatial quantization result are then dequantized and semantically reconstructed to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics, respectively. This realizes the temporal and spatial detail features of the reconstructed video. The fused reconstructed semantic feature sequence is input into the decoder to obtain the reconstructed video, achieving complete video reconstruction. This can approximate the visual quality of the original video to the greatest extent and maintain high-frequency details and motion smoothness.

[0142] In some embodiments, such as Figure 5 As shown, the compressed bitstream includes a first sub-compressed bitstream and a second sub-compressed bitstream. The first sub-compressed bitstream is generated by entropy encoding based on the time compression vectors corresponding to at least two time windows, and the second sub-compressed bitstream is generated by entropy encoding based on the spatial compression vectors corresponding to at least two spatial windows.

[0143] Entropy decoding is performed on the compressed bitstream to obtain the reconstructed temporal quantization result and the reconstructed spatial quantization result, including:

[0144] S510: Perform entropy decoding on the first sub-compressed bitstream to obtain the reconstructed time compression vectors corresponding to at least two time windows respectively, and perform entropy decoding on the second sub-compressed bitstream to obtain the reconstructed spatial compression vectors corresponding to at least two spatial windows respectively.

[0145] Entropy decoding can be performed on the first sub-compressed bitstream to reconstruct the time-condensed vector corresponding to each time window, and entropy decoding can be performed on the second sub-compressed bitstream to obtain the reconstructed spatial-condensed vector corresponding to each spatial window.

[0146] The reconstructed temporal quantization results and the reconstructed spatial quantization results are dequantized and semantically reconstructed respectively to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics, including:

[0147] S520: Perform inverse quantization and semantic reconstruction on the reconstructed time condensation vector to obtain the temporal reconstruction semantics corresponding to at least two time windows respectively; and perform inverse quantization and semantic reconstruction on the reconstructed spatial condensation vector to obtain the spatial reconstruction semantics corresponding to at least two spatial windows respectively.

[0148] The reconstructed time condensation vector obtained in step S510 is dequantized, for example, by looking up a codebook table, and semantic reconstruction is performed by LSTM, Transformer or deconvolution network, to obtain the temporal reconstruction semantics corresponding to the time window.

[0149] The reconstructed spatial condensation vector obtained in step S510 is dequantized and input into a transposed convolutional network for semantic reconstruction to obtain the spatial reconstruction semantics corresponding to the spatial window.

[0150] This application embodiment obtains the reconstructed temporal compression vector and the reconstructed spatial compression vector by performing entropy decoding on the first sub-compressed bitstream and the second sub-compressed bitstream respectively. Through hierarchical decoding, anti-Liang Hai, and semantic reconstruction, information distortion during the decoding process can be minimized, ensuring high fidelity of the reconstructed semantics. This allows the reconstructed video to retain the clarity and details of the original video to the greatest extent, thereby improving the quality of the reconstructed video.

[0151] In some embodiments, inputting the reconstructed semantic feature sequence into a preset decoder to obtain the reconstructed video may include:

[0152] Obtain the weight coefficients corresponding to the temporal quantization results and spatial quantization results. The weight coefficients represent the importance of the temporal quantization results and spatial quantization results. Input the weight coefficients and the reconstructed semantic feature sequence into the decoder to obtain the reconstructed video.

[0153] The weighting coefficients corresponding to the aforementioned temporal and spatial quantization results can be manually set by the user, taking into account the video's usage scenario, the user's current device's computing resources, and transmission bandwidth. In some examples, the temporal quantization results include temporal compression vectors corresponding to different time windows, and the spatial quantization results include spatial compression vectors corresponding to different spatial windows. The aforementioned weighting coefficients can include either the weighting coefficients of the temporal compression vectors corresponding to different time windows or the weighting coefficients of the spatial compression vectors corresponding to different spatial windows.

[0154] For example, when processing high-speed motion videos, the weight coefficients of the temporal quantization results are larger, and the decoder can prioritize ensuring the continuity and smoothness of the motion in the video reconstructed based on the weight coefficients; when processing high-resolution still images, the weight coefficients of the spatial quantization results are larger, and the video reconstructed by the decoder can better preserve texture details and sharpness.

[0155] For example, in static scenes, a larger time window corresponds to a larger weight coefficient for the temporal compression vector, which can significantly reduce the video bitrate and improve decoding efficiency. In video call scenarios, a larger time window corresponds to a larger weight coefficient for the temporal compression vector, while a smaller spatial window corresponds to a larger weight coefficient for the spatial compression vector, resulting in high-definition reconstructed video.

[0156] For example, when user devices have limited computing resources or low transmission bandwidth, the weight coefficient of the time compression vector corresponding to a smaller time window is larger, and the weight coefficient of the spatial compression vector corresponding to a smaller spatial window is larger, in order to achieve smooth video playback.

[0157] This application embodiment sets weight coefficients, enabling the decoder to reconstruct the video based on the weight coefficients and the reconstructed semantic feature sequence, thereby obtaining the reconstructed video. This achieves flexibility in video reconstruction, allowing it to adapt to different user scenarios, which is beneficial for improving user experience and reconstruction efficiency.

[0158] In some embodiments, before the encoder extracts semantic features from the video to be encoded to obtain a semantic feature sequence, the decoder can be trained using a training video. For example, some characters in the temporal quantization and spatial quantization results of the training video can be randomly flipped by a certain bit value to obtain modified temporal quantization and spatial quantization results. By performing inverse quantization, semantic reconstruction, and video reconstruction of the original decoder on the modified temporal quantization and spatial quantization results, the reconstructed video corresponding to the training video is obtained. The loss function of the decoder is trained using the preset reconstructed video and the reconstructed video until the preset conditions are met, and the above-mentioned decoder is obtained, so that the decoder can still reconstruct a high-quality video even when there is a lot of network noise or pronunciation errors.

[0159] Based on the same inventive concept, embodiments of this application also provide a video processing apparatus.

[0160] In some embodiments, such as Figure 6 As shown in the figure, this application provides a video processing apparatus, which may include:

[0161] The acquisition module 601 is used to acquire the video to be encoded and to extract semantic features from the video to be encoded by the encoder to obtain a semantic feature sequence.

[0162] The processing module 602 is used to perform sliding common semantic extraction and quantization processing on the semantic feature sequence step by step based on at least two time windows of different sizes from large to small, so as to obtain the time quantization result and time residual;

[0163] The processing module 602 is also used to perform a sliding operation based on at least two spatial windows of different sizes from large to small to extract and quantize the common semantics of each video frame in the same time window, so as to obtain the spatial quantization result.

[0164] The encoding module 603 is used to entropy encode the time quantization result and the spatial quantization result respectively to obtain a compressed bitstream.

[0165] This application embodiment acquires semantic feature sequences through an acquisition module, and then performs common semantic extraction and quantization processing on these sequences using different time windows and spatial windows. This eliminates spatiotemporal redundancy in the video, accurately separates and compresses global and local details, reduces the memory footprint of the generated quantization results, and improves compression efficiency. Simultaneously, the processing module performs entropy encoding on the temporal and spatial quantization results, increasing the overall video compression ratio. Furthermore, by extracting common semantics through time windows and extracting spatial windows from the quantized temporal residuals, residual propagation is achieved, preserving both commonalities and unique characteristics of the video, which is beneficial for video reconstruction and improves reconstruction quality.

[0166] In some embodiments, at least two time windows of different sizes include n time windows, where n≥2, and the processing module can be specifically used for:

[0167] For the first time window, a sliding common semantic extraction is performed on the semantic feature sequence based on the first time window to obtain the first time common semantic;

[0168] Based on a pre-defined codebook, the common semantics of the first time are quantized to obtain the first time condensed vector;

[0169] The first time-condensed vector is dequantized and semantically reconstructed to obtain the first reconstructed feature sequence.

[0170] Based on the semantic feature sequence and the first reconstructed feature sequence, determine the first residual corresponding to the first time window;

[0171] For the i-th time window, the common semantics of the i-1 residual corresponding to the (i-1)-th time window are extracted by sliding method to obtain the common semantics of the i-th time, where i≥2;

[0172] Based on a pre-defined codebook, the common semantics of time i are quantized to obtain the condensed vector of time i.

[0173] The i-th time-condensed vector is dequantized and semantically reconstructed to obtain the i-th reconstructed feature sequence.

[0174] Based on the (i-1)th residual and the ith reconstructed feature sequence, determine the ith residual corresponding to the ith time window;

[0175] Set i=i+1 and repeat the following steps: Extract the common semantics of the (i-1)th residual corresponding to the (i-1)th time window using a sliding method to obtain the common semantics of the i-th time; quantize the common semantics of the i-th time based on a preset codebook to obtain the condensed vector of the i-th time; perform inverse quantization and semantic reconstruction on the condensed vector of the i-th time to obtain the reconstructed feature sequence of the i-th time; determine the i-th residual corresponding to the i-th time window based on the (i-1)th residual and the reconstructed feature sequence of the i-th time window until i=n;

[0176] The time-condensed vector set corresponding to each time window is used as the time quantization result, and the residual corresponding to the nth time window is used as the time residual.

[0177] In some embodiments, at least two spatial windows of different sizes include m spatial windows, where m ≥ 2, and the processing module can be specifically used for:

[0178] For the first spatial window, based on the first temporal window, for each video frame in the temporal residual, a sliding common semantic extraction is performed on the semantic features corresponding to the video frame based on the first spatial window to obtain the first spatial common semantic.

[0179] Based on a pre-defined codebook, the common semantics of the first space are quantized to obtain a condensed vector of the first space.

[0180] The first spatial condensation vector is dequantized and semantically reconstructed to obtain the first reconstructed semantics.

[0181] Based on the semantic features corresponding to the video frames and the first reconstructed semantics, the first spatial residual is determined;

[0182] For the j-th spatial window, a sliding common semantic extraction is performed on the (j-1)-th spatial residual corresponding to the (j-1)-th spatial window to obtain the common semantics of the j-th space, where j≥2;

[0183] Based on a pre-defined codebook, the common semantics of the j-th space are quantized to obtain the condensed vector of the j-th space.

[0184] The j-th spatial condensation vector is dequantized and semantically reconstructed to obtain the j-th reconstructed semantics.

[0185] Based on the (j-1)th spatial residual and the jth reconstructed semantics, determine the jth spatial residual corresponding to the jth spatial window;

[0186] Set j to j+1 and repeat the following steps: extract common semantics from the (j-1)th spatial residual corresponding to the (j-1)th spatial window using a sliding method to obtain the common semantics of the j-th space; quantize the common semantics of the j-th space based on a preset codebook to obtain the j-th space condensed vector; dequantize and reconstruct the j-th space condensed vector to obtain the j-th reconstructed semantics; determine the j-th spatial residual corresponding to the j-th spatial window based on the (j-1)th residual and the j-th reconstructed semantics; until j=m is satisfied.

[0187] The set of spatially condensed vectors corresponding to each spatial window is used as the spatial quantization result.

[0188] In some embodiments, the temporal quantization result includes temporal compression vectors corresponding to at least two time windows, and the spatial quantization result includes spatial compression vectors corresponding to at least two spatial windows. The encoding module can specifically be used for:

[0189] The time-condensed vectors corresponding to at least two time windows are entropy encoded to obtain the first sub-compressed bitstream, and the spatial-condensed vectors corresponding to at least two spatial windows are entropy encoded to obtain the second sub-compressed bitstream. The first sub-compressed bitstream and the second sub-compressed bitstream constitute the compressed bitstream.

[0190] In some embodiments, the encoding module may specifically be used for:

[0191] According to the time sequence, the time compression vector corresponding to each time window is entropy encoded sequentially to obtain the first sub-compressed bitstream; wherein, when encoding the current time compression vector, the time compression vectors corresponding to all previously encoded time windows are used as conditions;

[0192] Entropy encoding is performed on the spatial compression vectors corresponding to at least two spatial windows to obtain the second sub-compressed bitstream, including:

[0193] According to spatial order, the spatial compression vector corresponding to each spatial window is entropy encoded sequentially to obtain the second sub-compressed bitstream; wherein, when encoding the current spatial compression vector, the spatial compression vectors corresponding to all encoded spatial windows are used as conditions.

[0194] In some embodiments, the apparatus further includes a decoding module for:

[0195] Entropy decoding is performed on the compressed bitstream to obtain the reconstructed temporal quantization result and the reconstructed spatial quantization result;

[0196] The reconstructed temporal quantization results and the reconstructed spatial quantization results are dequantized and semantically reconstructed to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics.

[0197] By fusing temporal and spatial reconstructed semantics, a reconstructed semantic feature sequence is obtained.

[0198] The reconstructed semantic feature sequence is input into a preset decoder to obtain the reconstructed video.

[0199] In some embodiments, the compressed bitstream includes a first sub-compressed bitstream and a second sub-compressed bitstream. The first sub-compressed bitstream is generated by entropy encoding based on time compression vectors corresponding to at least two time windows, and the second sub-compressed bitstream is generated by entropy encoding based on spatial compression vectors corresponding to at least two spatial windows.

[0200] The decoding module can be used specifically for:

[0201] Entropy decoding is performed on the first sub-compressed bitstream to obtain the reconstructed time compression vectors corresponding to at least two time windows respectively, and entropy decoding is performed on the second sub-compressed bitstream to obtain the reconstructed spatial compression vectors corresponding to at least two spatial windows respectively;

[0202] The reconstructed temporal condensation vector is dequantized and semantically reconstructed to obtain temporal reconstruction semantics corresponding to at least two time windows, and the reconstructed spatial condensation vector is dequantized and semantically reconstructed to obtain spatial reconstruction semantics corresponding to at least two spatial windows.

[0203] In some embodiments, the apparatus further includes a reconstruction module for:

[0204] Obtain the weight coefficients corresponding to the temporal quantization results and spatial quantization results. The weight coefficients represent the importance of the temporal quantization results and spatial quantization results.

[0205] The weighting coefficients and the reconstructed semantic feature sequence are input into the decoder to obtain the reconstructed video.

[0206] The apparatus of the above embodiments is used to implement the corresponding video processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0207] Figure 7 A schematic diagram of the hardware structure of an electronic device is provided in the application embodiment.

[0208] The electronic device 700 may include a processor 701 and a memory 702 storing computer program instructions.

[0209] Specifically, the processor 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0210] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.

[0211] In a particular embodiment, memory 702 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0212] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this application.

[0213] The processor 701 reads and executes computer program instructions stored in the memory 702 to implement any of the video processing methods in the above embodiments.

[0214] In one example, the electronic device may also include a communication interface 703 and a bus 704. Wherein, as... Figure 7 The processor 701, memory 702, and communication interface 703 are connected through bus 704 and complete communication with each other.

[0215] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0216] Bus 704 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 704 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0217] The electronic devices described above are used to implement the corresponding video processing methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0218] Furthermore, in conjunction with the video processing methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the video processing methods described in the above embodiments.

[0219] Furthermore, in conjunction with the video processing methods described in the above embodiments, this application can provide a computer program product for implementation. When the instructions of this computer program product are executed by the processor of an electronic device, they implement any of the video processing methods described in the above embodiments.

[0220] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0221] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0222] It should also be noted that the exemplary embodiments mentioned in this application describe methods or apparatuses based on a series of steps or devices. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0223] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0224] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A video processing method, characterized in that, include: The video to be encoded is acquired, and semantic features are extracted from the video to be encoded using an encoder to obtain a semantic feature sequence; Based on at least two time windows of different sizes from large to small, the semantic feature sequence is progressively extracted and quantized using a sliding common semantic method to obtain time quantization results and time residuals. Based on a sliding motion using at least two spatial windows of varying sizes from large to small, common semantics are extracted and quantized for each video frame within the same time window to obtain spatial quantization results. The temporal quantization result and the spatial quantization result are entropy encoded respectively to obtain a compressed bitstream.

2. The video processing method according to claim 1, characterized in that, The at least two time windows of different sizes include n time windows, where n≥2. The extraction and quantization of common semantics of the semantic feature sequence are performed step-by-step based on the at least two time windows of different sizes to obtain time quantization results and time residuals, including: For the first time window, a sliding common semantic extraction is performed on the semantic feature sequence based on the first time window to obtain the first time common semantic; The first temporal common semantics are quantized based on a preset codebook to obtain a first temporal condensation vector; The first time-condensed vector is dequantized and semantically reconstructed to obtain the first reconstructed feature sequence. Based on the semantic feature sequence and the first reconstructed feature sequence, determine the first residual corresponding to the first time window; For i ranging from 2 to n, perform the following steps in sequence: Based on the (i-1)th residual, a sliding common semantic extraction is performed to obtain the i-th time common semantic; The common semantics of the i-th time are quantized using the preset codebook to obtain the i-th time condensed vector; The i-th time-condensed vector is dequantized and semantically reconstructed to obtain the i-th reconstructed feature sequence; According to the i-th 1. The residual is determined by combining it with the i-th reconstructed feature sequence; The time condensation vector set corresponding to each time window is used as the time quantization result, and the residual corresponding to the nth time window is used as the time residual.

3. The video processing method according to claim 1, characterized in that, The at least two spatial windows of different sizes include m spatial windows, where m ≥ 2. For each video frame in the temporal residual, common semantics are extracted and quantized step by step based on the at least two spatial windows of different sizes to obtain spatial quantization results, including: For the first spatial window, based on the first temporal window, for each video frame in the temporal residual, a sliding common semantic extraction is performed on the semantic features corresponding to the video frame based on the first spatial window to obtain the first spatial common semantic. Based on a preset codebook, the common semantics of the first space are quantized to obtain a first space condensed vector; The first spatial condensation vector is dequantized and semantically reconstructed to obtain the first reconstructed semantics. Based on the semantic features corresponding to the video frame and the first reconstructed semantics, the first spatial residual is determined; For j ranging from 2 to m, perform the following steps in sequence: Based on the jth 1. The common semantics of the j-th space are extracted by sliding method using the spatial residuals. The common semantics of the j-th space are quantized using the preset codebook to obtain the j-th space condensed vector; The j-th spatial condensation vector is dequantized and semantically reconstructed to obtain the j-th reconstructed semantics; According to the j-th 1. Use the spatial residual and the j-th reconstructed semantics to determine the j-th spatial residual; The spatial condensation vector set corresponding to each spatial window is used as the spatial quantization result.

4. The video processing method according to claim 1, characterized in that, The temporal quantization result includes temporal compression vectors corresponding to at least two time windows, and the spatial quantization result includes spatial compression vectors corresponding to at least two spatial windows. The step of performing entropy encoding on the temporal quantization result and the spatial quantization result to obtain a compressed bitstream includes: The time-condensed vectors corresponding to at least two time windows are entropy encoded to obtain a first sub-compressed bitstream, and the spatial-condensed vectors corresponding to at least two spatial windows are entropy encoded to obtain a second sub-compressed bitstream. The first sub-compressed bitstream and the second sub-compressed bitstream constitute the compressed bitstream.

5. The video processing method according to claim 4, characterized in that, The step of entropy encoding the time-condensed vectors corresponding to at least two time windows to obtain the first sub-compressed bitstream includes: According to the time sequence, the time compression vector corresponding to each time window is entropy encoded sequentially to obtain the first sub-compressed bitstream; wherein, when encoding the current time compression vector, the time compression vectors corresponding to all encoded time windows are used as conditions; Entropy encoding is performed on the spatial compression vectors corresponding to at least two spatial windows to obtain the second sub-compressed bitstream, including: According to spatial order, the spatial compression vector corresponding to each spatial window is entropy encoded sequentially to obtain the second sub-compressed bitstream; wherein, when encoding the current spatial compression vector, the spatial compression vectors corresponding to all encoded spatial windows are used as conditions.

6. The video processing method according to claim 1, characterized in that, After performing entropy encoding on the temporal quantization result and the spatial quantization result to obtain a compressed bitstream, the processing method further includes: Entropy decoding is performed on the compressed bitstream to obtain the reconstructed temporal quantization result and the reconstructed spatial quantization result; The reconstructed temporal quantization result and the reconstructed spatial quantization result are respectively dequantized and semantically reconstructed to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics. By fusing the temporal reconstruction semantics and the spatial reconstruction semantics, a reconstructed semantic feature sequence is obtained; The reconstructed semantic feature sequence is input into a preset decoder to obtain the reconstructed video.

7. The video processing method according to claim 6, characterized in that, The compressed bitstream includes a first sub-compressed bitstream and a second sub-compressed bitstream. The first sub-compressed bitstream is generated by entropy encoding based on time compression vectors corresponding to at least two time windows, and the second sub-compressed bitstream is generated by entropy encoding based on spatial compression vectors corresponding to at least two spatial windows. The entropy decoding of the compressed bitstream to obtain the reconstructed temporal quantization result and the reconstructed spatial quantization result includes: Entropy decoding is performed on the first sub-compressed bitstream to obtain reconstructed time compression vectors corresponding to at least two time windows, and entropy decoding is performed on the second sub-compressed bitstream to obtain reconstructed spatial compression vectors corresponding to at least two spatial windows. The reconstructed temporal quantization result and the reconstructed spatial quantization result are respectively subjected to inverse quantization and semantic reconstruction to obtain the corresponding temporal reconstruction semantics and spatial reconstruction semantics, including: The reconstructed time-condensed vector is dequantized and semantically reconstructed to obtain the time reconstruction semantics corresponding to at least two time windows, and the reconstructed spatial-condensed vector is dequantized and semantically reconstructed to obtain the spatial reconstruction semantics corresponding to at least two spatial windows.

8. The video processing method according to claim 6 or 7, characterized in that, The step of inputting the reconstructed semantic feature sequence into a preset decoder to obtain the reconstructed video includes: Obtain the weight coefficients corresponding to the time quantization result and the spatial quantization result, wherein the weight coefficients characterize the importance of the time quantization result and the spatial quantization result; The weight coefficients and the reconstructed semantic feature sequence are input into the decoder to obtain the reconstructed video.

9. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire the video to be encoded and to perform sliding semantic feature extraction on the video to be encoded by the encoder to obtain a semantic feature sequence. The processing module is used to perform sliding common semantic extraction and quantization processing on the semantic feature sequence step by step based on at least two time windows of different sizes from large to small, so as to obtain time quantization results and time residuals; The processing module is also used to perform a sliding operation based on at least two spatial windows of different sizes from large to small, to extract and quantize the common semantics of each video frame in the same time window, and obtain the spatial quantization result. The encoding module is used to entropy encode the time quantization result and the spatial quantization result respectively to obtain a compressed bitstream.

10. An electronic device, characterized in that, The device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the video processing method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when executed by a processor, implement the video processing method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is processed by a processor, it implements the video processing method as described in any one of claims 1 to 8.

Images