A MVPS video encoding method, device and medium based on deep learning

Abstract

The present application discloses a MVPS video encoding method, device and medium based on deep learning, which relates to the field of video encoding technology. The method includes: obtaining a video stream, extracting features of the video stream in the time domain and the spatial domain through a feature extraction unit, and generating a joint feature map through an adaptive weight fusion layer; performing bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; generating a frame-level quantization parameter initial value based on the multi-scale feature set and combining the current network bandwidth feedback; performing temporal consistency correction on the quantization parameter initial value to generate a target quantization parameter; performing block encoding on the current frame according to the target quantization parameter to generate an encoded data stream. The present application solves the problems of low efficiency, insufficient real-time performance, unstable quality and poor adaptability of traditional solutions through the above-mentioned method, through dynamic parameter adjustment, hardware algorithm collaborative optimization, cross-frame consistency enhancement and network perception.

Description

Technical Field

[0001] The present application relates to the field of video coding technology, and in particular to a deep learning-based MVPS video coding method, device, and medium. Background Art

[0002] With the diversification of video application scenarios, including more and more ultra-high-definition live broadcasts, smart security, VR / AR, etc., traditional video encoding technology faces significant bottlenecks.

[0003] Traditional video coding technologies rely on fixed quantization tables and predefined coding block division rules, making it impossible to dynamically adjust coding strategies based on video content. This results in wasted bitrate in low-complexity scenes and reduced quality in high-dynamic scenes. Deep learning-based coding schemes primarily focus on software algorithm optimization and lack collaborative design with dedicated hardware, resulting in high computational latency, low energy efficiency, and difficulty meeting real-time requirements. Furthermore, traditional bitrate allocation models independently process single-frame data and ignore video timing correlations, leading to quality fluctuations between adjacent frames and impacting the subjective visual experience. Their real-time response to network bandwidth fluctuations can easily lead to lag or image quality degradation in weak network environments.

[0004] Through the above analysis, the problems and defects of the existing technology are as follows:

[0005] Existing video encoding technologies suffer from low efficiency, insufficient hardware coordination, cross-frame quality fluctuations, and poor network adaptability. Summary of the Invention

[0006] The embodiments of the present application provide a deep learning-based MVPS video encoding method, device, and medium, which can solve the problems of low efficiency, insufficient hardware coordination, cross-frame quality fluctuations, and poor network adaptability of video encoding technology in the prior art.

[0007] In the first aspect, an embodiment of the present application provides an MVPS video encoding method based on deep learning, the method comprising: obtaining a video stream, performing time domain and spatial domain feature extraction on the video stream through a feature extraction unit, and generating a joint feature map through an adaptive weight fusion layer; performing bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; based on the multi-scale feature set and combined with current network bandwidth feedback, generating a frame-level quantization parameter initial value; performing temporal consistency correction on the quantization parameter initial value to generate a target quantization parameter; performing block encoding on the current frame according to the target quantization parameter to generate an encoded data stream.

[0008] In one implementation of the present application, a video stream is obtained, and time domain and spatial domain feature extraction is performed on the video stream through a feature extraction unit, and a joint feature map is generated through an adaptive weight fusion layer, specifically including: deploying feature extraction units in parallel on an FPGA chip; parsing the video stream into MVPS video stream data in frames to obtain video frames; using a three-dimensional convolutional neural network to extract the time domain motion features of the video frames through the feature extraction unit, wherein the optical flow calculation result is used as the initial value of the convolution weight; using a deformable convolution kernel to extract the spatial domain texture features of the video frames, and adjusting the sampling position according to the edge gradient of the video frame; channel-splitting the time domain motion features and the spatial domain texture features, calculating the fusion weight of the time domain motion features and the spatial domain texture features, and generating a joint feature map.

[0009] In one implementation of the present application, the temporal motion features and the spatial texture features are channel-spliced, the fusion weights of the temporal motion features and the spatial texture features are calculated, and a joint feature map is generated, specifically including: obtaining spatial audio through a microphone array, obtaining a depth map through a ToF sensor, and parsing to obtain audio spectrum features and depth map edge features; aligning the audio spectrum features with the temporal motion features to obtain a cross-modal similarity matrix; fusing the depth map edge features with the spatial texture features to generate a geometrically perceived spatial feature field; inputting the cross-modal similarity matrix and the feature field into a differentiable neural architecture to generate a dynamic sparsely connected spatiotemporal attention network; performing multimodal collaborative compression through the spatiotemporal attention network to generate a joint feature map.

[0010] In one implementation of the present application, a bidirectional feature interaction is performed between the layers of the joint feature map to obtain a multi-scale feature set, specifically including: deploying a causal Transformer encoder and a non-causal Transformer decoder between adjacent layers of the feature pyramid; constraining the preset low-level features through the causal Transformer encoder; multiplying the preset high-level semantic features with the low-level features element-by-element through the Transformer decoder to obtain an interacted feature map; applying channel attention weights to the feature map to suppress redundant feature channels and obtain a multi-scale feature set.

[0011] In one implementation of the present application, based on a multi-scale feature set and combined with current network bandwidth feedback, an initial value of a frame-level quantization parameter is generated, specifically including: establishing a network bandwidth monitoring thread to obtain network bandwidth data in real time, the network bandwidth data including the TCP congestion window size and packet loss rate; encoding the network bandwidth data into a vector, splicing it with the multi-scale feature set, and inputting it into a gated attention network; generating an initial value of the quantization parameter based on the attention weight distribution so that the bit rate distribution conforms to the network transmission constraints.

[0012] In one implementation of the present application, after performing temporal consistency correction on the initial value of the quantization parameter and generating the target quantization parameter, the method also includes: initially dividing the video frame into coding blocks of maximum size; extracting the motion vector field of the coding block of the previous frame and constructing a motion consistency matrix; calculating the correlation coefficient between the coding block of the current frame and the motion consistency matrix, and generating a time domain smoothing constraint factor.

[0013] In one implementation of the present application, after generating the time domain smoothing constraint factor, the method also includes: performing nonlinear mapping on the initial value of the quantization parameter according to the time domain smoothing constraint factor; recursively calculating the rate-distortion cost of the coding block, and if the rate-distortion cost is higher than a preset threshold, continuing to segment the coding block according to the target quantization parameter.

[0014] In one implementation of the present application, after dividing the current frame into coding blocks according to the target quantization parameter and generating the encoded data stream, the method also includes: inputting the coding block into the rate-distortion optimization module, iteratively optimizing the coding block division mode through the rate-distortion cost function; performing entropy coding on the optimized coding block to generate the final compressed code stream; during the encoding process, monitoring the complexity of the video content in real time to update the weight parameters of the bit rate allocation.

[0015] In the second aspect, an embodiment of the present application also provides an MVPS video encoding device based on deep learning, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain a video stream, perform time domain and spatial domain feature extraction on the video stream through a feature extraction unit, and generate a joint feature map through an adaptive weight fusion layer; perform bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; based on the multi-scale feature set and combined with the current network bandwidth feedback, generate a frame-level quantization parameter initial value; perform temporal consistency correction on the quantization parameter initial value to generate a target quantization parameter; block encode the current frame according to the target quantization parameter to generate an encoded data stream.

[0016] On the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for MVPS video coding based on deep learning, which stores computer executable instructions, and the computer executable instructions are set to: obtain a video stream, perform time domain and spatial domain feature extraction on the video stream through a feature extraction unit, and generate a joint feature map through an adaptive weight fusion layer; perform bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; based on the multi-scale feature set, combined with the current network bandwidth feedback, generate a frame-level quantization parameter initial value; perform temporal consistency correction on the quantization parameter initial value to generate a target quantization parameter; block encode the current frame according to the target quantization parameter to generate an encoded data stream.

[0017] The embodiment of the present application provides a MVPS video encoding method, device and medium based on deep learning, with an adaptive weight fusion layer and bidirectional feature interaction between layers to achieve dynamic parameter adjustment for video content perception; FPGA parallel feature extraction unit and deformable convolution kernel hardware acceleration design to reduce video processing delay and power consumption; motion consistency matrix and time domain smoothing constraint factor to reduce the range of fluctuation between frames, significantly reducing block effects and flickering; bandwidth feedback mechanism and rate distortion optimization module to achieve smooth quality transition when bandwidth fluctuates, cross-modal similarity matrix and geometric perception spatial feature field to support collaborative encoding of audio, depth map and video. Through dynamic parameter adjustment, hardware-algorithm collaborative optimization, cross-frame consistency enhancement and network perception technology, the problems of low efficiency, lack of real-time performance, unstable quality and poor adaptability of traditional solutions are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A flowchart of a deep learning-based MVPS video encoding method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the internal structure of an MVPS video encoding device based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] The embodiments of the present application provide a deep learning-based MVPS video encoding method, device, and medium, which solve the problems of low efficiency, insufficient hardware coordination, cross-frame quality fluctuations, and poor network adaptability of video encoding technology in the prior art.

[0023] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 This is a flow chart of a MVPS video encoding method based on deep learning provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides an MVPS video encoding method based on deep learning, which specifically includes the following steps:

[0025] Step 10: Obtain the video stream, extract the temporal and spatial features of the video stream through the feature extraction unit, and generate a joint feature map through the adaptive weight fusion layer.

[0026] As an optional embodiment, a video stream is obtained, and temporal and spatial features of the video stream are extracted through a feature extraction unit, and a joint feature map is generated through an adaptive weight fusion layer. Specifically, the method may include: step 101: deploying feature extraction units in parallel on an FPGA chip; step 102: parsing the video stream into MVPS video stream data in frames to obtain video frames; step 103: using a three-dimensional convolutional neural network to extract temporal motion features of the video frames through the feature extraction unit, wherein the optical flow calculation result is used as the initial value of the convolution weight; step 104: using a deformable convolution kernel to extract spatial texture features of the video frames, and adjusting the sampling position according to the edge gradient of the video frame; step 105: channel-splicing the temporal motion features and the spatial texture features, calculating the fusion weight of the temporal motion features and the spatial texture features, and generating a joint feature map.

[0027] In this step, the temporal and spatial feature extraction modules are deployed in parallel on an FPGA (Field-Programmable Gate Array) chip. The hardware design uses a two-way data pipeline: the first uses a three-dimensional convolution kernel to process several consecutive frames of video data to extract motion features; the second uses a deformable convolution kernel to analyze the texture details of a single frame. The feature maps output by both pipelines are cached in high-speed on-chip memory and spliced ​​in real time by an adaptive weight fusion layer.

[0028] Optionally, temporal feature extraction uses a pre-trained 3D convolutional network, with initial weights optimized using optical flow. Specifically, the optical flow fields of adjacent frames are first calculated, and the displacements are converted into initial weights for the 3D convolution kernels. For example, if the optical flow field indicates horizontal motion in a region, the horizontal weight of the corresponding convolution kernel is increased. This allows the network to quickly capture motion trends in the video.

[0029] Optionally, spatial feature extraction uses a deformable convolution kernel, whose sampling positions are dynamically adjusted based on image gradients. For an input video frame, the Sobel edge gradient map is first calculated, and an offset control vector is generated based on the gradient magnitude. For example, in areas with complex textures, such as leaves or hair, where the gradient magnitude is high, the convolution kernel's sampling points are shifted toward the edge, enhancing detail capture. This process achieves offset prediction through a two-layer fully connected network.

[0030] As an optional embodiment, channel splicing of temporal motion features and spatial texture features is performed, and the fusion weights of the temporal motion features and spatial texture features are calculated to generate a joint feature map. Specifically, the following steps may be included: Step 1051: acquiring spatial audio through a microphone array, acquiring a depth map through a ToF (Time of Flight) sensor, and parsing to obtain audio spectrum features and depth map edge features; Step 1052: aligning the audio spectrum features with the temporal motion features to obtain a cross-modal similarity matrix;

[0031] In this step, the audio features are linearly interpolated based on the video frame rate to ensure that each frame corresponds to a set of audio features. A similarity matrix is ​​constructed through cosine similarity, the matrix is ​​row-normalized, and then elements with similarity greater than a preset value are retained to form a sparse correlation matrix.

[0032] Step 1053: Fuse the depth map edge features with the spatial texture features to generate a geometrically perceived spatial feature field.

[0033] In this step, depth edge weights are introduced into the sampling offset calculation of the deformable convolution. The fused features are then mapped to a geometrically aware spatial feature field through a 3×3 convolutional layer, with the channel dimensions consistent with the original spatial features. This way, for transparent objects, the geometric constraints provided by the depth edge prevent RGB (Red Green Blue) texture misjudgments, ensuring that the feature field more closely matches the boundaries of the real object.

[0034] Step 1054: Input the cross-modal similarity matrix and feature field into the differentiable neural architecture to generate a dynamic sparsely connected spatiotemporal attention network.

[0035] In this step, the cross-modal matrix and feature field are input into the neural architecture search module. The cross-modal matrix is ​​used as the node connection weight and the feature field is used as the node feature. The spatiotemporal graph structure is initialized, and the top k strongly correlated edges are selected through Gumbel-Softmax sampling to generate an attention network that only retains important connections. The searched network structure is compiled into FPGA-executable lookup table logic for dynamic reconstruction.

[0036] Step 1055: Perform multimodal collaborative compression through the spatiotemporal attention network to generate a joint feature map.

[0037] In this step, an audio-driven attention mask is applied to the temporal motion features to suppress motion estimation errors in silent areas, such as random background jitter. In the spatial feature field, the texture features of the depth edge area are losslessly compressed, while the non-edge area is losslessly compressed. The final joint feature map is output through weighted splicing of the channel dimension.

[0038] That is to say, by extracting the feature of temporal motion features and spatial texture features, the audio and depth map are combined to generate a joint feature map, providing high-dimensional input for subsequent feature interaction.

[0039] Step 20: Perform bidirectional feature interaction between layers on the joint feature map to obtain a multi-scale feature set;

[0040] As an optional embodiment, bidirectional feature interaction is performed between layers of the joint feature map to obtain a multi-scale feature set, which may specifically include: step 201: deploying a causal Transformer encoder and a non-causal Transformer decoder between adjacent layers of the feature pyramid; step 202: constraining the preset low-level features through the causal Transformer encoder; step 203: multiplying the preset high-level semantic features and the low-level features element-by-element through the Transformer decoder to obtain an interacted feature map; step 204: applying channel attention weights to the feature map to suppress redundant feature channels and obtain a multi-scale feature set.

[0041] In this step, bidirectional Transformers are deployed between adjacent levels of the feature pyramid. The encoder operates from the bottom up, employing a causal attention mechanism that allows only low-level features to be influenced by features from previous frames, preventing future information leakage. For example, when encoding frame t, layer P3 only references features from frame t-1 and earlier. The decoder operates from the top down, multiplying high-level semantic features such as object outlines element-wise with low-level detail features such as texture, and fusing them via residual connections. For example, weighting facial regions in high-level features enhances the texture of facial features at corresponding locations in the lower layers.

[0042] Furthermore, a channel attention mechanism is applied to the interacted feature maps, and global average pooling is performed on the feature maps of each channel to obtain a channel description vector. The importance weight of each channel is calculated through a two-layer fully connected network, and channels with weights below a threshold are soft-masked to reduce redundant calculations. For example, if a channel mainly responds to flat areas such as the sky, its weight will be reduced.

[0043] In other words, the joint feature map contains multimodal information. Through the interaction of bidirectional Transformers between layers, it fuses low-level details with high-level semantics and outputs a multi-scale feature set, providing a basis for the generation of dynamic quantization parameters.

[0044] Step 30: Generate an initial value of the frame-level quantization parameter based on the multi-scale feature set and the current network bandwidth feedback.

[0045] As an optional embodiment, based on the multi-scale feature set and combined with the current network bandwidth feedback, the initial value of the frame-level quantization parameter is generated, which may specifically include: Step 301: Establishing a network bandwidth monitoring thread to obtain network bandwidth data in real time, the network bandwidth data including the TCP congestion window size and packet loss rate; Step 302: Encoding the network bandwidth data into a vector, splicing it with the multi-scale feature set, and inputting it into the gated attention network; Step 303: Generating the initial value of the quantization parameter based on the attention weight distribution so that the bit rate distribution conforms to the network transmission constraints.

[0046] In this step, a bandwidth monitoring thread is deployed at the network transport layer to obtain real-time TCP (Congestion Window) congestion window and packet loss rate. In other words, when the TCP congestion window decreases and the packet loss rate increases, the network bandwidth is determined to be decreasing, triggering a bitrate compression strategy. The bandwidth data is encoded as a 16-dimensional vector, the TCP congestion window is normalized to [0, 1], and the packet loss rate is mapped to a logarithmic scale. This vector is concatenated with multi-scale features and input into a gated recurrent unit network to predict the initial quantization parameter. When the network bandwidth decreases, the initial quantization parameter value is increased by 2-3 units to reduce the bitrate.

[0047] Furthermore, the initial quantization parameter can also be dynamically adjusted according to the complexity of different regions in the feature map. For high-complexity regions, such as those with intense motion or rich textures, a lower initial quantization parameter is used to retain details; for flat or static regions, the initial quantization parameter is appropriately increased to reduce bit allocation; in the specific implementation, the regional complexity is calculated by the variance of the joint feature map and mapped to the initial quantization parameter offset.

[0048] In other words, the above multi-scale feature set integrates semantic information at different levels, and then combines network bandwidth feedback to generate the initial value of the quantization parameter through the gated attention network.

[0049] Step 40: Perform timing consistency correction on the initial value of the quantization parameter to generate a target quantization parameter;

[0050] As an optional embodiment, after performing temporal consistency correction on the initial value of the quantization parameter and generating the target quantization parameter, the method may also include: initially dividing the video frame into coding blocks of the maximum size; extracting the motion vector field of the coding block of the previous frame and constructing a motion consistency matrix; calculating the correlation coefficient between the coding block of the current frame and the motion consistency matrix, and generating a time domain smoothing constraint factor.

[0051] In this step, QP (Quantization Parameter Smoothing) smoothing is performed using the correlation between the motion vectors of the previous and next frames. The motion vectors of the previous frame's coding blocks are extracted and a motion consistency matrix is ​​constructed. For example, if the motion direction of the current block is the same as that of the previous frame, a higher consistency score is assigned. The correlation coefficient between each coding block in the current frame and the historical motion trajectory is calculated to generate a temporal smoothing factor (0.0-1.0). In other words, if a block's motion changes suddenly, such as the sudden appearance of an object, its smoothing factor is reduced to allow for local fluctuations in the QP value.

[0052] Furthermore, QP correction is performed on areas that are sensitive to the human eye. The focus areas are extracted through a pre-trained visual saliency detection model, and the QP values ​​of these areas are constrained to a lower limit, such as QP ≤ 32, to prevent blocking effects caused by excessive compression.

[0053] In summary, the initial value of the quantization parameter is adjusted based on content complexity and network status to ensure that the bitrate allocation meets both video content requirements and adapts to real-time bandwidth changes.

[0054] Step 50: Block-encode the current frame according to the target quantization parameter to generate an encoded data stream.

[0055] As an optional embodiment, after generating the time domain smoothing constraint factor, the method may also include: performing nonlinear mapping on the initial value of the quantization parameter according to the time domain smoothing constraint factor; recursively calculating the rate-distortion cost of the coding block, and if the rate-distortion cost is higher than a preset threshold, continuing to segment the coding block according to the target quantization parameter.

[0056] In this step, the initial quantization parameters may jump between frames due to network fluctuations or sudden changes in content. Therefore, they need to be smoothed using a motion consistency matrix and visual saliency detection. The coding block division can adopt a rate-distortion optimization (RDO) strategy. Assume that the rate-distortion cost J is initially calculated using 64×64 blocks. If J exceeds a threshold, for example, due to increased distortion caused by complex textures, the block is split into four 32×32 sub-blocks and re-evaluated. This recursive process is repeated until the minimum block size of 8×8 is reached, ensuring fine coding of complex areas.

[0057] As an optional embodiment, after dividing the current frame into coding blocks according to the target quantization parameter and generating the encoded data stream, the method may also include: inputting the coding blocks into a rate-distortion optimization module, iteratively optimizing the coding block division pattern through a rate-distortion cost function; performing entropy encoding on the optimized coding blocks to generate a final compressed code stream; during the encoding process, monitoring the complexity of the video content in real time to update the weight parameters of the bit rate allocation.

[0058] In this step, the target quantization parameter further considers content complexity, network status, and temporal smoothness. Rate-distortion optimization is used to recursively segment the coding blocks, achieving refined coding of complex regions. The final coded data stream is compressed via entropy coding, with the bitrate distribution aligned with the network bandwidth in real time while ensuring subjective quality stability. Context-adaptive binary arithmetic coding is employed during the entropy coding stage, and a probability model is dynamically updated based on the statistical characteristics of the feature maps. Motion vectors are differentially coded, and a prediction context is constructed using historical MVs. Residual coefficients are grouped by texture complexity, using different Huffman tables for each group.

[0059] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides an MVPS video encoding device based on deep learning, whose structure is as follows Figure 2 shown.

[0060] Figure 2 This is a schematic diagram of the internal structure of an MVPS video encoding device based on deep learning provided in an embodiment of the present application. Figure 2 As shown, the equipment includes:

[0061] at least one processor 201;

[0062] and, a memory 202 communicatively coupled to the at least one processor;

[0063] Among them, the memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 so that the at least one processor 201 can: obtain a video stream, perform time domain and spatial domain feature extraction on the video stream through a feature extraction unit, and generate a joint feature map through an adaptive weight fusion layer; perform bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; based on the multi-scale feature set, combined with current network bandwidth feedback, generate a frame-level quantization parameter initial value; perform temporal consistency correction on the quantization parameter initial value to generate a target quantization parameter; block encode the current frame according to the target quantization parameter to generate an encoded data stream.

[0064] Some embodiments of the present application provide corresponding Figure 1A non-volatile computer storage medium for MVPS video coding based on deep learning stores computer-executable instructions, wherein the computer-executable instructions are configured to: obtain a video stream, extract temporal and spatial features of the video stream through a feature extraction unit, and generate a joint feature map through an adaptive weight fusion layer; perform bidirectional feature interaction between layers of the joint feature map to obtain a multi-scale feature set; generate an initial value of a frame-level quantization parameter based on the multi-scale feature set and in combination with current network bandwidth feedback; perform temporal consistency correction on the initial value of the quantization parameter to generate a target quantization parameter; and perform block encoding on the current frame based on the target quantization parameter to generate an encoded data stream.

[0065] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the IoT device and media embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0066] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0067] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0071] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0072] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0073] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0074] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0075] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A MVPS video coding method based on deep learning, characterized in that: The method comprises: Obtain a video stream, extract temporal and spatial features of the video stream through a feature extraction unit, and generate a joint feature map through an adaptive weight fusion layer, specifically including: Deploying the feature extraction units in parallel on an FPGA chip; Parsing the video stream into MVPS video stream data in frames to obtain video frames; Extracting temporal motion features of the video frames using a three-dimensional convolutional neural network through the feature extraction unit, wherein the optical flow calculation result is used as the initial value of the convolution weight; Extracting spatial texture features of the video frame using a deformable convolution kernel, and adjusting a sampling position according to an edge gradient of the video frame; Perform channel splicing on the temporal motion features and the spatial texture features, calculate the fusion weight of the temporal motion features and the spatial texture features, and generate the joint feature map; Performing bidirectional feature interaction between levels on the joint feature map to obtain a multi-scale feature set; Generating an initial value of a frame-level quantization parameter based on the multi-scale feature set and current network bandwidth feedback; Performing timing consistency correction on the initial value of the quantization parameter to generate a target quantization parameter; The current frame is block-encoded according to the target quantization parameter to generate an encoded data stream.

2. The MVPS video encoding method based on deep learning according to claim 1, characterized in that: Channel-joining the temporal motion features and the spatial texture features, calculating the fusion weights of the temporal motion features and the spatial texture features, and generating the joint feature map specifically includes: Acquire spatial audio through a microphone array, obtain depth maps through a ToF sensor, and analyze the audio spectrum features and depth map edge features; Aligning the audio spectrum features with the time domain motion features to obtain a cross-modal similarity matrix; Fusing the depth map edge features with the spatial texture features to generate a geometrically perceived spatial feature field; Inputting the cross-modal similarity matrix and feature field into a differentiable neural architecture to generate a dynamic sparsely connected spatiotemporal attention network; Multimodal collaborative compression is performed through the spatiotemporal attention network to generate a joint feature map.

3. The MVPS video encoding method based on deep learning according to claim 1, characterized in that: Performing bidirectional feature interaction between layers on the joint feature map to obtain a multi-scale feature set, specifically including: Deploy causal Transformer encoders and non-causal Transformer decoders between adjacent levels of the feature pyramid; Constraining the preset low-level features through the causal Transformer encoder; Through the Transformer decoder, the preset high-level semantic features are multiplied element-by-element with the low-level features to obtain an interactive feature map; Channel attention weights are applied to the feature map to suppress redundant feature channels and obtain a multi-scale feature set.

4. The MVPS video encoding method based on deep learning according to claim 1, characterized in that: Based on the multi-scale feature set and in combination with current network bandwidth feedback, an initial value of a frame-level quantization parameter is generated, specifically including: Establish a network bandwidth monitoring thread to obtain network bandwidth data in real time, including TCP congestion window size and packet loss rate; Encoding the network bandwidth data into a vector, concatenating it with the multi-scale feature set, and inputting it into a gated attention network; The initial value of the quantization parameter is generated according to the attention weight distribution so that the bit rate distribution conforms to the network transmission constraints.

5. The MVPS video encoding method based on deep learning according to claim 1, characterized in that: After performing timing consistency correction on the initial value of the quantization parameter to generate the target quantization parameter, the method further includes: Initially dividing the video frame into coding blocks of maximum size; Extracting the motion vector field of the coding block of the previous frame and constructing a motion consistency matrix; The correlation coefficient between the coding block of the current frame and the motion consistency matrix is ​​calculated to generate a temporal smoothing constraint factor.

6. The MVPS video encoding method based on deep learning according to claim 5, characterized in that: After generating the time domain smoothing constraint factor, the method further includes: Performing nonlinear mapping on the initial value of the quantization parameter according to the time domain smoothing constraint factor; The rate-distortion cost of the coding block is recursively calculated, and if the rate-distortion cost is higher than a preset threshold, the coding block is further segmented according to the target quantization parameter.

7. The MVPS video encoding method based on deep learning according to claim 4, characterized in that: After dividing the current frame into coding blocks according to the target quantization parameter to generate a coded data stream, the method further includes: Input the coding block into the rate-distortion optimization module, and iteratively optimize the coding block division through the rate-distortion cost function; Perform entropy coding on the optimized coding blocks to generate the final compressed code stream; During the encoding process, the complexity of the video content is monitored in real time to update the weight parameters of the bit rate allocation.

8. A MVPS video encoding device based on deep learning, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least a processor to execute so that the at least one processor is capable of: Execute the steps of a deep learning-based MVPS video encoding method as described in any one of claims 1-7.

9. A non-volatile computer storage medium for MVPS video coding based on deep learning, storing computer-executable instructions, characterized in that: The computer executable instructions are configured to: Execute the steps of a deep learning-based MVPS video encoding method as described in any one of claims 1-7.

Images