Multifunctional video coding method and decoding method based on feature super-resolution technology

Abstract

The invention discloses a multifunctional video coding method and decoding method based on a feature super-resolution technology, and relates to the technical field of machine video coding and decoding. According to the invention, the feature super-resolution technology is applied to the video coding and decoding process under the VVC standard, and the low-resolution image or video is improved to the high-resolution level through the feature super-resolution algorithm, so that more detailed and accurate image information is provided for a VCC encoder, the compression efficiency in the coding process can be remarkably improved, and the coding efficiency is improved. Therefore, video transmission with higher quality is realized under the same bit rate, and the real-time performance and the stability of machine video coding are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine video encoding and decoding, and in particular to a multifunctional video encoding method and a decoding method based on feature super-resolution technology. Background Art

[0002] Video data has become one of the most important sources of information in global consumption. The current society is increasingly showing a trend of multimedia centralization, and most video traffic will be processed by machines. Video Coding for Machine (VCM) makes metadata machine-readable, searchable and operable by identifying, classifying and indexing objects in video content, and has broad application prospects in the future.

[0003] Traditionally, video coding uses the characteristics of the human visual system to improve compression efficiency while maintaining good subjective quality. Popular video coding standards such as MPEG2, H.264 / SVC, H.265 / HEVC, and the latest H.266 / VVC all follow this design principle. The Versatile Video Coding (VVC) standard can significantly reduce the bit rate of video data without losing visual quality. Therefore, under the same bandwidth conditions, higher quality video content can be transmitted, or bandwidth and storage space can be saved at the same quality.

[0004] Although VCM under the VVC standard performs well in compressing standard resolution videos, it still faces the following problems in practical applications: First, the computational complexity is high. VCM usually involves complex deep learning models, which require a lot of computing resources and time for training and inference. At the same time, the encoding and decoding process involves complex algorithms and a large amount of computing resources, making real-time video encoding and decoding more difficult, resulting in limited real-time performance and device compatibility in some application scenarios; second, when faced with different scenarios and content types, its performance may not be as stable and reliable as traditional encoding methods. Summary of the invention

[0005] The present invention provides a multifunctional video encoding method and a decoding method based on feature super-resolution technology to solve the problem that the real-time and stability of machine video encoding are insufficient, which affects its application in certain scenarios.

[0006] The present invention is achieved through the following technical solutions:

[0007] A first aspect of the present invention provides a multifunctional video encoding method based on feature super-resolution technology, comprising:

[0008] Extract super-resolution features from raw video data;

[0009] Downsampling, feature rearrangement and quantization processing are performed on the super-resolution features to obtain conversion features;

[0010] Encoding the conversion feature according to a multifunctional video coding standard to obtain a feature encoding bit stream;

[0011] The original video data and the characteristic coding bit stream are mixed and coded to obtain a video coding bit stream.

[0012] In view of the high computational complexity of VCM under the VVC standard and its limitations in scenarios such as huge video data volume and high transmission real-time requirements, the present invention proposes a multifunctional video coding method based on feature super-resolution technology, and applies the feature super-resolution technology to video coding and decoding under the VVC standard. First, a low-resolution image or video is upgraded to a high-resolution level through a feature super-resolution algorithm, thereby providing the VCC encoder with more detailed and accurate image information, avoiding a complex deep learning process, and super-resolution feature extraction has no type restriction on the original video data. This information can significantly improve the compression efficiency in the coding process, thereby achieving higher quality video transmission at the same bit rate, and improving the real-time performance and stability of the coding.

[0013] In some implementations, extracting super-resolution features of original video data includes:

[0014] Extracting features from the original video data using a shallow feature extraction network to obtain shallow features;

[0015] Extracting the shallow features through multiple multi-scale feature reuse hybrid attention networks to obtain multiple deep features;

[0016] A plurality of the deep features are fused through a feature fusion network to obtain super-resolution features.

[0017] In some implementations, the shallow features are extracted by using multiple multi-scale feature reuse hybrid attention networks to obtain multiple deep features, including:

[0018] For the first M / 2+1 multiple multi-scale feature reused hybrid attention networks, the output features of the i-1th multi-scale feature reused hybrid attention network are used as the input features of the i-th multi-scale feature reused hybrid attention network to obtain the i-th deep feature, which is expressed as:

[0019] F i =G i (F i-1 ), 1≤i <M / 2+1

[0020] Among them, M represents the number of multi-scale feature reuse hybrid attention networks, i represents the sequence number of multi-scale feature reuse hybrid attention networks, G i (·) represents the feature extraction function of the i-th multi-scale feature reuse hybrid attention network, F i represents the output feature of the i-th multi-scale feature reuse hybrid attention network, i.e., the i-th deep feature, F i-1 represents the output feature of the i-1th multi-scale feature reuse hybrid attention network. The input feature F0 of the first multi-scale feature reuse hybrid attention network is a shallow feature.

[0021] For the next M / 2+1 multi-scale feature reused hybrid attention networks, the output features of the i-1th and M-i+1th multi-scale feature reused hybrid attention networks are used as the input features of the i-th multi-scale feature reused hybrid attention network to obtain the i-th deep feature, which is expressed as:

[0022] F i =G i (F i-1 +F M-i+1 ), M / 2+1 <i≤M

[0023] Among them, G i (·) represents the feature extraction function of the i-th multi-scale feature reuse hybrid attention network, F i-1 represents the output feature of the i-1th multi-scale feature reuse hybrid attention network, F M-i+1 Represents the output features of the M-i+1th multi-scale feature reuse hybrid attention network.

[0024] In some embodiments, the multi-scale feature reuse hybrid attention network includes a multi-scale feature reuse module and a hybrid attention module;

[0025] The multi-scale feature reuse module adopts four-way convolution processing, each convolution processing is set with convolution kernels of different sizes and numbers, and each convolution processing is connected through short jumps. The output expression of the multi-scale feature reuse module is as follows:

[0026] f out =G conv1 (G link [f1',f2',f3',f4'])+f0

[0027] Among them, f0 is the input feature of the multi-scale feature reuse module, f out represents the output features of the multi-scale feature reuse module, G link (·) represents the channel concatenation function, which concatenates the four output features f1′, f2′, f3′, and f4′. conv1(·) represents a 1×1 convolution function, f1, f2, f3, and f4 are the outputs after four-way convolution processing;

[0028] The output of the first convolution process is: f1 = G conv1 (f0);

[0029] The output of the second convolution process is: f2 = G conv3 (f1+f0);

[0030] The output of the third convolution process is: f3 = G conv3 (G ReLU (G conv3 (f2+f0)));

[0031] The output of the fourth convolution process is: f4 = G conv3 (G ReLU (G conv3 (G ReLU (G conv3 (f3+f0))))), G conv3 (·) represents the 3×3 convolution function, G ReLU (·) represents the ReLU activation function;

[0032] The hybrid attention module includes a 5×5 depth-separable convolutional layer and a 1×1 convolutional layer. The output expression of the hybrid attention module is as follows:

[0033] f o ' ut =f out ×(G sigm (G conv1 (G ReLU (G d-conv5 (f out )))))

[0034] Among them, f o ' ut represents the output of the hybrid attention module, G sigm (·) represents the Sigmoid function, G d-conv5 (·) represents a 5×5 depthwise separable convolution function.

[0035] In some embodiments, the feature fusion network includes a channel series layer and a 1×1 convolution layer, the channel series layer performs channel series splicing on multiple deep features to obtain splicing features, and the 1×1 convolution layer performs dimensionality reduction on the splicing features to obtain fusion features.

[0036] In some embodiments, the shallow feature extraction network is a 3×3 convolutional layer.

[0037] A second aspect of the present invention provides a multifunctional video decoding method based on feature super-resolution technology, comprising:

[0038] Decode the video encoding bit stream to obtain the feature encoding bit stream and the original video data.

[0039] Decoding the feature encoded bit stream according to a multifunctional video decoding standard to obtain a decoding feature;

[0040] Dequantizing and rearranging the decoded features to obtain reconstructed features;

[0041] Performing super-resolution reconstruction on the reconstructed features to obtain a high-resolution feature image;

[0042] Decoding the high-resolution feature image and the original video data according to a multifunctional video decoding standard to obtain a high-resolution video;

[0043] The video encoding bit stream is encoded by the multifunctional video encoding method based on feature super-resolution technology described in any one of the embodiments of the first aspect of the present invention.

[0044] In some embodiments, performing super-resolution reconstruction on the reconstructed features to obtain a high-resolution feature image includes:

[0045] Convolutionally processing the reconstructed features through a 3×3 convolutional layer to obtain convolutional features;

[0046] The convolutional features are upsampled through a pixel rearrangement layer to obtain a high-resolution feature image.

[0047] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multifunctional video encoding method based on the feature super-resolution technology described in any one of the embodiments of the first aspect of the present invention and / or the multifunctional video decoding method based on the feature super-resolution technology described in any one of the embodiments of the second aspect of the present invention is implemented.

[0048] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the multifunctional video encoding method based on the feature super-resolution technology described in any one of the embodiments of the first aspect of the present invention and / or the multifunctional video decoding method based on the feature super-resolution technology described in any one of the embodiments of the second aspect of the present invention is implemented.

[0049] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0050] Feature super-resolution technology can enhance low-resolution images or videos to high-resolution levels, which can more accurately preserve image details, avoid unnatural effects caused by over-processing, encode high-resolution features, and improve compression efficiency during the encoding process. It can also perform mixed encoding of the original video frame and the bit stream obtained by feature encoding to improve the compression rate and reconstruction quality of the original video frame.

[0051] The super-resolution algorithm model adopts a convolutional neural network combined with a multi-scale feature reuse hybrid attention network. It takes advantage of the structural advantages of the convolutional neural network and reuses features at multiple scales through a multi-scale feature reuse hybrid attention network to enhance the understanding and reconstruction capabilities of the video content, ensuring the output image quality and the fidelity of visual details.

[0052] Through downsampling, feature arrangement and quantization processing, the original feature data is optimized, the spatial size of the feature is reduced, and a lower bit rate is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0054] Figure 1 It is a flowchart of a multifunctional video encoding method based on feature super-resolution technology according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of a model structure of a super-resolution algorithm model according to an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of a coding and decoding method using a super-resolution algorithm according to an embodiment of the present invention;

[0057] Figure 4 is a structural schematic diagram of a multi-scale feature multiplexing block in an embodiment of the present invention;

[0058] Figure 5 is a structural schematic diagram of a hybrid attention module according to an embodiment of the present invention;

[0059] Figure 6 It is a flowchart of a multifunctional video decoding method based on feature super-resolution technology according to an embodiment of the present invention;

[0060] Figure 7It is a schematic diagram of a complete encoding and decoding process based on feature super-resolution technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0062] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to or inherent to other steps or units of the device.

[0063] The terms used in the various embodiments of the application are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments of the application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise limited, all terms used here (including technical terms and scientific terms) have the same meaning as the meanings commonly understood by ordinary technicians in the field of the various embodiments of the application. The terms (such as the terms defined in the dictionary generally used) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the application.

[0064] Benefiting from the integration of emerging technologies (such as 5G, artificial intelligence, smart sensors, the Internet of Things, and connected autonomous vehicles), the future will see higher and higher requirements for the transmission and processing of massive video data in the fields of smart cities, autonomous driving, smart industry, intelligent transportation systems, immersive entertainment, etc. At the same time, more and more edge devices are able to capture video signals and transmit them through the Internet or private networks, which are ultimately analyzed and consumed by machines.

[0065] The emergence of the Versatile Video Coding (VVC) standard marks an important step forward in video coding technology in terms of compression efficiency and visual quality. Compared with its predecessor H.265 / HEVC, VCC has the flexibility to support a variety of application scenarios from ultra-high definition (UHD) to panoramic video (VR / AR), making it suitable for the transmission and storage needs of various visual data in the future. In addition, VVC was designed with machine learning application requirements in mind, such as the ability to optimize compression effects through more accurate object recognition and motion estimation, making it perform better in machine vision and artificial intelligence applications. In general, with the popularization of 5G networks and the increase in IoT devices, the demand for efficient video transmission will continue to grow. As an advanced video coding standard, VVC can effectively respond to the challenges of these emerging technologies to video data processing and transmission capabilities. It is not only a technological advancement in the field of video coding, but also an important foundation for supporting the development of future video applications and services. By improving compression efficiency and supporting diverse video content and application scenarios, VVC is expected to push video technology forward by a big step, providing users with a better and more efficient visual experience.

[0066] At present, although the VVC standard and machine video coding (VCM) perform well in compressing standard resolution videos, they face the following key problems in the current situation where the public has increasingly higher requirements for video viewing experience. (1) High computational complexity: Although the VVC standard has made significant progress in improving video quality and compression efficiency, because VCM usually involves complex deep learning models, these models require a lot of computing resources and time for training and inference. At the same time, the encoding and decoding process involves complex algorithms and a lot of computing resources, which makes real-time video encoding and decoding more difficult, resulting in limited real-time performance and device compatibility in some application scenarios; (3) Poor versatility and applicability: It may work well for specific types of video content, but when facing different scenarios and content types, its performance may not be as stable and reliable as traditional encoding methods. (3) Difficulty in matching codecs: Codecs need to maintain consistent model architecture and parameter settings, otherwise the decoder may not be able to correctly decode the video stream generated by the encoder, which increases the complexity of implementation and deployment.

[0067] In view of this, the present invention proposes a multifunctional video encoding method and a decoding method based on feature super-resolution technology, which combine feature super-resolution with machine video encoding based on a multifunctional video coding standard, provide key technologies for the combination of super-resolution networks and the latest multifunctional video codecs, effectively respond to the challenges of video processing and transmission brought about by the rapid development of machine vision and artificial intelligence, and provide users with more flexible and efficient video content transmission.

[0068] Super-resolution technology is a key image processing technology that can improve low-resolution images or videos to high-resolution levels through algorithms. The application of this technology is not limited to improving the visual experience, but also plays an important role in many fields such as medical imaging, surveillance systems, and satellite images. Feature super-resolution technology is an advanced form of super-resolution technology. It uses deep learning models to learn image features from a large amount of image data, thereby effectively enhancing the resolution of the image. Compared with traditional methods, feature super-resolution technology can more accurately preserve image details and avoid unnatural effects caused by over-processing. Therefore, it performs well in producing high-quality high-resolution images.

[0069] Combining feature super-resolution technology with VCM has several key advantages. First, feature super-resolution technology can provide the encoder with more detailed and accurate image information, which can significantly improve the compression efficiency during the encoding process, thereby achieving higher quality video transmission at the same bit rate. Second, through the application of machine learning in object recognition, motion estimation, etc., feature super-resolution technology can further optimize the encoding of video content and enhance the naturalness and realism of the visual experience. The combination of feature super-resolution technology and VCM is not only a technological advancement, but also a response to the needs of future video applications. With the popularization of 5G networks and the increase in IoT devices, the demand for efficient video transmission is growing. As an efficient video coding standard, VVC can work with feature super-resolution technology to push video technology forward by a big step. Whether it is ultra-high-definition video, panoramic video or virtual reality application, this combination can bring users a better and more efficient visual experience, laying an important foundation for the development of future video applications and services.

[0070] like Figure 1 As shown, Figure 1 It is a flowchart of a multifunctional video encoding method based on feature super-resolution technology according to an embodiment of the present invention, which includes the following steps.

[0071] S1, extracts super-resolution features of original video data.

[0072] S2, performs feature conversion on the super-resolution feature to obtain the conversion feature.

[0073] The purpose of this step is to optimize the extracted super-resolution features. Feature conversion includes downsampling, feature permutation and quantization. Downsampling reduces the spatial size of the features and achieves a lower bit rate.

[0074] In a preferred embodiment of the present invention, a dual-triple interpolation downsampling method is used to achieve a lower bit rate.

[0075] The purpose of feature rearrangement processing is to arrange the extracted depth features in time for subsequent compression processing. In a preferred embodiment of the present invention, a time arrangement method based on time similarity is adopted.

[0076] The range of super-resolution feature values ​​extracted by the super-resolution algorithm model covers positive and negative numbers. In order to use the traditional VVC compressor for compression, the feature value range must be an integer value from 0 to 255. Therefore, the features are quantized and the range of feature values ​​is adjusted to an integer interval from 0 to 255 to achieve the minimum error boundary. The quantization method can be used to analyze the value distribution or use k-means clustering.

[0077] In a preferred embodiment of the present invention, a uniform quantization method with equal step length in each interval is adopted to achieve minimum performance loss by mapping the feature value to an integer value with equal step length in each interval. Uniform quantization is defined as follows:

[0078]

[0079] Among them, F q represents the quantized eigenvalue, Round(·) is the quantization function, which rounds the eigenvalue to the nearest quantization level, F represents the original eigenvalue, t represents the quantized bit depth, V min and V max are the minimum and maximum values ​​of the entire feature map respectively.

[0080] S3, encoding the transformed features according to the multifunctional video coding standard to obtain a feature coding bit stream.

[0081] S4, hybrid encoding the original video data and the feature encoding bit stream to obtain a video encoding bit stream.

[0082] The multifunctional video standard encoding may adopt a VCC standard encoder, such as VVC EnC. The feature bit stream encoded by VCC and the original video data, that is, the low-resolution video features from which the super-resolution features are extracted, are then VCC encoded to obtain a hybrid encoded bit stream of the feature encoded bit stream and the original video frame, so as to improve the compression rate and reconstruction quality of the original video frame. The hybrid encoding may adopt the same VCC standard encoder as in step S3, such as VVC EnC.

[0083] The embodiment of the present invention further provides a super-resolution algorithm model, which is used to extract super-resolution features in step S1.

[0084] The super-resolution algorithm model of the present invention is based on a convolutional neural network (CNN), combined with a multi-scale feature reuse mixed attention block (MSAB), which aims to convert the input low-resolution feature map into high-quality high-resolution features. The model takes advantage of the structural advantages of CNN and reuses features at multiple scales through the MSAB module to enhance the understanding and reconstruction capabilities of video content. The super-resolution algorithm model is named Multi-Scale Feature Reuse Mixed Attention Video Feature Super-Resolution Network (MSAB Video Feature Super Resolution Network, MSAB-VFSRN), hereinafter referred to as "MSAB-VFSRN".

[0085] like Figure 2 The figure shows the model structure diagram of the super-resolution algorithm model (MSAB-VFSRN). MSAB-VFSRN consists of a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and a feature reconstruction module. It can be divided into two parts: the super-resolution feature extraction process and the feature reconstruction process.

[0086] like Figure 3 The schematic diagram of the encoding and decoding method using MSAB-VFSRN is shown in the figure. The super-resolution feature extraction process applies the shallow feature extraction module, the deep feature extraction module and the feature fusion module. The super-resolution feature extraction extracts super-resolution features from low-resolution features. The shallow feature extraction and deep feature extraction parts are responsible for extracting multi-level feature representations from the input low-resolution features. The global feature fusion part integrates these features and strengthens the global consistency. This part is applied to the feature encoding process of the original video.

[0087] The feature reconstruction process utilizes a feature reconstruction module, and the feature reconstruction utilizes the extracted high-resolution features to reconstruct high-resolution video, which is applied to the decoding process of the coded features.

[0088] MSAB-VFSRN is designed to achieve significant performance improvement in video super-resolution tasks through effective feature extraction, reuse and integration, while ensuring the output image quality and fidelity of visual details.

[0089] When extracting the super-resolution features of the original video data, the original video data is extracted through a shallow feature extraction network to obtain shallow features; then, multiple deep features are obtained by extracting the shallow features through multiple multi-scale feature reuse hybrid attention networks; finally, the multiple deep features are fused through a feature fusion network to obtain super-resolution features.

[0090] For the given low-resolution feature F of the input, the shallow feature extraction module LR , through shallow feature extraction, can obtain the shallow feature F SF , and its expression is as follows.

[0091] F SF = G SFE (F LR )

[0092] Among them, F LR is the low-resolution feature input to the shallow feature extraction module, that is, the original video data after sampling, and G SFE (·) represents the shallow feature extraction function, and F SF is the extracted shallow feature.

[0093] In one implementation, a 3×3 convolutional layer is used as the shallow feature extraction module, then the shallow feature F SF is expressed as: F SF = cov 3×3 (F LR )

[0094] In some implementations, the deep feature extraction module is composed of M multi-scale feature reuse hybrid attention modules (MSABs), with the shallow feature as the input. The output feature of the previous MSAB module is reused by the subsequent MSAB module for deep feature extraction. Let F i represent the output of the i-th MSAB, and its processing process can be expressed as follows.

[0095] For the first M / 2 + 1 MSAB modules, that is, when 1 ≤ i < M / 2 + 1, the output feature of the (i - 1)-th multi-scale feature reuse hybrid attention network is used as the input feature of the i-th multi-scale feature reuse hybrid attention network to obtain the i-th deep feature, and its expression is:

[0096] F i = G i (F i-1 )

[0097] Among them, G i (·) represents the feature extraction function of the i-th MSAB. Since each MSAB is the same, each G i (·) is also the same, and Fi denotes the output of the \(i\)-th MSAB, \(F\) i-1 denotes the output of the \((i - 1)\)-th MSAB and is also the input of the \(i\)-th MSAB. The input \(F_0\) of the first MSAB is the shallow feature \(F\). SF .

[0098] For the latter \(M / 2 + 1\) MSAB modules, that is, when \(M / 2+1 < i\leq M\), \(F\) i-1 and \(F\) M-i-1 \(F\) M-i+1 are passed as input features to the \(i\)-th MSAB module, and its expression is:

[0099] \(F\) i = \(G\) i (\(F\) i-1 +\(F\) M-i+1 )

[0100] where \(G\) i (·) represents the feature extraction function of the \(i\)-th multi-scale feature reuse hybrid attention network, \(F\) i \(^{-1}\) represents the output feature of the \((i - 1)\)-th multi-scale feature reuse hybrid attention network, and \(F\) M-i+1 represents the output feature of the \((M - i + 1)\)-th multi-scale feature reuse hybrid attention network.

[0101] Taking the depth extraction module including 5 MSABs as an example, substituting \(M = 5\), \(M / 2+1 = 3.5\). That is, for the first 3 MSABs, \(F\) i = \(G\) i (\(F\) i-1 ) is used for depth feature extraction to obtain \(F_1 = G_1(F\) SF ), \(F_2 = G_2(F_1)\), \(F_3 = G_3(F_2)\). For the subsequent 4th and 5th MSABs, \(F\) i = \(G\) i (\(F\) i-1 +\(F\) M-i+1 ) is used for depth feature extraction to obtain \(F_4 = G_4(F\) 3+ \(F_2\)), \(F_5 = G_5(F\) 4+ \(F_1\)).

[0102] It should be noted that when the depth extraction module includes 2 MSABs, substituting \(M = 5\), then \(M / 2 + 1 = 2\). That is, only the first feature extraction formula is used. Therefore, to ensure that more scale features are extracted, \(M\) is preferably 3 or more.

[0103] In some embodiments, a global feature fusion module is used to utilize the output features of all MSABs for cross-layer connection. The global feature fusion module uses a series layer and a convolution layer of size 1×1. The series layer is used to fuse the outputs of multiple MSAB modules in series, and the 1×1 convolution layer is used to reduce the dimension of the fused features, thereby eliminating redundancy and reducing computational costs. The processing process is expressed as follows.

[0104] F GF =G conv1 (G link [F M ,F M-1 ,...,F1])

[0105] Among them, F GF represents the output of the global feature fusion module, i.e., the fusion feature of the deep features, G conv1 (·) represents a 1×1 convolution function, G link (·) represents the channel series function, F1~F M is the output of M MSAB modules.

[0106] The deeper features in CNN have larger receptive fields and rich semantic information, but they have relatively few geometric details. In contrast, the shallow features in CNN have smaller receptive fields but rich geometric information, which can make up for the shortcomings of deep features and better utilize the geometric and semantic information in image features through global feature fusion.

[0107] Each MSAB module contains a multi-scale feature reuse block (MB) and a mixed attention (MA) mechanism.

[0108] See also Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a multi-scale feature reuse block (MB). In order to better capture feature information of different scales, we use convolution kernels of different sizes and numbers in each path, and add short jump connections to the diagram to further expand the network's perception field of view and increase the reuse of features of different scales. Taking four paths as an example, when the input image feature data enters the MB, the model can be described as:

[0109]

[0110] Where f0 represents the input of MSAB, G conv1 (·) represents a 1×1 convolution function, G conv3 (·) represents the 3×3 convolution function, G ReLU(·) represents the ReLU activation function, and f1, f2, f3, and f4 are the outputs of the four-way convolution process.

[0111] After passing through the four paths of the MB structure, the input features of each path are first fused with the calculated features through short jump connections. Subsequently, the channels of the four feature maps are concatenated together using a concatenation operation, and then the features are fused through a 1×1 convolutional layer, and the number of channels is restored to the number at the input. The specific process is as follows:

[0112]

[0113] The final input of MB is expressed as:

[0114] f out =G conv1 (G link [f1',f2',f3',f4'])+f0

[0115] Among them, G link (·) represents the channel concatenation function, G conv1 (·) represents a 1×1 convolution function, and f0 represents the input of MSAB.

[0116] See also Figure 5 As shown, Figure 5 It is a schematic diagram of the structure of the hybrid attention module (MA). First, a 5×5 depthwise separable convolution layer (DC) is used to extract the feature dependencies in each channel space, and then a 1×1 convolution layer is used to learn the correlation between each channel.

[0117] The MA we use here combines channel-wise attention and spatial attention (SA) on one channel, which can extract channel and spatial feature relationships respectively, simplifying the network and saving computational costs while achieving good attention effects. The model can be described as:

[0118] f o ' ut =f out ×(G sigm (G conv1 (G ReLU (G d-conv5 (f out )))))

[0119] In the formula, f o ' ut Represents the output of the MA module, which is also the output of the entire MSAB. sigm(·) represents the Sigmoid function, G d-conv5 (·) represents a 5×5 depthwise separable convolution function.

[0120] In some embodiments, the feature reconstruction part is completed by a high-resolution image feature reconstruction module, which consists of a 3×3 convolution layer and a pixel shuffle (Pixel Shuffle, PS) layer. Inspired by the Efficient Sub-Pixel Convolutional Neural Network (ESPCN), the pixel shuffle layer is used as an upsampling filter to obtain the deep feature F output from the global fusion part. SFGF First, it passes through a 3×3 convolutional layer to obtain the number of channels suitable for the pixel rearrangement layer, and then passes through the pixel rearrangement layer to generate the reconstructed high-resolution feature F HR . Its expression is as follows.

[0121] F HR =PS(G conv3 (F SFGF ))+BC(F LR )

[0122] Among them, F SFGF is the input feature of the high-resolution image feature reconstruction module, which is the shallow feature F SF The fusion feature F of the deep feature GF The addition of F SFGF =F SF +F GF . G conv3 (·) represents the 3×3 convolution function, PS(·) represents the pixel rearrangement function, and BC(·) represents the upsampling bicubic interpolation.

[0123] The embodiment of the present invention also provides a corresponding decoding method, see Figure 6 , Figure 7 As shown, Figure 6 It is a flowchart of a multifunctional video decoding method based on feature super-resolution technology. Figure 7 It is a complete schematic diagram of the encoding and decoding process. The decoding method is used to decode the video encoding bit stream obtained by the multifunctional video encoding method based on the feature super-resolution technology. It is the inverse process of the encoding process and includes the following steps.

[0124] S210, decoding the video encoding bit stream to obtain a feature encoding bit stream and original video data.

[0125] S220, decoding the feature coded bit stream by a multifunctional video standard decoder to obtain a decoded feature.

[0126] S230, dequantizing and re-arranging the decoded features to obtain reconstructed features.

[0127] S240, performing super-resolution reconstruction on the reconstructed features to obtain a high-resolution feature image.

[0128] S250, decoding the reconstructed high-resolution feature image and the original video data by a multifunctional video standard decoder to obtain a high-resolution video.

[0129] The video encoding bitstream is encoded by the multifunctional video encoding method based on the feature super-resolution technology of any of the above embodiments of the present invention. The video encoding bitstream is a mixture of the feature encoding bitstream of the super-resolution feature and the bitstream of the original video frame, so the video encoding bitstream is first decoded into the feature encoding bitstream and the original video data.

[0130] For the feature coding bit stream, the high-resolution feature image is reconstructed through the processing of steps S220 to S240. First, the feature coding bit stream is decoded by a multifunctional video standard decoder, such as a VVC DeC decoder. The dequantization process is the inverse process of the quantization process in step S2, and then the decoded features are restored to super-resolution features through feature rearrangement. Finally, the restored super-resolution features are super-reconstructed.

[0131] In some embodiments, super-resolution reconstruction uses the feature reconstruction part of the MSAB-VFSRN model of the present invention to reconstruct the super-resolution features into a high-resolution feature image through a feature reconstruction module.

[0132] The feature reconstruction method specifically includes: performing 3×3 convolution processing on the reconstructed features to obtain a number of channels suitable for the pixel rearrangement layer; passing the convolution features through the pixel rearrangement layer to generate a reconstructed high-resolution feature F HR .

[0133] In the encoding and decoding process of the invention, a consistent codec is adopted, and the same image feature extraction module is adopted, that is, the encoding end uses the super-resolution feature extraction part in MSAB-VFSRN, and the decoding end uses the super-resolution feature extraction part and feature reconstruction part in MSAB-VFSRN, which ensures the consistency of the model architecture and parameter settings and simplifies the implementation and deployment of the codec.

[0134] An embodiment of the present invention also provides an electronic device, which includes a processor, a memory, an input device, an output device and a communication device; the number of processors in the computer device can be one or more, and the processor, memory, input device and output device in the electronic device can be connected via a bus or other means.

[0135] The memory is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory to implement the multifunctional video encoding method based on feature super-resolution technology and / or the multifunctional video decoding method based on feature super-resolution technology of any of the above embodiments of the present invention.

[0136] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a multifunctional video encoding method based on feature super-resolution technology and / or a multifunctional video decoding method based on feature super-resolution technology according to any embodiment of the present invention is implemented. The storage medium may be a ROM / RAM, a magnetic disk, an optical disk, etc.

[0138] A fifth aspect of the present invention provides a computer program product, which, when running on a computer, enables the computer to execute a multifunctional video encoding method based on feature super-resolution technology and / or a multifunctional video decoding method based on feature super-resolution technology according to any of the above-mentioned embodiments of the present invention.

[0139] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multifunctional video encoding method based on feature super-resolution technology, characterized in that: include: Extract super-resolution features from raw video data; Downsampling, feature rearrangement and quantization processing are performed on the super-resolution features to obtain conversion features; Encoding the conversion feature according to a multifunctional video coding standard to obtain a feature encoding bit stream; The original video data and the characteristic coding bit stream are mixed and coded to obtain a video coding bit stream.

2. The multifunctional video encoding method based on feature super-resolution technology according to claim 1, characterized in that: The step of extracting super-resolution features of the original video data comprises: Extracting features from the original video data using a shallow feature extraction network to obtain shallow features; Extracting the shallow features through multiple multi-scale feature reuse hybrid attention networks to obtain multiple deep features; A plurality of the deep features are fused through a feature fusion network to obtain super-resolution features.

3. The multifunctional video encoding method based on feature super-resolution technology according to claim 2, characterized in that: The shallow features are extracted by using multiple multi-scale feature reuse hybrid attention networks to obtain multiple deep features, including: For the first M / 2+1 multi-scale feature reused hybrid attention networks, the output features of the i-1th multi-scale feature reused hybrid attention network are used as the input features of the i-th multi-scale feature reused hybrid attention network to obtain the i-th deep feature, which is expressed as: F i =G i (F i-1 ),1≤i<M / 2+1 Among them, M represents the number of multi-scale feature reuse hybrid attention networks, i represents the sequence number of multi-scale feature reuse hybrid attention networks, G i (·) represents the feature extraction function of the i-th multi-scale feature reuse hybrid attention network, F i represents the output feature of the i-th multi-scale feature reuse hybrid attention network, i.e., the i-th deep feature, F i-1 represents the output feature of the i-1th multi-scale feature reuse hybrid attention network. The input feature F0 of the first multi-scale feature reuse hybrid attention network is a shallow feature. For the next M / 2+1 multi-scale feature reused hybrid attention networks, the output features of the i-1th and M-i+1th multi-scale feature reused hybrid attention networks are used as the input features of the i-th multi-scale feature reused hybrid attention network to obtain the i-th deep feature, which is expressed as: F i =G i (F i- 1+F M-i+1 ),M / 2+1<i≤M Among them, G i (·) represents the feature extraction function of the i-th multi-scale feature reuse hybrid attention network, F i-1 represents the output feature of the i-1th multi-scale feature reuse hybrid attention network, F M-i+1 Represents the output features of the M-i+1th multi-scale feature reuse hybrid attention network.

4. The multifunctional video encoding method based on feature super-resolution technology according to claim 2, characterized in that: The multi-scale feature reuse hybrid attention network includes a multi-scale feature reuse module and a hybrid attention module; The multi-scale feature reuse module adopts four-way convolution processing, each convolution processing is set with convolution kernels of different sizes and numbers, and each convolution processing is connected through short jumps. The output expression of the multi-scale feature reuse module is as follows: f out =G conv1 (G link [f1′,f′2,f′3,f′4])+f0 Among them, f0 is the input feature of the multi-scale feature reuse module, f out represents the output features of the multi-scale feature reuse module, G link (·) represents the channel concatenation function, which concatenates the four output features f1′, f2′, f3′, and f4′. conv1 (·) represents a 1×1 convolution function, f1, f2, f3, and f4 are the outputs after four-way convolution processing; The output of the first convolution process is: f1 = G conv1 (f0); The output of the second convolution process is: f2 = G conv3 (f1+f0); The output of the third convolution process is: f3 = G conv3 (G ReLU (G conv3 (f2+f0))); The output of the fourth convolution process is: f4 = G conv3 (G ReLU (G conv3 (G ReLU (G conv3 (f3+f0))))), G conv3 (·) represents the 3×3 convolution function, G ReLU (·) represents the ReLU activation function; The hybrid attention module includes a 5×5 depth-separable convolutional layer and a 1×1 convolutional layer. The output expression of the hybrid attention module is as follows: f′ out =f out ×(G sigm (G conv1 (G ReLU (G d-conv5 (f out ))))) Among them, f′ out represents the output of the hybrid attention module, G sigm (·) represents the Sigmoid function, G d-conv5 (·) represents a 5×5 depthwise separable convolution function.

5. The multifunctional video encoding method based on feature super-resolution technology according to claim 2, characterized in that: The feature fusion network includes a channel series layer and a 1×1 convolution layer; The channel series layer performs channel series splicing on a plurality of the deep features to obtain a spliced ​​feature; The 1×1 convolutional layer reduces the dimension of the concatenated features to obtain fused features.

6. The multifunctional video encoding method based on feature super-resolution technology according to claim 2, characterized in that: The shallow feature extraction network is a 3×3 convolutional layer.

7. A multifunctional video decoding method based on feature super-resolution technology, characterized in that: include: Decode the video encoding bit stream to obtain the feature encoding bit stream and the original video data. Decoding the feature encoded bit stream according to a multifunctional video decoding standard to obtain a decoding feature; Dequantizing and rearranging the decoded features to obtain reconstructed features; Performing super-resolution reconstruction on the reconstructed features to obtain a high-resolution feature image; Decoding the high-resolution feature image and the original video data according to a multifunctional video decoding standard to obtain a high-resolution video; The video encoding bit stream is encoded by the multifunctional video encoding method based on feature super-resolution technology as described in any one of claims 1-6.

8. The multifunctional video decoding method based on feature super-resolution technology according to claim 7, characterized in that: The super-resolution reconstruction of the reconstructed features to obtain a high-resolution feature image includes: Convolutionally processing the reconstructed features through a 3×3 convolutional layer to obtain convolutional features; The convolutional features are upsampled through a pixel rearrangement layer to obtain a high-resolution feature image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multifunctional video encoding method based on the feature super-resolution technology described in any one of claims 1-6 and / or the multifunctional video decoding method based on the feature super-resolution technology described in any one of claims 7-8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multifunctional video encoding method based on feature super-resolution technology described in any one of claims 1-6 and / or the multifunctional video decoding method based on feature super-resolution technology described in any one of claims 7-8 are implemented.

Images