Video coding and decoding method and device based on feature fusion and fixed point

Abstract

The invention discloses a video coding and decoding method and device based on feature fusion and fixed-point processing, and the method comprises the steps: fusing the features of a large resolution and multiple channels into the features of a small resolution and few channels, and carrying out the fixed-point processing of feature values through employing a feature value step-by-step fixed-point processing method which accords with the features; fixed-point features are coded into compressed code streams, and original high-resolution multichannel features are effectively recovered at a decoding end through analysis, anti-fixed-point and feature anti-fusion, so that the compressed code streams can be effectively applied to the rear end of a machine task network to efficiently complete machine tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of video encoding and decoding, and more specifically, to a video encoding and decoding method and device based on feature fusion and fixed-point. Background Art

[0002] The existing image and video encoding and decoding processing includes traditional encoding and decoding methods and intelligent encoding and decoding methods based on neural networks. Traditional encoding and decoding methods eliminate redundant information of image and video through operations such as prediction, transformation, quantization, entropy coding and loop filtering. Intelligent encoding and decoding methods use neural networks to transform image and video, convert image and video into feature tensors, and perform operations such as downsampling, quantization and entropy coding to achieve compression coding.

[0003] The image intelligent coding and decoding method proposes many efficient neural network structures for image feature information extraction. Convolutional neural network (CNN) is the earliest network structure used for image coding and decoding. Based on CNN, many improved network structures and probability estimation models have been derived. Taking network structures as an example, including generative adversarial network (GAN) and recurrent neural network (RNN), they have greatly improved the end-to-end image compression performance based on neural networks. Among them, the image coding and decoding method based on generative adversarial network (GAN) has achieved significant results in improving the subjective quality of images.

[0004] Video intelligent coding and decoding methods are mainly concentrated in three aspects: 1) hybrid neural network coding, 2) neural network rate-distortion optimization coding, and 3) end-to-end video coding. Hybrid neural network coding embeds neural networks into the video framework instead of traditional coding modules. Generally, inter-frame prediction modules, loop filter modules, and entropy coding modules are used more. Neural network rate-distortion optimization coding uses the highly nonlinear characteristics of neural networks to train neural networks into efficient discriminators and classifiers, for example, in the video coding mode decision link. End-to-end video coding is currently generally divided into using CNN to replace all modules of traditional coding methods, or expanding the input dimension of the neural network to perform end-to-end compression on all frames.

[0005] In the image or video encoding and decoding method using an end-to-end neural network, the common operation is to first extract features from the image or video, then encode and decode it, and then restore it to the image. The process of extracting the feature tensor is E1, and the process of encoding the feature tensor obtained by E1 and obtaining the bitstream is E2. The encoding method consists of E1 and E2. Correspondingly, the decoding method consists of D1 and D2, where D1 refers to the process of transforming the feature tensor into a decoded image, and D2 refers to the process of decoding the bitstream into a feature tensor.

[0006] In the current mainstream application scenarios, the encoded and decoded images and videos are not only used for human viewing, but are also often used to complete machine intelligence tasks. The intelligent task network completes task objectives such as target detection, target tracking or behavior recognition by analyzing images or videos.

[0007] The existing traditional video coding is usually designed for pixel fidelity and cannot effectively complete machine intelligence tasks. However, the image intelligent coding and decoding based on neural networks can be trained for the accuracy of machine intelligence tasks, thereby achieving results that surpass traditional image coding methods in machine intelligence tasks. However, the current video intelligent coding and decoding methods based on neural networks are not good enough and cannot surpass the latest traditional video coding methods such as H.266 / VVC. This is mainly because the neural network-based methods cannot fully eliminate redundant information in the time domain.

[0008] In the encoding and decoding framework of feature data, it is first necessary to extract features from the image video, then use the traditional video codec to encode and decode the features, and finally use the decoded features as the input of the machine intelligence task and infer to complete the machine intelligence task. Since the extracted features are often floating-point numbers, and the traditional video codec requires fixed-point numbers as input, it is often necessary to fix the floating-point features before inputting them into the traditional encoder. The fixed-point conversion usually involves scaling the input floating-point numbers first and then quantizing them. Correspondingly, after the fixed-point features are reconstructed by the traditional decoder, they need to be further de-fixed to obtain floating-point features as the input of the next machine intelligence task network. The de-fixed-point conversion usually involves de-quantizing and de-scaling the input fixed-point numbers.

[0009] The existing fixed-point method is to set the fixed-point parameters offline in advance at the encoding and decoding ends. The problem with this method is that different features have different dynamic ranges, and using the same fixed-point parameters may result in a large difference between the floating-point features before fixed-point and the floating-point features after defixed-point. Summary of the invention

[0010] In order to solve the above-mentioned shortcomings of the prior art, the present invention proposes a video encoding and decoding method and device based on feature fusion and fixed-point, which fuses the features of large resolution and multiple channels into features of small resolution and few channels, and uses a step-by-step fixed-point method of feature value that conforms to the features to fix the feature values, encodes the fixed-point features into a compressed bit stream, and effectively restores the original large resolution and multiple channel features through parsing, de-fixation and feature de-fusion at the decoding end, so that the features can be effectively used in the machine task network backend to efficiently complete the machine task.

[0011] To this end, the first object of the present invention is to propose a video decoding method based on feature fusion and fixed-point, and for an input video bit stream, the following operations are performed:

[0012] Parsing the code stream to obtain first decoding feature data;

[0013] Performing inverse fixed-point processing on the first decoded feature data to obtain second decoded feature data;

[0014] The second decoded feature data is subjected to feature defusion processing to obtain third decoded feature data, wherein the third decoded feature data includes decoded feature data at at least two scales.

[0015] Furthermore, the operation of the de-fixed processing also includes:

[0016] Performing inverse quantization processing on the first decoded characteristic data to obtain fourth decoded characteristic data;

[0017] Performing inverse scaling processing on the fourth decoded characteristic data to obtain fifth decoded characteristic data;

[0018] The fifth decoded feature data is subjected to numerical offset processing to obtain second decoded feature data.

[0019] Furthermore, the operation method of the dequantization processing includes one or more quantization methods such as linear uniform dequantization, nonlinear function dequantization, histogram equalization dequantization and long-tail dequantization.

[0020] Furthermore, the method also includes:

[0021] A scaling parameter and a numerical offset parameter are obtained from the bitstream, wherein the scaling parameter is used for inverse scaling processing as a scaling ratio of the characteristic value, and the numerical offset parameter is used for numerical offset processing as an offset of the characteristic value.

[0022] Furthermore, the feature anti-fusion processing operation also includes:

[0023] Processing the second decoded feature data using the first neural network to obtain sixth decoded feature data conforming to the first scale;

[0024] Processing the second decoded feature data using a second neural network to obtain seventh decoded feature data conforming to a second scale;

[0025] Processing the sixth decoded feature data using a third neural network to obtain eighth decoded feature data conforming to the second scale;

[0026] Using a fourth neural network to fuse the seventh decoded feature data and the eighth feature data to obtain ninth decoded feature data that conforms to the second scale;

[0027] The sixth decoding feature data and the ninth decoding feature data are combined to obtain the third decoding feature data.

[0028] The second object of the present invention is to provide a video encoding method based on feature fusion and fixed-point, and for an input video, the following operations are performed:

[0029] Extracting first coding feature data from an image of a video, wherein the first coding feature data includes coding feature data of at least two scales;

[0030] Performing feature fusion processing on the first coded feature data to obtain second coded feature data;

[0031] performing fixed-point processing on the second coding feature data to obtain third coding feature data;

[0032] The third encoding feature data is encoded to obtain a code stream.

[0033] Furthermore, the fixed-point processing operation also includes:

[0034] Performing numerical offset processing on the second coded feature data to obtain fourth coded feature data;

[0035] Scaling the fourth encoded feature data to obtain fifth encoded feature data;

[0036] The fifth coding feature data is quantized to obtain third coding feature data.

[0037] Furthermore, the method further comprises:

[0038] The scaling parameter and the numerical offset parameter are put into the bitstream, wherein the scaling parameter is used for scaling processing as the scaling ratio of the characteristic value, and the numerical offset parameter is used for numerical offset processing as the offset of the characteristic value.

[0039] The third object of the present invention is to provide a video decoding device based on feature fusion and fixed-point, comprising:

[0040] processor;

[0041] A memory for storing code streams and image data to be decoded; and

[0042] One or more programs are used to complete the video decoding method based on feature fusion and fixed-point as described in the first purpose above.

[0043] The third object of the present invention is to provide a video encoding device based on feature fusion and fixed-point, comprising:

[0044] processor;

[0045] A memory for storing code streams and images to be encoded; and

[0046] One or more programs are used to complete the video encoding method based on feature fusion and fixed-point as described in the second purpose above.

[0047] The beneficial effects of the present invention are as follows: feature fusion can effectively reduce the data size of feature data extracted from video images, and maintain the accuracy of the results of the restored feature data completing the machine task in the machine task network backend through an efficient anti-fusion method, which can effectively improve the compression efficiency; at the same time, the fused features are further fixed-pointed according to their statistical characteristics, which can further reduce the amount of data required for compressing the features; in addition, before anti-fusion, the features are adaptively anti-scaled and quantized, which can adjust the features to a similar expression space, so that different features can be processed by the same anti-fusion method, thereby reducing resource consumption at the decoding end. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of feature de-fusion in a video encoding and decoding method based on feature fusion and fixed-point in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of feature fusion in a video encoding and decoding method based on feature fusion and fixed-point according to an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the overall framework of a video encoding and decoding method based on feature fusion and fixed-point encoding according to an embodiment of the present invention;

[0052] Figure 4 This is a flowchart of Example 1 of a video encoding and decoding method based on feature fusion and fixed-point according to an embodiment of the present invention;

[0053] Figure 5 This is a flowchart of Example 2 of a video encoding and decoding method based on feature fusion and fixed-point according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0055] Definition of terms:

[0056] VVC: Versatile Video Coding, also known as H.266, is a video coding standard that can compress the input video to obtain a bitstream. The bitstream can contain multiple layers of sub-bitstreams, and the sub-bitstreams can have reference relationships.

[0057] Feature fusion: It is an operation to fuse multi-channel features extracted by neural networks from video images. Multi-channel features are usually composed of a combination of two-dimensional feature maps of multiple channels, and the feature map of each channel contains part of the feature information in the video image. In order to fully reflect the semantic information in the video image, multi-channel features usually contain a very large number of channels, which makes the data volume of multi-channel features very large and inconvenient to transmit and compress. In some neural networks that complete machine tasks such as target detection, instance segmentation or target tracking, their feature extraction networks usually also contain multi-scale branch networks, which can extract multi-channel features of multiple scales from video images to fully capture objects or targets of different scales in video images, which further increases the data volume of features. Feature fusion can effectively fuse multi-scale multi-channel features or multi-channel features of different scales, reduce the number of channels or resolution of feature data, and at the same time maintain the semantic information contained in the feature data, so that when the fused data is restored to feature data, it can still effectively complete the subsequent semantic processing process in the machine task network and obtain higher task accuracy. Corresponding to feature fusion is feature anti-fusion.

[0058] Machine task network: a neural network that processes input video images and completes a certain machine task. The tasks that can be completed include but are not limited to classification, target detection, instance segmentation, target tracking or posture estimation. In a feature-based video encoding and decoding method, the codec is inserted into a node in the machine task network, and the feature data extracted by the network before the node is encoded to obtain a bit stream, and decoded and reconstructed to obtain feature data. The reconstructed feature data can be processed by the network after the node to complete the machine task and obtain the task result. For ease of description, in the present invention, the network before the node is called the machine task network front end, and the network after the node is called the machine task network back end.

[0059] The present embodiment discloses a video decoding method based on feature fusion and fixed-point, and the implementing body of the method can be a decoder. Specifically, the decoding method mainly includes the following operations: for the input bitstream, the decoder uses an entropy decoding method to restore the bitstream to obtain intermediate features, and parses the bitstream to obtain the dequantization method and descaling method that should be used for the feature value of the intermediate feature, and uses the given dequantization method and descaling method to dequantize and descale the intermediate feature to obtain a fused feature, which is obtained by the encoder using a feature fusion method to fuse the features extracted by the machine task network front end; the decoder uses a feature defusion method to process the fused feature to obtain a reconstructed feature, which can be processed by the machine task network back end to obtain a machine task result. The decoding method is described in detail below.

[0060] In one embodiment, the feature de-fusion method can restore the fused features to single-scale multi-channel reconstruction features. This feature de-fusion method uses a series of resolution amplification, feature value filling and multi-convolution kernel neural networks to transform the fused features with smaller resolution and fewer channels into the reconstruction features with larger resolution and more channels.

[0061] In another embodiment, the feature anti-fusion method can restore the fused features to multi-scale and multi-channel reconstructed features, such as Figure 1 As shown in . This method uses a sequence from small scale to large scale. First, a neural network is used to transform the fused features into intermediate features of multiple scales. Then, the small-scale intermediate features are downsampled or pooled to obtain large-scale downsampled features, which are then fused with the large-scale intermediate features to obtain large-scale reconstructed features. Since the smallest-scale intermediate features cannot be fused with features of other scales, the above operations are performed on the intermediate features except the smallest scale in sequence to obtain reconstructed features of all scales. For example, in Figure 1 In the figure, the minimum scale reconstruction feature P5' is obtained by a series of transformations of the fused features. After upsampling, P5' is fused with the second minimum scale intermediate feature P4' to obtain the reconstruction feature P4', and so on, and finally the reconstruction features P2'-P5' of all scales are obtained. In another embodiment, the feature anti-fusion method can also be anti-fused in the order of large scale to small scale to obtain the reconstruction features of all scales, which will not be repeated here. In one embodiment, the reconstruction features of the minimum scale are further downsampled to obtain the reconstruction features of a smaller scale, such as Figure 1 As shown in P6'.

[0062] In one embodiment, the entropy decoding method used to parse the bitstream can be an intelligent entropy decoding method based on super-prior probability estimation, the input of which is the bitstream and the output is the intermediate feature. In another embodiment, the entropy decoding method may further include a prediction method, such as a video decoding method of standards such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series or AV1 series, which can be used to decode the input bitstream to obtain the intermediate features. Among them, the image in the video can be obtained by splicing the feature maps of all channels of the intermediate feature, or each image in the video can be a feature map of a channel of the intermediate feature.

[0063] When the intermediate features are dequantized and descaled, a variety of different dequantization and descaling methods that adapt to the step-by-step characteristics of the feature data can be used. The advantage is that the quantization loss of the feature value can be avoided as much as possible and the value range of the feature value can be expressed more effectively using limited bits. Specifically, dequantization and descaling can be the following methods.

[0064] In one embodiment, inverse quantization and inverse scaling can be a unified operation method for all channels. Specifically, the method uniformly performs inverse quantization operations on all channels of feature data in a specific manner, and the specific inverse quantization method options include linear uniform inverse quantization, nonlinear function inverse quantization, and histogram equalization inverse quantization.

[0065] If the inverse quantization method is linear uniform inverse quantization, the options for the reconstruction level include two types: the center of the quantization interval and the center of the probability distribution of the quantization interval, depending on the reconstruction method.

[0066] For linear uniform inverse quantization with the reconstruction level being the center of the quantization interval, for example, the characteristic data of all channels can be uniformly linearly inverse quantized using the following inverse quantization formula:

[0067]

[0068] Among them, y cij is the quantized value of the cth channel, row i, column j of the feature data, s c is the scaling value of the feature data under all channels, x cmin is the minimum value of the feature data under all channels, x cij is the reconstructed value of the cth channel, row i, column j of the feature data. float[·] represents the floating point function.

[0069] For the linear uniform inverse quantization method where the reconstruction level is the probability distribution center of the quantization interval, for example, the probability distribution center of the corresponding quantization interval can be checked in the reconstruction codebook based on the reconstructed fixed-point feature data value as the quantization interval index, and the probability distribution center is used as the inverse quantization value.

[0070] If the inverse quantization method is nonlinear exponential function inverse quantization, the characteristic data of all channels can be uniformly nonlinearly inverse quantized using the following inverse quantization formula example:

[0071]

[0072] Where log_base is the base of the logarithmic function used in logarithmic quantization.

[0073] If the inverse quantization method is histogram equalization quantization, the probability distribution center of the corresponding quantization interval can be checked in the reconstructed codebook based on the reconstructed fixed-point feature data value as the quantization interval index, and the probability distribution center is used as the inverse quantization value.

[0074] In one embodiment, the scaling parameter s c and the offset parameter x cmin A set of fixed values ​​can be obtained from a large amount of statistics on the training data, which can be used for the inverse quantization and inverse scaling of any feature data. This has the advantage of reducing the data consumption of these parameters in the bitstream. It can also be explicitly recorded in the syntax elements of the bitstream and parsed by the decoder. This has the advantage of being able to adaptively inverse quantize and inverse scale feature data with large content differences, which can effectively reduce the distortion of inverse quantization and inverse scaling.

[0075] In another embodiment, inverse quantization and inverse scaling can be independent operation methods for each channel. Specifically, the method performs inverse quantization operations on each of the feature data independently in a specific manner. Specific inverse quantization method options include linear uniform inverse quantization, nonlinear function inverse quantization, and histogram equalization inverse quantization. For different inverse quantization methods, inverse quantization operations can be performed independently on each channel of the feature data based on a unified method for all channels.

[0076] In another embodiment, inverse quantization and inverse scaling can be an operation method of multi-channel organization. Specifically, the method groups each channel of the feature data, at least one group contains feature data of at least two channels, and performs inverse quantization operations on each group in a unified manner in a specific manner. Specific inverse quantization method options include linear uniform inverse quantization, nonlinear function inverse quantization, and histogram equalization inverse quantization. For different inverse quantization methods, inverse quantization operations can be uniformly performed on each group channel of the feature data based on a unified method for all channels.

[0077] This embodiment discloses a video encoding method based on feature fusion and fixed-point, and the implementation subject of the method can be an encoder. Specifically, the encoding method mainly includes the following operations: for the image in the input video, the encoder uses the machine task network front end to extract single-scale multi-channel features or multi-scale multi-channel features, and uses the feature fusion method to fuse the multi-channel features into a fusion feature with a smaller resolution and fewer channels; after the fusion feature is scaled and quantized, the encoder uses the entropy coding method to encode the quantized feature into a bit stream. The encoding method is described in detail below.

[0078] In one embodiment, the feature fusion method can transform the single-scale multi-channel features into fused features. This feature fusion method uses a series of resolution reduction, feature value pooling or downsampling and neural networks with fewer convolution kernels to achieve the transformation of fused features with larger resolution and more channels into reconstructed features with smaller resolution and fewer channels.

[0079] In another embodiment, the feature fusion method can transform the multi-scale multi-channel features into fused features, such as Figure 2 This method uses a neural network to transform multi-scale features into intermediate features of the same small scale, and then concatenates these intermediate features in the channel dimension or spatial dimension to obtain a merged feature. In order to reduce the resolution or number of channels of the merged feature, the neural network is used to process the merged feature to obtain a fusion feature with a more streamlined dimension. For example, Figure 2 In the above figure, P2-P4 features are downsampled or pooled to the same resolution as P5, and then concatenated with P5 in the channel dimension. They are passed through a network with an attention mechanism to adjust the weights of the feature values ​​on different channels, and finally processed by a series of neural networks to obtain fused features with a smaller number of channels.

[0080] In one embodiment, the entropy coding method used for encoding the bitstream can be an intelligent entropy coding method based on super-prior probability estimation, the input of which is a quantized and scaled fusion feature, and the output is a bitstream. In another embodiment, the entropy coding method may further include a prediction method, such as a video coding method of a standard specification such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, AVS series or AV1 series, which can be used to encode the input fusion feature to obtain a bitstream. Among them, one method is to splice the feature maps of all channels of the fusion feature to obtain an image in the video, or to use the feature map of each channel of the intermediate feature as an image in the video.

[0081] When quantizing and scaling the fused features, a variety of different quantization and scaling methods that adapt to the step-by-step characteristics of the feature data can be used. The advantage is that the quantization loss of the feature value can be avoided as much as possible and the range of the feature value can be expressed more effectively using limited bits. Specifically, quantization and scaling can be the following methods.

[0082] In one embodiment, the quantization and scaling operations may adopt a multi-channel organization operation method. Specifically, the method uniformly performs quantization operations on all channels of the feature data in a specific manner, and only transmits a set of quantization parameters for all channels. Specific quantization options include linear uniform quantization, nonlinear function quantization, and histogram equalization quantization.

[0083] If the quantization method is linear uniform quantization, for example, the characteristic data of all channels can be uniformly linearly uniformly quantized using the following quantization formula:

[0084]

[0085] Among them, x cij is the original value of the cth channel, row i, column j of the feature data, x cmax and x cmin are the maximum and minimum values ​​of the feature data under all channels, bitdepth is the quantization bit width, int[·] represents the integer function, and y cij is the value of the i-th row and j-th column of the c-th channel of the quantized feature data, and Δ is a minimum value used to map the floating-point feature data into a left-closed and right-open value interval.

[0086] In one embodiment, the encoding method requires not only the bitstream information generated after conventional video and image encoding, but also the transmission of feature data quantization and scaling information. Depending on the reconstruction method, the reconstruction level options include the center of the quantization interval and the center of the probability distribution of the quantization interval.

[0087] For the linear uniform quantization method where the reconstruction level is the center of the quantization interval, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the characteristic data under all channels cmin and the scaling value s c , the scaling value s of the feature data under all channels c It can be used to quantify the formula part; 2) quantization bitwidth bitdepth; 3) the maximum value x of feature data under all channels cmax For the linear uniform quantization method in which the reconstruction level is the probability distribution center of the quantization interval, the following additional information needs to be transmitted: a reconstruction codebook consisting of the quantization interval index and the probability distribution center of the corresponding quantization interval.

[0088] If the quantization method is nonlinear function quantization, the quantization method can be further divided into nonlinear logarithmic function quantization and nonlinear exponential function quantization according to the different nonlinear functions. For nonlinear logarithmic function quantization, for example, the feature data of all channels can be uniformly nonlinearly quantized using the following quantization formula:

[0089]

[0090] Where log_base is the base of the logarithmic function used in logarithmic quantization.

[0091] For nonlinear logarithmic function quantization, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the feature data under all channels cmin and the scaling value s c ; 2) the base of the logarithmic function used in logarithmic quantization, log_base, the scaling value s of the feature data under all channels c It can be used to quantify the formula part; 3) quantization bit width bitdepth; 4) the maximum value x of feature data under all channels cmax .

[0092] If the quantization method is histogram equalization quantization, for example, the characteristic data of all channels can be uniformly histogram-equalized and quantized through the following steps: sort the characteristic data of all channels according to the value size to obtain the sorted characteristic data; divide the sorted characteristic data into multiple quantization intervals containing an equal number of characteristic data values, and use the index that can be represented by the corresponding quantization bit width to represent each quantization interval, and the index of each quantization interval is the characteristic data value after histogram equalization quantization. The following additional information also needs to be transmitted: the reconstruction codebook information composed of the quantization interval index and the probability distribution center of the corresponding quantization interval. For the histogram equalization quantization method, since the 0 value accounts for a large proportion of the characteristic data value, the characteristic data except the 0 value after sorting can be divided into intervals containing an equal number of characteristic data values, that is, all the 0 values ​​of the characteristic data after sorting are recorded as index 0, and the corresponding reconstruction level is also set to 0.

[0093] In one embodiment, the quantization and scaling operations can be performed using a channel-by-channel independent operation method. Specifically, the method performs quantization operations on each channel of the feature data independently in a specific manner, and transmits a set of quantization parameters for each channel. Specific quantization options include linear uniform quantization, nonlinear function quantization, and histogram equalization quantization. The various quantization methods are similar to the various quantization methods under the unified fixed-point of all channels, as follows:

[0094] If the quantization method is linear uniform quantization, for example, the feature data of each channel can be independently linearly uniformly quantized by the linear uniform quantization formula under the unified fixed-point of all channels, where x cmax and x cmin are the maximum and minimum values ​​of the feature data in each channel respectively.

[0095] For the linear uniform quantization method where the reconstruction level is the center of the quantization interval, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the characteristic data under each channel cmin and the scaling value s c ; 2) The scaling value s of the characteristic data under each channel c It can be used to quantify the formula part; 3) quantization bit width bitdepth; 4) the maximum value x of the feature data under each channel cmax For the linear uniform quantization method in which the reconstruction level is the probability distribution center of the quantization interval, the following additional information needs to be transmitted: a reconstruction codebook consisting of the quantization interval index and the probability distribution center of the corresponding quantization interval.

[0096] If the quantization method is nonlinear function quantization, the quantization method can be further divided into nonlinear logarithmic function quantization and nonlinear exponential function quantization according to the different nonlinear functions. For example, the feature data of each channel can be independently nonlinearly quantized by the nonlinear logarithmic quantization formula under the unified fixed-point of all channels or the nonlinear logarithmic quantization formula, where x cmax and x cmin are the maximum and minimum values ​​of the feature data in each channel respectively.

[0097] For nonlinear logarithmic function quantization, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the feature data under each channel cmin and the scaling value s c ; 2) the base of the logarithmic function used in logarithmic quantization, log_base, the scaling value s of the feature data under each channel c It can be used to quantify the formula part; 3) quantization bit width bitdepth; 4) the maximum value x of the feature data under each channel cmax .

[0098] If the quantization method is histogram equalization quantization, for example, the feature data of each channel can be quantized independently by histogram equalization steps under unified fixed-point quantization of all channels. For histogram equalization quantization, the following additional information needs to be transmitted: a reconstructed codebook consisting of a quantization interval index and a probability distribution center of the corresponding quantization interval.

[0099] In one embodiment, the quantization and scaling operations can also be performed using a multi-channel organization operation method. Specifically, the method groups the channels of the feature data, and uniformly performs quantization operations on each group in a specific manner, and transmits a set of quantization parameters for each group. Specific quantization options include linear uniform quantization, nonlinear function quantization, and histogram equalization quantization. The various quantization methods are similar to the various quantization methods under the unified fixed-point of all channels, as follows:

[0100] If the quantization method is linear uniform quantization, for example, the characteristic data of each group channel can be independently linearly uniformly quantized by the linear uniform quantization formula under the unified fixed-point of all channels, where x cmax and x cmin are the maximum and minimum values ​​of the characteristic data under each group channel respectively.

[0101] For the linear uniform quantization method where the reconstruction level is the center of the quantization interval, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the characteristic data under each grouping channel cmin and the scaling value s c ; 2) The scaling value s of the characteristic data under each grouping channel c It can be used to quantify the formula part; 3) quantization bit width bitdepth; 4) the maximum value x of the feature data under each group channel cmax For the linear uniform quantization method in which the reconstruction level is the probability distribution center of the quantization interval, the following additional information needs to be transmitted: a reconstruction codebook consisting of the quantization interval index and the probability distribution center of the corresponding quantization interval.

[0102] If the quantization method is nonlinear function quantization, the quantization method can be further divided into nonlinear logarithmic function quantization and nonlinear exponential function quantization according to different nonlinear functions. For example, the feature data of each group channel can be independently nonlinearly quantized by the nonlinear logarithmic quantization formula under the unified fixed-point of all channels or the nonlinear logarithmic quantization formula, where x cmax and x cmin are the maximum and minimum values ​​of the feature data in each channel respectively.

[0103] For nonlinear logarithmic function quantization, one or more of the following additional information needs to be transmitted: 1) The minimum value x of the characteristic data under each grouping channel cmin and the scaling value s c ; 2) the base of the logarithmic function used in logarithmic quantization, log_base, the scaling value s of the characteristic data under each grouping channel c It can be used to quantify the formula part; 3) quantization bit width bitdepth; 4) the maximum value x of the feature data under each group channelcmax .

[0104] If the quantization method is histogram equalization quantization, for example, the feature data of each group channel can be independently histogram-equalized and quantized by the step of histogram equalization under the unified fixed-point of all channels. For histogram equalization quantization, the following additional information needs to be transmitted: a reconstructed codebook consisting of a quantization interval index and a probability distribution center of the corresponding quantization interval.

[0105] In one embodiment, when the statistical distribution of the feature data to be quantized shows a high aggregation effect, that is, the values ​​of the feature data are concentrated in a certain value range with a high probability, and appear in other value ranges with a low probability, for example, the statistical distribution of the feature data shows a long tail effect or conforms to a Gaussian distribution or a Laplace distribution. At this time, using a limited number of bits to represent all the values ​​of the feature data will cause the high bits to be mainly used to represent large eigenvalues ​​that appear with a low probability, which will cause the limited bits to be unable to fully exert the expression effect, and the small eigenvalues ​​that appear with a high probability can only use a few of the limited bits, which ultimately makes the quantization distortion larger. In fact, these large eigenvalues ​​that appear with a low probability contribute less to the semantic information contained in the feature data, and these large eigenvalues ​​can be discarded to a certain extent, which can reduce the quantization distortion of small eigenvalues ​​while making full use of the limited bits. In one embodiment, the encoder can truncate the eigenvalues ​​that appear with a low probability in the feature data to a certain threshold, and then scale and quantize all the eigenvalues.

[0106] In one embodiment, Figure 3 The schematic diagram of the overall framework of a codec is shown, and the overall framework includes a feature extraction module 101, a fixed-point conversion 102, an encoder 103, a decoder 104, a de-fixed-point conversion 105, and a task network 106, wherein the fixed-point conversion includes scaling and quantization operations, and the de-fixed-point conversion includes de-quantization and de-scaling operations; wherein the feature extraction module 101, the fixed-point conversion 102, and the encoder 103 belong to the encoding end device, and the encoder 104, the de-fixed-point conversion 105, and the task network 106 belong to the decoding end device, and the encoding end device and the decoding end device exchange information by establishing a wireless connection. The feature extraction module 101 extracts floating-point features from the input video, the fixed-point conversion 102 converts the floating-point features into fixed-point features, and the encoder 103 encodes the fixed-point features into a bitstream. The decoder 104 decodes the encoded bitstream to obtain reconstructed fixed-point features, and the de-fixed-point conversion 105 de-fixes the reconstructed fixed-point features to obtain reconstructed floating-point features. The task network 106 performs a machine learning task according to the reconstructed floating-point number features to obtain a task result.

[0107] The overall framework of the video encoding and decoding method is introduced above. The details of the fixed-point conversion 102 and the de-fixed-point conversion 105 will be described in detail below with reference to specific examples.

[0108] In one embodiment, if Figure 4 As shown, at the encoding end, the method of 201 is first used to count the eigenvalue distribution of the input floating-point features, such as a histogram distribution. According to the distribution, the minimum threshold and the maximum threshold are set to determine the distribution range of the eigenvalue. It should be noted that the threshold area does not necessarily include all eigenvalues, and the eigenvalues ​​outside the threshold area are directly set to the nearest threshold. Then, according to the method of 202, the minimum threshold and the maximum threshold are used to scale the floating-point eigenvalues, and their range is scaled to the eigenvalue range of the fixed-point number. Finally, the encoder 103 is used to encode the fixed-point number features into a bit stream. The minimum threshold and the maximum threshold obtained in 201 also need to be written into the bit stream.

[0109] At the decoding end, the decoder 104 extracts the reconstructed fixed-point features from the coded bitstream. At the same time, the minimum threshold and the maximum threshold parsed in the bitstream are passed as fixed-point parameters to 203. According to the obtained fixed-point parameters, 203 numerically scales the reconstructed fixed-point features to obtain reconstructed floating-point features.

[0110] In another embodiment, if Figure 5 As shown, the method 301 is first used to offline count the distribution of the eigenvalues ​​of the entire data set, such as the histogram distribution. The minimum threshold of the data set and the maximum threshold of the data set are set according to the distribution to determine the distribution range of the eigenvalues ​​of the data set, and the eigenvalues ​​outside the threshold area are directly set to the nearest threshold. The data set threshold serves as a benchmark for subsequent fixed-point and de-fixed-point conversion.

[0111] At the encoding end, the method of 201 is first used to count the characteristic value distribution of the input floating-point feature, such as the histogram distribution. According to the distribution, the minimum threshold and the maximum threshold are set to determine the distribution range of the characteristic value. It should be noted that the threshold area does not necessarily include all characteristic values, and the characteristic values ​​outside the threshold area are directly set to the nearest threshold. Then, according to the method of 202, the minimum threshold and the maximum threshold of the feature are used to scale the floating-point characteristic values, and its range is scaled to the characteristic value range of the fixed-point number. Finally, the encoder 103 is used to encode the fixed-point number feature into a bit stream. The threshold offset is obtained by calculating the difference between the threshold of the feature and the threshold of the data set through 302, that is, the threshold offset = the feature threshold - the threshold of the data set. The minimum threshold offset and the maximum threshold offset calculated by 302 also need to be written into the bit stream.

[0112] At the decoding end, the decoder 104 extracts the reconstructed fixed-point features from the coded bitstream. At the same time, the minimum threshold offset and the maximum threshold offset are parsed from the bitstream and passed to 303, and added to the data set threshold in 301 to obtain the threshold of the feature, that is, the feature threshold = threshold offset + data set threshold. The minimum threshold and maximum threshold calculated by 303 are passed to 203 as fixed-point parameters. According to the obtained fixed-point parameters, 203 performs numerical scaling on the reconstructed fixed-point features to obtain reconstructed floating-point features.

[0113] This embodiment discloses a decoding device, which includes a processor and a memory, and is used to execute the decoding method disclosed in the present invention.

[0114] This embodiment discloses a coding device, which includes a processor and a memory, and is used to execute the coding method disclosed in the present invention.

[0115] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A video decoding method based on feature fusion and fixed-point, characterized in that: For the input video stream, perform the following operations: Parsing the code stream to obtain first decoding feature data; Performing inverse fixed-point processing on the first decoded feature data to obtain second decoded feature data; The second decoded feature data is subjected to feature defusion processing to obtain third decoded feature data, wherein the third decoded feature data includes decoded feature data at at least two scales.

2. The method according to claim 1, characterized in that The operations of de-fixed processing also include: Performing inverse quantization processing on the first decoded characteristic data to obtain fourth decoded characteristic data; Performing inverse scaling processing on the fourth decoded characteristic data to obtain fifth decoded characteristic data; The fifth decoded feature data is subjected to numerical offset processing to obtain second decoded feature data.

3. The method according to claim 2, characterized in that The operation method of the dequantization processing includes one or more quantization methods such as linear uniform dequantization, nonlinear function dequantization, histogram equalization dequantization and long tail dequantization.

4. The method according to claim 2 or 3, characterized in that: Also includes: A scaling parameter and a numerical offset parameter are obtained from the bitstream, wherein the scaling parameter is used for inverse scaling processing as a scaling ratio of the characteristic value, and the numerical offset parameter is used for numerical offset processing as an offset of the characteristic value.

5. The method according to claim 1, characterized in that The feature anti-fusion processing operation also includes: Processing the second decoded feature data using the first neural network to obtain sixth decoded feature data conforming to the first scale; Processing the second decoded feature data using a second neural network to obtain seventh decoded feature data conforming to a second scale; Processing the sixth decoded feature data using a third neural network to obtain eighth decoded feature data conforming to the second scale; Using a fourth neural network to fuse the seventh decoded feature data and the eighth feature data to obtain ninth decoded feature data that conforms to the second scale; The sixth decoding feature data and the ninth decoding feature data are combined to obtain the third decoding feature data.

6. A video encoding method based on feature fusion and fixed-point, characterized in that: For the input video, do the following: Extracting first coding feature data from an image of a video, wherein the first coding feature data includes coding feature data of at least two scales; Performing feature fusion processing on the first coded feature data to obtain second coded feature data; performing fixed-point processing on the second coding feature data to obtain third coding feature data; The third encoding feature data is encoded to obtain a code stream.

7. The method according to claim 6, characterized in that The fixed-point processing operations also include: Performing numerical offset processing on the second coded feature data to obtain fourth coded feature data; Scaling the fourth encoded feature data to obtain fifth encoded feature data; The fifth coded feature data is quantized to obtain third coded feature data.

8. The method according to claim 7, characterized in that Also includes: The scaling parameter and the numerical offset parameter are put into the bitstream, wherein the scaling parameter is used for scaling processing as the scaling ratio of the characteristic value, and the numerical offset parameter is used for numerical offset processing as the offset of the characteristic value.

9. A video decoding device based on feature fusion and fixed-point, comprising: processor; A memory for storing code streams and image data to be decoded; as well as One or more programs are used to complete the video decoding method based on feature fusion and fixed-point as described in any one of claims 1-5.

10. A video encoding device based on feature fusion and fixed-point, comprising: processor; A memory for storing code streams and images to be encoded; as well as One or more programs are used to complete the video encoding method based on feature fusion and fixed-point as described in any one of claims 6-8.

Images