Data encoding method and apparatus, data decoding method and apparatus, computer device, and storage medium
By filtering and quantizing the scale parameter values of feature data, only important feature elements are encoded and decoded, which solves the problem of high complexity of entropy encoding and entropy decoding in the prior art, and improves the efficiency and accuracy of data encoding and decoding.
Patent Information
- Application Number
- PCT/CN2023/131492
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2023-11-14
- Publication Date
- 2024-11-21
AI Technical Summary
In the existing data encoding and decoding technology, the entropy encoding and entropy decoding processes are complex, resulting in low data encoding and decoding efficiency.
By obtaining the auxiliary encoding information of feature data, determining the scale parameter values of feature elements, quantizing and filtering, extracting important feature elements for entropy encoding, generating an encoded data stream, and quantizing the filtered scale parameter values during the decoding process Entropy decoding, merging feature elements and alternative elements to restore data.
It significantly reduces the complexity of entropy encoding and entropy decoding, improves data encoding and decoding efficiency, reduces the number of feature elements to be executed, and improves the speed and accuracy of encoding and decoding.
Smart Images

Figure CN2023131492_21112024_PF_FP_ABST
Abstract
Description
Data encoding, decoding method, device, computer equipment and storage medium
[0001] Related applications
[0002] This application claims priority to Chinese patent application number 2023105493790, filed on May 15, 2023, entitled “Data encoding, decoding method, device, computer equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of computer technology, and in particular to a data encoding method, apparatus, computer equipment, storage medium and computer program product, as well as a data decoding method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0004] With the development of computer technology, data encoding and decoding technologies have emerged. Encoding can compress data, while decoding can restore the compressed data. Data encoding and decoding are widely used in various scenarios, especially cross-platform data transmission scenarios, such as real-time conversation applications such as video or voice chat and video or voice conferencing.
[0005] In the data encoding process, entropy coding and entropy decoding are often required. The purpose of entropy coding is to estimate the number of bits required for encoding based on the input feature data. Entropy coding is usually implemented based on a statistical model. It analyzes and models the encoded data in order to minimize the number of bits required during the entropy coding process. Entropy decoding is the inverse process of entropy coding. However, the entropy coding and entropy decoding processes in related technologies are highly complex, resulting in low data encoding and decoding efficiency.
[0006] Summary of the Invention
[0007] According to various embodiments of the present application, a data encoding method, apparatus, computer device, computer-readable storage medium, and computer program product, as well as a data decoding method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.
[0008] In one aspect, the present application provides a data encoding method. The method comprises:
[0009] Acquiring auxiliary coding information of the feature data, and determining, based on the auxiliary coding information, scale parameter values corresponding to respective feature elements included in the feature data;
[0010] Quantizing each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value;
[0011] From the scale parameter quantization values, a portion of the scale parameter quantization values is screened out according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values;
[0012] Extracting and screening the characteristic elements corresponding to the quantized values of the scale parameters from the plurality of characteristic elements included in the characteristic data;
[0013] For each extracted characteristic element, entropy coding is performed on the characteristic element based on a quantized value of a scale parameter corresponding to the characteristic element to obtain coded data corresponding to the characteristic element; and
[0014] According to the encoded data, an encoded data stream of the feature data is determined.
[0015] On the other hand, the present application also provides a data encoding device. The device includes:
[0016] a scale parameter value determination module, configured to obtain auxiliary coding information of the feature data, and determine, based on the auxiliary coding information, the scale parameter value corresponding to each feature element included in the feature data;
[0017] a quantization value screening module, configured to quantize each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; and to screen out a portion of the scale parameter quantization values according to a preset screening condition from the scale parameter quantization values; wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values.
[0018] a feature element screening module, configured to extract, from the plurality of feature elements included in the feature data, feature elements corresponding to the quantized values of the scale parameters obtained by screening;
[0019] an entropy coding module, configured to perform entropy coding on each extracted characteristic element based on a quantized value of a scale parameter corresponding to the characteristic element, to obtain coded data corresponding to the characteristic element; and
[0020] The coded data stream determining module is used to determine the coded data stream of the feature data according to the coded data.
[0021] In another aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned data encoding method when executing the computer-readable instructions.
[0022] On the other hand, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the above-mentioned data encoding method when executed by a processor.
[0023] On the other hand, the present application further provides a computer program product, which includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, implements the steps of the above-mentioned data encoding method.
[0024] In another aspect, the present application provides a data decoding method. The method comprises:
[0025] Obtaining auxiliary coding information corresponding to a coded data stream, wherein the coded data stream is obtained by encoding a portion of characteristic elements of the characteristic data according to the auxiliary coding information;
[0026] Determining, according to the auxiliary coding information, a scale parameter value corresponding to each characteristic element included in the characteristic data;
[0027] Quantizing each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value;
[0028] From the scale parameter quantization values, a portion of the scale parameter quantization values is screened out according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values;
[0029] Performing entropy decoding on the encoded data stream based on the screened scale parameter quantization value to obtain corresponding feature elements;
[0030] Determining positions of characteristic elements constituting the characteristic data that have not been entropy-encoded based on the quantized values of the scale parameters that have not been screened out, and determining replacement elements at the positions of the characteristic elements that have not been entropy-encoded; and
[0031] The characteristic element obtained by entropy decoding and the replacement element are combined to obtain restored characteristic data.
[0032] On the other hand, the present application also provides a data decoding device. The device includes:
[0033] an auxiliary coding information acquisition module, configured to acquire auxiliary coding information corresponding to a coded data stream, wherein the coded data stream is obtained by encoding a portion of characteristic elements of the characteristic data according to the auxiliary coding information;
[0034] a scale parameter value determination module, configured to determine, based on the auxiliary coding information, a scale parameter value corresponding to each characteristic element included in the characteristic data;
[0035] a quantization value screening module, configured to perform quantization processing on each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; and to screen out a portion of the scale parameter quantization values from the scale parameter quantization values according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values;
[0036] an entropy decoding module, configured to perform entropy decoding on the coded data stream based on the screened scale parameter quantization values to obtain corresponding feature elements; determine positions of feature elements constituting the feature data that have not been entropy-encoded based on the unscreened scale parameter quantization values, and determine replacement elements at the positions of the feature elements that have not been entropy-encoded; and
[0037] The feature data recovery module is used to merge the feature element obtained by entropy decoding with the replacement element to obtain recovered feature data.
[0038] On the other hand, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned data decoding method when executing the computer-readable instructions.
[0039] In another aspect, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the above-mentioned data decoding method when executed by a processor.
[0040] On the other hand, the present application further provides a computer program product, which includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, implements the steps of the above-mentioned data decoding method. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.
[0042] FIG1 is a diagram illustrating an application environment of a data encoding method and a data decoding method in some embodiments;
[0043] FIG2 is a schematic flow chart of a data encoding method in some embodiments;
[0044] FIG3 is a schematic flow chart of a data encoding method in some other embodiments;
[0045] FIG4 is a schematic flow chart of a data decoding method in some other embodiments;
[0046] FIG5 is a schematic diagram of the overall process of video encoding and decoding in some embodiments;
[0047] FIG6 is a framework diagram of a coding / decoding model in some embodiments;
[0048] FIG7 is a schematic diagram of a flow chart of an entropy coding module in some embodiments;
[0049] FIG8 is a schematic diagram of an encoding process of a first device in some embodiments;
[0050] FIG9 is a schematic diagram of a decoding process of a first device in some embodiments;
[0051] FIG10 is a schematic diagram of a decoding process of a first device in some embodiments;
[0052] FIG11 is a schematic diagram of comparison results under different thresholds in some embodiments;
[0053] FIG12 is a block diagram of a data encoding device in some embodiments;
[0054] FIG13 is a block diagram of a data decoding device in some embodiments;
[0055] FIG14 is a diagram illustrating the internal structure of a computer device in some embodiments;
[0056] FIG15 is a diagram illustrating the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0058] The data encoding method and data decoding method provided in the embodiments of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other servers. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc.
[0059] The data encoding method provided in the embodiments of the present application can be executed by a first device, which can be the server 104 or the terminal 102 in Figure 1. During the encoding process, the first device can obtain auxiliary encoding information of the feature data, determine the scale parameter value corresponding to each feature element included in the feature data based on the auxiliary encoding information, quantize each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value, filter out a portion of the scale parameter quantization values from the scale parameter quantization values according to preset filtering conditions, extract feature elements corresponding to each of the filtered scale parameter quantization values from multiple feature elements included in the feature data, perform entropy encoding on each extracted feature element based on the scale parameter quantization value corresponding to the feature element, obtain encoded data corresponding to the feature element, and determine an encoded data stream of the feature data based on the encoded data.
[0060] The data encoding method provided in the embodiments of the present application can be performed by a second device, which can be the server 104 or the terminal 102 in Figure 1. During the decoding process, the second device can obtain auxiliary coding information corresponding to the encoded data stream. The encoded data stream is obtained by encoding a portion of feature elements of the feature data based on the auxiliary coding information. The second device further determines, based on the auxiliary coding information, a scale parameter value corresponding to each feature element included in the feature data, quantizes each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value, and selects a portion of the scale parameter quantization values from the scale parameter quantization values according to a preset screening condition. The selected scale parameter quantization values are greater than the unselected scale parameter quantization values. The encoded data stream is entropy decoded based on the selected scale parameter quantization values to obtain corresponding feature elements. The positions of the feature elements that have not been entropy encoded and that constitute the feature data are determined based on the unselected scale parameter quantization values, and replacement elements are determined for the positions of the feature elements that have not been entropy encoded. The feature elements obtained by entropy decoding and the replacement elements are merged to obtain restored feature data.
[0061] In some embodiments, as shown in FIG2 , a data encoding method is provided. The method is performed by a first device, which may be the server 104 or the terminal 102 in FIG1 . In the embodiments of the present application, the method is described by applying it to the server in FIG1 , and includes the following steps:
[0062] Step 202: Obtain auxiliary coding information of the feature data, and determine the scale parameter value corresponding to each feature element included in the feature data according to the auxiliary coding information.
[0063] Among them, feature data refers to the data obtained by pre-encoding the original data. The original data can be video frame data or image data, such as original video frame, original image, motion estimation data of video frame, residual compensation data of video frame, and the corresponding feature data is feature map. The original data can also be audio data, and the corresponding feature data can be audio feature variables. The original data can also be point cloud data, and the corresponding feature data can be point cloud feature data. Pre-coding includes transformation, quantization, inverse transformation and other processing, which can be implemented based on traditional mathematical methods or based on deep learning neural networks. Auxiliary coding information refers to information used to assist the coding process, that is, side information. Side information can be feature information obtained by further encoding feature data, for example, it can be feature information obtained by further extracting feature data by inputting feature data into a neural network. The number of feature elements contained in the side information is less than the number of feature elements of the feature data. For example, taking the feature data as a feature graph, the feature graph can be subjected to super-prior encoding to reduce the feature dimension through the video codec model to obtain auxiliary coding information of the feature graph. The auxiliary coding information can be further input into the entropy model in the video codec model, and the entropy model outputs the scale parameter value and position parameter value corresponding to each feature element contained in the feature data. The video codec model can include a VAE (Variational AutoEncoder) model. The entropy model corresponds to the prior of the VAE hidden layer representation, and the auxiliary coding information is used to assist the entropy model encoding, that is, the prior of the prior, and is therefore called super-prior coding.
[0064] The feature data includes feature elements, which are data units that constitute the feature data. Taking the feature data as a feature map as an example, the feature elements refer to the pixels of the feature map, and each pixel represents a feature element.
[0065] The scale parameter value corresponding to a feature element is the probability distribution parameter value corresponding to the feature element. The probability distribution parameter value corresponding to the feature element is the parameter value of the probability distribution function corresponding to the feature element. The probability distribution function corresponding to the feature element refers to the function used to describe the probability distribution of the feature element. The probability distribution of the feature element refers to the distribution of the probability of the feature element taking a range of possible values relative to these values. In some embodiments, the probability of a feature element taking a value can reflect the probability of that value appearing in the feature graph.
[0066] The relevant parameters of a probability distribution function include the scale parameter and the location parameter. The scale parameter determines the shape of the curve describing the probability distribution function, such as its flatness. A larger scale parameter value results in a flatter curve; conversely, a smaller scale parameter value results in a taller and thinner curve. The abscissa of the probability distribution function curve represents the values of the characteristic elements, while the ordinate represents the probability values. Therefore, a larger scale parameter value indicates a higher degree of dispersion in the probability distribution. In specific applications, the probability distribution of the characteristic elements can be a Gaussian distribution or a Laplace distribution. When the probability distribution of the characteristic elements is Gaussian, the scale parameter can be the variance or standard deviation of the probability distribution. The variance of a probability distribution can be the average of the sum of the squared differences between each probability value and the mean of all probability values in the probability distribution, while the standard deviation is the square root of the variance. The location parameter describes the location of the central tendency of the probability distribution. When the probability distribution of the characteristic elements is Gaussian, the location parameter can be the mean, or mathematical expectation, of the probability distribution of the characteristic elements. In the embodiment of the present application, the position parameter value of the feature element can be used as the predicted feature value of the feature element.
[0067] The scale parameter value of the feature element can also represent the degree of deviation of the feature element from the overall distribution of the feature data, and the scale parameter value can be positively correlated with the degree of deviation, that is, the larger the scale parameter value, the greater the degree of deviation of the feature element from the overall distribution of the feature data. This is because the larger the scale parameter value, the higher the discrete degree of the probability distribution, then the larger the range of the feature element value, and its degree of certainty is also agreed. Therefore, the scale parameter value can better reflect the degree of certainty of the feature element value. In the data encoding process, there is usually a lot of redundant information between the original data to be encoded between the previous and next frames. These redundant information usually show aggregation in the original data. The amount of information carried by these redundant information is small, and it can be accurately predicted in the encoding and decoding process with high certainty. Correspondingly, the feature elements corresponding to the redundant information also show aggregation in the feature data. When the degree of deviation of a feature element from the overall distribution of the feature data is greater, the certainty of the feature element is lower, that is, the scale parameter value of the feature element is larger, so the degree of deviation of the feature element from the overall distribution of the feature data can be represented by the scale parameter value of the feature element.
[0068] In some embodiments, the server may obtain feature data, perform super-a priori encoding on the feature data to reduce feature dimensions, obtain auxiliary encoding information of the feature data, and then determine the scale parameter values corresponding to each feature element included in the feature data based on the auxiliary encoding information.
[0069] For example, when the original data to be encoded is a video frame, the data encoding method of the present application can be implemented through a video coding model, which includes an entropy model. The entropy model refers to a model used to predict the data required for the entropy coding process. The entropy model can predict the probability distribution parameter value corresponding to each feature element in the feature map.
[0070] In step 204 , each scale parameter value is quantized to obtain a scale parameter quantization value corresponding to each scale parameter value. From each scale parameter quantization value, a portion of scale parameter quantization values is selected according to a preset selection condition. The selected scale parameter quantization values are larger than the unselected scale parameter quantization values.
[0071] Quantization refers to the process of discretizing scale parameter values. The scale parameter values determined based on the auxiliary coding information are floating-point numbers. Floating-point numbers are values within a continuous interval and are subject to uncertainty in different computing environments. Therefore, quantization is required. For example, quantization can be the process of representing the scale parameter values as integers. Preset screening conditions refer to conditions that can be used to select larger values from the scale parameter quantization values obtained through quantization, and can be set as needed. Optionally, the preset screening conditions can require that the scale parameter quantization values exceed a preset quantization threshold; optionally, the preset screening conditions can require that the scale parameter quantization values be ranked in the top N when sorted from smallest to largest; optionally, the preset screening conditions can require that the scale parameter quantization values be the largest N% of all scale parameter quantization values. The selected scale parameter quantization values are larger than the unselected scale parameter quantization values, and the scale parameter quantization values can be positively correlated with the degree of deviation they represent, meaning that the selected scale parameter quantization values can represent a greater degree of deviation than the unselected scale parameter quantization values.
[0072] In some embodiments, the server may determine a quantization step size, and perform quantization processing on each scale parameter value according to the quantization step size to obtain a scale parameter quantization value corresponding to each scale parameter value.
[0073] Considering the large amount of redundant information between data frames during the encoding process, most feature elements are clustered within the overall distribution of the feature data. If a feature element has a small degree of deviation, then its certainty is high and it is likely redundant information. Since redundant information can be accurately predicted during the encoding process, encoding and decoding of these elements can be skipped during the encoding process, significantly accelerating the entropy encoding and decoding process without compromising encoding and decoding quality, thereby improving encoding and decoding efficiency. The scale parameter quantization value is obtained by quantizing the scale parameter value and can also represent the degree of certainty of the feature element. Therefore, the server can filter the scale parameter quantization values according to preset filtering conditions to select a portion of scale parameter quantization values with larger values. In the hope of entropy encoding, these scale parameter quantization values can be used to select feature elements corresponding to non-redundant information for entropy encoding, while skipping entropy encoding of feature elements with redundant information.
[0074] In some specific embodiments, when the feature data is obtained by pre-encoding motion-compensated data, the server may further perform a secondary screening on the unfiltered scale parameter quantization values. In the secondary screening, the server may further obtain motion estimation data of the data frame to which the feature data belongs, determine, based on the motion estimation data, motion estimation values that match the positions of the feature elements corresponding to each scale parameter quantization value in the unfiltered scale parameter quantization values, and further filter, from the unfiltered scale parameter quantization values, scale parameter quantization values whose matching motion estimation values are greater than a motion estimation threshold.
[0075] In some specific embodiments, each scale parameter value is a floating-point number, and each scale parameter quantization value is an integer. During the quantization process, the first device maps continuous floating-point numbers to discrete integers, so that when a second device performs quantization using the same method, the same scale parameter quantization value can be obtained. The second device can further filter the scale parameter quantization value using the same filtering method as the first device. This allows the first and second devices to perform entropy encoding and decoding on the same feature element, avoiding inaccurate reconstructed data obtained by decoding due to inconsistent calculation results of scale parameter values when the encoding and decoding processes are performed across devices.
[0076] Step 206 : Extracting and screening the feature elements corresponding to the quantized values of the scale parameters from the plurality of feature elements included in the feature data.
[0077] The correspondence between the scale parameter quantization value and the feature element refers to a position correspondence. Taking the feature data as a feature map as an example, the scale parameter quantization value obtained by quantizing the scale parameter value determined for the feature data can be a map of the same size as the feature map. Thus, for each scale parameter quantization value in the map of the same size, the feature element at the same position in the feature map is the feature element corresponding to the scale parameter quantization value.
[0078] In some embodiments, the server can extract the feature elements corresponding to each scale parameter quantization value obtained by screening from the multiple feature elements included in the feature data. Since only a part of the scale parameter quantization value is screened out, the screened feature elements also only include part of the feature elements of the feature data. In the subsequent entropy coding process, the server only performs entropy coding on the screened feature elements and skips the entropy coding on the feature elements that are not screened out.
[0079] Step 208 : For each extracted characteristic element, entropy coding is performed on the characteristic element based on the quantized value of the scale parameter corresponding to the characteristic element to obtain coded data corresponding to the characteristic element.
[0080] In some embodiments, since the original scale parameter value is quantized, when performing entropy coding, for each feature element screened out, the server can re-determine the scale parameter value of the feature element based on the scale parameter quantization value corresponding to the feature element. According to the re-determined scale parameter value, the probability distribution function required for entropy coding can be determined, so that the feature map can be compressed into as few byte streams as possible through the probability distribution function. Entropy coding can be implemented by arithmetic coding or range coding. Taking arithmetic coding as an example, after obtaining the probability distribution function required for arithmetic coding, the probability value of each feature element in the feature map can be calculated, and then the feature elements are read in one by one. Each time a feature element is read in, the range of the feature map on [0,1] is reduced to the latest obtained interval according to the ratio. The value of the ratio is determined by the probability value of each feature element. Then, it is iterated in sequence until all feature elements are read. Any decimal in the obtained interval is output in binary form to obtain the encoded data stream. In some embodiments and specific applications, re-determining the scale parameter value according to the scale parameter quantization value can be implemented by formula (1), where scale QS is the quantization step size, scale max The maximum value of the scale parameter, which can be 64.0. minis the minimum value of the scale parameter, specifically 0.11, L is the maximum quantization level, specifically 256, index is the scale parameter quantization value, ranging from 0 to 255, and scale_q represents the re-determined scale parameter value. scale_q=exp(log(scale min )+scale QS ×index) Formula (1)
[0081] Optionally, the server can also construct a mapping relationship between the scale parameter quantization value index and scale_q using the above formula (1) to construct a probability distribution function lookup table, such as: index = 0 corresponds to scale_q = 0.1100, index = 1 corresponds to scale_q = 0.1128. Therefore, after filtering and obtaining the scale parameter quantization value, the scale parameter value scale_q can be directly searched in the mapping relationship table, thereby improving arithmetic coding efficiency.
[0082] Optionally, after obtaining the scale parameter value scale_q corresponding to a certain feature element, the server can determine the probability distribution function required for arithmetic coding based on the scale parameter value scale_q and position parameter corresponding to the feature element, and then determine the probability value of the feature element by substituting the value of the feature element into the probability distribution function to calculate the probability value; also optionally, the server can subtract the value of the corresponding position parameter value from the value at each feature element position in the feature graph y, and the feature graph y_0 obtained after the subtraction can be regarded as a zero-mean distribution, so that the server can search a pre-established probability value lookup table based on the value of each feature element in the zero-mean distribution feature graph y_0 to obtain the probability value, wherein the pre-established probability value lookup value is established in the following manner: under each scale parameter value scale_q and the probability distribution function determined by the mean 0, the probability value of each possible value of the feature element is calculated, and a corresponding relationship between the possible value of the feature element and the calculated probability value under each probability distribution function is established. Step 210, based on the encoded data, determines the encoded data stream of the feature data.
[0083] In some embodiments, the server may directly use the obtained coded data as the coded data stream of the feature data, or further process the coded data and use the further processed feature data as the coded data stream of the feature data.
[0084] In a specific application, the server can send the encoded data stream to the second device, and the second device can decode the encoded data stream to reconstruct the data frame to which the feature data belongs. Optionally, the server can encode the auxiliary encoding information into the encoded data stream of the feature data and send it together to the second device.
[0085] In the above-mentioned data encoding method, auxiliary encoding information of feature data is obtained, and scale parameter values corresponding to each feature element included in the feature data are determined based on the auxiliary encoding information. Quantization processing is performed on each scale parameter value, and a portion of scale parameter quantization values obtained from the quantization processing is screened out according to a preset screening condition, where the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values. Feature elements corresponding to each of the screened scale parameter quantization values are extracted from multiple feature elements included in the feature data. For each extracted feature element, entropy encoding is performed on the feature element based on the scale parameter quantization value corresponding to the feature element to obtain encoded data corresponding to the feature element. Furthermore, an encoded data stream of the feature data can be determined based on the encoded data. Since, during the entropy encoding process, the scale parameter quantization values can be screened according to the preset screening condition, and the feature elements can be screened based on the screened scale parameter quantization values, entropy encoding is performed only on the screened feature elements, and the entropy encoding process for the unscreened feature elements is skipped. This significantly reduces the number of feature elements that require entropy encoding, greatly reduces the complexity of entropy encoding, and significantly improves data encoding efficiency.
[0086] In some embodiments, selecting a portion of the scale parameter quantization values according to a preset screening condition from the scale parameter quantization values includes: comparing the scale parameter quantization values obtained by the quantization process with a preset quantization threshold value to obtain a comparison result corresponding to each scale parameter quantization value; and selecting the scale parameter quantization values whose corresponding comparison results indicate that the scale parameter quantization values exceed the preset quantization threshold value to obtain a portion of the scale parameter quantization values.
[0087] In this embodiment, the preset screening condition requires that the quantized value of the scale parameter exceed a preset quantization threshold. The preset quantization threshold can be set and adjusted as needed. It is understood that the larger the preset quantization threshold, the more feature elements are skipped during the entropy coding process, the better the entropy coding acceleration effect, and the more significant the improvement in coding efficiency. Of course, different preset quantization thresholds also have a certain impact on the reconstruction effect during the decoding process, so a trade-off can be made based on actual needs.
[0088] In some embodiments, during the scale parameter quantization value screening process, the server may compare each scale parameter quantization value obtained by the quantization processing with a preset quantization threshold value to obtain a comparison result corresponding to each scale parameter quantization value. If the comparison result of a certain scale parameter quantization value indicates that the scale parameter quantization value exceeds the preset quantization threshold value, the scale parameter quantization value is screened out. If the comparison result of a certain scale parameter quantization value indicates that the scale parameter quantization value does not exceed the preset quantization threshold value, the scale parameter quantization value is filtered out. Finally, a portion of scale parameter quantization values can be screened out, and these portion of scale parameter quantization values are greater than the scale parameter quantization values that have not been screened out.
[0089] In this embodiment, by comparing the quantized values of each scale parameter obtained by the quantization process with a preset quantization threshold, the quantized values of the scale parameter are screened to find those whose comparison results indicate that they exceed the preset quantization threshold. This allows for rapid screening of the quantized values of the scale parameter, improves screening efficiency, and thus improves coding efficiency.
[0090] In some embodiments, extracting and filtering the feature elements corresponding to the respective scale parameter quantization values from the plurality of feature elements included in the feature data includes: determining motion estimation values corresponding to the respective scale parameter quantization values in a portion of the scale parameter quantization values, where the motion estimation values match the positions of the feature elements corresponding to the corresponding scale parameter quantization values; filtering the scale parameter quantization values whose corresponding motion estimation values are greater than a motion estimation threshold from the portion of the scale parameter quantization values; and extracting and filtering the feature elements corresponding to the respective scale parameter quantization values from the plurality of feature elements included in the feature data.
[0091] In this embodiment, the feature data is a feature map corresponding to the video frame data, the motion estimation data is a motion vector map obtained by performing motion estimation, and the motion estimation value of the motion estimation data can represent the magnitude of the motion between video frames. Generally speaking, pixels at positions with larger motion estimation values contain more important information, while pixels at positions with larger motion estimation values are usually redundant information. Based on this, in this embodiment, a portion of the scale parameter quantization values that have been screened can also be secondary screened based on the motion estimation value.
[0092] In some embodiments, the vector map obtained by motion estimation is the same size as the original size of the video frame, while the feature data is equivalent to the feature map after feature extraction and is smaller than the size of the motion estimation map. Therefore, in practical applications, the vector map obtained by motion estimation is typically scaled down to match the size of the feature map. This allows for determining, from the scaled down vector map, motion estimation values that match the positions of the feature elements corresponding to each scale parameter quantization value in the portion of scale parameter quantization values that have been screened out. In the vector map obtained by motion estimation, each position has two values: an x value in the horizontal direction and a y value in the numerical direction. The motion estimation value may be a statistical value of the values in these two directions. For example, the x value and the y value at each position may be added to obtain the motion estimation value at that position. Since the vector map obtained by motion estimation is the same size as the feature map, for each pixel in the feature map, the motion estimation value at the same position in the vector map is the matching motion estimation value. The server may continue to screen for scale parameter quantization values whose matching motion estimation values are greater than a motion estimation threshold from the portion of scale parameter quantization values obtained by the first screening. Furthermore, the server may screen for feature elements corresponding to each scale parameter quantization value obtained by the second screening from the plurality of feature elements included in the feature map.
[0093] It is understandable that in the video encoding process, the original data can be a video frame used as a reference frame, a vector diagram obtained by motion estimation, and motion compensation data, and only for entropy encoding of motion compensation data can the method provided in this embodiment be used for secondary screening.
[0094] In the above embodiment, since a second screening can be performed using the motion estimation value in the motion estimation data, it is possible to avoid screening out feature elements corresponding to redundant information during the first screening process, thereby skipping more feature elements during the entropy coding process, further reducing the complexity of entropy coding.
[0095] In some embodiments, each scale parameter value is quantized separately, including: determining a quantization step size, where the quantization step size is positively correlated with the difference between the maximum value and the minimum value of the scale parameter value, and the quantization step size is negatively correlated with a maximum quantization level set for the quantization process; determining the difference between each scale parameter value and the minimum value of the scale parameter value; and performing rounding operations on the quantization step sizes according to the differences corresponding to each scale parameter value to obtain scale parameter quantization values corresponding to each scale parameter value.
[0096] In some embodiments, the quantization step size can be calculated by the following formula (2), where scale QS is the quantization step size.
[0097] The scale parameter quantization value can be calculated using the following formula (3), where index is the scale parameter quantization value, and its value range is an integer between 0 and 255.
[0098] It can be understood that the above formula (3) represents the Rounding is performed, that is, rounding to the nearest integer. In other embodiments, Round up or round down. Rounding up can be to the nearest integer value, rounding down can be to the nearest integer value, and rounding to the nearest integer value can be to the nearest integer value by rounding up. For example, the value obtained by rounding up 71.97 can be 72, the value obtained by rounding down 71.97 can be 71, and the value obtained by rounding to the nearest integer can be 72.
[0099] In this embodiment, by determining the quantization step size, respectively determining the difference between each scale parameter value and the minimum value, and performing a rounding operation on the quantization step size according to the difference corresponding to each scale parameter value, respectively, the scale parameter quantization value corresponding to each scale parameter value is obtained, so that each scale parameter value can be quantized more accurately.
[0100] In some embodiments, the feature data is a feature map; the data encoding method of the present application further includes: determining, from a plurality of feature elements included in the feature map, feature elements corresponding to respective unscreened scale parameter quantization values; dividing the feature elements corresponding to respective unscreened scale parameter quantization values into regions according to positions of the feature elements corresponding to respective unscreened scale parameter quantization values in the feature map to obtain a plurality of feature element regions; for each of the plurality of feature element regions, selecting at least one feature element located at a center position of the targeted feature element region as a region representative element; and performing entropy encoding on the representative feature elements based on the scale parameter quantization values of the representative feature elements to obtain additional encoded data of the feature data.
[0101] The region representative element refers to a characteristic element used to represent other elements in the region. There can be one or more region representative elements, where more than one refers to at least two.
[0102] In some embodiments, considering that the redundant information between video frames usually has a certain degree of aggregation, that is, the characteristic elements corresponding to the unscreened scale parameter quantization values are clustered in a certain area in the feature map, therefore, during the encoding process, the characteristic elements corresponding to the unscreened scale parameter quantization values can also be divided into regions. The division process can specifically be to divide the characteristic elements whose distance is less than a preset threshold into a region, for each characteristic element region divided, select at least one characteristic element at the center position of the targeted characteristic element region as the region representative element, and perform entropy encoding on the representative characteristic element based on the scale parameter quantization value representing the characteristic element to obtain additional encoded data of the characteristic data, and then the additional encoded data can be encoded into the encoded data stream of the characteristic data and participate in the decoding process together with the encoded data stream.
[0103] In the above embodiment, by selecting the region representative element and performing entropy coding on the region representative element, as much information as possible can be encoded into the encoded data stream while ensuring that the entropy coding of some characteristic elements is skipped, so as to further ensure the accuracy of the image obtained by decoding and reconstruction.
[0104] In some embodiments, auxiliary coding information of feature data is obtained, and the scale parameter value corresponding to each feature element included in the feature data is determined based on the auxiliary coding information, including: obtaining feature data, performing super-prior coding on the feature data to reduce the feature dimension, and obtaining auxiliary coding information of the feature data; using the auxiliary coding information, predicting the probability distribution parameter value of each feature element included in the feature data, and obtaining the scale parameter value corresponding to each feature element.
[0105] In this embodiment, the entropy coding process can be implemented by an AI model. In a specific application, the AI model here can be a VAE (Variational AutoEncoder) model, wherein the entropy model is the prior of the VAE hidden layer representation, and the auxiliary coding information is used to assist the entropy model encoding, that is, the prior of the prior, so it is called super prior coding.
[0106] In some embodiments, after the server obtains the feature data, it can input the feature data into a neural network, perform super-prior coding to reduce the feature dimension through the neural network, obtain auxiliary coding information of the feature data, input the auxiliary coding information as a priori parameters into the entropy model, and use the entropy model to estimate the probability distribution of each feature element. The process of estimating the probability distribution is equivalent to super-prior decoding, and obtains the scale parameter value corresponding to each feature element included in the feature data and the predicted feature value corresponding to each feature element. Among them, the scale parameter value can specifically be the variance of the probability distribution function of the feature element, and the predicted feature value can specifically be the mean of the probability distribution function of the feature element.
[0107] In the above embodiment, auxiliary coding information is obtained through super-prior coding, and the probability distribution of each feature element is estimated using the auxiliary coding information to obtain a scale parameter value, which can capture the spatial correlation of the hidden layer representation and make the obtained scale parameter value more accurate.
[0108] In some embodiments, the characteristic data corresponds to the current data frame; the method also includes: obtaining the encoded data stream of the previous frame data frame, and reconstructing the previous frame data frame based on the encoded data stream of the previous frame data frame; performing motion estimation on the current data frame based on the reconstructed previous frame data frame to obtain motion estimation data of the current data frame; performing residual compensation estimation on the current data frame based on the motion estimation data to obtain residual compensation data of the current data frame; and using the motion estimation data and the residual compensation data as the characteristic data corresponding to the current data frame.
[0109] The coded data stream of the previous data frame is obtained by encoding the feature data of the previous data frame. The specific steps include: obtaining auxiliary coding information of the feature data of the previous data frame, determining the scale parameter value corresponding to each feature element included in the feature data based on the auxiliary coding information; quantizing each scale parameter value, and filtering out a portion of the scale parameter quantization values obtained from the quantization process according to preset filtering conditions; the filtered scale parameter quantization values are larger than the unfiltered scale parameter quantization values; extracting feature elements corresponding to each of the filtered scale parameter quantization values from multiple feature elements included in the feature data; performing entropy coding on each extracted feature element based on the scale parameter quantization value corresponding to the feature element to obtain coded data corresponding to the feature element; and determining the coded data stream of the previous data frame based on the coded data.
[0110] In some embodiments, when encoding is required for the current data frame, the server can obtain the encoded data stream of the previous frame data frame, reconstruct the previous frame data frame based on the encoded data stream of the previous frame data frame, perform motion estimation on the current data frame based on the reconstructed previous frame data frame, and obtain motion estimation data of the current data frame, perform residual compensation estimation on the current data frame based on the motion estimation data, and obtain residual compensation data of the current data frame. The obtained motion estimation data and residual compensation data are the original data corresponding to the current data frame. The server can obtain feature data by pre-encoding these original data. After obtaining the feature data, the server can perform entropy encoding by executing the steps of the previous embodiment to obtain the encoded data stream of the current data frame.
[0111] In the above embodiment, the encoded data stream of the previous frame data frame is obtained, the previous frame data frame is decoded and reconstructed based on the encoded data stream of the previous frame data frame, and the current data frame is encoded based on the reconstructed data frame, which can effectively ensure that the data decoded and reconstructed by the second device is consistent with the first device.
[0112] In some embodiments, reconstructing a previous frame of data based on the encoded data stream of the previous frame of data includes: entropy decoding the encoded data stream of the previous frame of data based on the scale parameter quantization value screened out during the encoding process of the feature data of the previous data frame to obtain corresponding feature elements; determining replacement elements at the positions of the feature elements that constitute the feature data of the previous data frame and have not been entropy encoded; merging the feature elements and the replacement elements obtained by entropy decoding to obtain feature data restored for the previous data frame; and reconstructing data based on the feature data restored for the previous data frame to obtain a reconstructed previous frame of data.
[0113] In some embodiments, during the encoding of the previous data frame, the server may filter out a portion of the scale parameter quantization values and then perform entropy encoding only on the feature elements corresponding to the portion of the scale parameter quantization values. The server may further store the filtered scale parameter quantization values and the unfiltered scale parameter quantization values separately. During the decoding and reconstruction of the previous data frame by the first device, to ensure the accuracy of the reconstruction result, a distribution function required for entropy decoding may be determined based on the scale parameter quantization values stored during the encoding of the previous data frame. Entropy decoding may then be performed based on the determined distribution function to obtain corresponding feature elements. The obtained feature elements are data recovered from the feature elements encoded during the entropy encoding process. Feature elements not encoded during the entropy encoding process cannot be obtained through entropy decoding during the decoding process. Therefore, during the decoding process, the server may determine, based on the positions corresponding to the unfiltered scale parameter quantization values, replacement elements at the positions of the feature elements not entropy encoded that constitute the feature data. The feature elements obtained through entropy decoding and the replacement elements may then be merged to obtain feature data recovered for the previous data frame. Finally, data reconstruction may be performed based on the feature data recovered for the previous data frame to obtain a reconstructed previous data frame. The data reconstruction here refers to the inverse process of the above-mentioned pre-encoding.
[0114] In the above embodiment, in the first device, the coded data stream of the previous frame of data is entropy decoded using the scale parameter quantization value screened out during the encoding process based on the feature data of the previous frame of data to obtain the corresponding feature elements, thereby ensuring that the encoding process and the decoding process perform entropy encoding and entropy decoding on the same feature elements, thereby ensuring the accuracy of the previous frame of data reconstructed by decoding.
[0115] In some specific embodiments, as shown in FIG3 , a data encoding method is provided. The method is performed by a first device, which may be the server 104 or the terminal 102 in FIG1 . In the embodiments of the present application, the method is described by applying it to the server in FIG1 as an example, and includes the following steps:
[0116] Step 302 : Entropy decoding is performed on the coded data stream of the previous data frame based on the scale parameter quantization value selected during the encoding process of the feature data of the previous data frame to obtain corresponding feature elements.
[0117] Step 304: Determine a replacement element at the location of the feature element that constitutes the feature data of the previous data frame and has not been entropy encoded.
[0118] Step 306: Merge the characteristic elements and replacement elements obtained by entropy decoding to obtain characteristic data restored for the previous data frame.
[0119] Step 308 : reconstructing data based on the feature data recovered for the previous data frame to obtain a reconstructed previous data frame.
[0120] Step 310 : performing motion estimation on the current data frame based on the reconstructed previous data frame to obtain motion estimation data of the current data frame.
[0121] Step 312 : performing residual compensation estimation on the current data frame based on the motion estimation data to obtain residual compensation data of the current data frame, and performing pre-encoding on the residual compensation data to obtain feature data.
[0122] Step 314 : Perform super-a priori coding on the feature data to reduce the feature dimension, and obtain auxiliary coding information of the feature data.
[0123] Step 316 : Using the auxiliary coding information, predict the probability distribution parameter value of each feature element included in the feature data, and obtain the scale parameter value corresponding to each feature element.
[0124] Step 318: Determine the quantization step size, and determine the difference between each scale parameter value and the minimum value of the scale parameter value.
[0125] The quantization step size is positively correlated with the difference between the maximum value and the minimum value of the scale parameter value, and the quantization step size is negatively correlated with the maximum quantization level set for the quantization process.
[0126] In step 320 , a rounding operation is performed on the quantization step size according to the differences corresponding to the respective scale parameter values to obtain the scale parameter quantization value corresponding to each scale parameter value.
[0127] Step 322 : Compare the quantized values of the scale parameters obtained by the quantization process with the preset quantization threshold value, select the quantized values of the scale parameters whose comparison results indicate that they exceed the preset quantization threshold value, and obtain a portion of the quantized values of the scale parameters.
[0128] Step 324 : Determine, based on the motion estimation data of the current data frame, motion estimation values that match the positions of the feature elements corresponding to the respective scale parameter quantization values in the portion of scale parameter quantization values obtained by screening.
[0129] Step 326 : Filter out, from the portion of scale parameter quantization values obtained through screening, scale parameter quantization values whose matching motion estimation values are greater than a motion estimation threshold.
[0130] Step 328 : Extracting and screening the feature elements corresponding to the quantized values of the scale parameters from the multiple feature elements included in the feature data.
[0131] Step 330 , for each extracted characteristic element, entropy coding is performed on the characteristic element based on the quantized value of the scale parameter corresponding to the characteristic element to obtain coded data corresponding to the characteristic element, and a coded data stream of the characteristic data is determined based on the coded data.
[0132] It should be noted that, in actual applications, the server also needs to pre-encode the motion estimation data to obtain feature data. After obtaining the feature data corresponding to the motion estimation data, the server can further filter and obtain the scale parameter quantization value through steps 314 to 322, and then obtain the encoded data stream through steps 328 to 330.
[0133] In the above embodiment, during the entropy coding process, the scale parameter quantization value can be filtered according to the preset filtering conditions, and the characteristic elements can be filtered according to the filtered scale parameter quantization value, and only the filtered characteristic elements are entropy coded, and the entropy coding process of the unfiltered characteristic elements is skipped, thereby significantly reducing the number of characteristic elements that need to be entropy coded, greatly reducing the complexity of entropy coding, and significantly improving data coding efficiency.
[0134] In some embodiments, as shown in FIG4 , a data decoding method is provided. The method is performed by a second device, which may be the server 104 or the terminal 102 in FIG1 . In the embodiments of the present application, the method is described by applying it to the terminal in FIG1 , and includes the following steps:
[0135] Step 402: Obtain auxiliary coding information corresponding to the coded data stream.
[0136] Auxiliary coding information refers to information used to assist in the encoding process, namely side information. Side information can be feature information obtained by further encoding the feature data. For example, it can be feature information obtained by inputting the feature data into a neural network for further extraction. The number of feature elements contained in the side information is fewer than the feature elements of the feature data. The encoded data stream is obtained by encoding a portion of the feature elements of the feature data based on the auxiliary coding information. The specific encoding process can be referred to in the above embodiment and will not be repeated here.
[0137] In some embodiments, the first device may encode auxiliary coding information into the coded data stream during the encoding process, so that after receiving the coded data stream, the terminal can obtain the auxiliary coding information from the received coded data stream, and further can implement data decoding through the auxiliary coding information.
[0138] Step 404: Determine the scale parameter value corresponding to each feature element included in the feature data according to the auxiliary coding information.
[0139] In some embodiments, the terminal may estimate the probability distribution parameter value of each feature element included in the feature data using the auxiliary coding information to obtain the scale parameter value corresponding to each feature element, and may also obtain the predicted feature value corresponding to each feature element.
[0140] In step 406 , each scale parameter value is quantized to obtain a scale parameter quantization value corresponding to each scale parameter value. From each scale parameter quantization value, a portion of scale parameter quantization values is selected according to a preset selection condition; the selected scale parameter quantization values are larger than the unselected scale parameter quantization values.
[0141] Among them, quantization refers to the process of discretizing the scale parameter value. The scale parameter value determined according to the auxiliary coding information is a floating point number. The floating point number is a value within a continuous interval and has uncertainty in different computing environments. Therefore, the scale parameter value needs to be quantized. The preset filtering conditions can be set as needed. Optionally, the preset filtering condition can be to require that the scale parameter quantization value exceeds a preset quantization threshold; optionally, the preset filtering condition can also be to require that the scale parameter quantization value is ranked in the top N when sorted from small to large; optionally, the preset filtering condition can also be to require that the scale parameter quantization value is the data with the largest N% ratio among all scale parameter quantization values.
[0142] In some embodiments, the terminal may determine a quantization step size and perform quantization processing on each scale parameter value according to the quantization step size to obtain a scale parameter quantization value corresponding to each scale parameter value. Since entropy coding is performed only on some feature elements by screening the scale parameter quantization values during the data encoding process, i.e., some feature elements are skipped during the entropy coding process, the second device needs to screen out some scale parameter quantization values according to the same preset screening conditions as the first device.
[0143] In some specific embodiments, the preset screening condition is that a scale parameter quantization value is required to exceed a preset quantization threshold; and screening out a portion of the scale parameter quantization values obtained from the quantization process according to the preset screening condition includes: comparing the scale parameter quantization values obtained from the quantization process with the preset quantization threshold to obtain a comparison result corresponding to each scale parameter quantization value; and screening out, from the scale parameter quantization values, scale parameter quantization values whose comparison results indicate that they exceed the preset quantization threshold, to obtain a portion of the scale parameter quantization values.
[0144] In some specific embodiments, when the feature data is obtained by pre-encoding motion compensation data, the terminal may further perform a secondary screening for the unfiltered scale parameter quantization values. In the secondary screening, the terminal may further obtain motion estimation data of the data frame to which the feature data belongs, determine, based on the motion estimation data, motion estimation values that match the positions of the feature elements corresponding to each scale parameter quantization value in the unfiltered scale parameter quantization values, and further filter, from the unfiltered scale parameter quantization values, scale parameter quantization values whose matching motion estimation values are greater than a motion estimation threshold.
[0145] In some specific embodiments, each scale parameter value is a floating-point number, and each scale parameter quantization value is an integer. During the quantization process, the first device maps continuous floating-point numbers to discrete integers, so that when a second device performs quantization using the same method, the same scale parameter quantization value can be obtained. The second device can further filter and obtain the scale parameter quantization value in the same manner as the first device, so that the first and second devices can perform entropy encoding and decoding on the same feature elements, avoiding inconsistent calculation results of scale parameter values and inaccurate reconstructed data obtained by decoding when the encoding and decoding processes are performed across devices.
[0146] Step 408 : Perform entropy decoding on the encoded data stream based on the screened scale parameter quantization value to obtain corresponding feature elements.
[0147] Specifically, the terminal may re-determine the scale parameter value based on the quantization of the screened scale parameter values, determine the probability distribution function required for arithmetic coding according to the re-determined scale parameter value, and then perform entropy decoding according to the probability distribution function to obtain the corresponding feature element.
[0148] In a specific application, the terminal can construct a probability distribution function lookup table between the scale parameter quantization value and the scale parameter value through the above formula (1), so that after obtaining the scale parameter quantization value, the terminal can look up the probability distribution function lookup table to obtain the scale parameter value cale_q, and determine the probability distribution function required for arithmetic coding according to cale_q.
[0149] Step 410 : determining the positions of the feature elements constituting the feature data that have not been entropy-coded based on the quantized values of the scale parameters that have not been screened out, and determining replacement elements at the positions of the feature elements that have not been entropy-coded.
[0150] The scale parameter quantization values that are not screened out during entropy decoding are also not screened out during the entropy coding process. Therefore, the characteristic elements corresponding to these scale parameter values are feature elements that are not entropy coded. Therefore, during the decoding process, the terminal can determine the positions corresponding to the unscreened scale parameter quantization values as the positions of the feature elements that are not entropy coded. Since the feature elements at these positions are not entropy coded, these feature elements cannot be restored during the entropy decoding process. Therefore, in this embodiment, the terminal can further determine replacement elements for these positions.
[0151] Step 412: Merge the characteristic elements and replacement elements obtained by entropy decoding to obtain restored characteristic data.
[0152] In some embodiments, during the decoding process, the terminal can determine the replacement element at the position of the characteristic element that has not been entropy-encoded and constitutes the feature data based on the position corresponding to the quantization value of the scale parameter that has not been screened out, and can further merge the characteristic element and the replacement element obtained by entropy decoding, that is, combine the characteristic element and the replacement element according to their respective positions to obtain restored feature data. The terminal can further perform data reconstruction based on the restored feature data to obtain a reconstructed data frame. The data reconstruction here refers to the inverse process of the above-mentioned pre-encoding.
[0153] The data decoding method obtains auxiliary coding information corresponding to the coded data stream, determines the scale parameter value corresponding to each characteristic element included in the characteristic data according to the auxiliary coding information, quantizes each scale parameter value, and selects a portion of the scale parameter quantization values obtained from the quantization process according to a preset screening condition, wherein the selected scale parameter quantization values are larger than the unselected scale parameter quantization values, entropy decoding is performed on the coded data stream based on the selected scale parameter quantization values to obtain the corresponding characteristic elements, determines the replacement elements for the positions of the characteristic elements that are not entropy-coded and constitute the characteristic data, and replaces the characteristic elements obtained by entropy decoding with the replacement elements. The characteristic elements and the replacement elements are merged to obtain the restored characteristic data. During the decoding process, since larger scale parameter quantization values can be screened out, entropy decoding is performed only based on the screened scale parameter quantization values. Entropy decoding can be performed only on some characteristic elements, which greatly reduces the complexity of entropy coding and significantly improves data decoding efficiency. In addition, since the screening process screens the scale parameter quantization values obtained by quantization processing according to preset screening conditions, the scale parameter values are discretized during the quantization process, which increases the probability that the screened scale parameter quantization values are consistent with the scale parameter quantization values screened out during the encoding process, thereby improving the accuracy of the characteristic data restored by decoding.
[0154] In some embodiments, each scale parameter value is quantized separately, including: determining a quantization step size, where the quantization step size is positively correlated with the difference between the maximum value and the minimum value of the scale parameter value, and negatively correlated with a quantization level set for the quantization process; determining the difference between each scale parameter value and the minimum value; and rounding the quantization step size according to the difference corresponding to each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value.
[0155] As can be seen from the above embodiments, for feature elements that have not been entropy-coded, replacement elements need to be determined. The following embodiments provide two different methods for determining replacement elements.
[0156] In some embodiments, the quantization values of the scale parameters obtained by the quantization process correspond one-to-one to the positions of the feature elements in the feature data, and the auxiliary coding information is obtained by performing super-prior coding on the feature data; determining the replacement element at the position of the feature element that has not been entropy coded includes: obtaining the predicted feature value corresponding to each feature element, the predicted feature value is obtained by predicting the probability distribution parameter value of the corresponding feature element using the auxiliary coding information; for the position of each feature element that has not been entropy coded, determining a predicted feature value that matches the position from the predicted feature value, and determining the replacement element at the position of the feature element based on the determined predicted feature value.
[0157] In this embodiment, when the terminal uses the auxiliary coding information to predict the probability distribution parameter value of each feature element, it can also obtain a predicted feature value. The predicted feature value can specifically be the mean value of the probability distribution function corresponding to the feature element. Since the feature elements skipped in the entropy coding process are basically feature elements corresponding to redundant information, and the predicted feature values corresponding to these feature elements are usually more accurate, during the decoding process, for feature elements that have not been entropy coded, they can be replaced by the predicted feature value obtained by predicting the probability distribution parameter value of the corresponding feature element. Based on this, after obtaining the predicted feature value, the terminal screens the predicted feature value that matches the position of each feature element that has not been entropy coded from each predicted feature value, and determines the replacement element at the position of the feature element that has not been entropy coded, which constitutes the feature data, based on the screened predicted feature values.
[0158] Taking the feature data as an example of a feature map, the probability distribution of each feature element can be estimated using auxiliary coding information to obtain a predicted feature map. For the quantized values of each scale parameter that have not been screened out, their positions in the map are determined as the positions of the feature elements that have not been entropy coded. Then, the predicted feature value at the same position can be determined from the obtained predicted feature map, and the predicted feature value can be used as a replacement element for the feature element at the same position in the feature map.
[0159] In the above embodiment, considering that the elements skipped in the encoding process are usually highly certain, the predicted feature values estimated by the probability distribution can be used as real values to replace them, which can reduce the burr phenomenon in the encoding and decoding process.
[0160] In some embodiments, the feature data is a feature map, and the quantized values of the scale parameters obtained by the quantization process correspond one-to-one to the positions of the feature elements in the feature map; determining the replacement elements at the positions of the feature elements that have not been entropy encoded includes: obtaining additional encoded data of the feature map from the encoded data stream, entropy decoding the additional encoded data, and obtaining corresponding region representative elements; in the feature map, dividing the regions based on the positions of the feature elements that have not been entropy encoded; and for each divided region, determining the region representative elements located in the targeted region as the replacement elements for the positions of each feature element that has not been entropy encoded in the targeted region.
[0161] In this embodiment, when performing encoding, the first device may further encode additional encoded data into the encoded data stream. The method for determining the additional encoded data may refer to the above embodiment and will not be described in detail here. Thus, during the decoding process, the terminal may obtain the additional encoded data of the feature map from the encoded data stream, perform entropy decoding on the additional encoded data, and obtain the corresponding region representative elements. Considering that during the encoding process, the feature elements selected for entropy encoding are the feature elements corresponding to the selected scale parameter quantization values, and the feature elements not entropy encoded are the feature elements corresponding to the unselected scale parameter quantization values, the positions corresponding to the unselected scale parameter quantization values may be determined as the positions of the feature elements not entropy encoded. In the feature map, the positions of the feature elements not entropy encoded are divided into regions in the same manner as the encoding process. For each divided region, the region representative element located in the targeted region is determined as the replacement element for the position of each feature element not entropy encoded in the targeted region.
[0162] In the above embodiment, by determining the position of the characteristic elements that have not been entropy encoded, and performing region division based on the position of the characteristic elements that have not been entropy encoded, for the position of each characteristic element that has not been entropy encoded in each divided region, the region representative element located in the targeted region is used as the replacement element for the targeted position, so that a more accurate replacement element can be determined.
[0163] In some specific embodiments, the present application also provides an application scenario in which the data encoding method and data decoding method of the present application can be used to encode and decode a video, wherein the terminal can be installed with a video application, and the terminal can play the video through the video application. Referring to Figure 5, the server can act as an encoding end to encode each video frame in the original video to obtain an encoded data stream and then send it to the terminal. The terminal can act as a decoding end to decode the encoded data stream to obtain a reconstructed video frame, so that the video can be played based on the reconstructed video frame.
[0164] Continuing to refer to Figure 5, in this embodiment, the encoding at the encoding end and the decoding at the decoding end are both implemented based on the video coding model, and the video coding model is an AI model. In the encoding and decoding process of the video coding and decoding model, it mainly includes two links, namely I frame encoding and decoding (intra-frame coding) and P frame encoding and decoding (inter-frame coding). Usually, I frame encoding and decoding is implemented using the AI image coding and decoding algorithm, and the P frame encoding and decoding model needs to be designed according to the characteristics of inter-frame coding, and is usually divided into two modules: motion estimation and residual compensation. The core idea of these models is to convert the original image into some feature maps that need to be transmitted, and reduce the amount of bytes transmitted by the feature map through the entropy coding module, thereby greatly reducing the byte size of the video transmission.
[0165] During encoding, the original image is converted into a feature map for transmission in the I-frame model. For the P-frame model, the original image is typically converted into a motion-estimated feature map and a residual-compensated feature map for transmission. During decoding, the I-frame model receives the feature map and reconstructs the I-frame image. The P-frame model receives the motion-estimated feature map and reconstructs the motion estimate. This is applied to the reference I-frame reconstructed image to obtain the P-frame intermediate result of the motion estimate. Finally, the residual-compensated feature map is used to reconstruct the residual compensation information and applied to the P-frame intermediate result to obtain the reconstructed P-frame image.
[0166] Entropy coding estimation is commonly used in AI video codecs to transmit feature maps. Entropy coding is a common data compression technique and a crucial component of video codecs. In video coding, entropy coding is often used to compress residual data, motion vectors, and other encoding parameters in the video encoder to reduce storage space and transmission bandwidth.
[0167] The entropy coding estimation module is a key module in AI video encoding and decoding. Its purpose is to estimate the number of bits required for entropy coding based on the input encoded data. This module is usually implemented based on a statistical model. It analyzes and models the encoded data to minimize the number of bits required during the entropy coding process. Common entropy coding algorithms include Huffman coding and arithmetic coding. Taking arithmetic coding as an example, during the entropy coding process, arithmetic coding calculations need to be performed on each feature element (which can be understood as each value in the feature map). In order to improve the reconstruction quality at the decoding end, the feature map usually contains a large number of feature elements. Using arithmetic coding to serially encode each element one by one consumes a lot of time.
[0168] The data encoding and data decoding methods provided in the embodiments of the present application can realize arithmetic coding acceleration. The data encoding method and data decoding method provided in the embodiments of the present application, by converting the scale value (i.e., the scale parameter value) estimated by entropy coding into an index (i.e., the scale parameter quantization value), the index value can reflect the degree of certainty of the position. For values with lower index values, the certainty is very high and they can be skipped, thereby reducing the number of elements actually coded. For example, the original arithmetic coding needs to encode 10,000 elements, but after the skip strategy, only 2,000 elements with larger index values need to be encoded. The arithmetic coding time will be reduced by 4 / 5, greatly improving the speed of arithmetic coding. In the decoding process, the value estimated by the entropy coding model can be used as the true value of the skipped element, which can effectively reduce the burr phenomenon in the encoding and decoding process. The following is a detailed introduction:
[0169] Referring to Figure 6, for the original data (which may be an image, motion estimation, residual estimation, etc.), after passing through the encoding module, the original data can be compressed in the first stage to obtain a feature map; the decoding end obtains the restored feature map, and the reconstructed data can be obtained through the decoding module.
[0170] As for how to transmit the feature map, it is necessary to introduce a second stage of compression, namely the entropy coding module. Referring to Figure 7, in the related art, the feature map to be encoded is first super-a priori encoded and super-a priori decoded to obtain the scale value corresponding to each feature element, and then the scale value is quantized into an index. Finally, the probability distribution function is obtained by looking up the mapping relationship table through the index, thereby achieving information compression of the feature map to be encoded. In this process, since there are many feature elements in the feature map to be encoded, if some unimportant feature elements are not screened and skipped, arithmetic coding will take a long time to calculate. Based on this, in the specific implementation of the embodiment of the present application, the amount of actual encoding required can be adjusted according to actual needs, and the entropy coding link can be greatly accelerated without basically affecting the encoding and decoding quality, thereby promoting the real-time implementation of the AI video encoding and decoding algorithm. During encoding, according to the characteristics of the index, only the more important parts are encoded. When skipping 80%-95% of the calculations, the accuracy of the model can still be maintained, thereby significantly reducing the encoding time of the entropy coding link. During decoding, the important feature maps are obtained from the byte stream. For the unimportant feature map parts, the predicted feature maps obtained by super prior decoding are used to replace them, thereby obtaining the real decoding feature map y_hat for the subsequent decoding process. The detailed description of each step is as follows:
[0171] Referring to Figure 8, the encoding process of the encoding end is first introduced:
[0172] The feature map y obtained in the encoding and decoding model is the feature map to be encoded. The super-prior encoding module can encode the encoded feature map to obtain the super-prior z (i.e., auxiliary encoding information), and the super-prior z will also be transmitted to the decoding end in a certain way. At the encoding end, after obtaining the super-prior z, z can be further super-prior decoded to obtain the predicted feature map means and scale parameter values scales corresponding to the feature map y to be encoded. Means and scales can be understood as estimates of the probability distribution of y, where each value in means is the mean of the probability distribution of each feature element in the feature map y, and scales is the variance of the probability distribution of each feature element in the feature map y. Means and scales help to achieve higher compression rates in the subsequent entropy coding link. The scale parameter value scales obtained from the super-prior decoding is first quantized using the above formula (3) to obtain the scale parameter quantization value indexes.
[0173] For the feature map y to be encoded and the scale parameter quantization value indexes, both are feature maps of hxwxc, set the indexes threshold parameter th, the value range of th is 0-255, for the position (i, j, k) where the index value exceeds th, the feature value y to be encoded at that position is retained at the same time i,j,kand scale parameter quantization value index i,j,k , thereby forming the real coding feature map y1 and the real coding indexes1, and for the position where the index value is less than or equal to th, the skip coding feature map y2 and the skip coding indexes2 are formed, wherein the real coding indexes1 has the same size as y1, and the skip coding indexes2 has the same size as y2, that is, the real coding feature map y1 records the feature elements obtained by screening, and the real coding indexes1 records the scale parameter quantization values corresponding to each feature element obtained by screening, the skip coding feature map y2 records the feature elements that are not screened out, and the skip coding indexes2 records the scale parameter quantization values corresponding to each feature element that is not screened out.
[0174] Due to the large amount of redundant information between video frames, most positions can be predicted more accurately. The index values of the positions with more accurate predictions will be smaller, which allows for skipping. By selecting different thresholds th, different proportions of index values can be skipped, resulting in different degrees of acceleration. Of course, different thresholds th will also affect the final reconstruction effect, and a trade-off needs to be made. In the embodiment of this application, when the skip ratio is between 80% and 95%, the acceleration strategy will not have a significant impact on the reconstruction effect. The actual threshold can be adjusted appropriately as needed.
[0175] In the entropy coding module, firstly, a mapping relationship table between the scale parameter quantization value and the scale parameter value is constructed using the above formula (1). The mapping relationship table is used as a probability distribution function lookup table. For the predicted scale parameter value scales, each element thereof corresponds to the value in y. For each scale, the scale parameter quantization value index can be obtained according to the above formula (2) and formula (3). The corresponding scale_q value can be found from the mapping relationship table using the index. Since the predicted feature map means is obtained by introducing the super prior coding, the feature elements in the feature map can be converted into a probability distribution with a mean of 0 by predicting the feature map means during the coding process. Therefore, scale_q can be directly used as the probability distribution function required for arithmetic coding, so that y can be compressed into as few byte streams as possible, which is convenient for subsequent transmission.
[0176] Referring to Figure 9, the decoding process of the encoding end is introduced:
[0177] Since the decoding module needs to obtain the reconstructed data from the complete y_hat decoding, but the encoding process only encodes a part of the elements of the feature map y into a byte stream for transmission, the decoding can only decode a part of y. For the other parts that are not encoded into the byte stream, the values of the corresponding positions in the predicted feature map means obtained by super prior decoding can be used to replace them. This not only ensures the consistency of the encoding and decoding links, but also ensures that the real decoding feature map y_hat_new used during decoding can be capable of reconstructing the reconstructed data of equivalent quality. For details, refer to formula (4).
[0178] Corresponding to entropy coding, the entropy decoding step can decode each index_i value in the obtained real encoding indexes1 from the encoded byte stream one by one to obtain the corresponding y1_i_hat value, thereby forming the real decoding feature map y1_hat. This is the first line of formula (4).
[0179] For the skip coding indexes2 obtained during the encoding process, all element positions (i, j, k) corresponding to indexes2 are the positions of feature elements that have not been entropy coded. The values of these feature elements in the decoded feature map are not the values in the original skip coding feature map y2, but are replaced by the values at the corresponding positions in the predicted feature map means. The values in the predicted feature map means used to replace the feature element positions (i, j, k) corresponding to indexes2 are the replacement elements corresponding to the feature elements that have not been entropy coded, thereby constructing the skip decoding feature map y2_hat. This is the second line in formula (4).
[0180] For the decoded y1_hat and the constructed y2_hat, the two feature maps are merged (i.e., the method of obtaining y_hat_new using formula (4)), and the true decoded feature map y_hat_new is obtained. The decoding module shown in Figure 6 is applied to this feature map to obtain the reconstructed data.
[0181] Referring to Figure 10, the decoding process of the decoding end is introduced:
[0182] In the decoding process at the decoding end, the super-prior z is first decoded to obtain the predicted feature map means and scale parameter value scales corresponding to the feature map to be encoded y. The scale parameter value scales obtained from the super-prior decoding is first calculated using formula (3) above to obtain indexes, and the indexes threshold parameter th is obtained. The value range of th can be 0-255. The positions (i, j, k) with index values exceeding th are screened to form the real encoding indexes1, and for the positions with index values less than or equal to th, the skip encoding indexes2 is formed. According to the real coding indexes1, the mapping relationship table is searched to obtain the probability distribution function required for entropy decoding, and then the corresponding y1_i_hat values are obtained by decoding one by one from the encoded byte stream, thereby forming the real decoding feature map y1_hat, and the positions (i, j, k) with index values not exceeding th are screened to form the skip coding indexes2. All element positions (i, j, k) corresponding to indexes2 are the positions of feature elements that have not been entropy coded. The values of these feature elements in the decoded and restored feature map do not use the values in the original skip coding feature map y2, but are replaced by the values at the corresponding positions in the predicted feature map means. The values in the predicted feature map means used to replace the feature element positions (i, j, k) corresponding to indexes2 are the replacement elements corresponding to the feature elements that have not been entropy coded. Based on these replacement elements, the skip decoding feature map y2_hat can be constructed. For the decoded y1_hat and the constructed y2_hat, the two feature maps are merged (that is, the method of obtaining y_hat_new by formula (4)), and the true decoding feature map y_hat_new can be obtained. That is, for the feature elements whose scale parameter quantization value in the feature map y exceeds the preset threshold th, the value at the corresponding position in the true decoding feature map y_hat_new is obtained by decoding the encoded byte stream, and for the feature elements whose scale parameter quantization value in the feature map y does not exceed the preset threshold th, a replacement element is used at the corresponding position in the true decoding feature map y_hat_new, and the replacement element is the value at the same position as the feature element in the predicted feature map means.
[0183] Furthermore, the decoding end performs the decoding module shown in FIG6 on the feature map to obtain reconstructed data.
[0184] The data encoding and decoding methods provided by this application are highly adaptable, independent of the hardware environment, and can handle various types of video encoding and decoding tasks. For AI video encoding and decoding algorithms that use arithmetic coding for entropy compression, the present invention can effectively reduce the algorithm's entropy coding computational complexity, thereby improving encoding and decoding speed, making a significant contribution to the real-time implementation of AI video encoding and decoding. Furthermore, when using estimated values instead of true values, some glitches can often be avoided, thereby improving the smoothness of the reconstructed data. The present invention selects an AI video encoding and decoding algorithm for experimental comparison, presenting a comparison of the reduction in algorithm computational complexity and the impact on the algorithm's final results after adopting the strategy of the present invention. In actual use, different thresholds can be selected as needed to achieve a trade-off between speed and performance. The comparison results are shown in Figure 11. The different curves represent the results of different thresholds. Skip-0% represents the result without skip acceleration, while skip-74%, skip-81%, skip-86%, and skip-88% represent the reduction in arithmetic coding computational complexity by different percentages at different thresholds, and the impact on the algorithm results. When skipping -74%, the speed of the arithmetic coding link can be increased by 5 times while the accuracy remains basically unchanged, which is very impressive.
[0185] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.
[0186] Based on the same inventive concept, embodiments of the present application further provide a data encoding device for implementing the aforementioned data encoding method, and a data decoding device for implementing the aforementioned data decoding method. The implementation solutions provided by these devices are similar to the implementation solutions described in the aforementioned methods. Therefore, the specific limitations of one or more data encoding device and data decoding device embodiments provided below can be found in the above-mentioned limitations of the data encoding method and data decoding method, and will not be repeated here.
[0187] In some embodiments, as shown in FIG12 , a data encoding apparatus 1200 is provided, including:
[0188] The scale parameter value determination module 1202 is configured to obtain auxiliary coding information of the feature data and determine the scale parameter value corresponding to each feature element included in the feature data according to the auxiliary coding information;
[0189] The quantization value screening module 1204 is configured to perform quantization processing on each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; screen out a portion of the scale parameter quantization values according to a preset screening condition; and the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values.
[0190] The feature element screening module 1206 is configured to extract, from the plurality of feature elements included in the feature data, the feature elements corresponding to the quantized values of the scale parameters obtained by screening;
[0191] An entropy coding module 1208 is configured to perform entropy coding on each extracted feature element based on a quantized value of a scale parameter corresponding to the feature element to obtain coded data corresponding to the feature element;
[0192] The coded data stream determining module 1210 is configured to determine the coded data stream of the feature data according to the coded data.
[0193] The above-mentioned data encoding device can screen larger scale parameter quantization values according to preset screening conditions during the entropy coding process, and screen feature elements based on the screened scale parameter quantization values. Entropy coding is performed only on the screened feature elements, and the entropy coding process of unscreened feature elements is skipped. This can significantly reduce the number of feature elements that need to be entropy coded, greatly reduce the complexity of entropy coding, and significantly improve data coding efficiency.
[0194] In some embodiments, the preset screening condition requires that the scale parameter quantization value exceed a preset quantization threshold. The quantization value screening module 1204 is further configured to compare each scale parameter quantization value with the preset quantization threshold to obtain a comparison result corresponding to each scale parameter quantization value; and to screen the scale parameter quantization values for which the corresponding comparison results indicate that the scale parameter quantization value exceeds the preset quantization threshold, thereby obtaining a portion of the scale parameter quantization values.
[0195] In some embodiments, the feature element screening module 1206 is further configured to obtain motion estimation data for a data frame to which the feature data belongs, determine, based on the motion estimation data, a motion estimation value corresponding to each scale parameter quantized value in a portion of the scale parameter quantized values, wherein the motion estimation value matches a position of a feature element corresponding to the corresponding scale parameter quantized value; screen out, from the portion of the scale parameter quantized values, scale parameter quantized values whose corresponding motion estimation values are greater than a motion estimation threshold; and extract, from the plurality of feature elements included in the feature data, feature elements corresponding to each of the screened scale parameter quantized values.
[0196] In some embodiments, each scale parameter value is a floating point number, and each scale parameter quantized value is an integer.
[0197] In some embodiments, the quantization value screening module 1204 is further configured to determine a quantization step size, where the quantization step size is positively correlated with the difference between the maximum and minimum values of the scale parameter value, and negatively correlated with the maximum quantization level set for the quantization process; determine the difference between each scale parameter value and the minimum value; and perform rounding operations on the quantization step sizes based on the differences corresponding to each scale parameter value to obtain scale parameter quantization values corresponding to each scale parameter value.
[0198] In some embodiments, the feature data is a feature map; the data encoding device further includes: an additional encoding module for determining, from a plurality of feature elements included in the feature map, feature elements corresponding to respective unscreened scale parameter quantization values; dividing the feature elements corresponding to respective unscreened scale parameter quantization values into regions according to positions of the feature elements corresponding to respective unscreened scale parameter quantization values in the feature map to obtain a plurality of feature element regions; for each of the plurality of feature element regions, selecting at least one feature element located at a center position of the targeted feature element region as a region representative element; and performing entropy encoding on the representative feature elements based on the scale parameter quantization values of the representative feature elements to obtain additional encoded data of the feature data.
[0199] In some embodiments, the scale parameter value determination module 1202 is further used to obtain feature data, perform super-prior coding on the feature data to reduce the feature dimension, and obtain auxiliary coding information of the feature data; and use the auxiliary coding information to predict the probability distribution parameter value of each feature element included in the feature data, and obtain the scale parameter value corresponding to each feature element.
[0200] In some embodiments, the feature data is obtained by pre-encoding the original data of the current data frame; the above-mentioned data encoding device also includes: an original data determination module, used to obtain the encoded data stream of the previous frame data frame, and reconstruct the previous frame data frame based on the encoded data stream of the previous frame data frame; based on the reconstructed previous frame data frame, motion estimation is performed on the current data frame to obtain motion estimation data of the current data frame; based on the motion estimation data, residual compensation estimation is performed on the current data frame to obtain residual compensation data of the current data frame; the motion estimation data and the residual compensation data are used as the original data of the current data frame.
[0201] In some embodiments, the original data determination module is further used to perform entropy decoding on the encoded data stream of the previous frame of data based on the scale parameter quantization value screened out during the encoding process of the feature data of the previous frame of data to obtain corresponding feature elements; determine replacement elements at the positions of the feature elements that constitute the feature data of the previous data frame and have not been entropy encoded; merge the feature elements and replacement elements obtained by entropy decoding to obtain feature data restored for the previous data frame; and perform data reconstruction based on the feature data restored for the previous data frame to obtain a reconstructed previous frame of data.
[0202] In some embodiments, as shown in FIG13 , a data decoding apparatus 1300 is provided, including:
[0203] The auxiliary coding information acquisition module 1302 is used to obtain auxiliary coding information corresponding to the coded data stream, where the coded data stream is obtained by encoding a portion of feature elements of the feature data according to the auxiliary coding information;
[0204] The scale parameter value determining module 1304 is configured to determine the scale parameter value corresponding to each feature element included in the feature data according to the auxiliary coding information;
[0205] The quantization value screening module 1306 is configured to perform quantization processing on each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; and to screen out a portion of the scale parameter quantization values according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values.
[0206] The entropy decoding module 1308 is configured to perform entropy decoding on the coded data stream based on the selected scale parameter quantization values to obtain corresponding feature elements; determine the positions of the feature elements that have not been entropy-coded and constitute the feature data based on the unselected scale parameter quantization values, and determine replacement elements for the positions of the feature elements that have not been entropy-coded;
[0207] The feature data recovery module 1310 is configured to combine the feature elements obtained by entropy decoding with the replacement elements to obtain recovered feature data.
[0208] In the above-mentioned data decoding device, during the decoding process, since larger scale parameter quantization values can be screened out and entropy decoding is performed only based on the screened scale parameter quantization values, entropy decoding can be performed only on some characteristic elements, greatly reducing the complexity of entropy coding and significantly improving data decoding efficiency. In addition, since the screening process screens the scale parameter quantization values obtained by quantization processing according to preset screening conditions, the scale parameter values are discretized during the quantization process, which increases the probability that the screened scale parameter quantization values are consistent with the scale parameter quantization values screened out during the encoding process, thereby improving the accuracy of the characteristic data recovered by decoding.
[0209] In some embodiments, the entropy decoding module is further configured to obtain motion estimation data reconstructed for a data frame to which the characteristic data belongs, determine, based on the motion estimation data, motion estimation values corresponding to respective scale parameter quantization values in a portion of scale parameter quantization values, wherein the motion estimation values match positions of characteristic elements corresponding to the corresponding scale parameter quantization values; screen out scale parameter quantization values whose corresponding motion estimation values are greater than a motion estimation threshold from the portion of scale parameter quantization values; and perform entropy decoding on the encoded data stream using the screened scale parameter quantization values to obtain corresponding characteristic elements.
[0210] In some embodiments, each scale parameter value is a floating point number, and each scale parameter quantized value is an integer.
[0211] In some embodiments, the quantization value screening module is used to determine a quantization step size, where the quantization step size is positively correlated with the difference between the maximum value and the minimum value of the scale parameter value, and is negatively correlated with the quantization level set for the quantization process; respectively determine the difference between each scale parameter value and the minimum value; and round the quantization step size according to the difference corresponding to each scale parameter value to obtain the scale parameter quantization value corresponding to each scale parameter value.
[0212] In some embodiments, the entropy decoding module is further used to obtain the predicted characteristic value corresponding to each feature element, and the predicted characteristic value is obtained by predicting the probability distribution parameter value of the corresponding feature element using auxiliary coding information; for the position of each feature element that has not been entropy encoded, a predicted characteristic value that matches the position is determined from the predicted characteristic values, and a replacement element at the position of the feature element is determined based on the determined predicted characteristic value.
[0213] In some embodiments, the entropy decoding module further obtains additional encoded data of the feature map from the encoded data stream, performs entropy decoding on the additional encoded data, and obtains corresponding region representative elements; in the feature map, regions are divided based on the positions of the feature elements that have not been entropy encoded; and for each divided region, the region representative element located in the targeted region is determined as a replacement element for the position of each feature element that has not been entropy encoded in the targeted region.
[0214] Each module in the above-mentioned data encoding device and data decoding device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.
[0215] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 14. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer device is used to store encoded data streams or reconstructed data obtained by decoding. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer-readable instructions are executed by the processor, a data encoding method or a data decoding method is implemented.
[0216] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 15. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means may be implemented via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer-readable instructions implement a data encoding method or a data decoding method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0217] Those skilled in the art will understand that the structures shown in Figures 14 and 15 are merely block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different component arrangement.
[0218] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data encoding method or the data decoding method in any of the above embodiments are implemented.
[0219] In some embodiments, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the data encoding method or the data decoding method in any of the above embodiments are implemented.
[0220] In some embodiments, a computer program product is provided, comprising computer-readable instructions, which, when executed by a processor, implement the steps of the data encoding method or the data decoding method in any of the above embodiments.
[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0222] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.
[0223] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0224] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A data encoding method, performed by a first device, include: Acquire auxiliary coding information of the feature data, and determine, according to the auxiliary coding information, the scale parameter value corresponding to each feature element included in the feature data; Quantizing each scale parameter value respectively to obtain a scale parameter quantization value corresponding to each scale parameter value; From the scale parameter quantization values, a portion of the scale parameter quantization values are screened out according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values; Extracting and screening the characteristic elements corresponding to the quantized values of the scale parameters from the plurality of characteristic elements included in the characteristic data; For each feature element obtained by extraction, entropy encoding is performed on the feature element based on the quantized value of the scale parameter corresponding to the feature element to obtain encoded data corresponding to the feature element; and According to the encoded data, an encoded data stream of the feature data is determined.
2. According to the method of claim 1, the preset screening condition is to require that the quantization value of the scale parameter exceeds a preset quantization threshold; The step of selecting a portion of the scale parameter quantization values from the scale parameter quantization values according to a preset screening condition includes: Compare the quantized values of the scale parameters with the preset quantization thresholds respectively to obtain the comparison results corresponding to the quantized values of the scale parameters; and From the scale parameter quantization values, the scale parameter quantization values corresponding to the comparison results indicating that they exceed the preset quantization threshold are screened to obtain a part of the scale parameter quantization values.
3. The method according to any one of claims 1 to 2, wherein the feature elements corresponding to the quantized values of the scale parameters obtained by extraction and screening are extracted from the plurality of feature elements included in the feature data, include: Acquire motion estimation data of a data frame to which the feature data belongs, and determine, based on the motion estimation data, motion estimation values corresponding to each scale parameter quantization value in the portion of scale parameter quantization values, wherein the motion estimation values match positions of feature elements corresponding to the corresponding scale parameter quantization values; Filtering out, from the part of the scale parameter quantized values, scale parameter quantized values corresponding to motion estimation values greater than a motion estimation threshold; and Feature elements corresponding to the screened quantized values of the scale parameters are extracted from the plurality of feature elements included in the feature data.
4. According to the method according to any one of claims 1 to 3, each of the scale parameter values is a floating point number, and each of the scale parameter quantization values is an integer.
5. The method according to claim 4, wherein each scale parameter value is quantized respectively. include: Determining a quantization step size, wherein the quantization step size is positively correlated with a difference between a maximum value and a minimum value of the scale parameter value, and the quantization step size is negatively correlated with a maximum quantization level set for quantization processing; Determine the difference between each scale parameter value and the minimum value respectively; and The quantization step size is rounded according to the differences corresponding to the values of each scale parameter to obtain the values of each scale parameter. The values correspond to the quantized values of the scale parameters.
6. The method according to any one of claims 1 to 5, wherein the characteristic data is a characteristic graph; include: Determining, from a plurality of feature elements included in the feature graph, feature elements corresponding to respective quantized values of each scale parameter that have not been screened out; According to the positions of the characteristic elements corresponding to the respective quantized values of the scale parameters that have not been screened out in the characteristic map, the characteristic elements corresponding to the respective quantized values of the scale parameters that have not been screened out are divided into regions to obtain a plurality of characteristic element regions; For each of the plurality of characteristic element regions, selecting at least one characteristic element located at the center of the characteristic element region as a region representative element; and Based on the quantized value of the scale parameter of the representative feature element, the representative feature element is entropy encoded to obtain additional encoded data of the feature data.
7. The method according to any one of claims 1 to 6, wherein the auxiliary coding information of the feature data is obtained, and the scale parameter value corresponding to each feature element included in the feature data is determined according to the auxiliary coding information. include: Acquire feature data, perform super-a priori coding on the feature data to reduce feature dimensions, and obtain auxiliary coding information of the feature data; and The auxiliary coding information is used to predict the probability distribution parameter value of each feature element included in the feature data, and the scale parameter value corresponding to each feature element is obtained.
8. The method according to any one of claims 1 to 7, wherein the feature data is obtained by pre-encoding the original data of the current data frame; include: Acquire a coded data stream of a previous data frame, and reconstruct the previous data frame based on the coded data stream of the previous data frame; Performing motion estimation on the current data frame based on the reconstructed previous data frame to obtain motion estimation data of the current data frame; Performing residual compensation estimation on the current data frame based on the motion estimation data to obtain residual compensation data of the current data frame; and The motion estimation data and the residual compensation data are used as original data of the current data frame.
9. The method according to claim 8, wherein the previous data frame is reconstructed based on the coded data stream of the previous data frame, include: Based on the scale parameter quantization value screened out in the encoding process of the feature data of the previous data frame, entropy decoding is performed on the encoded data stream of the previous data frame to obtain the corresponding feature element; Determine a replacement element at a position where a feature element constituting feature data of the previous data frame and not entropy encoded is located; Merging the characteristic element obtained by entropy decoding with the replacement element to obtain characteristic data restored for the previous data frame; and Data reconstruction is performed based on the feature data restored for the previous data frame to obtain a reconstructed previous data frame.
10. A data decoding method, performed by a second device, include: Acquire auxiliary coding information corresponding to a coded data stream, wherein the coded data stream is obtained by encoding a portion of feature elements of feature data according to the auxiliary coding information; Determining, according to the auxiliary coding information, scale parameter values corresponding to the respective feature elements included in the feature data; Quantizing each scale parameter value respectively to obtain a scale parameter quantization value corresponding to each scale parameter value; From the scale parameter quantization values, a portion of the scale parameter quantization values are screened out according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values; Performing entropy decoding on the encoded data stream based on the screened scale parameter quantization value to obtain corresponding feature elements; Determining the positions of the feature elements constituting the feature data that are not entropy encoded according to the quantized values of the scale parameters that are not screened out, and determining replacement elements at the positions of the feature elements that are not entropy encoded; and The characteristic element obtained by entropy decoding and the replacement element are combined to obtain restored characteristic data.
11. The method according to claim 10, wherein entropy decoding is performed on the coded data stream based on the selected scale parameter quantization value to obtain corresponding feature elements, include: Obtaining motion estimation data reconstructed from a data frame to which the feature data belongs, and determining, based on the motion estimation data, motion estimation values corresponding to each scale parameter quantization value in the portion of scale parameter quantization values, wherein the motion estimation values match positions of feature elements corresponding to the corresponding scale parameter quantization values; Filtering out, from the part of the scale parameter quantized values, scale parameter quantized values corresponding to motion estimation values greater than a motion estimation threshold; and The coded data stream is entropy decoded using the screened scale parameter quantization value to obtain corresponding feature elements.
12. The method according to any one of claims 10 to 11, wherein each of the scale parameter values is a floating point number, and each of the scale parameter quantization values is an integer.
13. The method according to claim 12, wherein each scale parameter value is quantized. include: Determining a quantization step size, wherein the quantization step size is positively correlated with a difference between a maximum value and a minimum value of the scale parameter value, and the quantization step size is negatively correlated with a quantization level set for the quantization process; Determine the difference between each scale parameter value and the minimum value respectively; and The quantization step lengths are rounded according to the differences corresponding to the scale parameter values to obtain scale parameter quantization values corresponding to the scale parameter values.
14. The method according to any one of claims 10 to 13, wherein determining a replacement element at a position where a feature element that has not been entropy encoded is located, include: Obtaining a predicted characteristic value corresponding to each characteristic element, wherein the predicted characteristic value is obtained by predicting a probability distribution parameter value of the corresponding characteristic element using the auxiliary coding information; For each position of a feature element that has not been entropy encoded, a predicted feature value with a matching position is determined from the predicted feature values, and a replacement element at the position of the feature element is determined according to the determined predicted feature value.
15. The method according to any one of claims 10 to 13, wherein the feature data is a feature map ; The determining of the replacement element at the position of the feature element that has not been entropy encoded includes: Acquire additional coded data of the feature map from the coded data stream, perform entropy decoding on the additional coded data, and obtain corresponding region representative elements; In the feature map, region division is performed based on the positions of feature elements that have not been entropy encoded; and For each divided region, the region representative element located in the targeted region is determined as a replacement element for the position of each feature element that is not entropy encoded in the targeted region.
16. A data encoding device, include: A scale parameter value determination module, used to obtain auxiliary coding information of feature data, and determine the scale parameter value corresponding to each feature element included in the feature data according to the auxiliary coding information; A quantization value screening module is used to quantize each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; From the scale parameter quantization values, a portion of the scale parameter quantization values are screened out according to a preset screening condition; the screened out scale parameter quantization values are larger than the unscreened scale parameter quantization values; A feature element screening module, used to extract, from the plurality of feature elements included in the feature data, feature elements corresponding to the quantized values of the scale parameters obtained by screening; An entropy coding module is used to perform entropy coding on each feature element obtained by extraction based on a quantized value of a scale parameter corresponding to the feature element to obtain coded data corresponding to the feature element; and The coded data stream determining module is used to determine the coded data stream of the characteristic data according to the coded data.
17. A data decoding device, include: An auxiliary coding information acquisition module, used to acquire auxiliary coding information corresponding to a coded data stream, wherein the coded data stream is obtained by encoding a part of feature elements of feature data according to the auxiliary coding information; A scale parameter value determination module, used to determine the scale parameter value corresponding to each characteristic element included in the characteristic data according to the auxiliary coding information; A quantization value screening module is used to quantize each scale parameter value to obtain a scale parameter quantization value corresponding to each scale parameter value; From the scale parameter quantization values, a portion of the scale parameter quantization values are screened out according to a preset screening condition, wherein the screened scale parameter quantization values are larger than the unscreened scale parameter quantization values; an entropy decoding module, configured to perform entropy decoding on the coded data stream based on the screened scale parameter quantization values to obtain corresponding feature elements; determine positions of feature elements constituting the feature data that have not been entropy encoded according to the scale parameter quantization values that have not been screened out, and determine replacement elements at the positions of the feature elements that have not been entropy encoded; and The feature data recovery module is used to merge the feature element obtained by entropy decoding with the replacement element to obtain recovered feature data.
18. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, It is characterized in that When the processor executes the computer-readable instructions, the steps of the method of any one of claims 1 to 9 or 10 to 15 are implemented.
19. A computer-readable storage medium having computer-readable instructions stored thereon, It is characterized in that When the computer-readable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 or 10 to 15 are implemented.
20. A computer program product comprising computer readable instructions, It is characterized in that When the computer readable instructions are executed by a processor, the steps of the method of any one of claims 1 to 9 or 10 to 15 are implemented.
Citation Information
Patent Citations
Perceptual high-definition video coding method based on salient target detection and saliency guidance
CN111432207A
Image coding method and device, electronic equipment and storage medium
CN113613004A
Feature data encoding and decoding method and device
CN115442609A
Data encoding method and device, data decoding method and device, computer equipment and storage medium
CN116979971A
Image encoding method, image decoding method, image encoding device and image decoding device
WO2011052216A1
Cited By
Video decoding method and apparatus, and video encoding method and apparatus
WO2026179123A1
