Video codec model processing method, video encoding method, video decoding method and related device

Abstract

A video codec model processing method, executed by a computing device and comprising: acquiring a training video frame, extracting a feature map of the training video frame by means of a video codec model, and determining scale parameter values respectively corresponding to feature elements in the feature map (202); on the basis of a preset mapping relationship, respectively mapping the scale parameter values corresponding to the feature elements to obtain scale parameter mapping values respectively corresponding to the feature elements, the scale parameter mapping values being within a preset mapping value range (204); acquiring a constraint reference mapping value corresponding to each scale parameter mapping value, wherein the distance between the constraint reference mapping value and a rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is greater than the distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value (206); on the basis of the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, determining a quantization constraint loss of the training video frame (208); and updating at least some of model parameters of the video codec model on the basis of the quantization constraint loss, so as to obtain a trained video codec model (210).

Description

Video encoding and decoding model processing, video encoding, decoding methods and related equipment

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023112566635, filed on September 25, 2023, entitled “Video encoding and decoding model processing, video encoding, decoding methods and related equipment,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of video coding and decoding technology, and in particular to video coding and decoding model processing, video encoding, decoding methods and related equipment. Background Art

[0004] With the development of computer technology, video codecs have emerged. Encoding compresses video frames, while decoding restores compressed frames. Video codecs are widely used in various scenarios, especially cross-platform video transmission scenarios, such as video chat, video conferencing, and other real-time conversation applications.

[0005] In related technologies, video encoding and decoding can be implemented with the help of video encoding and decoding models. However, during cross-platform video transmission, since encoding and decoding through video encoding and decoding models are performed by different computer devices, there is often a problem of low accuracy of video frames obtained by decoding and reconstruction.

[0006] Summary of the Invention

[0007] The present application provides a video coding and decoding model processing method, apparatus, computer equipment, computer-readable storage medium and computer program product, as well as a video encoding method, apparatus, computer equipment, computer-readable storage medium and computer program product and a video decoding method, apparatus, computer equipment, computer-readable storage medium and computer program product.

[0008] In one aspect, the present application provides a video encoding and decoding model processing method, which is executed by a computer device and includes:

[0009] Obtaining a training video frame, extracting a feature map of the training video frame using a video codec model, and determining a scale parameter value corresponding to each feature element in the feature map;

[0010] According to a preset mapping relationship, each of the characteristic elements is mapped into a preset mapping value range to obtain a scale parameter mapping value corresponding to each of the characteristic elements;

[0011] Obtaining a constraint reference mapping value corresponding to each scale parameter mapping value, wherein a distance between the constraint reference mapping value and a rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is farther than a distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value;

[0012] determining a quantization constraint loss for the training video frame based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values; and

[0013] At least a portion of model parameters of the video codec model is updated based on the quantization constraint loss to obtain a trained video codec model.

[0014] On the other hand, the present application also provides a video encoding and decoding model processing device. The device includes:

[0015] A video frame acquisition module is used to acquire a training video frame, extract a feature map of the training video frame through a video codec model, and determine the scale parameter value corresponding to each feature element in the feature map;

[0016] A numerical mapping module is used to map each of the characteristic elements into a preset mapping value range according to a preset mapping relationship, and obtain a scale parameter mapping value corresponding to each of the characteristic elements;

[0017] a loss determination module, configured to obtain a constraint reference mapping value corresponding to each of the scale parameter mapping values, and determine a quantization constraint loss for the training video frame based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values, wherein a distance between the constraint reference mapping value and a rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is greater than a distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value; and

[0018] A parameter updating module is used to update at least a portion of model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0019] On the other hand, the present application provides another video codec model processing method, which is executed by a computer device, and the method includes:

[0020] Obtaining a training video frame, extracting a feature map of the training video frame using a video codec model, and determining a scale parameter value corresponding to each feature element in the feature map;

[0021] Obtaining a constraint reference value corresponding to each scale parameter value, wherein the constraint reference value is farther from a rounding boundary value of a scale parameter mapping value obtained by mapping the corresponding scale parameter value after being mapped to a preset mapping value range according to a preset mapping relationship than the scale parameter value corresponding to the constraint reference value;

[0022] determining a quantization constraint loss for the training video frame based on a difference between each of the scale parameter values ​​and the corresponding constraint reference value; and

[0023] At least a portion of model parameters of the video codec model is updated based on the quantization constraint loss to obtain a trained video codec model.

[0024] On the other hand, the present application also provides another video encoding and decoding model processing device. The device includes:

[0025] A video frame acquisition module is used to acquire a training video frame, extract a feature map of the training video frame through a video codec model, and determine the scale parameter value corresponding to each feature element in the feature map;

[0026] a loss determination module, configured to obtain a constraint reference value corresponding to each of the scale parameter values, and determine a quantization constraint loss for the training video frame based on a difference between each of the scale parameter values ​​and the corresponding constraint reference value, wherein the constraint reference value is farther from a rounding boundary value of a scale parameter mapping value obtained by mapping the corresponding scale parameter value after being mapped to a preset mapping value range according to a preset mapping relationship than the scale parameter value corresponding to the constraint reference value; and

[0027] A parameter updating module is used to update at least a portion of model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0028] In another aspect, the present application provides a video encoding method, performed by a computer device, the method comprising:

[0029] Obtaining a target video frame, extracting a feature map of the target video frame using a trained video codec model, and determining scale parameter values ​​corresponding to respective feature elements in the feature map, wherein the video codec model is trained using the video codec model processing method described in any of the above embodiments;

[0030] For each characteristic element of the characteristic graph, mapping the scale parameter value of the characteristic element to a preset mapping value range according to a preset mapping relationship, thereby obtaining a scale parameter mapping value;

[0031] Rounding the scale parameter mapping value obtained by mapping according to its rounding boundary value for quantization to obtain the scale parameter quantization value of the feature element; and

[0032] The feature map is entropy-coded according to the quantized value of the scale parameter of each feature element of the feature map, and a transmission data stream is determined according to the coded data stream obtained by the entropy coding.

[0033] On the other hand, the present application also provides a video encoding device. The device includes:

[0034] a video frame acquisition module, configured to acquire a target video frame, extract a feature map of the target video frame using a trained video codec model trained using the video codec model processing device described in any of the above embodiments, and determine a scale parameter value corresponding to each feature element in the feature map;

[0035] a numerical quantization module, configured to map the scale parameter value of each feature element of the feature map to a preset mapping value range according to a preset mapping relationship, thereby obtaining a scale parameter mapping value, and rounding the mapped scale parameter mapping value according to its rounding boundary value for quantization to obtain a scale parameter quantized value of the feature element; and

[0036] The entropy coding module is used to perform entropy coding on the feature map according to the quantized value of the scale parameter of each feature element of the feature map, and determine the transmission data stream according to the coded data stream obtained by the entropy coding.

[0037] In another aspect, the present application provides a video decoding method, performed by a computer device, the method comprising:

[0038] Acquire a transmission data stream, and obtain a coded data stream from the transmission data stream, wherein the coded data stream is obtained by encoding a feature map of a target video frame;

[0039] Determining the scale parameter value corresponding to each feature element in the feature map using a trained video codec model, wherein the video codec model is trained using the video codec model processing method described in any of the above embodiments;

[0040] For each characteristic element of the characteristic graph, mapping the scale parameter value of the characteristic element to a preset mapping value range according to a preset mapping relationship, thereby obtaining a scale parameter mapping value;

[0041] Rounding the scale parameter mapping value obtained by mapping according to its rounding boundary value for quantization to obtain the scale parameter quantization value of the feature element; and

[0042] The encoded data stream is entropy decoded according to the quantized value of the scale parameter of each feature element of the feature map, and the target video frame is reconstructed based on the feature map restored by entropy decoding.

[0043] On the other hand, the present application also provides a video decoding device. The device includes:

[0044] A data stream acquisition module is used to acquire a transmission data stream, and obtain a coded data stream from the transmission data stream, wherein the coded data stream is obtained by encoding a feature map of a target video frame;

[0045] a scale parameter prediction module, configured to determine a scale parameter value corresponding to each feature element in the feature map using a trained video codec model, wherein the video codec model is trained using the video codec model processing device described in any of the above embodiments;

[0046] a numerical quantization module, configured to map the scale parameter value of each feature element to a preset mapping value range according to a preset mapping relationship, obtain a scale parameter mapping value, and round the mapped scale parameter mapping value according to its rounding boundary value for quantization to obtain a scale parameter quantized value of each feature element; and

[0047] The entropy decoding module is used to perform entropy decoding on the encoded data stream according to the quantized value of the scale parameter of each feature element of the feature map, and reconstruct the target video frame based on the feature map restored by entropy decoding.

[0048] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0050] FIG1 is a diagram illustrating an application environment of a video coding and decoding model processing method, a video coding method, and a video decoding method in some embodiments;

[0051] FIG2 is a schematic flow chart of a video coding model processing method in some embodiments;

[0052] FIG3 is a flow chart of a video coding model processing method in some other embodiments;

[0053] FIG4 is a schematic flow chart of a video encoding method in some embodiments;

[0054] FIG5 is a schematic flow chart of a video decoding method in some embodiments;

[0055] FIG6 is a schematic diagram of the overall process of video encoding and decoding in some embodiments;

[0056] FIG7 is a schematic diagram of the overall process of video encoding and decoding in some embodiments;

[0057] 8a and 8b are schematic diagrams of a decoding process failure in some embodiments;

[0058] FIG9 is a schematic diagram of constraints of an entropy model in some embodiments;

[0059] FIG10 is a framework diagram of a coding / decoding model in some embodiments;

[0060] FIG11 is a schematic diagram of a flow chart of an entropy coding module in some embodiments;

[0061] FIG12 is a schematic diagram showing a comparison between a video encoding and decoding method in some embodiments and related technologies;

[0062] FIG13 is a schematic diagram of a process flow at a decoding end in some embodiments;

[0063] FIG14 is a schematic diagram of a video encoding and decoding method in some embodiments;

[0064] FIG15 is a schematic diagram of the results of cross-platform scenario verification in some embodiments;

[0065] FIG16 is a structural block diagram of a video coding model processing apparatus in some embodiments;

[0066] FIG17 is a structural block diagram of a video coding model processing apparatus in some embodiments;

[0067] FIG18 is a structural block diagram of a video encoding apparatus in some embodiments;

[0068] FIG19 is a structural block diagram of a video decoding apparatus in some embodiments;

[0069] FIG20 is a diagram illustrating the internal structure of a computer device in some embodiments;

[0070] FIG21 is a diagram showing the internal structure of a computer device in some other embodiments. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0072] The video encoding and decoding model processing method, video encoding method and video decoding method provided in the embodiments of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Among them, the terminal or server can obtain training video frames, calculate the quantization constraint loss for the training video frames, and then update the model parameters of the video codec model based on the quantization constraint loss. After obtaining the trained video codec model, it can be deployed locally or sent to other computer devices, so that the trained video codec model can be used to realize encoding and decoding of target video frames.

[0073] In some embodiments, as shown in FIG2 , a video codec model processing method is provided. The method is performed by a computer device, which may be the server 104 or the terminal 102 in FIG1 . In the embodiments of the present application, the method is described by taking the server in FIG1 as an example, and includes the following steps:

[0074] Step 202: obtain a training video frame, extract a feature map of the training video frame using a video encoding and decoding model, and determine the scale parameter value corresponding to each feature element in the feature map.

[0075] The training video frame can be any type of frame in the training video, including a P-frame or an I-frame. An I-frame, also known as a keyframe or intra-coded frame, is a complete image frame that exists independently of other frames. A P-frame, also known as a predictive frame, is generated based on the previous I-frame or P-frame. A feature map refers to the data obtained by extracting features from the to-be-encoded data of the training video frame. The to-be-encoded data refers to the data required for encoding during the video encoding process. For example, when the training video frame is an I-frame, the to-be-encoded data can be the training video frame. When the training video frame is a P-frame, the to-be-encoded data can include motion estimation data and residual compensation data for the training video frame. A video codec model refers to a machine learning model that can be used for video encoding and decoding. The video codec model includes a feature extraction model, which extracts features from the to-be-encoded data of the training video frame to generate a feature map. A feature map includes feature elements, which are pixels in the feature map, with each pixel representing a feature element.

[0076] The video codec model also includes an entropy model, which is a model used to predict the data required for the entropy encoding process. In specific applications, the entropy model can predict the relevant parameter values ​​of the probability distribution function corresponding to each feature element in the feature map. The probability distribution function corresponding to a feature element refers to the function used to describe the probability distribution of the feature element. The probability distribution of a feature element refers to the distribution of the probability of the feature element taking a range of possible values ​​relative to these values. In some embodiments, the probability of a feature element's value can reflect the probability of that value appearing in the feature map.

[0077] The relevant parameters of a probability distribution function include the scale parameter and location parameter. The scale parameter is an indicator of the degree of dispersion or variation in the data within the probability distribution function, while the scale parameter value is the value of the scale parameter. Scale parameters can include standard deviation, variance, or range. The scale parameter value determines the shape of the probability distribution function's curve, such as its flatness. A larger scale parameter value results in a flatter curve; conversely, a smaller scale parameter value results in a taller and thinner curve. The abscissa of the probability distribution function curve represents the value of the characteristic element, and the ordinate represents the probability value. Therefore, a larger scale parameter value indicates a higher degree of dispersion in the probability distribution. In specific applications, the probability distribution of the characteristic element can be a Gaussian distribution or a Laplace distribution. When the probability distribution of the characteristic element is a Gaussian distribution, the scale parameter can specifically be the variance or standard deviation of the probability distribution. The variance of a probability distribution can be the average of the squares of the differences between each probability value and the mean of all probability values ​​in the probability distribution. The standard deviation is the square root of the variance. The location parameter value is used to describe the central tendency position of the probability distribution. When the probability distribution of the characteristic element is Gaussian distribution, the location parameter can specifically be the mean value of the probability distribution of the characteristic element, or the mathematical expectation.

[0078] It can be understood that the video coding and decoding model of the present application can be an end-to-end model, and its specific structure can be implemented using a known video coding and decoding model structure. The present application does not limit the specific structure of the video coding and decoding model.

[0079] Specifically, after acquiring the training video frame, the server inputs the training video frame into the video codec model. The feature extraction model in the video codec model then extracts features from the to-be-encoded data of the training video frame to obtain a feature map of the training video frame. The server then uses an entropy model to predict the probability distribution function of each feature element contained in the feature map to obtain the scale parameter value corresponding to each feature element. Optionally, while obtaining the scale parameter value corresponding to each feature element through the entropy model, the server can also obtain the position parameter value of each feature element.

[0080] In some embodiments, for the obtained feature map, the server can also perform super-prior encoding on the feature map to reduce the feature dimension through the video codec module to obtain auxiliary coding information of the feature map, which refers to information used to assist the encoding process, i.e., side information. The auxiliary coding information is then input into the entropy model, and the entropy model outputs the scale parameter value and position parameter value corresponding to each feature element contained in the feature map. Optionally, the entropy model can be a VAE (Variational AutoEncoder) model, in which the entropy model corresponds to the prior represented by the VAE hidden layer, and the auxiliary coding information is used to assist the entropy model encoding, i.e., the prior of the prior, and is therefore called super-prior encoding.

[0081] Step 204 : Mapping the scale parameter values ​​corresponding to the characteristic elements according to a preset mapping relationship to obtain scale parameter mapping values, wherein the scale parameter mapping values ​​are within a preset mapping value range.

[0082] The preset mapping relationship is a preset mapping relationship between scale parameter values ​​and scale parameter mapping values. Based on this preset mapping relationship, any scale parameter value can be mapped to a scale parameter mapping value. The computer device can map each characteristic element to a preset mapping value range according to the preset mapping relationship, thereby obtaining a scale parameter mapping value corresponding to each characteristic element.

[0083] During the video encoding and decoding process, the scale parameter value output by the video encoding and decoding model needs to be further quantized. Quantization here refers to representing the scale parameter value by an integer in an integer set. The integer set includes multiple consecutive integers. The number of integers in the integer set is determined by the required maximum quantization level. For example, assuming the maximum quantization level is 32, the integer set includes 32 integers. During the quantization process, the scale parameter value is first mapped to a preset mapping value range according to a preset mapping relationship to obtain a scale parameter mapping value. The scale parameter value is then rounded. The rounding method can be any of rounding up, rounding down, or rounding to the nearest integer. The scale parameter mapping value is rounded up, that is, the nearest rounding boundary value is taken upward. For example, assuming the scale parameter mapping value is 71.97, the value obtained by rounding up 71.97 can be 72; the scale parameter mapping value is rounded down, that is, the nearest rounding boundary value is taken downward. For example, the value obtained by rounding down 71.97 can be 71; the scale parameter mapping value is rounded to the nearest integer, that is, the scale parameter mapping value is rounded to the nearest integer boundary value. For example, the value obtained by rounding 71.97 to the nearest integer can be 72.

[0084] The preset mapping value range is a numerical range determined by the maximum quantization level. For example, if the maximum quantization level is 32, the preset mapping value range can be set to [0, 31]. Every two adjacent integer quantization levels within the preset mapping value range can determine a mapping value interval. For example, the mapping value intervals can be [0, 1], [1, 2], [2, 3].

[0085] A preset mapping relationship is a pre-set relationship used to implement numerical mapping. This relationship can be used to map a value within a larger range to a value within a smaller range. The preset mapping relationship can be determined by the maximum and minimum values ​​within the scale parameter mapping value range, as well as the maximum quantization level. The scale parameter mapping value range can be set based on actual needs after statistical analysis of the data output by the pre-trained entropy model.

[0086] Optionally, the preset mapping relationship between the scale parameter value and the scale parameter mapping value can be calculated by calculating the distance between the scale parameter mapping value and the minimum value, further calculating the ratio of the distance to the interval distance of the preset value range, and then multiplying the ratio by the maximum quantization level to obtain the scale parameter mapping value, wherein the interval distance of the preset value range is the distance between the maximum value and the minimum value.

[0087] Optionally, the preset mapping relationship between the scale parameter value and the scale parameter mapping value can refer to the following formulas (1) and (2), where I is the scale parameter mapping value, σ max is the maximum value of the scale parameter, which can be 64, for example. step is the quantization step size, σ min is the minimum value of the scale parameter, which can be, for example, 0.11; L is the maximum quantization level, which can be set as needed, for example, L can be 32. Formula (1) can be regarded as truncating the input value with 0 as the lower bound and L-1 as the upper bound, so as to achieve mapping the scale parameter value to the preset mapping value range of [0, L-1].

[0088] Specifically, the server may obtain a preset mapping relationship between scale parameter values ​​and scale parameter mapping values, and map each scale parameter value to a preset mapping value range according to the preset mapping relationship to obtain a scale parameter mapping value corresponding to the scale parameter value corresponding to each feature element.

[0089] Step 206: Obtain a constraint reference mapping value corresponding to each of the scale parameter mapping values, wherein the distance between the constraint reference mapping value and the rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is farther than the distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value.

[0090] Step 208 : determining a quantization constraint loss of the training video frame based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values.

[0091] Among them, the rounding boundary value within the preset mapping value range can be the boundary that determines the rounding direction, and the values ​​on both sides of the rounding boundary value can be rounded in different directions when rounded. The preset mapping value range includes multiple rounding boundary values, which divide the preset mapping value range into multiple intervals. The values ​​in the same interval have the same rounding results when rounded according to the same rounding method, and the values ​​in different intervals have different rounding results when rounded according to the same rounding method. The rounding boundary value is determined by the rounding method set when the scale parameter mapping value is quantized. When the rounding method is rounding up or rounding down, the rounding boundary value can be an integer value within the preset mapping value range. Optionally, integers at a certain distance within the preset mapping value range can be used as rounding boundary values. For example, assuming the preset mapping value range is [0, 32], the rounding boundary value can be an integer value that is an odd number such as 1, 3, 7, etc. Optionally, each integer value within the preset mapping value range can also be used as a rounding boundary value; when the rounding method is [0, 32], the rounding boundary value can be an odd integer value such as 1, 3, 7, etc. Optionally, each integer value within the preset mapping value range can be used as a rounding boundary value; when the rounding method is [0, 32], the rounding boundary value can be an odd integer value such as 1, 3, 7, etc. In the case of rounding to the nearest integer, the rounding boundary value can be the average of two adjacent integers within the preset mapping value range. Optionally, the average of every two adjacent integer values ​​within the preset mapping value range can be used as the rounding boundary value, that is, the center of each interval x.5 (x is greater than or equal to 0) within the preset mapping value range is the rounding boundary value. For example, 2.5, 3.5, 4.5, etc. can be used as the rounding boundary value. Optionally, the average of every two adjacent odd and even numbers within the preset mapping value range can be used as the rounding boundary value, such as 1.5, 3.5, 5.5, etc.

[0092] The distance between the constraint reference mapping value and the rounding boundary value of the scale parameter mapping value corresponding to the constraint mapping reference value is farther than the distance between the scale parameter mapping value corresponding to the constraint mapping reference value and the rounding boundary value. That is, for each scale parameter mapping value, assuming that the scale parameter mapping value is x, the constraint reference mapping value corresponding to the scale parameter mapping value is y, and the rounding boundary value of the scale parameter mapping value is z, then the distance between y and z is farther than the distance between x and z.

[0093] Optionally, the rounding boundary value of the scale parameter mapping value may be a rounding boundary value adjacent to the scale parameter mapping value. When the rounding mode is rounding up or rounding down, and each integer within the preset mapping value range is a rounding boundary value, the rounding boundary value adjacent to a certain scale parameter mapping value is the integer closest to the scale parameter mapping value in the mapping value interval to which the scale parameter mapping value belongs. For example, assuming that the scale parameter mapping value is 72.22, then under the rounding up or rounding down mode, the rounding boundary value adjacent to the scale parameter mapping value in the mapping value interval [72,73] to which it belongs is 72; when the rounding mode is rounding to the nearest integer, and the center of each mapping value interval within the preset mapping value range is a rounding boundary value, the rounding boundary value adjacent to a certain scale parameter mapping value is the center of the interval in the mapping value interval to which the scale parameter mapping value belongs. Assuming that the scale parameter mapping value is 72.22, then under the rounding up or rounding down mode, the rounding boundary value adjacent to the scale parameter mapping value in the mapping value interval [72,73] to which it belongs is 72.5.

[0094] The constraint reference mapping value corresponding to a scale parameter mapping value refers to a value used as a reference to constrain the numerical value of the scale parameter mapping value. Specifically, the constraint reference mapping value can be a value that is far from the rounding boundary value in the mapping value interval to which the scale parameter mapping value belongs. Optionally, if the rounding method set by the preset quantization method is rounding up and rounding down, and each integer in the preset mapping value range is a rounding boundary value, then the constraint reference mapping value corresponding to the scale parameter mapping value can be the center of the mapping value interval to which the scale parameter mapping value belongs. For example, assuming that the scale parameter mapping value is 2.2, the constraint reference mapping value of the scale parameter mapping value in the mapping value interval [2, 3] to which it belongs can be 2.5; if the rounding method set by the preset quantization method is rounding to the nearest integer, and the center of each mapping value interval is a rounding boundary value, then the constraint reference mapping value corresponding to the scale parameter mapping value can be the nearest integer boundary in the mapping value interval to which the scale parameter mapping value belongs. For example, assuming that the scale parameter mapping value is 2.2, under the rounding method, the constraint reference mapping value of the scale parameter mapping value in the mapping value interval [2, 3] to which it belongs is 2; assuming that the scale parameter mapping value is 2.7, under the rounding method, the constraint reference mapping value of the scale parameter mapping value in the mapping value interval [2, 3] to which it belongs is 3.

[0095] The quantization constraint loss of the training video frame refers to the loss generated in the process of constraining the scale parameter mapping value of each feature element of the training video frame. The constraint here refers to the numerical size of the scale parameter mapping value obtained by the constraint, so that the scale parameter mapping value is offset in the direction away from the rounding boundary value. The constrained reference mapping value is farther away from the rounding boundary value than the corresponding scale parameter mapping value, that is, the constrained reference mapping value can represent the direction away from the rounding boundary value. The quantization constraint loss is positively correlated with the difference between the scale parameter mapping value and the corresponding constraint reference mapping value. During the model training process, the video codec model can be trained in the direction where the loss is reduced, so that the difference between the scale parameter mapping value and the constraint reference mapping value is reduced. In this way, the scale parameter mapping value output by the video codec model can be as far away from the rounding boundary value as possible.

[0096] Specifically, the server can obtain the constraint reference mapping value corresponding to each scale parameter mapping value, determine the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, and perform statistics on the quantization constraint sub-loss corresponding to each scale parameter mapping value, thereby determining the quantization constraint loss of the training video frame. The statistics can be any one of summation, averaging, or median. The summation can be direct addition or weighted summation, that is, first determine the loss weight corresponding to each quantization constraint sub-loss, and then accumulate the multiplication results of each quantization constraint sub-loss and its respective loss weight.

[0097] In some embodiments, since the quantization constraint sub-loss is positively correlated with the difference between the scale parameter mapping value and the corresponding constraint reference mapping value, that is, after determining the difference between the scale parameter mapping value and the corresponding constraint reference mapping value, the server can determine the difference in various ways that can ensure a positive correlation, for example, squaring the difference, multiplying the difference by a preset multiple, reducing the difference by a preset multiple, adding a constant to the difference, etc.

[0098] In some embodiments, the difference between the scale parameter mapping value and the corresponding constraint reference mapping value may specifically be the difference between the scale parameter mapping value and the corresponding constraint reference mapping value. In other embodiments, the server may calculate function values ​​for the scale parameter mapping value and the corresponding constraint reference mapping value using the same function, and then calculate the difference between the two obtained function values ​​as the difference between the scale parameter mapping value and the corresponding constraint reference mapping value.

[0099] In some embodiments, a constraint parameter value may be preset for each mapping value interval within a preset mapping value range. Specifically, the server may obtain the constraint reference mapping value corresponding to each scale parameter mapping value by determining, for each scale parameter mapping value, the mapping value interval to which the scale parameter mapping value belongs, and using the constraint parameter value preset in the corresponding mapping value interval as the constraint parameter value corresponding to the scale parameter mapping value. In other embodiments, if the difference between the scale parameter mapping value and the corresponding constraint reference mapping value is obtained by calculating the difference in function values ​​of the two under the same function, and the function has the same value at the constraint reference mapping value of each mapping value interval, then the server may also use the constraint reference mapping value of any mapping value interval as the constraint reference mapping value corresponding to all scale parameter mapping values. Thus, after mapping and obtaining the scale parameter mapping value, the server can directly obtain the constraint reference. Since the constraint reference mapping value can be determined without determining the mapping value interval to which the scale parameter mapping value belongs, the efficiency of loss calculation can be improved, thereby improving training efficiency.

[0100] Step 210 : Update at least a portion of model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0101] Herein, at least a part of the model parameters refers to a part of the model parameters or all the model parameters.

[0102] In some embodiments, the trained video codec model is obtained by initializing the model structure of a preset structure model. After all parameters of the video codec model are trained and updated by the video codec model processing method of the present application, the video codec model has normal encoding and decoding capabilities. In the encoding and decoding process, the scale parameter value output by the model is a scale parameter mapping value obtained after mapping, and the scale parameter value is offset in the direction away from the adjacent rounding boundary value. In this case, the bit rate loss and reconstruction distortion loss need to be added to the training process, that is, each time training, the server further calculates the bit rate loss and reconstruction distortion loss, and calculates the bit rate loss, reconstruction distortion loss and quantization constraint loss to obtain a total loss, and the video codec model is trained by the total loss. Among them, the bit rate loss is used to characterize the loss of the bit rate obtained by encoding the video codec model, and the reconstruction distortion loss is used to characterize the loss of the video frame reconstructed by the model.

[0103] In other embodiments, the trained video codec model is obtained through pre-training, and the pre-trained video codec already has normal encoding and decoding capabilities. The video codec model of the embodiment of the present application is used to fine-tune the pre-trained video codec model so that the scale parameter value output by the video codec model is a scale parameter mapping value obtained after mapping, and the scale parameter value is offset in the direction away from the adjacent rounding boundary value. Since the scale parameter mapping value is output by the entropy model of the video codec model, during the training process, other model parameters other than the model parameters of the entropy model in the video codec model can be frozen, and only the model parameters of the entropy model are updated. In this way, the model can be globally optimized, and then the optimal subspace for the quantization constraint loss is found near the optimal subspace, which can not only achieve the best constraint effect, but also well maintain the original encoding and decoding performance of the video codec model. Freezing other model parameters other than the model parameters of the entropy model in the video codec model means that during the training process, other model parameters are kept unchanged and only the model parameters of the entropy model are updated. Optionally, during this process, the server may directly use the quantization constraint loss as the target loss, or combine the quantization constraint loss with other losses that may occur in the video encoding and decoding process, such as bit rate loss and reconstruction distortion loss, to obtain the target loss.

[0104] During the specific training process, the server can determine a target loss for parameter update based on the quantization constraint loss, and then update the model parameters of the entropy model based on the target loss. After the update is completed, the training is completed. The server can then obtain the next training video frame to continue training. During the continued training process, the following steps are repeatedly performed: extracting a feature map of the training video frame through the video codec model and determining the scale parameter value corresponding to each feature element in the feature map; mapping each feature element to a preset mapping value range according to a preset mapping relationship to obtain the scale parameter mapping value corresponding to each feature element; obtaining the constraint reference mapping value corresponding to each scale parameter mapping value; determining the quantization constraint loss of the training video frame based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value; and updating at least a portion of the model parameters of the video codec model based on the quantization constraint loss. The continued training process is iterative training, that is, after each training, the video codec model with adjusted parameters is used as the model for the next training process. When the training process meets the training stop condition, the trained video codec model is obtained. The training stopping condition can be that the model parameters of the entropy model no longer change, the target loss reaches the minimum value, or the number of training iterations reaches the maximum number, etc.

[0105] During the training process, the purpose of updating the model parameters of the video codec model is to continuously reduce the loss obtained during the training process, that is, the training process updates the model parameters of the video codec model in the direction of reducing the loss. Therefore, after the server determines the target loss for updating the model parameters based on the quantization constraint loss, it can use a preset algorithm to calculate the updated value of each parameter that needs to be updated in the video codec model according to the target loss, and then replace the current value of the parameter in the model with the calculated updated value to achieve the update of the model parameters. The preset algorithm can, for example, use the gradient (derivative) of the parameter as a clue to update the parameter along the gradient direction. Each time the parameter is updated, the loss obtained in the next training process can be reduced, and repeated multiple times gradually approach the optimal parameter. In specific applications, the preset algorithm can be any one of the stochastic gradient descent algorithm, Adagrad (Adaptive Gradient) algorithm, Adadelta (an improvement of the AdaGrad algorithm), RMSprop (an improvement of the AdaGrad algorithm), and Adam (Adaptive Moment Estimation) algorithm.

[0106] In the above-mentioned video codec model processing method, since the constraint reference mapping value is farther away from the rounding boundary value than the corresponding scale parameter mapping value, the quantization constraint loss determined based on the difference between the scale parameter mapping value and the corresponding constraint reference mapping value can reflect the degree of distance of the scale parameter mapping value from the rounding boundary value. When the parameters of the video codec model are updated based on the quantization constraint loss, the scale parameter mapping value corresponding to the scale parameter value output by the video codec model can be constrained, thereby indirectly realizing the constraint on the scale parameter value output by the video codec model, so that the scale parameter value output by the video codec model is offset in the direction away from the rounding boundary value after mapping, that is, the probability of the scale parameter mapping value being close to the rounding boundary value is reduced, which alleviates the problem of inconsistent quantization results obtained in the encoding process and the decoding process in the cross-platform coding and decoding scenario, and thus can improve the accuracy of the video frames reconstructed by the decoding process.

[0107] In some embodiments, based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, the quantization constraint loss of the training video frame is determined, including: based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, determining the quantization constraint sub-loss corresponding to each scale parameter mapping value; determining the loss weight corresponding to each scale parameter mapping value; and determining the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value.

[0108] Among them, the loss weight is used to characterize the importance of the quantization constraint sub-loss to the quantization constraint loss of the training video frame. The larger the loss weight, the greater the importance of the quantization constraint sub-loss to the quantization constraint loss of the training video frame. Considering that the purpose of training a video codec model is to constrain the scale parameter value output by the video codec model, so that the scale parameter mapping value obtained after mapping the scale parameter value output by the video codec model is as far away from the rounding boundary value as possible, the attributes of the scale parameter mapping value itself can determine the importance of the quantization constraint sub-loss of the scale parameter mapping value to the quantization constraint loss of the training video frame. Optionally, the attribute of the scale parameter mapping value itself can be a numerical attribute, i.e., the numerical size, or a position attribute, i.e., the specific position of the scale parameter mapping value within a preset mapping value range. For example, if a scale parameter mapping value is relatively close to a rounding boundary value within the preset mapping value range, then the quantization constraint sub-loss of the scale parameter mapping value is relatively important to the quantization constraint loss of the training video frame, and a larger loss weight can be determined for this scale parameter value. On the other hand, if a scale parameter mapping value is relatively far from each constraint reference mapping value within the preset mapping value range, then the quantization constraint sub-loss of the scale parameter mapping value is relatively less important, and a smaller loss weight can be determined for this scale parameter value.

[0109] Specifically, the server can determine the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, and determine the loss weight corresponding to each scale parameter mapping value. The quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value are multiplied, and then the multiplication results are added to obtain the quantization constraint loss of the training video frame.

[0110] In some specific embodiments, the server can determine the loss weight corresponding to the scale parameter mapping value based on the distance between the scale parameter mapping value and the rounding boundary value within the mapping value interval to which it belongs. In specific applications, the distance between the scale parameter mapping value and the rounding boundary value within the mapping value interval to which it belongs can be negatively correlated with the loss weight, that is, the larger the distance, the smaller the loss weight, and the smaller the distance, the larger the loss weight. The negative correlation here means that the distance and the loss weight change in different directions, but it does not mean that the loss weight must change when the distance changes. For example, it can be set that when the distance is less than the first distance, the loss weight is the first value, when the distance is greater than the first distance and less than the second distance, the loss weight is the second value, and when the distance is greater than the second distance, the loss weight is the third value. Among them, the first distance and the second distance are different and can be set as needed. For example, the first distance can be 0.1 and the second distance can be 0.3; the first value is greater than the second value, and the second value is greater than the third value. The first value, the second value and the third value can be set as needed. For example, the first value can be 1, the second value can be 0.6, and the third value can be 0.2.

[0111] In the above embodiment, by determining a loss weight for each scale parameter mapping value, the loss weight can reflect the importance of the scale parameter mapping value. Therefore, based on the loss weight of each scale parameter mapping value and the quantization constraint sub-loss, a more accurate quantization constraint loss can be obtained.

[0112] In some embodiments, determining the loss weight corresponding to each scale parameter mapping value includes: determining the attribution relationship between each scale parameter mapping value and a preset numerical interval; determining the loss weight corresponding to each scale parameter mapping value according to the attribution relationship between each scale parameter mapping value and the preset numerical interval; and determining the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value.

[0113] In this embodiment, the attribute of the scale parameter mapping value may be its relationship with a preset numerical range. Scale parameter mapping values ​​falling within the preset numerical range have consistent rounding results when encoding and decoding the corresponding feature elements. The rounding result here refers to the final integer value obtained using the preset quantization method, i.e., the scale parameter quantization value corresponding to the scale parameter mapping value.

[0114] The relationship between the scale parameter mapping value and the preset numerical range includes belonging and not belonging. When a scale parameter mapping value belongs to the preset numerical range, it means that the scale parameter quantization value obtained by the characteristic element corresponding to the scale parameter mapping value during the decoding process will not jump. When a scale parameter mapping value does not belong to the preset numerical range, it means that the scale parameter quantization value obtained by the characteristic element corresponding to the scale parameter mapping value during the decoding process may jump. The jump will cause inconsistent rounding results at the decoding end, thereby causing decoding failure during the decoding process. The jump here refers to the situation in which, during the cross-platform video encoding and decoding process, for the same feature element, the scale parameter mapping value obtained in the decoding process crosses a certain rounding boundary value relative to the scale parameter mapping value obtained in the encoding process. For example, assuming that for a feature element y1 in the feature map, the encoding process is performed by the first device, the scale parameter value output by the video codec model during the encoding process is 0.115629, and the scale parameter mapping value obtained after mapping the scale parameter value is 1.9998, while the decoding process is performed by the second device, the scale parameter value output by the video codec model during the decoding process is 0.115628, and the scale parameter mapping value obtained after mapping the scale parameter value is 2.0001. It can be seen that the scale parameter mapping value obtained in the decoding process jumps at the rounding boundary value of 2. In this case, when the rounding method set by the preset quantization method is rounding down, the rounding result obtained when the first device performs encoding is 1, while the rounding result obtained when the second device performs decoding is 2, so that the decoding end will have a decoding error at this feature element during decoding and reconstruction.

[0115] In order to avoid the occurrence of this jump phenomenon as much as possible, the loss weight of the scale parameter mapping value corresponding to the characteristic element that jumps is relatively important for the training process and is a loss that needs special attention during the training process. Therefore, the loss weight corresponding to the scale parameter mapping value can be determined based on the relationship between the scale parameter mapping value and the preset numerical range. Then, the quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value can be multiplied together, and then the various multiplication results are added together to obtain the quantization constraint loss of the training video frame.

[0116] In some embodiments, according to the relationship between each scale parameter mapping value and the preset numerical range, the loss weight corresponding to each scale parameter mapping value is determined, including: for the scale parameter mapping value belonging to the preset numerical range, the corresponding loss weight is determined to be a first weight; for the scale parameter mapping value not belonging to the preset numerical range, the corresponding loss weight is determined to be a second weight; the second weight is greater than the first weight.

[0117] Specifically, the server can judge the scale parameter mapping value of each feature element in the feature map of the training video frame separately. When the scale parameter mapping value of a feature element belongs to a preset numerical range, the loss weight corresponding to the scale parameter mapping value of the feature element is determined to be the first weight. When the scale parameter mapping value of a feature element does not belong to the preset numerical range, the loss weight corresponding to the scale parameter mapping value of the feature element is determined to be the second weight. The values ​​of the second weight and the first weight can be determined as needed. It only needs to ensure that the second weight is greater than the first weight. For example, the second weight can be set to 1 and the first weight can be set to 0.

[0118] In some specific embodiments, the distance between the scale parameter mapping value belonging to the preset numerical interval and the corresponding constraint reference mapping value is less than a preset threshold.

[0119] The distance between the scale parameter mapping value and the corresponding constraint reference mapping value is less than the preset threshold. This can be understood as the scale parameter mapping value being close to the constraint reference mapping value within the corresponding mapping value interval. When the scale parameter mapping value is close to the constraint reference mapping value, the corresponding feature element will not jump during decoding. Therefore, the distance between the scale parameter mapping value belonging to the preset numerical interval and the corresponding constraint reference mapping value can be set to be less than the preset threshold.

[0120] In a specific application, when the rounding method set by the preset quantization method is rounding up or rounding down, and the constraint reference mapping value can be the center of the mapping value interval, the preset numerical interval may include: the distance between the scale parameter mapping value belonging to the preset numerical interval and the interval center of the mapping value interval to which it belongs is less than a preset threshold. Taking the preset quantization method setting the rounding method as rounding up as an example, the scale parameter mapping value belonging to the preset numerical interval satisfies: Wherein, x represents the scale parameter mapping value, η is a preset threshold, and the specific value of η can be set as needed, for example, it can be 0.3.

[0121] In some specific embodiments, the distance between the scale parameter mapping value belonging to the preset numerical interval and the lower limit value of the preset mapping value range is less than a preset threshold.

[0122] The lower limit of the preset mapping value range is the minimum quantization level. For example, assuming that the preset mapping value range is [0, 31], the lower limit of the preset mapping value range is 0. The distance between the scale parameter mapping value and the corresponding constrained reference mapping value is less than the preset threshold. This can be understood as the scale parameter mapping value being close to the lower limit of the preset mapping value range within the preset mapping value range. When the scale parameter mapping value is close to the lower limit of the preset mapping value range, the corresponding feature element will not jump during decoding. Therefore, the distance between the scale parameter mapping value belonging to the preset numerical interval and the lower limit of the preset mapping value range can be set to be less than the preset threshold.

[0123] Taking the lower limit of the preset mapping value range as 0 as an example, the scale parameter mapping value belonging to the preset numerical interval meets the following condition: x<η, where x represents the scale parameter mapping value.

[0124] In the above embodiment, by setting a preset numerical interval, the loss weight can be quickly determined according to the attribution relationship between the scale parameter mapping value and the preset numerical interval, thereby improving the loss calculation efficiency and thus improving the training efficiency.

[0125] In some embodiments, the quantization constraint loss of the training video frame is determined based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, including: obtaining a preset probability distribution function; determining the expected probability value of each scale parameter mapping value and the expected probability value of the constraint reference mapping value corresponding to each scale parameter mapping value through the probability distribution function; determining the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the expected probability value of each scale parameter mapping value and the expected probability value of the corresponding constraint reference mapping value; and determining the quantization constraint loss of the training video frame according to the quantization constraint sub-loss corresponding to each scale parameter mapping value.

[0126] Among them, the probability distribution function is used to input a scale parameter mapping value within a preset mapping value range and output an expected probability value of the input scale parameter mapping value. Optionally, within the range from the constraint reference mapping value to its nearest rounding boundary value in the preset mapping value range, the closer the scale parameter mapping value input to the probability distribution function is to the constraint reference mapping value, the greater the expected probability value output by the probability distribution function, that is, the expected probability value decreases from the expected probability value at the constraint reference mapping value to the expected probability value at the rounding boundary value, that is, the expected probability value output by the probability distribution function gradually decreases as the distance between the input scale reference mapping value and the constraint reference mapping value increases. Here, the expected probability value of the scale parameter mapping value is the probability of occurrence of the scale parameter mapping value expected during the training process. That is to say, in the training process of the present application, the expected scale parameter mapping value is distributed in the direction close to the constraint reference mapping value.

[0127] In some specific embodiments, when the rounding method of the preset quantization method is rounding up or rounding down, the constrained reference mapping value of each mapping value interval is the center of the interval, then the expected probability value of each mapping value interval can obtain the maximum value at the center of the interval, and the expected probability value shows a monotonically decreasing trend from the center of the interval toward the integer boundary of the mapping value interval. Optionally, the preset probability distribution function can be a Gaussian distribution or a Laplace distribution. Taking the Gaussian distribution as an example, the preset probability distribution function within the entire preset mapping value range can be represented by a piecewise Gaussian function as shown in the following formula (3):

[0128] Where x represents the value for which probability calculation is required. For example, if the expected probability value needs to be calculated for the scale parameter mapping value, x is the scale parameter mapping value. If the expected probability value needs to be calculated for the value at the center of the interval, x is the value at the center of the interval. Represents the center of the interval of the mapping value to which the scale parameter mapping value belongs, that is, the probability distribution described by the piecewise Gaussian function is the same in each mapping value interval, and in each mapping value interval, it is expressed as a Gaussian distribution with the interval center as the mean. δ is the standard deviation of the Gaussian distribution, and δ can be 1.

[0129] Considering that the distribution of the piecewise Gaussian function in each mapping value interval is the same, the probability at the center of each mapping value interval is also the same. Therefore, the probability value G(0.5,δ) of the interval center in the [0,1] interval can be used to represent the probability value of the interval center of all mapping value intervals. The quantization constraint loss can be calculated by the following formula (4):

[0130] Among them, G(0.5,δ) is the expected probability value of the interval center, G(I i ,δ) is the expected probability value of the scale parameter mapping value corresponding to the i-th feature element in the feature map, H and W represent the height and width of the feature map respectively, and C represents the number of channels. Represents the quantization constraint loss. M(I i ) represents the loss weight of the scale parameter mapping value corresponding to the i-th feature element. i ) can be a binary function of the following formula (5):

[0131] That is, when the I value is in the range of the mapped value interval far from the integer boundary, I can be unconstrained, and when the I value is close to the boundary of the mapped value interval, the Gaussian constraint shown above is applied; in addition, when the I value is close to 0, no constraint is required, because I will be truncated to a minimum of 0 in the process of obtaining I, so there will be no jump in the I value near 0.

[0132] In the above embodiment, by determining the expected probability value of each scale parameter mapping value in the expected probability distribution of the preset mapping value range, and determining the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the expected probability value of each scale parameter mapping value and the first reference probability value of the corresponding constraint reference mapping value, a more accurate quantization constraint sub-loss can be obtained, thereby making the performance of the trained entropy model better.

[0133] In some embodiments, based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, the quantization constraint loss of the training video frame is determined, including: determining the difference between each scale parameter mapping value and the corresponding constraint reference mapping value; determining the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between each scale parameter mapping value and the corresponding constraint reference mapping value; and determining the quantization constraint loss of the training video frame based on the quantization constraint sub-loss corresponding to each scale parameter mapping value.

[0134] Specifically, in this embodiment, the server can calculate the mean square error (MSE), that is, after calculating the difference between each scale parameter mapping value and the corresponding constraint reference mapping value, calculate the square of the difference, and calculate the average of all squares of the difference to obtain the quantization constraint sub-loss corresponding to each scale parameter mapping value, and then calculate the average of all quantization constraint sub-losses to obtain the quantization constraint loss of the training video frame.

[0135] In the above embodiment, the quantization constraint sub-loss corresponding to each scale parameter mapping value is determined based on the difference between each scale parameter mapping value and the corresponding constraint reference mapping value, which can quickly determine the quantization constraint loss of the training video frame, thereby improving training efficiency.

[0136] In some embodiments, at least a portion of the model parameters of the video codec model are updated based on quantization constraint loss, bitrate loss and reconstruction distortion loss, and the bitrate loss and reconstruction distortion loss are calculated through a loss calculation step, and the loss calculation step includes: determining the probability value of each feature element according to the scale parameter value and position parameter value of the probability distribution of each feature element; the position parameter value of each feature element is obtained by predicting the probability distribution of each feature element; determining the probability value of the training video frame according to the probability value of each feature element, and performing cross entropy calculation on the probability value of the training video frame to obtain the bitrate loss; determining the reconstructed video frame of the training video frame, and determining the reconstruction distortion loss according to the difference between the training video frame and the reconstructed video frame of the training video frame.

[0137] Specifically, the probability distribution q predicted by the entropy model is shown in the following formula:

[0138] Where Etp represents the entropy model, is the super prior of the t-frame video frame, c is the conditional input of the entropy model (when the t-frame video frame is an I-frame, there is no c as a condition; when the t-frame video frame is a p-frame, the reference information c of the previous frame needs to be input into the entropy model as a condition), (μ, σ) represents the parameter value of the probability distribution predicted by the entropy model, represents a normal distribution, Represents a uniform distribution with width 1.

[0139] Therefore, the bit-rate loss can be calculated by the following formula (7):

[0140] Among them, p represents the actual probability distribution of the feature map of the t-frame video frame, is the function used to calculate the cross entropy.

[0141] The reconstruction distortion loss is determined based on the difference between the reconstructed video frame obtained by reconstructing the training video frame through the video codec model and the training video frame. Optionally, the server can refer to the following formulas (8) and (9) to calculate the reconstruction distortion loss, where x represents the original frame, Represents the original frame reconstructed frame, H and W represent the height and width of the video frame respectively, and C represents the number of channels:

[0142] After obtaining the reconstruction distortion loss and bit rate loss, the two losses can be statistically analyzed using the following formula (10) to obtain the loss:

[0143] Where D represents the reconstruction distortion loss between the reconstructed video frame and the original frame, and λ is a weight used to adjust the trade-off between reconstruction quality and transmission bit-rate.

[0144] Furthermore, the server can calculate the loss obtained by statistically analyzing the reconstruction distortion loss and bit rate loss. and quantized constraint loss The final target loss is calculated together, referring to the following formula (11), where β is used to adjust the constraint strength applied to the output I, and β can be set as needed, for example, it can be set to 1.0.

[0145] Then, the server may update at least a portion of the model parameters of the video coding model according to the target loss.

[0146] In the above embodiment, during the training process of the entropy model, bit rate loss and reconstruction distortion loss are also combined, so that the video encoding and decoding model containing the trained entropy model can maintain the original performance as much as possible, and the output value of the entropy model can be offset in the direction away from the rounding boundary value after mapping.

[0147] The video codec model processing method provided in the above embodiment is to achieve indirect constraint on the scale parameter value output by the video codec model by constraining the scale parameter mapping value obtained by mapping the scale parameter value output by the video codec model to a preset mapping value range, thereby achieving the constraint in the direction away from the rounding boundary value after mapping the output scale parameter value. In the following embodiment, this application also provides another video codec model processing method. In this video codec model processing method, the same purpose as the above embodiment is achieved by directly constraining the scale parameter value output by the video codec model. It can be understood that the process of constraining the scale parameter value output by the video codec model is similar to the above. Therefore, the specific explanation of some steps in the following embodiment can refer to the description of the above embodiment, and will not be repeated in this application.

[0148] In some embodiments, another video codec model processing method is provided. The method is performed by a computer device, which may be the server 104 or the terminal 102 in FIG. 1 . As shown in FIG. 3 , in the embodiment of the present application, the method is described by applying the method to the server in FIG. 1 as an example, including the following steps:

[0149] Step 302: obtain a training video frame, extract a feature map of the training video frame using a video encoding and decoding model, and determine the scale parameter value corresponding to each feature element in the feature map.

[0150] Step 304 : Obtain a constraint reference value corresponding to each scale parameter value, and determine a quantization constraint loss for the training video frame based on a difference between each scale parameter value and the corresponding constraint reference value.

[0151] The constraint reference value is further away from the rounding boundary value of the scale parameter mapping value obtained by mapping the corresponding scale parameter value after the constraint reference value is mapped to the preset mapping value range according to the preset mapping relationship than the scale parameter value corresponding to the constraint reference value. In other words, the distance between the value obtained by mapping the constraint reference value to the preset mapping value range according to the preset mapping relationship and the rounding boundary value of the scale parameter mapping value obtained by mapping the scale parameter value corresponding to the constraint reference value is further away from the rounding boundary value of the scale parameter mapping value obtained by mapping the scale parameter value corresponding to the constraint reference value according to the preset mapping relationship.

[0152] Optionally, each scale parameter value and the corresponding constraint reference value belong to the same scale parameter value interval; the constraint reference value is farther from a rounding reference point adjacent to the corresponding scale parameter value in the scale parameter value interval than the corresponding scale parameter value; the two boundary values ​​of the scale parameter value interval can be mapped to two adjacent integer quantization levels under the preset quantization method; and the rounding reference point can be mapped to the rounding boundary values ​​of two adjacent integer quantization levels under the preset quantization method. The rounding boundary values ​​of adjacent integer quantization levels are the rounding boundary values ​​in the mapping value interval determined by the two adjacent integer quantization levels. Taking the rounding method set by the preset quantization method as rounding up as an example, if a scale parameter value x1 can be mapped to 2 under the preset quantization method, and another scale parameter value x2 can be mapped to 3 under the preset quantization method, then these two scale parameter values ​​can serve as two boundary values ​​to determine a scale parameter value interval [x1, x2]. Since the rounding boundary values ​​in the mapping value interval [2, 3] are 2 and 3, x1 and x2 in the scale parameter value interval [x1, x2] are the rounding reference points of the interval.

[0153] The constraint reference value corresponding to the scale parameter value is used as a reference to constrain the scale parameter value. Because the constraint reference value is farther from the rounding reference point adjacent to the corresponding scale parameter value in the mapping value range than the corresponding scale parameter value, constraining the scale parameter value using the constraint reference value as a reference can cause the scale parameter value to tend away from the rounding reference point. The constraint reference value can be a numerical point that is less likely to jump after being mapped to the preset mapping value range. Optionally, if the rounding method set by the preset quantization method is rounding up and rounding down, the constraint reference value corresponding to a certain scale parameter value can be the center of the scale parameter value interval to which the scale parameter value belongs, or the value in the scale parameter value interval to which the scale parameter value belongs that can be mapped to the center of the corresponding mapping value interval. For example, assuming that a certain scale parameter value x0 in the scale parameter value interval [x1, x2] above can be mapped to the interval center 2.5 of the corresponding mapping value interval [2, 3], then x0 is the constraint reference value in the scale parameter value interval [x1, x2]; if the rounding method set by the preset quantization method is rounding to the nearest integer, the constraint reference value can be the nearest boundary value in the scale parameter value interval to which the scale parameter value belongs.

[0154] Specifically, the server can determine the constraint reference value corresponding to each scale parameter value, determine the quantization constraint sub-loss corresponding to each scale parameter value based on the difference between each scale parameter value and the corresponding constraint reference value, and perform statistics on the quantization constraint sub-loss corresponding to each scale parameter value to determine the quantization constraint loss of the training video frame, wherein the quantization constraint sub-loss can be positively correlated with the difference between the scale parameter value and the corresponding constraint reference value.

[0155] Step 306: Update at least a portion of the model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0156] In some embodiments, the trained video codec model is obtained by initializing the model structure of a preset structure model, and all parameters of the video codec model are trained and updated using the video codec model processing method of the present application. In other embodiments, the trained video codec model is obtained through pre-training, and the pre-trained video codec already has normal encoding and decoding capabilities. The video codec model of an embodiment of the present application is used to fine-tune the pre-trained video codec model. During the training process, only the model parameters of the entropy model can be updated, and the remaining model parameters in the video codec model can be frozen.

[0157] The above-mentioned video codec model processing method, since the constraint reference value is farther away from the rounding boundary value than the corresponding scale parameter value after being mapped to the preset mapping value range according to the preset mapping relationship, the quantization constraint loss determined based on the difference between the scale parameter value and the corresponding constraint reference value can reflect the degree of distance of the scale parameter mapping value from the rounding boundary value. Therefore, when the parameters of the video codec model are updated based on the quantization constraint loss, the scale parameter value output by the video codec model can be constrained, so that the scale parameter value output by the video codec model is offset in the direction away from the rounding boundary value after mapping, that is, the probability of the scale parameter mapping value being close to the rounding boundary value is reduced, which alleviates the problem of inconsistent quantization results obtained in the encoding process and the decoding process in the cross-platform coding and decoding scenario. Therefore, the accuracy of the video frame reconstructed by the decoding process can be improved, and since the quantization constraint loss directly acts on the scale parameter value, the numerical value directly output by the video codec model can be constrained, and the training process does not need to perform other additional mapping steps, and the training efficiency is high.

[0158] In some embodiments, based on the difference between each scale parameter value and the corresponding constraint reference value, the quantization constraint loss of the training video frame is determined, including: determining the quantization constraint sub-loss corresponding to each scale parameter value based on the difference between each scale parameter value and the corresponding constraint reference value; determining the loss weight corresponding to each scale parameter value; and determining the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter value.

[0159] In some embodiments, determining the loss weight corresponding to each scale parameter value includes: determining the relationship between each scale parameter value and a preset numerical range; the scale parameter values ​​belonging to the preset numerical range and the scale parameter values ​​obtained by encoding and decoding the corresponding feature elements have consistent quantization results; determining the loss weight corresponding to each scale parameter value based on the relationship between each scale parameter value and the preset numerical range; and determining the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter value. The quantization result here is the integer value finally obtained under the preset quantization method, that is, the scale parameter quantization value corresponding to the scale parameter value.

[0160] In some embodiments, according to the relationship between each scale parameter value and the preset numerical range, the loss weight corresponding to each scale parameter value is determined, including: for the scale parameter value belonging to the preset numerical range, the corresponding loss weight is determined to be a first weight; for the scale parameter value not belonging to the preset numerical range, the corresponding loss weight is determined to be a second weight; the second weight is greater than the first weight.

[0161] In some embodiments, the distance between the scale parameter value belonging to the preset numerical interval and the corresponding constraint reference value is less than a preset threshold.

[0162] In some embodiments, the distance between the scale parameter value belonging to the preset numerical interval and the lower limit value of the scale parameter value range is less than a preset threshold.

[0163] In some embodiments, based on the difference between each scale parameter value and the corresponding constraint reference value, the quantization constraint loss of the training video frame is determined, including: obtaining a preset probability distribution function, the probability distribution function is used to input the scale parameter value and output the expected probability value of the input scale parameter value; through the probability distribution function, the expected probability value of each scale parameter value and the expected probability value of the constraint reference value corresponding to each scale parameter value are determined; based on the difference between the expected probability value of each scale parameter value and the expected probability value of the corresponding constraint reference value, the quantization constraint sub-loss corresponding to each scale parameter value is determined; according to the quantization constraint sub-loss corresponding to each scale parameter value, the quantization constraint loss of the training video frame is determined.

[0164] Among them, in the definition domain of the scale parameter value, within the range from the constraint reference value to the scale parameter value closest to it that is used to map to the rounding boundary value, the closer the scale parameter value of the input probability distribution function is to the constraint reference value, the greater the expected probability value output by the probability distribution function.

[0165] In some embodiments, based on the difference between each scale parameter value and the corresponding constraint reference value, the quantization constraint loss of the training video frame is determined, including: determining the difference between each scale parameter value and the corresponding constraint reference value; determining the quantization constraint sub-loss corresponding to each scale parameter value according to the difference between each scale parameter value and the corresponding constraint reference value; determining the quantization constraint loss of the training video frame according to the quantization constraint sub-loss corresponding to each scale parameter value.

[0166] In some embodiments, the model parameters of the entropy model are updated based on the quantization constraint loss, bitrate loss and reconstruction distortion loss. The bitrate loss and reconstruction distortion loss are calculated through a loss calculation step. The loss calculation step includes: determining the probability value of each feature element according to the scale parameter value and position parameter value of the probability distribution of each feature element; the position parameter value of each feature element is obtained by predicting the probability distribution of each feature element; determining the probability value of the training video frame according to the probability value of each feature element, and performing cross-entropy calculation on the probability value of the training video frame to obtain the bitrate loss; determining the reconstructed video frame of the training video frame, and determining the reconstruction distortion loss according to the difference between the training video frame and the reconstructed video frame of the training video frame.

[0167] In some embodiments, as shown in FIG4 , a video encoding method is provided. The method is performed by an encoding end, which may be the server 104 or the terminal 102 in FIG1 . In the embodiments of the present application, the method is described by applying it to the server in FIG1 , and includes the following steps:

[0168] Step 402 : Obtain a target video frame, extract a feature map of the target video frame using the trained video codec model, and determine the scale parameter value corresponding to each feature element in the feature map.

[0169] The video codec model is trained, and after the output scale parameter value is mapped according to a preset mapping relationship, the obtained scale parameter mapping value is offset in a direction away from the adjacent rounding boundary value.

[0170] It is understandable that the trained video codec model can be obtained by training using the codec model processing method provided in any of the above embodiments. The specific training process can be referred to the above embodiments and will not be described in detail here.

[0171] Step 404 : Map the scale parameter values ​​corresponding to the characteristic elements according to the preset mapping relationship to obtain scale parameter mapping values, where the scale parameter mapping values ​​are within the preset mapping value range.

[0172] Step 406 : The scale parameter mapping value obtained by mapping is rounded according to its rounding boundary value for quantization to obtain the scale parameter quantization value of the feature element.

[0173] Specifically, after the video codec model is trained, the scale parameter values ​​output by the video codec model are mapped according to a preset mapping relationship, and the scale parameter mapping values ​​obtained are offset in the direction away from the adjacent rounding boundary values. That is, the overall trend of the scale parameter mapping values ​​obtained by mapping the scale parameter values ​​output by the video codec model under the preset quantization method is to move away from the adjacent rounding boundary values ​​under the preset quantization method. If the mean is used to represent the scale parameter mapping value in each mapping value interval under the preset quantization method, then for a certain video frame, among the scale parameter mapping values ​​obtained by the trained video codec model, the actual mean of the scale parameter mapping values ​​corresponding to the same adjacent rounding boundary value is offset compared to the reference mean of the corresponding rounding boundary value, and the direction of the offset is away from the rounding boundary value, that is, the distance between the actual mean and the rounding boundary value is greater than the distance between the reference mean and the rounding boundary value.

[0174] Here, the reference mean of a rounding boundary value refers to, for the same video frame, the mean of all scale parameter mapping values ​​obtained by the pre-trained video codec model with the rounding boundary value as the adjacent rounding boundary value. The pre-trained video codec model can be understood as a video codec model that has not been trained using the video codec model processing method of the present application. For example, assuming that for a certain video frame, the scale parameter mapping values ​​obtained by the pre-trained video codec model are evenly distributed, and all scale parameter mapping values ​​with the integer boundary 30 as the adjacent rounding boundary value are in the interval [29.5, 30], and their mean is 29.75. Then, among the scale parameter mapping values ​​obtained by the trained video codec model, for the same video frame, the mean of all scale parameter mapping values ​​with the integer boundary 30 as the adjacent rounding boundary value will be less than 29.75, for example, it may be 29.6, that is, among the scale parameter mapping values ​​obtained by the trained video codec model, the mean of all scale parameter mapping values ​​with the integer boundary 30 as the adjacent rounding boundary value has shifted relative to 29.75, thereby being further away from the integer boundary 30.

[0175] It can also be understood from another perspective that if the distance between the definition and the rounding boundary value is less than the preset threshold value, it is close to the rounding boundary value. Then, the scale parameter mapping value obtained by the video codec model trained by the present application will be close to the rounding boundary value with a smaller probability than the scale parameter mapping value obtained under random probability. For example, assuming that the preset threshold is b and the rounding boundary value is x, for a certain target video frame, the scale parameter mapping value obtained by the video codec model before training will fall into the intervals [xb, x] and [x, x+b] with random probability. Assuming that 50 will fall into [xb, x] and [x, x+b], for the same target video frame, the scale parameter value output by the video codec model trained by the present application is mapped under the preset quantization method. Only 20 scale parameter mapping values ​​will fall into [xb, x] and [x, x+b], that is, the probability of the scale parameter mapping value falling into [xb, x] and [x, x+b] is greatly reduced.

[0176] Specifically, the server may map the scale parameter values ​​of all feature elements in the feature map into scale parameter mapping values ​​according to a preset mapping relationship, and then round them according to their respective rounding boundary values ​​for quantization to obtain a scale parameter quantization value of each feature element.

[0177] Optionally, the server may round the scale parameter values ​​of all feature elements according to the rounding boundary value set in the preset quantization method. Also optionally, the server may round a portion of the feature elements according to the rounding boundary value set in the preset quantization method.

[0178] Step 408 : Entropy coding is performed on the feature map according to the quantized values ​​of the scale parameters of the feature elements of the feature map, and a transmission data stream is determined according to the coded data stream obtained by the entropy coding.

[0179] Specifically, since the original scale parameter value is quantized, when performing entropy coding, for each feature element, the server can re-determine the scale parameter value of the feature element based on the quantized scale parameter value corresponding to the feature element, and determine the probability distribution function required for entropy coding based on the re-determined scale parameter value, thereby compressing the feature map into as few byte streams as possible through the probability distribution function, thereby obtaining the encoded data stream corresponding to the feature element. Entropy coding can be implemented using arithmetic coding or range coding. Taking arithmetic coding as an example, after obtaining the probability distribution function required for arithmetic coding, the probability value of each feature element in the feature map can be calculated, and then the feature elements are read in one by one. For each feature element read in, the range of the feature map on [0,1] is reduced to the newly obtained interval according to a ratio determined by the probability value of each feature element. Then, it is iterated in sequence until all feature elements have been read, and any decimal in the obtained interval is output in binary form to obtain the encoded data stream.

[0180] In specific applications, re-determining the scale parameter value according to the scale parameter quantization value can be achieved by formula (12), where θ is the re-determined scale parameter value, σ step is the quantization step size, σ min is the minimum value of the scale parameter, which can be 0.11, and L is the maximum quantization level, which can be set as needed, for example, L can be 32. It is the quantitative value of the scale parameter, and its value range is 0-L-1.

[0181] Optionally, the server can also construct the above formula (12) The mapping relationship between and θ is used to construct a probability distribution function lookup table, such as: Corresponding to θ=0.1100, Corresponding to θ = 0.1128. Therefore, after obtaining the quantized value of the scale parameter of each feature element, θ can be directly obtained by looking up the probability distribution function lookup table, thereby improving the arithmetic coding efficiency.

[0182] Optionally, after obtaining the probability distribution function required for arithmetic coding, the server determines the probability value of the feature element by substituting the probability value of the feature element into the probability distribution function to calculate the probability value; optionally, the server can subtract the corresponding position parameter value from the value at each feature element position in the feature graph y, and the feature graph y_0 obtained after subtraction can show a 0 mean distribution, so that the server can look up a pre-established probability value lookup table according to the value of each feature element in the 0 mean distribution feature graph y_0 to obtain the probability value, wherein the pre-established probability lookup value is established in the following manner: under each probability distribution function determined by θ and mean 0, the probability value of the possible value of the feature element is calculated, and a correspondence between the possible value of the feature element and the calculated probability value under each probability distribution function is established.

[0183] In the above-mentioned video encoding method, since the video encoding and decoding model is trained using the video encoding and decoding model processing method of any of the above embodiments, the scale parameter value output by the video encoding and decoding model is mapped according to a preset mapping relationship, and the scale parameter mapping value obtained is offset in the direction away from the adjacent rounding boundary value. Therefore, when the target video frame is encoded using this video encoding and decoding model, the scale parameter value in the encoding process, after mapping, the scale parameter mapping value obtained will be as far away from the rounding boundary value as possible, thereby reducing the probability of rounding jumps occurring in the quantization process at the decoding end, so that the encoding and decoding processes can be encoded and decoded based on the same quantization results, thereby improving the accuracy of the video frame reconstructed by the decoding process.

[0184] In other specific embodiments, the scale parameter mapping values ​​obtained by mapping are rounded according to their rounding boundary values ​​for quantization to obtain the scale parameter quantization values ​​of the feature elements, including: screening feature elements that meet preset screening conditions from each feature element of the feature map; rounding the scale parameter mapping values ​​of each feature element of the feature map according to their respective rounding boundary values ​​for quantization to obtain the scale parameter quantization values ​​of each feature element, wherein the scale parameter mapping values ​​of the unscreened feature elements are rounded according to their rounding boundary values ​​under a preset quantization method, and the scale parameter mapping values ​​of the screened feature elements are rounded according to their rounding boundary values ​​under other quantization methods; and determining the transmission data stream based on the coded data stream obtained by entropy coding includes: determining the transmission data stream of the target video frame based on the coded data stream obtained by entropy coding and the feature element positions of each of the screened feature elements.

[0185] Among them, the preset screening condition is: the scale parameter mapping value of the screened feature element is close to the rounding boundary value of the scale parameter mapping value of the screened feature element, and the distance between the scale parameter mapping value and the rounding boundary value is less than or equal to the preset threshold value. That is, the feature elements screened by the screening condition are feature elements whose scale parameter mapping values ​​after the scale parameter value is mapped are close to the rounding boundary value. These feature elements may jump during the decoding process.

[0186] The preset quantization method and other quantization methods use different rounding methods for the same scale parameter mapping value. Therefore, when the preset quantization method and other quantization methods are used to quantize the same scale parameter value, the quantization results obtained are different. Specifically, the rounding boundary values ​​of the rounding method under the other quantization method and the rounding boundary values ​​of the rounding method under the preset quantization method are different. Therefore, under the preset quantization method, due to the scale parameter mapping value being close to the rounding boundary value, the decoding process may produce characteristic elements that jump. When quantization is performed using other quantization methods, the jump phenomenon can be avoided, ensuring that the encoding process and the decoding process can obtain consistent quantization results.

[0187] Optionally, the rounding method under the preset quantization method is rounding up, and the rounding method under other quantization methods is rounding to the nearest integer; further optionally, the rounding method under the preset quantization method is rounding down, and the rounding method under other quantization methods is rounding to the nearest integer; further optionally, the rounding method under the preset quantization method is rounding to the nearest integer, and the rounding method under other quantization methods is rounding up; further optionally, the rounding method under the preset quantization method is rounding to the nearest integer, and the rounding method under other quantization methods is rounding down. Among them, rounding up refers to rounding up to the nearest integer, for example, rounding 2.3 up to 3, rounding down refers to rounding down to the nearest integer, for example, rounding 2.3 up to 2, and rounding to the nearest integer refers to rounding up to the nearest integer, for example, rounding 2.3 to the nearest integer.

[0188] Optionally, the rounding method under the preset quantization method is any one of rounding up, rounding down, or rounding to the nearest integer, and the rounding method under other quantization methods is to take a fixed integer value. Since the filtered feature elements usually only account for a small part of all feature elements in the feature map, setting a fixed integer as the scale parameter quantization value of the filtered feature elements has little effect on the data compression process, but it can save quantization calculation process to a certain extent and improve quantization efficiency.

[0189] It can be understood that in this embodiment, since a part of the characteristic elements are screened out and quantized using other quantization methods with different preset quantization methods, the characteristic element positions of these characteristic elements also need to be transmitted to the decoding end, so that the decoding end uses the same other quantization methods as the encoding end to quantize the positions of these characteristic elements. Therefore, after obtaining the encoded data stream, the server can also encode the position information of these screened characteristic elements into a calibration information byte stream, and package the encoded data stream and the calibration information byte stream together to obtain a transmission data stream for transmission to the decoding end. For example, the feature map can be stretched into a one-dimensional vector, in which each value represents a position coordinate in the feature element. For the feature elements for which position information needs to be transmitted, the value representing the position coordinate of the feature element in the one-dimensional vector is encoded to obtain a calibration byte stream. For example, assuming that the size of the feature map is 3*3*1, the numbers 0-8 can be used to represent the position information of each feature element in the feature map.

[0190] In the above embodiment, on the one hand, since the characteristic elements whose scale parameter mapping values ​​are close to the rounding boundary value can be screened out during the encoding process, other quantization methods can be used for special processing, and the calibration byte stream is transmitted to the decoding end so that the decoding end can perform the same special processing on the characteristic elements, the consistency of the quantization results obtained in the encoding and decoding process across platforms is ensured, and the accuracy of the decoded and reconstructed image is improved. On the other hand, since the scale parameter value output by the video codec model is constrained during the training process, the scale parameter value output by the video codec model during the encoding process will be close to the rounding boundary value with a smaller probability after mapping, thereby ensuring that fewer characteristic elements are screened out, and thus the accuracy of the decoded and reconstructed image can be improved with less consumption.

[0191] In some embodiments, characteristic elements that meet preset screening conditions are screened from each characteristic element of the characteristic graph, including: for each characteristic element of the characteristic graph, the scale parameter value mapping of the characteristic element is increased by a preset threshold value, and rounded according to the rounding boundary value under the preset quantization method to obtain a floating quantization upper limit value of the characteristic element; the scale parameter value mapping of the characteristic element is reduced by a preset threshold value, and rounded according to the rounding boundary value under the preset quantization method to obtain a floating quantization lower limit value of the characteristic element; according to the floating quantization upper limit value and the floating quantization lower limit value of each characteristic element, the characteristic elements that meet the preset screening conditions are screened from each characteristic element of the characteristic graph.

[0192] The floating quantization upper limit of a feature element is a value obtained by rounding the value of the scale parameter mapping value of the feature element by adding a preset threshold value to the value of the feature element according to the rounding method under the preset quantization method, which can represent the maximum quantization result that can be obtained for the feature element during the decoding process; the floating quantization lower limit of a feature element is a value obtained by rounding the value of the scale parameter mapping value of the feature element by reducing the preset threshold value according to the rounding method under the preset quantization method, which can represent the minimum quantization result that can be obtained for the feature element during the decoding process. Assuming the preset precision is ε, the floating quantization upper limit of the feature element is I+ε, the floating quantization lower limit of the feature element is I-ε, the floating quantization upper limit is an integer obtained by rounding I+ε according to the preset rounding method, that is, Q(I+ε), and the floating quantization lower limit is an integer obtained by rounding I-ε according to the preset rounding method, Q(I-ε), where Q represents the rounding function. The floating quantization upper limit value and the floating quantization lower limit value can represent the maximum and minimum values ​​of the estimated quantization values ​​obtained by quantizing the scale parameter value of a certain feature element in a preset quantization method during the decoding process. If a certain feature element meets the preset screening conditions, that is, after the feature element is mapped to the scale parameter mapping value according to the preset mapping relationship in the preset quantization method, the distance between the feature element and the rounding boundary value adjacent to the scale parameter mapping value is less than or equal to the preset threshold, then the maximum and minimum values ​​are definitely not equal, and at least one of them is not equal to the value directly quantized from the scale parameter mapping value of the feature element. Therefore, the server can determine whether the feature element meets the preset screening conditions based on the floating quantization upper limit value and the floating quantization lower limit value of the feature element.

[0193] In the above embodiment, by obtaining the floating quantization upper limit and the floating quantization lower limit of the scale parameter mapping value, the feature elements can be screened according to the floating quantization upper limit and the floating quantization lower limit. The screening judgment process is relatively simple, and the screening efficiency is improved.

[0194] In some embodiments, based on the floating quantization upper limit value and the floating quantization lower limit value of each feature element, feature elements that meet preset screening conditions are screened from the feature elements of the feature map, including: for each feature element, determining the distance between the floating quantization upper limit value and the floating quantization lower limit value of the feature element; when the distance is greater than zero, screening the feature element from the feature map.

[0195] Specifically, for each characteristic element, if the distance between the floating quantization upper limit value and the floating quantization lower limit value of the characteristic element is greater than 0, that is, |(Q(I+ε)-Q(I-ε))|>0, the maximum and minimum values ​​of the estimated quantization values ​​obtained by quantizing the scale parameter value of the characteristic element by the preset quantization method are not equal, then the quantization result obtained in the decoding process may contain a value that is inconsistent with the quantization result obtained by quantizing the scale parameter value itself by the preset quantization method, that is, there is a problem of inconsistent quantization results between the encoding process and the decoding process. The server can filter out the characteristic element and perform special quantization according to other quantization methods.

[0196] It can be understood that if |(Q(I+ε)-Q(I-ε))|=0 is calculated for a certain characteristic element, it means that the maximum and minimum values ​​of the estimated quantization values ​​obtained by quantizing the scale parameter value of the characteristic element in a preset quantization method are equal. In this case, it means that the quantization result obtained by the decoding process is unique, that is, the quantization results of the encoding process and the decoding process are consistent, so there is no need to screen the characteristic element.

[0197] Since it is possible to judge whether a characteristic element meets a preset screening condition according to the distance between the floating quantization upper limit value and the floating quantization lower limit value, the screening efficiency is improved.

[0198] In some embodiments, based on the floating quantization upper limit value and the floating quantization lower limit value of each feature element, feature elements that meet preset screening conditions are screened from the feature elements of the feature map, including: for each feature element, obtaining a first distance between the floating quantization upper limit value of the feature element and a reference quantization value corresponding to the feature element; the reference quantization value corresponding to the feature element is obtained by quantizing the scale parameter value of the feature element through a preset quantization method; obtaining a second distance between the floating quantization lower limit value of the feature element and the reference quantization value; and screening the feature element from the feature map when either the first distance or the second distance is greater than zero.

[0199] Specifically, a first distance between the floating quantization upper limit and the scale parameter quantization value corresponding to the feature element, namely, |(Q(I+ε)-Q(I))|, is calculated. If the first distance calculated for a feature element is greater than zero, it indicates that the maximum estimated quantization value obtained after the scale parameter mapping value of the feature element is positively floated at the decoding end is inconsistent with the quantization result obtained by quantizing the scale parameter mapping value of the feature element itself using the preset quantization method, thereby potentially causing a jump at the decoding end. The server can determine that the feature element meets the preset screening condition. A second distance between the floating quantization lower limit and the scale parameter quantization value corresponding to the feature element, namely, |(Q(I-ε)-Q(I))|, is calculated. If the second distance is greater than zero, it indicates that the minimum estimated quantization value obtained after the scale parameter mapping value of the feature element is negatively floated at the decoding end is inconsistent with the quantization result obtained by quantizing the scale parameter mapping value of the feature element itself using the preset quantization method, thereby potentially causing a jump at the decoding end. The server can determine that the feature element meets the preset screening condition.

[0200] It is understandable that if the first distance is equal to zero and the second distance is equal to zero, the server may determine that the feature element does not meet the preset screening condition.

[0201] Since a first distance between the upper limit value of the floating quantization and the quantized value of the scale parameter corresponding to the feature element can be obtained, and a second distance between the lower limit value of the floating quantization and the quantized value of the scale parameter corresponding to the feature element can be obtained, if either the first distance or the second distance corresponding to a certain feature element is greater than zero, the feature element is screened out. Since the first distance and the second distance are calculated simultaneously, the accuracy of the screening process can be improved.

[0202] In some embodiments, as shown in FIG5 , the method is executed by a decoding end, and the encoding end may be the server 104 or the terminal 102 in FIG1 . In the embodiment of the present application, the method is described by taking the terminal in FIG1 as an example, and includes the following steps:

[0203] Step 502: Acquire a transmission data stream, and obtain a coded data stream from the transmission data stream. The coded data stream is obtained by encoding a feature map of a target video frame.

[0204] Step 504: Determine the scale parameter value corresponding to each feature element in the feature map using the trained video encoding and decoding model.

[0205] The video codec model is trained using the video codec model processing method of any of the above embodiments. After the video codec model is trained, the output scale parameter value is mapped according to a preset mapping relationship, and the scale parameter mapping value obtained is offset in a direction away from the adjacent rounding boundary value.

[0206] Step 506 : Map the scale parameter values ​​corresponding to the characteristic elements according to the preset mapping relationship to obtain scale parameter mapping values, where the scale parameter mapping values ​​are within the preset mapping value range.

[0207] Step 508 : rounding the scale parameter mapping value obtained by mapping according to its rounding boundary value to perform quantization, thereby obtaining the scale parameter quantization value of the feature element.

[0208] Step 510 : Entropy decoding is performed on the coded data stream according to the quantized values ​​of the scale parameters of the feature elements of the feature map, and the target video frame is reconstructed based on the feature map restored by entropy decoding.

[0209] Specifically, the terminal can redetermine the scale parameter value based on the quantization of each scale parameter value, determine the probability distribution function required for arithmetic coding according to the redetermined scale parameter value, and then perform arithmetic decoding according to the probability distribution function, and reconstruct the target video frame based on the feature map restored by entropy decoding.

[0210] In a specific application, the terminal can construct a probability distribution function lookup table between the scale parameter quantization value and the scale parameter value through the above formula (12), so that after obtaining the scale parameter quantization value, the terminal can look up the probability distribution function lookup table to obtain the scale parameter value θ, and determine the probability distribution function required for arithmetic coding based on θ.

[0211] In some embodiments, the transmission data stream also includes a calibration information byte stream, which is used to determine the positions of multiple feature elements; the scale parameter mapping values ​​obtained by mapping are rounded according to their rounding boundary values ​​for quantization to obtain the scale parameter quantization values ​​of the feature elements, including: from the scale parameter mapping values ​​of each feature element in the feature graph, the scale parameter values ​​of the feature elements at the determined feature element positions are screened; the scale parameter mapping values ​​of each feature element in the feature graph are rounded according to their respective rounding boundary values ​​for quantization to obtain the scale parameter quantization values ​​of each feature element, wherein the scale parameter mapping values ​​that are not screened out are rounded according to their rounding boundary values ​​under a preset quantization method, and the screened scale parameter mapping values ​​are rounded according to their rounding boundary values ​​under other quantization methods.

[0212] In the above-mentioned video decoding method, since the video codec model is trained using the video codec model processing method of any of the above embodiments, the scale parameter value output by the video codec model is mapped according to a preset mapping relationship, and the scale parameter mapping value obtained is offset in the direction away from the adjacent rounding boundary value. Therefore, when the target video frame is encoded using this video codec model, the scale parameter value in the decoding process, after mapping, the scale parameter mapping value obtained will be as far away from the rounding boundary value as possible, thereby reducing the probability of rounding jumps occurring during the quantization process, so that the decoding process can perform entropy decoding based on the same quantization result as the encoding process, thereby improving the accuracy of the video frame reconstructed by the decoding process.

[0213] It can be understood that the encoding end in the embodiment of the present application refers to the first device that can perform video encoding, and the decoding end refers to the second device that can perform video decoding. The encoding end and the decoding end are not used to limit the first device and the second device. When the first device is used for video decoding, the first device can be used as a decoding end. When the second device is used for video encoding, the second device can be used as an encoding end.

[0214] In some specific embodiments, the present application also provides an application scenario, in which the video encoding method and video decoding method of the present application are used to implement cross-platform video encoding and decoding, wherein the terminal can be installed with a video application, and the terminal can play video through the video application. The rounding method set by the preset quantization method is rounding down, and each integer value is a rounding boundary value. The constraint reference mapping value corresponding to the size parameter mapping value is the center value of each mapping value interval, that is, the average value of each adjacent two integers.

[0215] Referring to Figure 6, the server acts as an encoding end to encode each video frame in the original video to obtain a transmission data stream and then sends it to the terminal. The terminal acts as a decoding end to decode the transmission data stream to obtain a reconstructed video frame, so that the video can be played based on the reconstructed video frame.

[0216] Continuing to refer to Figure 6, in this embodiment, the encoding at the encoding end and the decoding at the decoding end are both implemented based on the video coding model. The video coding model is an artificial intelligence (AI) model and can be implemented through a neural network. In the encoding and decoding process of the video coding and decoding model, it mainly includes two links, namely I frame encoding and decoding (intra-frame coding) and P frame encoding and decoding (inter-frame coding). Usually, I frame encoding and decoding is implemented using the AI ​​image coding and decoding algorithm, and the P frame encoding and decoding model needs to be designed according to the characteristics of inter-frame coding, and is usually divided into two modules: motion estimation and residual compensation. The core idea of ​​video coding is to convert the original image into some feature maps that need to be transmitted, and reduce the amount of bytes transmitted by the feature maps through entropy coding, thereby greatly reducing the byte size of the video transmission.

[0217] During encoding, the original image is converted into a feature map for transmission in the I-frame model. For the P-frame model, the original image is typically converted into a motion-estimated feature map and a residual-compensated feature map for transmission. During decoding, the I-frame model receives the feature map and reconstructs the I-frame image. The P-frame model receives the motion-estimated feature map and reconstructs the motion estimate. This is applied to the reference I-frame reconstructed image to obtain the P-frame intermediate result of the motion estimate. Finally, the residual-compensated feature map is used to reconstruct the residual compensation information and applied to the P-frame intermediate result to obtain the reconstructed P-frame image.

[0218] Entropy coding is a common method for transmitting feature maps. Entropy coding is a common data compression technique and a crucial component of video encoding and decoding. In video encoding, entropy coding is often used to compress residual data, motion vectors, and other encoding parameters in the video encoder, reducing storage space and transmission bandwidth.

[0219] The purpose of the entropy coding estimation module is to estimate the number of bits required for entropy coding based on the input coded data stream. This module is usually implemented based on a statistical model. It analyzes and models the coded data stream in order to minimize the number of bits required during the entropy coding process. Common entropy coding algorithms include Huffman coding, arithmetic coding, etc. Taking arithmetic coding as an example, during the entropy coding process, arithmetic coding calculations need to be performed on each feature element (which can be understood as each value in the feature map). In order to achieve a higher compression rate, the entropy coding process often introduces a high-precision probability estimation function. In order for the decoding end to be able to completely and correctly decode the corresponding element from the byte stream encoded by the encoding end, it is necessary to use a probability estimation function that is exactly the same as that of the encoding end.

[0220] During codec calculations, single-precision floating-point numbers are often used. When the codec runs on the same machine and in the same computing environment, it's easy to ensure that the codec uses consistent probability estimation functions or that calculation errors are within the codec's tolerable range. However, when the codec runs on different machines or in different computing environments, single-precision float calculations performed under different conditions may have significant precision errors, resulting in low image accuracy at the decoder and causing decoding failure. An example of a decoding failure is shown in Figure 7, where the mosaic indicates the pixel where decoding failed.

[0221] To address the problem of decoding failures caused by precision errors due to cross-platform calculations on the codec side, related technologies require converting all submodules of the video codec model from nondeterministic single-precision floating-point calculations to deterministic integer calculations. This conversion process must follow certain rules and requires some alignment training. This conversion process can lead to a loss of precision in the video codec model, resulting in reduced performance.

[0222] In this embodiment, for a pre-trained video codec model, while freezing all model parameters except the entropy model parameters, a piecewise Gaussian function constraint is applied to the output of the entropy model to fine-tune the model parameters of the entropy model, thereby obtaining a trained video coding model. During encoding using the trained video coding model, the encoder selects feature elements close to integer boundaries and then transmits additional redundant byte streams to represent the positions of these feature elements. When decoding using the trained video coding model, the decoder processes non-deterministic elements that may have errors with the encoder based on the received redundant information, consistent with the encoder, thereby avoiding inconsistent calculation results caused by calculation accuracy errors at the encoder and decoder, thereby aligning the calculation results at the encoder and decoder. Due to the piecewise Gaussian constraint, the output value of the entropy model can be mapped to an integer boundary with a relatively low probability during the encoding process. Therefore, only redundant information needs to be transmitted for a small number of feature elements, thereby achieving cross-platform encoding and decoding with minimal overhead at the encoder. At the decoding end, the output value of the entropy model can also be close to the integer boundary with a relatively small probability after mapping. Therefore, during the decoding process, not only can the encoding and decoding consistency of the feature elements with transmitted position information be guaranteed, but also the feature elements of other non-transmitted positions can be guaranteed to avoid jumps as much as possible, thereby greatly improving the accuracy of the image obtained by decoding and reconstruction.

[0223] The video encoding and decoding methods implemented by the video encoding and decoding models trained by the embodiments of this application can be provided as a software interface to solve the computational inconsistency problem that exists in cross-platform scenarios on the encoding and decoding side, thereby enabling the decoding side to decode and obtain the correct decoded video frames. The specific implementation of this application is introduced below:

[0224] 1. Training video codec model

[0225] Referring to Figure 8a, the essence of the entropy model is to evaluate the difference between the probability distribution q predicted by the model and the true distribution p of the feature map y. When the two are closer, the feature map y can be compressed with smaller bytes. When the two are farther apart, more bytes are required to compress the feature map y. In the related art, when training a video codec model, the loss of the training video codec model is the loss mentioned above. The trained video codec model does not impose any constraints on the scale parameter value σ output by the entropy model. Therefore, the scale parameter mapping value I calculated based on the scale parameter value σ will randomly approach the integer boundary, and the probability of approaching the integer boundary will increase as the calculation error between platforms increases.

[0226] Referring to Figure 8b, after pre-training a video codec model with normal encoding and decoding capabilities, the present application can freeze other model parameters except the model parameters of the entropy model in the video codec model, and then impose additional constraints based on the piecewise Gaussian function on the scale parameter mapping value after the scale parameter value output by the entropy model is mapped, thereby indirectly constraining the scale parameter value output by the entropy model. Specifically, first calculate the quantization constraint loss The specific calculation steps are as follows:

[0227] 1. Obtain a training video frame, extract a feature map of the training video frame using the feature extraction model in the video codec model, and determine the scale parameter value corresponding to each feature element in the feature map using the entropy model in the video codec model.

[0228] 2. According to the preset mapping relationship, each characteristic element is mapped to a preset mapping value range to obtain the scale parameter mapping value corresponding to each characteristic element.

[0229] The preset mapping relationship may refer to (1) and (2) above.

[0230] 3. Obtain the center value of the interval of the mapping value interval to which each scale parameter mapping value belongs, as the constraint reference mapping value of each scale parameter mapping value.

[0231] 4. Get the preset Gaussian distribution function.

[0232] Here, the formula of the preset Gaussian distribution function is the above formula (3), and the output of the Gaussian distribution function is the expected probability value. The Gaussian distribution function is a piecewise function. Within each integer interval, it is expressed as a mean with the center of the interval as the mean and 1 as the standard deviation.

[0233] 5. Input each scale parameter mapping value into a preset Gaussian distribution function to calculate the expected probability value of each scale parameter mapping value, and input the constraint reference mapping value corresponding to each scale parameter mapping value into the preset Gaussian distribution function to obtain the expected probability value of each constraint reference mapping value.

[0234] 6. Determine the quantized constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the expected probability value of each scale parameter mapping value and the expected probability value of the corresponding constraint reference mapping value.

[0235] 7. Determine the loss weight corresponding to each scale parameter mapping value.

[0236] Specifically, the attribution relationship between the scale parameter mapping value and the preset numerical range can be determined with reference to the above formula (5), and the loss weight can be determined according to the attribution relationship.

[0237] 8. Determine the quantization constrained loss for the training video frame based on the quantization constrained sub-loss and loss weight corresponding to each scale parameter mapping value.

[0238] The specific calculation process here can refer to the above formula (4).

[0239] Furthermore, the reconstruction distortion loss and bit rate loss can be calculated according to formulas (6) to (9), and then the total loss can be calculated according to formulas (10) to (11). The model parameters of the entropy model are updated according to the total loss to obtain the trained video codec model. The trained video codec model can be deployed on the encoder side to implement video encoding, and deployed on the decoder side to implement video decoding.

[0240] 2. Video Encoding and Video Decoding

[0241] First, the encoding and decoding model of the present application is introduced: referring to Figure 9, for the data to be encoded of the target video frame (which may be a video frame, an image, motion estimation data, residual estimation data, etc.), after pre-encoding by the encoding module, feature extraction can be performed on the data to be encoded to perform the first stage of compression to obtain a feature map; after the decoding end obtains the restored feature map, the target video frame can be reconstructed after passing through the decoding module.

[0242] As for how to transmit the feature map, it is necessary to introduce the second stage of encoding, that is, the arithmetic coding module. Referring to Figure 10, the feature map y obtained in the video coding and decoding model is the feature map to be encoded. The coded feature map can be encoded to obtain the super prior z through the super prior coding module, and the super prior will also be transmitted to the decoding end in a certain way. At the encoding end, after obtaining the super prior z, it is necessary to perform super prior decoding on z (that is, the super prior decoding in Figure 10), and the (μ, σ) of each feature element in the feature map y to be encoded will be obtained. (μ, σ) can be understood as a probability estimate of y, which helps to perform higher compression in the subsequent entropy coding link. Among them, μ refers to the scale parameter value of the probability distribution of the feature element, and σ refers to the position parameter value of the probability distribution of the feature element. Then σ is quantized into the scale parameter quantization value. Last passed The re-determined scale parameter value θ is obtained by searching the probability distribution function lookup table. The probability distribution function required for arithmetic coding can be determined based on θ. The probability value of the feature element can be determined through the probability distribution function, so that the information compression of the feature map to be encoded can be achieved through arithmetic coding to obtain the encoded byte stream (that is, the encoded data stream mentioned above).

[0243] In the entropy coding module, for each feature element, the scale parameter value σ i , the scale parameter mapping value I can be obtained by mapping according to the preset mapping relationship of formulas (1) and (2), and then the scale parameter quantization value can be obtained by rounding according to the following formula (13). Wherein, Q represents a floor quantization function. In this embodiment, for floor rounding, each integer can be used as a rounding boundary value, so floor rounding means rounding the nearest integer.

[0244] use The corresponding θ can be found from the probability distribution function lookup table i value, using θ i By determining the probability distribution function required for arithmetic coding, y can be compressed into as few byte streams as possible, that is, the encoded byte stream (i.e., the encoded data stream mentioned above) is obtained to facilitate subsequent transmission. The probability distribution function lookup table can be constructed using the above formula (12).

[0245] Referring to FIG11 , in the encoding process at the encoding end, in the related art, as shown in the left block diagram in FIG11 , the entropy model estimates the probability distribution based on the super prior and conditional information and quantizes the scale parameter value, and can obtain (μ e ,θ e ), in the encoding process at the decoding end, the entropy model estimates the probability distribution based on the super prior and conditional information and quantizes the scale parameter value to obtain (μd ,θ d ), and obtain θ e and θ d Due to the cross-platform problem, the calculation inconsistency is caused. The subscripts e and d represent the encoder and decoder respectively. The conditional information can be the reference information of the previous frame, for example, the feature map of the previous frame. Referring to the right side of Figure 11, this application improves the process of obtaining θ by quantization and introduces calibration information C b , the codec side can calculate consistent (μ′, θ′), so that the same codec parameters can be used in arithmetic coding, and finally the correct reconstruction result can be obtained.

[0246] Specifically, when the scale parameter value of the feature element is calculated at the encoding end with reference to the above formula (1) and formula (2), the scale parameter mapping value I is obtained. e After that, the scale parameter quantization value can be calculated using formula (14), formula (15) and formula (16): Finally, we get θ by looking up the probability distribution function table. e , as shown below: C b ={(x, y, z)|(|(Q(I e +ε)-Q(I e -ε))|>0)} (14)

[0247] Wherein, ε is the precision parameter, which is also the preset threshold in the preset screening condition above. ε can be set as needed, for example, it can be set to 1e-4. Q is the quantization function in formula (13), and Q in formula (15) is D In this embodiment, for the nearest integer, the average value of every two adjacent integers can be used as the rounding boundary value. Therefore, the nearest integer here is to round the quantized value of the scale parameter of the input feature element to the nearest integer. The above formulas (14) to (16) can be simply summarized as follows: by presetting the screening condition |(Q(I e +ε)-Q(I e -ε))|>0 for all characteristic elements I eA judgment is made to filter out the feature elements that meet the preset screening conditions, that is, the scale parameter mapping value is close to the integer boundary, and the position information of these feature elements is passed to the decoding end as calibration information. In addition, for these feature elements that meet the preset screening conditions, rounding is used to replace the rounding down set by the preset quantization method, so as to ensure that these elements are calculated with consistent values ​​at the codec end; for feature elements that do not meet the preset screening conditions, the original rounding down is still used, which can also ensure the consistency of these feature elements calculated at the codec end. In this way, the codec end can obtain completely consistent θ, thereby solving the decoding error problem at the decoding end and improving the accuracy of the image reconstructed at the decoding end. However, since the output value of the entropy model is constrained to be away from the integer boundary during the training process, the scale parameter quantization value obtained by the entropy model during the encoding process will shift in the direction away from the integer boundary under the adjacent preset quantization method, and the filtered I e The number of values ​​will be relatively small, that is, only the position information of a small amount of characteristic elements needs to be transmitted to achieve consistency in the calculation at the codec end.

[0248] Refer to Figure 12, which is a flowchart of the decoding end. The super-prior decoding at the decoding end is consistent with the super-prior decoding at the encoding part, and will not be repeated here. When the decoding end obtains the transmission data stream, it can obtain the super-prior z (i.e., the auxiliary coding information mentioned above), the encoded data stream, and the calibration byte stream from the transmission data stream. The decoding end can then perform super-prior decoding on z, and will obtain the scale parameter value σ and position parameter value μ corresponding to each feature element in the feature map to be encoded. The decoding end can further quantize the σ obtained by super-prior decoding to obtain a quantized value Then, the probability distribution parameter θ is obtained by looking up the probability distribution function table. Through θ, the feature map y_hat can be decoded from the byte stream encoded by the encoding end, and the final reconstructed image can be obtained by subsequent decoding. In the process, first refer to the above formulas and obtain the scale parameter mapping value I according to formulas (1) and (2), then determine the characteristic elements of the position information transmitted by the encoding end according to the calibration information byte stream, and use formula (13) to round the I of the characteristic elements of the position information transmitted by the encoding end, so as to obtain the quantization result consistent with the encoding end. For the I of other characteristic elements, the original rounding method, i.e., formula (15), is used for rounding. Since the output value of the entropy model at the decoding end can also be close to the integer boundary with a relatively small probability after mapping, the characteristic elements of other non-transmitted positions can also be guaranteed to not jump as much as possible, thereby greatly improving the accuracy of the image obtained by decoding and reconstruction.

[0249] For example, referring to Figure 13, when calibration information transmission is not used (as shown on the left side of Figure 13), the encoding and decoding ends have slight differences in σ due to platform differences, resulting in different values ​​in subsequent calculations. For example, for the example on the left side of Figure 13, the encoding end obtains is 1, the decoder gets =2, which results in inconsistency between θ used for arithmetic coding and decoding, resulting in decoding errors and ultimately an erroneous reconstructed image. e The preset screening condition shown in formula (14) is judged to determine the characteristic elements that may produce integer edge jumps, and the position information representing these characteristic elements is transmitted to the decoding end through the calibration information byte stream. For example, the position information of the characteristic element at the coordinate (3,5) is transmitted to the decoding end through the calibration information byte stream. The codec end will perform special processing on the characteristic element at the coordinate (3,5) according to the quantization method shown in formula (15), thereby obtaining a completely consistent codec end. (As shown in the right side of Figure 13, the encoding and decoding are consistent 2), and finally the decoding end will be able to correctly decode and obtain the reconstructed image.

[0250] The embodiment of the present application experiments on the encoding and decoding process under different calculation precision errors. As shown in Figure 14, wPGC indicates that the solution provided by the embodiment of the present application is adopted, that is, piecewise Gaussian constraint is adopted for the entropy model and the calibration information byte stream is transmitted. The horizontal axis in Figure 14 is BPP (bit per pixel), and the vertical axis is PSNR (Peak Signal to Noise Ratio). The 1e-4, 1e-3, and 1e-4 in the brackets represent different calculation precision errors. It can be seen from the curve in Figure 14 that under different calculation precision errors, the PSNR is significantly improved after adopting the solution of the present application compared with the H.265 encoder and the H.264 encoder.

[0251] As shown in Figure 15, the embodiment of the present application further compares the solution of only transmitting the calibration information byte stream (without Gaussian constraint on the entropy model). It can be seen from Figure 15 that under different precision errors, after adopting the present application, the model indicators (BD-rate) can mostly be improved, mainly due to the reduction of the byte consumption of additional transmission; and from the statistical analysis of the coordinate quantity (Quantity of CIT) transmitted by the calibration information, it can be seen that under different calculation precision errors, after adopting the present application, the calibration coordinate quantity can be reduced by an average of 28%.

[0252] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0253] Based on the same inventive concept, the embodiments of the present application also provide a video codec model processing device for implementing the above-mentioned video codec model processing method, a video encoding device for the above-mentioned video encoding method, and a video decoding device for the above-mentioned video decoding method. The implementation solutions provided by these devices are similar to the implementation solutions described in the above-mentioned methods. Therefore, the specific limitations in the embodiments of one or more codec model processing devices, video encoding devices, and video decoding devices provided below can be found in the above-mentioned limitations on the codec model processing method, and will not be repeated here.

[0254] In some embodiments, as shown in FIG16 , a video coding model processing apparatus 1600 is provided, including:

[0255] The video frame acquisition module 1602 is used to acquire a training video frame, extract a feature map of the training video frame through a video codec model, and determine the scale parameter value corresponding to each feature element in the feature map;

[0256] The value mapping module 1604 is configured to map the scale parameter value corresponding to each of the feature elements according to a preset mapping relationship to obtain a scale parameter mapping value, wherein the scale parameter mapping value is within a preset mapping value range.

[0257] A loss determination module 1608 is configured to obtain a constraint reference mapping value corresponding to each scale parameter mapping value, and determine a quantization constraint loss for the training video frame based on a difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value, wherein the distance between the constraint reference mapping value and the rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is greater than the distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value;

[0258] The parameter updating module is used to update at least a portion of the model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0259] The above-mentioned video codec model processing device, since the constraint reference mapping value is farther away from the rounding boundary value than the corresponding scale parameter mapping value, the quantization constraint loss determined based on the difference between the scale parameter mapping value and the corresponding constraint reference mapping value can reflect the degree of distance of the scale parameter mapping value from the rounding boundary value. When the parameters of the video codec model are updated based on the quantization constraint loss, the scale parameter mapping value corresponding to the scale parameter value output by the video codec model can be constrained, thereby indirectly realizing the constraint on the scale parameter value output by the video codec model, so that the scale parameter value output by the video codec model is offset in the direction away from the rounding boundary value after mapping, that is, the probability of the scale parameter mapping value approaching the rounding boundary value is reduced, which alleviates the problem of inconsistent quantization results obtained in the encoding process and the decoding process in the cross-platform coding and decoding scenario, and thus can improve the accuracy of the video frame reconstructed by the decoding process.

[0260] In some embodiments, the loss determination module is further used to determine the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between the scale parameter mapping value corresponding to each feature element and the constraint reference mapping value corresponding to each scale parameter mapping value; determine the loss weight corresponding to each scale parameter mapping value; and determine the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value.

[0261] In some embodiments, the loss determination module is further used to determine the relationship between each scale parameter mapping value and a preset numerical range; the scale parameter mapping value belonging to the preset numerical range and the scale parameter mapping value obtained by encoding and decoding the corresponding feature element have a consistent rounding result; according to the relationship between each scale parameter mapping value and the preset numerical range, the loss weight corresponding to each scale parameter mapping value is determined; based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter mapping value, the quantization constraint loss of the training video frame is determined.

[0262] In some embodiments, the loss determination module is further used to determine the corresponding loss weight as a first weight for the scale parameter mapping value belonging to the preset numerical interval; and to determine the corresponding loss weight as a second weight for the scale parameter mapping value not belonging to the preset numerical interval; the second weight is greater than the first weight.

[0263] In some embodiments, the distance between the scale parameter mapping value belonging to the preset numerical interval and the corresponding constraint reference mapping value is less than a preset threshold.

[0264] In some embodiments, the distance between the scale parameter mapping value belonging to the preset numerical interval and the lower limit value of the preset mapping value range is less than a preset threshold.

[0265] In some embodiments, the loss determination module is further used to obtain a preset probability distribution function, which is used to input a scale parameter mapping value within a preset mapping value range and output an expected probability value of the input scale parameter mapping value; through the probability distribution function, the expected probability value of each scale parameter mapping value and the expected probability value of the constraint reference mapping value corresponding to each scale parameter mapping value are determined; based on the difference between the expected probability value of each scale parameter mapping value and the expected probability value of the corresponding constraint reference mapping value, the quantization constraint sub-loss corresponding to each scale parameter mapping value is determined; and according to the quantization constraint sub-loss corresponding to each scale parameter mapping value, the quantization constraint loss of the training video frame is determined.

[0266] In some embodiments, within the range from the constrained reference mapping value to its nearest rounded boundary value in a preset mapping value range, the closer the scale parameter mapping value of the input probability distribution function is to the constrained reference mapping value, the greater the expected probability value output by the probability distribution function.

[0267] In some embodiments, the loss determination module is further used to determine the difference between each scale parameter mapping value and the corresponding constraint reference mapping value; determine the quantization constraint sub-loss corresponding to each scale parameter mapping value based on the difference between each scale parameter mapping value and the corresponding constraint reference mapping value; and determine the quantization constraint loss of the training video frame based on the quantization constraint sub-loss corresponding to each scale parameter mapping value.

[0268] In some embodiments, at least a portion of the model parameters of the video codec model are updated based on quantization constraint loss, bitrate loss and reconstruction distortion loss. The bitrate loss and reconstruction distortion loss are calculated through a loss calculation step. The loss determination module is further used to determine the probability value of each feature element based on the scale parameter value and position parameter value of the probability distribution of each feature element; the position parameter value of each feature element is obtained by predicting the probability distribution of each feature element; the probability value of the training video frame is determined based on the probability value of each feature element, and the probability value of the training video frame is cross-entropy calculated to obtain the bitrate loss; the reconstructed video frame of the training video frame is determined, and the reconstruction distortion loss is determined based on the difference between the training video frame and the reconstructed video frame of the training video frame.

[0269] In some embodiments, as shown in FIG17 , another video coding model processing apparatus 1700 is provided, including:

[0270] The video frame acquisition module 1702 is used to acquire a training video frame, extract a feature map of the training video frame through a video codec model, and determine the scale parameter value corresponding to each feature element in the feature map;

[0271] A loss determination module 1704 is configured to obtain a constraint reference value corresponding to each scale parameter value, and determine a quantization constraint loss for the training video frame based on a difference between each scale parameter value and the corresponding constraint reference value, wherein the constraint reference value is further from a rounding boundary value of a scale parameter mapping value obtained by mapping the corresponding scale parameter value to the corresponding scale parameter value after being mapped to a preset mapping value range according to a preset mapping relationship than the scale parameter value corresponding to the constraint reference value.

[0272] The parameter updating module 1706 is configured to update at least a portion of the model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

[0273] The above-mentioned video codec model processing device, since the constraint reference value is farther away from the rounding boundary value than the corresponding scale parameter value after being mapped to the preset mapping value range according to the preset mapping relationship, the quantization constraint loss determined based on the difference between the scale parameter value and the corresponding constraint reference value can reflect the degree of distance of the scale parameter mapping value from the rounding boundary value. Therefore, when the parameters of the video codec model are updated based on the quantization constraint loss, the scale parameter value output by the video codec model can be constrained, so that the scale parameter value output by the video codec model is offset in the direction away from the rounding boundary value after mapping, that is, the probability of the scale parameter mapping value being close to the rounding boundary value is reduced, which alleviates the problem of inconsistent quantization results obtained in the encoding process and the decoding process in the cross-platform coding and decoding scenario. Therefore, the accuracy of the video frame reconstructed by the decoding process can be improved. Moreover, since the quantization constraint loss directly acts on the scale parameter value, the numerical value directly output by the video codec model can be constrained. The training process does not need to perform other additional mapping steps, and the training efficiency is high.

[0274] In some embodiments, the loss determination module is further used to determine the quantization constraint sub-loss corresponding to each scale parameter value based on the difference between each scale parameter value and the corresponding constraint reference value; determine the loss weight corresponding to each scale parameter value; and determine the quantization constraint loss of the training video frame based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter value.

[0275] In some embodiments, the loss determination module is further used to determine the relationship between each scale parameter value and a preset numerical range; the scale parameter values ​​belonging to the preset numerical range and the scale parameter values ​​obtained by encoding and decoding the corresponding feature elements have consistent quantization results; according to the relationship between each scale parameter value and the preset numerical range, the loss weight corresponding to each scale parameter value is determined; based on the quantization constraint sub-loss and loss weight corresponding to each scale parameter value, the quantization constraint loss of the training video frame is determined.

[0276] In some embodiments, the loss determination module is also used to determine the corresponding loss weight as a first weight for the scale parameter value belonging to the preset numerical range; and to determine the corresponding loss weight as a second weight for the scale parameter value not belonging to the preset numerical range; the second weight is greater than the first weight.

[0277] In some embodiments, the distance between the scale parameter value belonging to the preset numerical interval and the corresponding constraint reference value is less than a preset threshold.

[0278] In some embodiments, the distance between the scale parameter value belonging to the preset numerical interval and the lower limit value of the scale parameter value range is less than a preset threshold.

[0279] In some embodiments, the loss determination module is also used to obtain a preset probability distribution function, which is used to input a scale parameter value and output an expected probability value of the input scale parameter value; through the probability distribution function, the expected probability value of each scale parameter value and the expected probability value of the constraint reference value corresponding to each scale parameter value are determined; based on the difference between the expected probability value of each scale parameter value and the expected probability value of the corresponding constraint reference value, the quantization constraint sub-loss corresponding to each scale parameter value is determined; according to the quantization constraint sub-loss corresponding to each scale parameter value, the quantization constraint loss of the training video frame is determined.

[0280] In some embodiments, in the domain of the scale parameter value, within the range from the constraint reference value to the scale parameter value most recently used to map to the rounding boundary value, the closer the scale parameter value of the input probability distribution function is to the constraint reference value, the greater the expected probability value output by the probability distribution function.

[0281] In some embodiments, the loss determination module is further used to determine the difference between each scale parameter value and the corresponding constraint reference value; determine the quantization constraint sub-loss corresponding to each scale parameter value based on the difference between each scale parameter value and the corresponding constraint reference value; and determine the quantization constraint loss of the training video frame based on the quantization constraint sub-loss corresponding to each scale parameter value.

[0282] In some embodiments, the model parameters of the entropy model are updated based on the quantization constraint loss, bitrate loss and reconstruction distortion loss. The bitrate loss and reconstruction distortion loss are calculated through the loss calculation step. The loss determination module is also used to determine the probability value of each feature element according to the scale parameter value and position parameter value of the probability distribution of each feature element; the position parameter value of each feature element is obtained by predicting the probability distribution of each feature element; the probability value of the training video frame is determined according to the probability value of each feature element, and the probability value of the training video frame is cross-entropy calculated to obtain the bitrate loss; the reconstructed video frame of the training video frame is determined, and the reconstruction distortion loss is determined according to the difference between the training video frame and the reconstructed video frame of the training video frame.

[0283] In some embodiments, as shown in FIG18 , a video encoding apparatus 1800 is provided, including:

[0284] A video frame acquisition module 1802 is configured to acquire a target video frame, extract a feature map of the target video frame using a trained video codec model, and determine the scale parameter values ​​corresponding to each feature element in the feature map. The video codec model is trained using the video codec model processing device of any of the above embodiments.

[0285] The numerical quantization module 1804 is configured to map the scale parameter value of each feature element of the feature graph to a preset mapping value range according to a preset mapping relationship, thereby obtaining a scale parameter mapping value, and to quantize the mapped scale parameter mapping value by rounding the mapped scale parameter mapping value according to its rounding boundary value to obtain a scale parameter quantized value of the feature element.

[0286] The entropy coding module 1806 is configured to perform entropy coding on the feature map according to the quantized value of the scale parameter of each feature element of the feature map, and determine a transmission data stream according to the coded data stream obtained by the entropy coding.

[0287] In the above-mentioned video encoding device, since the video codec model is trained using the video codec model processing method of any of the above embodiments, the scale parameter values ​​output by the video codec model are mapped according to a preset mapping relationship, and the scale parameter mapping values ​​obtained are offset in a direction away from the adjacent rounding boundary values. Therefore, when the target video frame is encoded using this video codec model, the scale parameter values ​​in the encoding process are mapped to the scale parameter mapping values ​​obtained as far away from the rounding boundary values ​​as possible, thereby reducing the probability of rounding jumps occurring during the quantization process at the decoding end, allowing the encoding and decoding processes to be encoded and decoded based on the same quantization results, thereby improving the accuracy of the video frames reconstructed by the decoding process.

[0288] In some embodiments, the numerical quantization module 1804 is further configured to: screen feature elements that meet preset screening conditions from the feature elements of the feature map, where the preset screening conditions are: the distance between the scale parameter mapping value of the screened feature element and the rounding boundary value adjacent to the scale parameter mapping value of the screened feature element is less than or equal to a preset threshold; round the scale parameter mapping value of each feature element of the feature map according to its respective rounding boundary value for quantization to obtain a scale parameter quantization value of each feature element, wherein the scale parameter mapping value of the unscreened feature element is rounded according to its rounding boundary value under the preset quantization mode, and the scale parameter mapping value of the screened feature element is rounded according to its rounding boundary value under other quantization modes; and the entropy coding module is further configured to determine the transmission data stream of the target video frame based on the coded data stream obtained by entropy coding and the feature element positions of each of the screened feature elements.

[0289] In some embodiments, the numerical quantization module 1804 is further used to: for each feature element of the feature graph, increase the numerical value after the preset threshold value is mapped to the scale parameter value of the feature element, round it according to the rounding boundary value under the preset quantization method, and obtain the floating quantization upper limit value of the feature element; reduce the numerical value after the preset threshold value is mapped to the scale parameter value of the feature element, round it according to the rounding boundary value under the preset quantization method, and obtain the floating quantization lower limit value of the feature element; and filter the feature elements that meet the preset filtering conditions from the feature elements of the feature graph according to the floating quantization upper limit value and the floating quantization lower limit value of each feature element.

[0290] In some embodiments, the numerical quantization module 1804 is further configured to: determine, for each feature element, the distance between the floating quantization upper limit and the floating quantization lower limit of the feature element; and filter out the feature element from the feature map when the distance is greater than zero.

[0291] In some embodiments, as shown in FIG19 , a video decoding apparatus 1900 is provided, including:

[0292] The data stream acquisition module 1902 is used to acquire a transmission data stream and obtain a coded data stream from the transmission data stream, where the coded data stream is obtained by encoding a feature map of a target video frame;

[0293] a scale parameter prediction module 1904 for determining scale parameter values ​​corresponding to respective feature elements in a feature map using a trained video codec model, wherein the video codec model is trained using the apparatus of claim 14 or 15;

[0294] The numerical quantization module 1906 is configured to map the scale parameter value of each feature element of the feature graph to a preset mapping value range according to a preset mapping relationship, thereby obtaining a scale parameter mapping value, and to quantize the mapped scale parameter mapping value by rounding the mapped scale parameter mapping value according to its rounding boundary value to obtain a scale parameter quantized value of the feature element.

[0295] The entropy decoding module 1908 is used to perform entropy decoding on the encoded data stream according to the quantized value of the scale parameter of each feature element of the feature map, and reconstruct the target video frame based on the feature map restored by entropy decoding.

[0296] In some embodiments, the transmission data stream also includes a calibration information byte stream, which is used to determine the positions of multiple feature elements; the numerical quantization module 1906 is further used to filter the scale parameter values ​​of the feature elements at the determined feature element positions from the scale parameter mapping values ​​of each feature element in the feature graph; the scale parameter mapping values ​​of each feature element in the feature graph are rounded according to their respective rounding boundary values ​​for quantization to obtain the scale parameter quantization value of each feature element, wherein the scale parameter mapping values ​​that are not filtered out are rounded according to their rounding boundary values ​​under the preset quantization method, and the scale parameter mapping values ​​that are filtered out are rounded according to their rounding boundary values ​​under other quantization methods.

[0297] In the above-mentioned video decoding device, since the video codec model is trained using the video codec model processing method of any of the above embodiments, the scale parameter value output by the video codec model is mapped according to a preset mapping relationship, and the scale parameter mapping value obtained is offset in a direction away from the adjacent rounding boundary value. Therefore, when the target video frame is encoded using this video codec model, the scale parameter value in the decoding process, after mapping, will be as far away from the rounding boundary value as possible, thereby reducing the probability of rounding jumps occurring during the quantization process. This allows the decoding process to perform entropy decoding based on the same quantization result as the encoding process, thereby improving the accuracy of the video frame reconstructed by the decoding process.

[0298] Each module in each of the above-mentioned devices may be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0299] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 20. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device can be used to store video data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a video coding model processing method, a video coding method or a video decoding method.

[0300] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 21. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner may be achieved through WiFi, a mobile cellular network, NFC (near field communication), or other technologies. When the computer program is executed by the processor, it implements a video codec model processing method, a video encoding method, or a video decoding method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0301] Those skilled in the art will understand that the structures shown in Figures 20 and 21 are merely block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different component arrangement.

[0302] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of any of the above-mentioned video coding model processing methods are implemented.

[0303] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned video coding model processing methods are implemented.

[0304] In some embodiments, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned video coding model processing methods.

[0305] In some embodiments, another computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned video encoding method when executing the computer program.

[0306] In some embodiments, another computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video encoding method are implemented.

[0307] In some embodiments, another computer program product is provided, comprising a computer program, which implements the steps of the above video encoding method when executed by a processor.

[0308] In some embodiments, another computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned video decoding method when executing the computer program.

[0309] In some embodiments, another computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video decoding method are implemented.

[0310] In some embodiments, another computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned video decoding method when executed by a processor.

[0311] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0312] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of an attributed relational database and a non-attributed relational database. The non-attributed relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited thereto.

[0313] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0314] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A video coding model processing method, executed by a computer device, comprising: Acquire a training video frame, extract a feature map of the training video frame through a video encoding and decoding model, and determine a scale parameter value corresponding to each feature element in the feature map; According to a preset mapping relationship, the scale parameter values ​​corresponding to the characteristic elements are respectively mapped to obtain scale parameter mapping values ​​corresponding to the characteristic elements, and the scale parameter mapping values ​​are within a preset mapping value range; Obtaining a constraint reference mapping value corresponding to each of the scale parameter mapping values, wherein a distance between the constraint reference mapping value and a rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is farther than a distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value; Determining a quantization constraint loss of the training video frame based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values; and At least a portion of model parameters of the video codec model are updated based on the quantization constraint loss to obtain a trained video codec model.

2. The method according to claim 1, wherein determining the quantization constraint loss of the training video frame based on the difference between the scale parameter mapping value corresponding to each of the feature elements and the constraint reference mapping value corresponding to each of the scale parameter mapping values ​​comprises: Determining a quantized constraint sub-loss corresponding to each scale parameter mapping value based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values; Determining a loss weight corresponding to each of the scale parameter mapping values; Based on the quantization constraint sub-loss and the loss weight respectively corresponding to each of the scale parameter mapping values, the quantization constraint loss of the training video frame is determined.

3. According to the method of claim 2, the determining the loss weight corresponding to each of the scale parameter mapping values ​​comprises: Determining the attribution relationship between each of the scale parameter mapping values ​​and a preset numerical range; The scale parameter mapping value belonging to the preset numerical range and the scale parameter mapping value obtained by encoding and decoding the corresponding characteristic element have a consistent rounding result; Determining the loss weight corresponding to each scale parameter mapping value according to the attribution relationship between each scale parameter mapping value and the preset numerical range; Based on the quantization constraint sub-loss and the loss weight respectively corresponding to each of the scale parameter mapping values, the quantization constraint loss of the training video frame is determined.

4. The method according to claim 3, wherein determining the loss weight corresponding to each scale parameter mapping value according to the attribution relationship between each scale parameter mapping value and the preset numerical range comprises: For the scale parameter mapping value belonging to the preset numerical range, determining the corresponding loss weight as the first weight; For the scale parameter mapping value that does not belong to the preset numerical range, the corresponding loss weight is determined to be a second weight; the second weight is greater than the first weight.

5. The method according to any one of claims 1 to 4, wherein determining the quantization constraint loss of the training video frame based on the difference between the scale parameter mapping value corresponding to each of the feature elements and the constraint reference mapping value corresponding to each of the scale parameter mapping values ​​comprises: Obtaining a preset probability distribution function, wherein the probability distribution function is used to input a scale parameter mapping value within a preset mapping value range and output an expected probability value of the input scale parameter mapping value; Determining, by means of the probability distribution function, an expected probability value of each of the scale parameter mapping values ​​and an expected probability value of a constraint reference mapping value corresponding to each of the scale parameter mapping values; Determining a quantized constraint sub-loss corresponding to each of the scale parameter mapping values ​​based on a difference between an expected probability value of each of the scale parameter mapping values ​​and an expected probability value of a corresponding constraint reference mapping value; Determine the quantization constraint sub-loss of the training video frame according to each of the scale parameter mapping values. Bundle loss.

6. According to the method according to claim 5, within the range from the constrained reference mapping value to its nearest rounded boundary value in the preset mapping value range, the closer the scale parameter mapping value input to the probability distribution function is to the constrained reference mapping value, the greater the expected probability value output by the probability distribution function.

7. The method according to any one of claims 1 to 6, wherein obtaining the constraint reference mapping value corresponding to each of the scale parameter mapping values ​​comprises: When the rounding mode is rounding up and rounding down, and each integer in the preset mapping value range is a rounding boundary value, the interval center of the mapping value interval to which each scale parameter mapping value belongs is used as the constraint reference mapping value corresponding to each scale parameter mapping value; and a mapping value interval is determined for every two adjacent integer quantization levels in the preset mapping value range.

8. The method according to any one of claims 1 to 7, wherein obtaining the constraint reference mapping value corresponding to each of the scale parameter mapping values ​​comprises: In the case where the rounding method is rounding to the nearest integer and the center of each mapping value interval is a rounding boundary value, for each scale parameter mapping value, the integer boundary value closest to the scale parameter mapping value in the mapping value interval to which the scale parameter mapping value belongs is used as the constrained reference mapping value corresponding to the scale parameter mapping value; and a mapping value interval is determined for every two adjacent integer quantization levels within the mapping value range.

9. A video coding model processing method, executed by a computer device, the method comprising: Acquire a training video frame, extract a feature map of the training video frame through a video encoding and decoding model, and determine a scale parameter value corresponding to each feature element in the feature map; Obtaining a constraint reference value corresponding to each of the scale parameter values, wherein the constraint reference value is farther from a rounding boundary value of a scale parameter mapping value obtained by mapping the corresponding scale parameter value after being mapped to a preset mapping value range according to a preset mapping relationship, compared to the scale parameter value corresponding to the constraint reference value; Determining a quantization constraint loss for the training video frame based on a difference between each of the scale parameter values ​​and the corresponding constraint reference value; and At least a portion of model parameters of the video codec model are updated based on the quantization constraint loss to obtain a trained video codec model.

10. The method according to claim 9, wherein determining the quantization constraint loss of the training video frame based on the difference between each of the scale parameter values ​​and the corresponding constraint reference value comprises: Determine a quantized constraint sub-loss corresponding to each scale parameter value based on a difference between each scale parameter value and the corresponding constraint reference value; Determine the loss weight corresponding to each of the scale parameter values; Based on the quantization constraint sub-loss and the loss weight respectively corresponding to each of the scale parameter values, the quantization constraint loss of the training video frame is determined.

11. The method according to claim 10, wherein determining the loss weight corresponding to each of the scale parameter values ​​comprises: Determining the attribution relationship between each of the scale parameter values ​​and a preset numerical range; The scale parameter value belonging to the preset numerical range and the scale parameter value obtained by encoding and decoding the corresponding characteristic element have consistent quantization results; Determining the loss weight corresponding to each scale parameter value according to the attribution relationship between each scale parameter value and the preset numerical range; Based on the quantization constraint sub-loss and the loss weight respectively corresponding to each of the scale parameter values, the quantization constraint loss of the training video frame is determined.

12. The method according to any one of claims 9 to 11, wherein determining the quantization constraint loss of the training video frame based on the difference between each of the scale parameter values ​​and the corresponding constraint reference value comprises: Obtaining a preset probability distribution function, wherein the probability distribution function is used to input a scale parameter value and output an expected probability value of the input scale parameter value; By using the probability distribution function, the expected probability value of each scale parameter value and the expected probability value of each scale parameter value are determined. The expected probability value of the constraint reference value corresponding to the parameter value; Determine the quantized constraint sub-loss corresponding to each of the scale parameter values ​​based on the difference between the expected probability value of each of the scale parameter values ​​and the expected probability value of the corresponding constraint reference value; The quantization constraint loss of the training video frame is determined according to the quantization constraint sub-loss corresponding to each of the scale parameter values.

13. According to the method of claim 12, in the definition domain of the scale parameter value, within the range of the constraint reference value to the scale parameter value that is most recently used to map to the rounding boundary value, the closer the scale parameter value input to the probability distribution function is to the constraint reference value, the greater the expected probability value output by the probability distribution function.

14. A video encoding method, performed by a computer device, the method comprising: Obtain a target video frame, extract a feature map of the target video frame using a trained video codec model, and determine a scale parameter value corresponding to each feature element in the feature map, wherein the video codec model is trained using the method according to any one of claims 1 to 13; For each characteristic element of the characteristic graph, according to a preset mapping relationship, the scale parameter value of the characteristic element is mapped to a preset mapping value range to obtain a scale parameter mapping value; The scale parameter mapping value obtained by mapping is rounded according to its rounding boundary value for quantization, so as to obtain the scale parameter quantization value of the feature element; The feature graph is entropy encoded according to the quantized values ​​of the scale parameters of the feature elements of the feature graph, and a transmission data stream is determined according to the encoded data stream obtained by the entropy encoding.

15. A video decoding method, performed by a computer device, the method comprising: Acquire a transmission data stream, and obtain a coded data stream from the transmission data stream, wherein the coded data stream is obtained by encoding a feature map of a target video frame; Determining the scale parameter values ​​corresponding to the feature elements in the feature graph by using a trained video codec model, wherein the video codec model is trained by the method according to any one of claims 1 to 13; For each characteristic element of the characteristic graph, according to a preset mapping relationship, the scale parameter value of the characteristic element is mapped to a preset mapping value range to obtain a scale parameter mapping value; The scale parameter mapping value obtained by mapping is rounded according to its rounding boundary value for quantization, so as to obtain the scale parameter quantization value of the feature element; The encoded data stream is entropy decoded according to the quantized values ​​of the scale parameters of each feature element of the feature map, and the target video frame is reconstructed based on the feature map restored by entropy decoding.

16. A video coding model processing device, the device comprising: A video frame acquisition module is used to acquire a training video frame, extract a feature map of the training video frame through a video encoding and decoding model, and determine the scale parameter value corresponding to each feature element in the feature map; A numerical mapping module, used to map the scale parameter values ​​corresponding to the characteristic elements according to a preset mapping relationship, to obtain scale parameter mapping values ​​corresponding to the characteristic elements, wherein the scale parameter mapping values ​​are within a preset mapping value range; a loss determination module, configured to obtain a constraint reference mapping value corresponding to each of the scale parameter mapping values, and determine a quantization constraint loss of the training video frame based on a difference between a scale parameter mapping value corresponding to each of the feature elements and a constraint reference mapping value corresponding to each of the scale parameter mapping values, wherein a distance between the constraint reference mapping value and a rounding boundary value of the scale parameter mapping value corresponding to the constraint reference mapping value is farther than a distance between the scale parameter mapping value corresponding to the constraint reference mapping value and the rounding boundary value; A parameter updating module is used to update at least a part of the model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

17. A video coding model processing device, the device comprising: A video frame acquisition module is used to acquire a training video frame, extract a feature map of the training video frame through a video encoding and decoding model, and determine the scale parameter value corresponding to each feature element in the feature map; The loss determination module is used to obtain the constraint reference value corresponding to each scale parameter value, based on each scale parameter value. The quantization constraint loss of the training video frame is determined by the difference between the parameter value and the corresponding constraint reference value, wherein the constraint reference value is farther from the rounding boundary value of the scale parameter mapping value obtained by mapping the corresponding scale parameter value after being mapped to the preset mapping value range according to the preset mapping relationship respectively compared with the scale parameter value corresponding to the constraint reference value; A parameter updating module is used to update at least a part of the model parameters of the video codec model based on the quantization constraint loss to obtain a trained video codec model.

18. A video encoding device, the device comprising: A video frame acquisition module, used to acquire a target video frame, extract a feature map of the target video frame through a trained video codec model, and determine the scale parameter value corresponding to each feature element in the feature map, wherein the video codec model is trained by the device according to claim 16 or 17; A numerical quantization module is used to map the scale parameter value of each feature element to a preset mapping value range according to a preset mapping relationship, obtain a scale parameter mapping value, and round the mapped scale parameter mapping value according to its rounding boundary value for quantization to obtain a scale parameter quantization value of each feature element; The entropy coding module is used to perform entropy coding on the feature map according to the quantized value of the scale parameter of each feature element of the feature map, and determine the transmission data stream according to the coded data stream obtained by entropy coding.

19. A video decoding device, the device comprising: A data stream acquisition module, used to acquire a transmission data stream, and obtain a coded data stream from the transmission data stream, wherein the coded data stream is obtained by encoding a feature map of a target video frame; A scale parameter prediction module, used to determine the scale parameter value corresponding to each feature element in the feature map through a trained video codec model, wherein the video codec model is trained by the apparatus according to claim 16 or 17; A numerical quantization module is used to map the scale parameter value of each feature element to a preset mapping value range according to a preset mapping relationship, obtain a scale parameter mapping value, and round the mapped scale parameter mapping value according to its rounding boundary value for quantization to obtain a scale parameter quantization value of each feature element; The entropy decoding module is used to perform entropy decoding on the encoded data stream according to the quantized value of the scale parameter of each feature element of the feature map, and reconstruct the target video frame based on the feature map restored by entropy decoding.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 15 when executing the computer program.

21. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15.

22. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images