Video encoding and decoding method, electronic device, and computer-readable storage medium

Abstract

The present application discloses a video encoding and decoding method, an electronic device, and a computer-readable storage medium. The method includes: obtaining a video sequence to be processed, where the video sequence to be processed includes at least one representative image; obtaining a quantization parameter offset value of the video sequence to be processed before and after resampling, where the quantization parameter offset value enables the rate-quantization balance cost of the representative image before resampling and the rate-quantization balance cost after resampling to satisfy a first constraint relationship; offsetting the quantization parameter before resampling based on the quantization parameter offset value to obtain a quantization parameter after resampling; performing resampling on at least one image in the video sequence to be processed; and performing encoding and decoding on at least one image based on the quantization parameter after resampling. Through the above manner, it is possible to avoid distortion of the video sequence caused by resampling.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a video encoding and decoding method, an electronic device, and a computer-readable storage medium. Background Art

[0002] For considerations such as improving the transmission speed and adapting to transmission conditions, it is often necessary to enable the Reference Picture Resampling (RPR) technology during the video encoding and decoding process. This resampling technology downsamples an image of the original size into a small image and then encodes and transmits it, and upsamples the small image reconstructed after decoding at the decoding end to the original size image, so as to save the bit consumption of encoding under poor transmission conditions or to improve the encoding efficiency, etc.

[0003] However, enabling the resampling technology during the video encoding and decoding process may cause video distortion. Summary of the Invention

[0004] This application provides a video encoding and decoding method, an electronic device, and a computer-readable storage medium, which can solve the problem that enabling the resampling technology during the video encoding and decoding process may cause video distortion.

[0005] To solve the above technical problem, a technical solution adopted by this application is: to provide a video encoding and decoding method. The method includes: obtaining a video sequence to be processed, where the video sequence to be processed includes at least one representative image; obtaining the quantization parameter offset value of the video sequence to be processed before and after resampling, where the quantization parameter offset value makes the rate-quantization balance cost of the representative image before resampling and the rate-quantization balance cost after resampling satisfy a first constraint relationship; offsetting the quantization parameter before resampling based on the quantization parameter offset value to obtain the quantization parameter after resampling; resampling at least one image in the video sequence to be processed; encoding and decoding at least one image based on the quantization parameter after resampling.

[0006] To solve the above technical problem, another technical solution adopted by this application is: to provide a video encoding and decoding method. The method includes: obtaining a video sequence to be processed, where the video sequence to be processed includes at least one representative image; making a resampling decision based on the rate-distortion balance cost of the representative image after resampling at the resampling ratio, where the rate-distortion balance cost is obtained by fusing the actual rate and the distortion degree of the representative image after resampling at the resampling ratio; if the decision is to perform resampling, resampling at least one image in the video sequence to be processed; encoding and decoding the video sequence to be processed.

[0007] To solve the above technical problems, another technical solution adopted by this application is: to provide a video encoding and decoding method. The method includes: obtaining a video sequence to be processed; determining a target resampling strategy from multiple candidate resampling strategies; determining a representative image and a corresponding image to be resampled and decided from the video sequence to be processed based on the target resampling strategy; making a resampling decision on the image to be resampled and decided based on the resampling decision result of the representative image; wherein, the corresponding relationships between the representative images and the images to be resampled and decided in different candidate resampling strategies are different; if the decision is to perform resampling, then resample the image to be resampled and decided; perform encoding and decoding on the video sequence to be processed.

[0008] To solve the above technical problems, another technical solution adopted by this application is: to provide an electronic device, which includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0009] To solve the above technical problems, another technical solution adopted by this application is: to provide a computer-readable storage medium storing program instructions, which can implement the above method when executed.

[0010] By the above method, when resampling the images in the video sequence to be processed, this application offsets the quantization parameter before resampling based on the quantization reference offset value to obtain the quantization parameter after resampling, and performs encoding and decoding on the resampled image based on the quantization parameter after resampling. The quantization parameter offset value makes the rate-quantization balance cost between the representative images before and after resampling satisfy the first constraint relationship. The rate-quantization balance cost before resampling is obtained by fusing the rate before resampling and the quantization parameter, and the rate-quantization balance cost after resampling is obtained by fusing the expected rate after resampling and the quantization parameter. Since the representative image is used as the basis for resampling the images in the video sequence to be processed and satisfies the first constraint relationship, it means that resampling the images in the video sequence to be processed will not cause the imbalance between the rate and the quantization parameter corresponding to the images. Thus, through the encoding and decoding method provided by this application, the problem of imbalance between the rate and the quantization parameter caused by resampling can be avoided, the quality of the resampled image will not be excessively reduced, the continuity between the quality of the resampled image and the non-resampled image is ensured, and the distortion of the video sequence caused by resampling is avoided. Description of the Drawings

[0011] Figure 1 is a schematic flowchart of an embodiment of the video encoding and decoding method of this application;

[0012] Figure 2 is Figure 1 a schematic flowchart of S12 in

[0013] Figure 3 is a schematic flowchart of another embodiment of the video encoding and decoding method of the present application;

[0014] Figure 4 is a schematic flowchart of yet another embodiment of the video encoding and decoding method of the present application;

[0015] Figure 5 is a schematic flowchart of yet another embodiment of the video encoding and decoding method of the present application;

[0016] Figure 6 is a schematic flowchart of yet another embodiment of the video encoding and decoding method of the present application;

[0017] Figure 7 is a schematic structural diagram of an embodiment of an electronic device of the present application;

[0018] Figure 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. Detailed Embodiments

[0019] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.

[0020] The terms "first", "second", and "third" in the present application are only used for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0021] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that, without conflict, the embodiments described herein may be combined with other embodiments.

[0022] Before introducing the video encoding and decoding method of the present application, the relevant names will be explained first:

[0023] Coding and decoding: The amount of image data in a video sequence is relatively large. It is necessary to encode the image pixel data to obtain a video bitstream. The video bitstream is transmitted to the decoding end through a wired or wireless network and then decoded for viewing. The entire encoding process includes processes such as block partitioning, prediction, transformation, quantization, and encoding.

[0024] Quantization parameter (QP): It characterizes the distortion degree of the video sequence caused by quantization. Generally, the larger the QP, the higher the distortion degree caused by quantization, and vice versa.

[0025] Bitrate (Rate): Also known as the bit rate, it refers to the number of bits transmitted per second. The higher the bitrate, the lower the image distortion degree and the higher the quality, and the larger the required bandwidth. When the bitrate decreases, the image distortion degree becomes higher and the quality deteriorates, that is, the bitrate and the quantization parameter are inversely correlated.

[0026] Reference Picture Resampling (RPR) technology: This technology downsamples the image of the original size into a small image and then encodes and transmits it, and upsamples the small image reconstructed after decoding at the decoding end to the original size image, so as to save the bit consumption of encoding under poor transmission conditions or to improve the encoding efficiency, etc. This technology will also be simply referred to as resampling in the following text of this application.

[0027] Scaling ratio: From the perspective of shrinking and magnifying, the scaling ratio includes the shrinking ratio and the magnifying ratio. The shrinking ratio is the ratio of downsampling in the resampling process, and the magnifying ratio is the ratio of upsampling in the resampling process. From the perspective of the scaling direction, the scaling ratio includes the horizontal and vertical scaling ratios. The horizontal scaling ratio is the ratio of the spatial resolution of the scaled image in the horizontal direction to that of the image of the original size, and the vertical scaling ratio is the ratio of the spatial resolution of the scaled image in the vertical direction to that of the image of the original size. The horizontal scaling ratio and the vertical scaling ratio can be the same or different. The scaling ratio will also be referred to as the resampling ratio in the following text of this application.

[0028] Peak Signal to Noise Ratio (PSNR): An index used to evaluate the image quality. Generally, the larger its value, the higher the image quality, and the smaller its value, the worse the image quality.

[0029] Video sequence: A sequence is an image set composed of several images that are continuous in time, and can be further divided into at least one Group Of Pictures (GOP). Each GOP contains at least one continuous image, and each image is also called a frame.

[0030] Figure 1It is a schematic flowchart of an embodiment of the video encoding and decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 1 the process sequence shown. For example, Figure 1 as shown, this embodiment may include:

[0031] S11: Obtain the video sequence to be processed.

[0032] The video sequence to be processed includes at least one representative image.

[0033] The video sequence to be processed may include several groups of images, and one group of images includes several images. The representative image is used as the basis for resampling at least one image in the video sequence to be processed.

[0034] S12: Obtain the quantization parameter offset value of the video sequence to be processed before and after resampling.

[0035] Among them, the quantization parameter offset value makes the rate-quantization balance cost of the representative image before resampling satisfy the first constraint relationship with the rate-quantization balance cost after resampling. The rate-quantization balance cost before resampling is obtained by fusing the rate before resampling and the quantization parameter, and the rate-quantization balance cost after resampling is obtained by fusing the expected rate after resampling and the quantization parameter.

[0036] The rate and the quantization parameter (QP) are inversely correlated. Satisfying the first constraint relationship means that the quantization parameter offset value is within expectations. Subsequently, the quantization parameter after resampling offset based on the quantization parameter offset value is also within expectations, and the balance degree between the rate and the quantization parameter before and after resampling is within expectations. Thus, resampling will not cause the imbalance of the rate and the quantization parameter.

[0037] The first constraint relationship can be a functional relationship (linear function, non-linear function), a mapping table, etc. For example, the first constraint relationship is: the rate-quantization balance cost before resampling satisfies a proportional relationship with the rate-quantization balance cost after resampling. That is, the first constraint relationship is:

[0038] keep1 = S·keep0 (1)

[0039] keep0 = f0(qp0,rate0) (2)

[0040] keep1 = f1(qp1,rate1) (3)

[0041] Among them, keep0 represents the rate-quantization balance cost before resampling, keep1 represents the rate-quantization balance cost after resampling, S represents the third coefficient, that is, the cost compensation factor after resampling, rate0 and qp0 respectively represent the code rate and quantization parameter before resampling, and rate1 and qp1 respectively represent the expected code rate and quantization parameter after resampling.

[0042] The way to fuse the code rate and quantization parameter to obtain the rate-quantization balance cost can be weighted, multiplied, etc.

[0043] In some embodiments, a number of candidate quantization parameter offset values can be preset in S12, and each candidate quantization parameter offset value is tested one by one to see if it can make the rate-quantization balance cost of the representative image before and after resampling satisfy the first constraint relationship. The candidate quantization parameter offset value that makes the rate-quantization balance cost of the representative image before and after resampling satisfy the first constraint relationship is used as the final quantization parameter offset value.

[0044] Or, in some embodiments, in S12, the quantization parameter offset value can be obtained according to the code rate and quantization parameter before resampling, the second constraint relationship between the code rate before resampling and the expected code rate after resampling, and the first constraint relationship. Based on this, with reference to Figure 2 , S12 may include the following sub-steps:

[0045] S121: Determine the expected code rate after resampling according to the code rate before resampling and the second constraint relationship between the code rate before resampling and the expected code rate after resampling.

[0046] The second constraint relationship can be used to constrain the code rate difference, code rate ratio, code rate difference ratio, etc. between the code rate before resampling and the expected code rate after resampling. The code rate difference ratio is the ratio of the code rate difference to the code rate before resampling.

[0047] Under the expected code rate after resampling, the subsequent encoding and decoding of the video sequence to be processed will not cause the situation of code rate and quantization parameter imbalance. The determined expected code rate after resampling can be a single value or a range value. For example, the second constraint relationship is: the code rate before resampling and the expected code rate after resampling satisfy a proportional range, which can be expressed as:

[0048]

[0049] The code rate after resampling can be obtained by transforming formula (4):

[0050]

[0051] Therefore, the expected code rate after resampling can be expressed as:

[0052]

[0053] Among them, represents the bitrate difference ratio threshold, |rate1 - rate0| / rate0 represents the bitrate difference ratio, and k represents the bitrate compensation factor after resampling.

[0054] S122: Calculate the quantization parameter offset value according to the bitrate and quantization parameter before resampling, the expected bitrate after resampling, and the first constraint relationship.

[0055] There is a first functional relationship between the bitrate - quantization balance cost before resampling and the bitrate and quantization parameter before resampling, and there is a second functional relationship between the bitrate - quantization balance cost after resampling and the expected bitrate and quantization parameter after resampling.

[0056] The first functional relationship / the second functional relationship can be a linear functional relationship or a non - linear functional relationship.

[0057] The parameters in the first functional relationship include the bitrate - quantization balance cost before resampling, the bitrate and quantization parameter before resampling, and the first coefficient (bitrate - quantization balance factor before resampling). The parameters in the second functional relationship include the bitrate - quantization balance cost after resampling, the expected bitrate after resampling and the quantization parameter, and the second coefficient (bitrate - quantization balance factor after resampling). The parameters in the first constraint relationship include the bitrate and quantization parameter before resampling, the expected bitrate and quantization parameter after resampling, and the third coefficient (cost balance factor after resampling).

[0058] Thus, the quantization parameter offset value can be calculated based on the first functional relationship, the second functional relationship, and the first constraint relationship:

[0059] qp offset = qp1 - qp0 = f offset (qp0, rate0, λ0, λ1, S) (7)

[0060] Among them, λ0, λ1, and S represent the first coefficient, the second coefficient, and the third coefficient respectively.

[0061] For example, in the first functional relationship, the bitrate - quantization balance cost before resampling is the linear summation of the product of the bitrate before resampling and the first coefficient and further with the quantization parameter before resampling. The first functional relationship can be expressed as:

[0062] keep0 = qp0 + λ0·rate0 (8)

[0063] In the second functional relationship, the resampled rate-quantization balance cost is a linear summation between the resampled expected rate multiplied by the second coefficient and the resampled quantization parameter. The second functional relationship can be expressed as:

[0064] keep1 = qp1 + λ1·rate1 (9)

[0065] Substituting Equation (8) and Equation (9) into Equation (1) gives:

[0066] qp1 + λ1·rate1 = S·(qp0 + λ0·rate0) (10)

[0067] Rearranging Equation (10) gives:

[0068] qp offset = qp1 - qp0 = (S - 1)·qp0 + S·λ0·rate0 - λ1·rate1 (11)

[0069] Furthermore, λ0, λ1, and S can all be constants, and thus λ0, λ1, and S can be directly substituted into Equation (7) / Equation (11) to calculate the quantization parameter offset value. Alternatively, at least one of λ0, λ1, and S is a variable, and thus after solving for the variable, it needs to be substituted into Equation (7) / Equation (11) to calculate the quantization parameter offset value.

[0070] In the case where λ0 and λ1 are variables, a third functional relationship can be set between the rate and quantization parameter before resampling, and a fourth functional relationship can be set between the resampled expected rate and quantization parameter. The third functional relationship / fourth functional relationship can be a linear functional relationship or a non-linear functional relationship. The third functional relationship and the fourth functional relationship can be statistically obtained based on the rates and quantization parameters of the historical images that have already been encoded and decoded.

[0071] For example, in the third functional relationship, the quantization parameter before resampling is a power function of the rate before resampling, and in the fourth functional relationship, the quantization parameter after resampling is a power function of the resampled expected rate. Thus, the third functional relationship can be expressed as:

[0072]

[0073] The fourth functional relationship can be expressed as:

[0074]

[0075] where α0, β0, α1, and β1 are all constants. For example, α0, β0, α1, and β1 are 230, -0.23, 235, and -0.26 respectively.

[0076] Thus, the first coefficient of the first functional relationship can be calculated based on the third functional relationship, and the second coefficient of the second functional relationship can be calculated based on the fourth functional relationship; the quantization parameter offset value is calculated based on the first coefficient, the second coefficient, the code rate before resampling and the quantization parameter, the expected code rate after resampling, and the first constraint relationship. In a specific embodiment, the absolute value of the partial derivative of the third functional relationship at the code rate before resampling can be used as the first coefficient, and the absolute value of the partial derivative of the fourth functional relationship at the expected code rate after resampling can be used as the second coefficient.

[0077] Based on the formulas (8)-(10) and formulas (12)-(13), an example of obtaining the quantization parameter offset value is illustrated as follows:

[0078] For the solution of λ0, the partial derivative of the formula is obtained to get keep0′, and let keep0′ = 0, then we can get:

[0079]

[0080] It can be known that λ0 is the negative partial derivative of qp0 with respect to rate0. Since qp0 is inversely proportional to rate0 and qp0 is a unary function of rate0, λ0 is the absolute value of the derivative of qp0 with respect to rate0:

[0081]

[0082] Similarly, we can get:

[0083]

[0084] Substituting formulas (15) and (16) into formula (11), we can get:

[0085] qp offset =(S - 1)·qp0 + S·|f0′(rate0)|·rate0 - |f1′(rate1)|·rate1 (17)

[0086] Furthermore, if the expected code rate after resampling is a single value, the obtained quantization parameter offset value is a single value. If the expected code rate after resampling is a range value, the range of the quantization parameter offset value can be determined according to the value range of the expected code rate after resampling; at least two candidate values are selected from the range of the quantization parameter offset value and weighted and summed to be used as the quantization parameter offset value. For example, the selected candidate values are the maximum value and the minimum value of the quantization parameter offset value.

[0087] For example, substituting formula (6) into formula (17), we can get:

[0088] qp offset=(S - 1)·qp0+(|S·f0′(rate0)| - k·|f1′(k·rate0)|)·rate0 (18)

[0089] Substituting the values of α0, β0, α1, and β1 into formula (18), we get:

[0090] qp offset =(S - 1)·qp0+(S·52.9·rate0 -1.23 -k·61.1·(k·rate0) -1.26 )·rate0(19)

[0091] The maximum value qp max_offset and the minimum value qp min_offset of the quantization parameter offset value can be determined through formula (19). After weighting the two and taking the integer, the final quantization parameter offset value can be obtained:

[0092] qp offset =round(ω0·qp min_offset +ω1·qp max_offset ) (20)

[0093] where round(·) represents rounding to the nearest integer, and ω0 and ω1 are the weights of qp max_offset and qp min_offset respectively, and ω0 + ω1 = 1.

[0094] In the above example, S = 0.95 can be set, i.e., k ∈ [0.95, 1.05], ω0 = ω1 = 0.5. Thus, when the video sequence Tango2 to be processed has qp0 = 42 and rate0 = 938 kbps, the obtained quantization parameter offset value qp offset = -2.

[0095] Alternatively, in some embodiments, S12 may include: obtaining the quantization parameter offset value by looking up a table based on the difference characterization value between the bitrate before resampling and the actual bitrate after resampling.

[0096] Wherein, the actual bitrate after resampling refers to the bitrate for encoding and decoding the image based on the quantization parameter before resampling after the image is resampled. The difference characterization value can be a bitrate difference, a bitrate ratio, a bitrate difference ratio, etc. The calculation method of the bitrate difference ratio can be referred to the previous description and will not be elaborated here.

[0097] A number of difference characterization value thresholds can be preset, and a number of difference characterization value ranges are obtained based on the difference characterization value thresholds, and the difference characterization value ranges and the quantization parameter offset values are stored in a mapping table in a one-to-one correspondence. Thus, the difference characterization value range to which the difference characterization value belongs can be determined, and the corresponding quantization parameter offset value can be obtained by looking up the table. The correspondence between the difference characterization value range and the quantization parameter offset value can be statistically obtained based on the historical images that have been encoded and decoded.

[0098] For example, the difference characterization value is the bitrate difference ratio, and the bitrate difference ratio thresholds include 75%, 50%, 25%, 10%, 5%. The pre-established mapping table is as follows:

[0099] Bit rate difference ratio range Quantization parameter offset value (0.50,0.75] -6 (0.25,0.50] -5 (0.10,0.25] -3 (0.05,0.10] -2 [0,0.05] -1

[0100] Determine the bitrate difference ratio range to which the bitrate difference ratio between the bitrate before resampling and the actual bitrate after resampling belongs, and obtain the quantization parameter offset value corresponding to the bitrate difference ratio range by looking up the table.

[0101] It can be understood that, on the one hand, compared with the method of testing each candidate quantization parameter offset value one by one to obtain the quantization parameter offset value, it can reduce the difficulty of obtaining the quantization parameter offset value, thereby reducing the encoding and decoding complexity. On the other hand, compared with the method of constraining the quantization parameter offset value by the bitrate difference before and after resampling, in S12, the quantization parameter offset value is constrained by the bitrate-quantization balance cost before and after resampling, which can avoid the problem of imbalance between the bitrate and the quantization parameter caused by resampling.

[0102] S13: Offset the quantization parameter before resampling based on the quantization parameter offset value to obtain the quantization parameter after resampling.

[0103] For example, the quantization parameter before resampling is 42, the quantization parameter offset value is -2, and the quantization parameter after resampling is 40.

[0104] S14: Resample at least one image in the video sequence to be processed.

[0105] The at least one image is an image resampled based on the representative image.

[0106] S15: Encode and decode at least one image based on the quantization parameter after resampling.

[0107] During the encoding and decoding process, the quantization parameter of the resampled image in the video sequence to be processed is the quantization parameter after resampling, while the quantization parameter of the image that has not been resampled remains unchanged, that is, the quantization parameter of the image that has not been resampled is the quantization parameter before resampling.

[0108] It can be understood that resampling the image by enabling the resampling technique during the encoding and decoding process may cause visual distortion of the video sequence. Specifically, if the image in the video sequence is resampled during the encoding and decoding process, the situation of imbalance between the bit rate and quantization parameter of the image may occur, resulting in too low quality of the resampled image in the video sequence, and the discontinuity of the quality between the resampled image and the non-resampled image, causing visual distortion.

[0109] By implementing this embodiment, when the present application resamples the image in the video sequence to be processed, the quantization parameter after resampling is obtained by offsetting the quantization parameter before resampling based on the quantization reference offset value, and the resampled image is encoded and decoded based on the quantization parameter after resampling. The quantization parameter offset value enables the bit rate-quantization balance cost representing the image before and after resampling to satisfy the first constraint relationship. The bit rate-quantization balance cost before resampling is obtained by fusing the bit rate and quantization parameter before resampling, and the bit rate-quantization balance cost after resampling is obtained by fusing the expected bit rate and quantization parameter after resampling. Since the representative image is used as the basis for resampling the image in the video sequence to be processed and satisfies the first constraint relationship, it means that resampling the image in the video sequence to be processed will not cause the imbalance between the corresponding bit rate and quantization parameter of the image. Therefore, through the encoding and decoding method provided by the present application, the problem of imbalance between the bit rate and quantization parameter caused by resampling can be avoided, the quality of the resampled image will not be excessively reduced, the continuity between the quality of the resampled image and the non-resampled image is ensured, and the distortion caused by resampling of the video sequence is avoided.

[0110] Furthermore, before performing the steps (S12-S15) related to resampling in the above embodiment, resampling decision can also be performed to decide whether to resample the video sequence to be processed, and only when it is decided to resample the video sequence to be processed, S12-S15 are executed. Specifically, it can be as follows:

[0111] Figure 3 It is a schematic flowchart of another embodiment of the video encoding and decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 3 the shown flowchart order. As Figure 3 shown, before S12, this embodiment may include:

[0112] S21: Based on the quality characterization value of the representative image after resampling at the resampling ratio, perform resampling decision.

[0113] The quality characterization value after resampling can be used to evaluate the quality of an image in the case of resampling during the encoding process. The quality characterization value can be the bitrate of the image at the resampling ratio, the distortion (peak signal-to-noise ratio), or the rate-distortion balance cost. Of course, the quality characterization value can also be other metrics for evaluating image quality.

[0114] The distortion can be obtained by converting the peak signal-to-noise ratio, and the rate-distortion balance cost can be obtained by fusing the actual bitrate and distortion after resampling of the image at the resampling ratio. The fusion method can be multiplication, weighting, etc. In the case of weighting, the rate-distortion balance cost is the weighted sum of the actual bitrate and distortion after resampling, which can be expressed as:

[0115] cost = d + μ·rate′1 (21)

[0116] Where cost represents the rate-distortion balance cost, d represents the distortion, μ represents the Lagrange factor, and rate′1 represents the actual bitrate after resampling at the resampling ratio.

[0117] The resampling ratio is single or at least two. If the resampling ratio is single, in S21, it can be determined whether the quality characterization value corresponding to the single resampling ratio is less than the characterization threshold; if it is less, the decision is to perform resampling, otherwise the decision is not to perform resampling. If the resampling ratio is at least two, in S21, it can be determined whether the minimum value among the quality characterization values corresponding to at least two resampling ratios is less than the characterization threshold; if it is less, the decision is to perform resampling and perform resampling based on the resampling ratio corresponding to the minimum value, otherwise the decision is not to perform resampling.

[0118] For example, for the case of a single resampling ratio, the expression of the resampling decision result is as follows:

[0119]

[0120] Where S1 represents the resampling decision result of the representative image, scale represents a single scaling ratio, non represents not performing resampling, cost1 represents the rate-distortion balance cost corresponding to the single scaling ratio, and cost th represents the cost threshold.

[0121] In this example, scale = 2.0 can be set, and cost th = τ·cost0. Where τ = 1.05 and cost0 is the rate-distortion cost before resampling.

[0122] For the case of at least two resampling ratios, the expression of the resampling decision result is as follows:

[0123]

[0124] Among them, φ represents the set of scaling ratios, and scale n represents the nth scaling ratio, and cost(scale n ) represents the rate-distortion balance cost corresponding to the nth scaling ratio.

[0125] In this example, φ can be set to {1.5, 2.0}, and cost th = τ·cost0.

[0126] S22: Determine whether the resampling decision result of the representative image is to perform resampling.

[0127] If it is to perform resampling, execute S23; otherwise, execute S24.

[0128] S23: Execute S12 to S15.

[0129] S24: Do not execute S12 to S15.

[0130] Through the implementation of this embodiment, resampling decision is made based on the quality characterization value of the representative image after resampling at the resampling ratio. Only when the decision result is to perform resampling, the resampling technology is enabled in the encoding and decoding process. Since the decision result is to perform resampling, it means that the image quality after resampling at the resampling ratio meets the requirements, and the video sequence to be processed is suitable for resampling at the resampling ratio. Thus, it is possible to determine whether to enable the resampling technology in the encoding and decoding process according to the actual situation of the video sequence to be processed. And when the quality characterization value is the rate-distortion balance cost, resampling decision is combined with the rate and distortion degree. Compared with the method of making resampling decision only using one of the rate and distortion degree, it can effectively balance the encoding and decoding performance.

[0131] Figure 4 is a schematic flowchart of another embodiment of the video encoding and decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 4 the shown process sequence. This embodiment is a further expansion of the above embodiment. As Figure 4 shown, before S14, this embodiment may include:

[0132] S31: Determine the target resampling strategy from multiple candidate resampling strategies.

[0133] The corresponding relationships between the representative images and the images to be resampled in different candidate resampling strategies are different, that is, the resampling decision methods corresponding to different candidate resampling strategies are different. The image to be resampled corresponding to the representative image is the image for which it is decided whether to perform resampling based on the resampling decision result of the representative image.

[0134] The resampling decision methods corresponding to different candidate resampling strategies are different, which can be reflected in different resampling decision complexities, different adapted coding levels, and so on. Among them, when the number of resampling decisions to be made remains unchanged, the more the number of representative images, the higher the resampling decision complexity is considered.

[0135] In some embodiments, the candidate resampling strategies may include at least two of a frame-level resampling strategy, a group-level resampling strategy, and a sequence-level resampling strategy. The frame-level resampling strategy is adapted to the frame-level coding level, the group-level resampling strategy is adapted to the group-of-pictures coding level, and the sequence-level resampling strategy is adapted to the sequence-level coding level.

[0136] Under the frame-level resampling strategy, the representative image and the image to be resampled can be the same image. That is, for each image in the video sequence to be processed, the resampling decision of the image is based on the image itself.

[0137] Under the group-level resampling strategy, the representative image is at least part of the images in the group of pictures included in the video sequence to be processed, and the images to be resampled are all the images in the group of pictures to which the representative image belongs. The determination of the representative image in the group of pictures can be random or according to a preset rule. For example, several representative images are randomly selected from the group of pictures. Another example is that, according to the playback order, several images with earlier rankings are selected from the group of pictures as the representative images. Still another example is that starting from the first image in the group of pictures, one image is selected as the representative image every several images.

[0138] Under the sequence-level resampling strategy, the representative image is the image in at least part of the groups of pictures included in the video sequence to be processed, and the images to be resampled are all the images in the video sequence to be processed. The determination of the group of pictures to which the representative image belongs can be random or according to a preset rule. For example, according to the playback order, several groups of pictures with earlier rankings are selected from the video sequence to be processed as the groups of pictures to which the representative image belongs, and representative images are respectively selected from the selected groups of pictures.

[0139] S32: Determine the representative image and the corresponding images to be resampled from the video sequence to be processed based on the target resampling strategy.

[0140] S33: Make a resampling decision on the images to be resampled based on the resampling decision result of the representative image.

[0141] For the method of obtaining the resampling decision result of the representative image, please refer to the relevant description of S21 above and will not be elaborated here.

[0142] Under the group-level resampling strategy, if there are multiple representative images, the resampling decision result of the images to be resampled can be obtained by voting on the resampling decision results of the multiple representative images.

[0143] Under the sequence-level resampling strategy, if the representative image is an image in multiple image groups, the resampling decision result of the video sequence to be processed can be obtained by voting on the resampling decision results of multiple image groups to which the representative image belongs. For the resampling decision result of an image group, please refer to the resampling decision method under the group-level resampling strategy.

[0144] Taking S31 to S33 as an example, the video frame sequence to be processed = {image group 1, image group 2,..., image group k}, and image group 1 = {image 1, image 2,..., image m}.

[0145] When the target resampling strategy is the group-level resampling strategy, select n representative images from image group 1 and vote on the resampling decision results of the n representative images to obtain the resampling decision result of image group 1. The expression of the resampling decision result can be:

[0146]

[0147] where S2 represents the resampling decision result of image group 1, yes means resampling is performed, non means resampling is not performed, s i represents the resampling decision result of the i-th representative image in image group 1, and s th represents the voting threshold.

[0148] In this example, n = 1, m = 32, and s th = 0.5 can be set.

[0149] When the target resampling strategy is the sequence-level resampling strategy, select image groups 1 to n to which the representative image belongs, and vote on the resampling decision results of image groups 1 to n to obtain the resampling decision result of the video frame sequence to be processed. The expression of the resampling decision result can be:

[0150]

[0151] where G represents the resampling decision result of the video frame sequence to be processed, g i represents the i-th image group to which the representative image belongs, and g th represents the voting threshold.

[0152] In this example, n = 1 and g th = 0.6 can be set.

[0153] Through the implementation of this embodiment, the present application selects a target resampling strategy from multiple candidate resampling strategies and makes a resampling decision under the guidance of the target resampling strategy. Since the correspondence between the representative images in different candidate resampling strategies and the images to be resampled and decided is different, that is, the resampling decision methods corresponding to different candidate resampling strategies are different, the resampling decision method (complexity, applicable coding level, etc.) can be made controllable, effectively promoting the application of the resampling technology in the encoding and decoding process.

[0154] Figure 5 It is a schematic flowchart of another embodiment of the video encoding and decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 5 the shown process sequence.

[0155] As Figure 5 shown, this embodiment may include:

[0156] S41: Obtain the video sequence to be processed.

[0157] The video sequence to be processed includes at least one representative image.

[0158] S42: Make a resampling decision based on the rate-distortion balance cost after resampling the representative image at the resampling ratio.

[0159] The rate-distortion balance cost is obtained by fusing the actual rate and distortion degree after resampling the representative image at the resampling ratio. The fusion method can be multiplication, weighting, etc. In the case of the weighting method, the rate-distortion balance cost is the weighted sum of the actual rate and distortion degree after resampling, and can be expressed as:

[0160] cost = d + μ·rate′1(26)

[0161] The resampling ratio is single or at least two. If the resampling ratio is single, in S42, it can be determined whether the rate-distortion balance cost corresponding to the single resampling ratio is less than the cost threshold; if it is less, the decision is to perform resampling, otherwise the decision is not to perform resampling. If the resampling ratio is at least two, in S42, it can be determined whether the minimum value among the rate-distortion balance costs corresponding to at least two resampling ratios is less than the cost threshold; if it is less, the decision is to perform resampling, and resampling is performed based on the resampling ratio corresponding to the minimum value, otherwise the decision is not to perform resampling.

[0162] S43: Determine whether the resampling decision result of the representative image is to perform resampling.

[0163] If it is to perform resampling, execute S44~S45; otherwise, do not execute S44 and directly enter S45.

[0164] S44: Resample at least one image in the video sequence to be processed.

[0165] The at least one image is an image determined to be resampled based on a representative image.

[0166] S45: Encode and decode the video sequence to be processed.

[0167] For the resampled image, perform encoding and decoding based on the resampling result; otherwise, perform encoding and decoding based on the original image.

[0168] For other detailed descriptions of this embodiment, please refer to the previous embodiments and will not be elaborated here.

[0169] By implementing this embodiment, the present application does not directly enable the resampling technique during the encoding and decoding process of the video sequence to be processed. Instead, it makes a resampling decision based on the rate-distortion balance cost after resampling at the resampling ratio based on the representative image. Only when the decision result is to perform resampling, the resampling technique is enabled during the encoding and decoding process. Since the decision result is to perform resampling, it means that the image quality after resampling at the resampling ratio meets the requirements, and the video sequence to be processed is suitable for resampling at the resampling ratio. Thus, it is possible to decide whether to enable the resampling technique during the encoding and decoding process according to the actual situation of the video sequence to be processed. Moreover, using the rate-distortion balance cost (combining the bit rate and distortion degree) to make the resampling decision can effectively balance the encoding and decoding performance compared with the method of making the resampling decision using only one of the bit rate and distortion degree.

[0170] Figure 6 It is a schematic flowchart of another embodiment of the video encoding and decoding method of the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 6 the shown process sequence.

[0171] As Figure 6 shown, this embodiment may include:

[0172] S51: Obtain the video sequence to be processed.

[0173] S52: Determine the target resampling strategy from multiple candidate resampling strategies.

[0174] Among them, the corresponding relationships between the representative images and the images to be resampled and decided in different candidate resampling strategies are different. That is, the resampling decision methods corresponding to different candidate resampling strategies are different.

[0175] Candidate resampling strategies may include at least two of frame-level resampling strategies, group-level resampling strategies, and sequence-level resampling strategies. The frame-level resampling strategy adapts to the frame-level coding level, the group-level resampling strategy adapts to the group-of-pictures coding level, and the sequence-level resampling strategy adapts to the sequence-level coding level.

[0176] Among them, under the frame-level resampling strategy, the representative image and the image to be resampled and decided can be the same image.

[0177] Under the group-level resampling strategy, the representative image is at least part of the images included in the group of pictures of the video sequence to be processed, and the image to be resampled and decided is all the images in the group of pictures to which the representative image belongs. The determination of the representative image in the group of pictures can be random or according to a preset rule. For example, several representative images are randomly selected from the group of pictures. Another example is that, according to the playback order, several images with earlier rankings are selected from the group of pictures as the representative images. Still another example is that starting from the first image in the group of pictures, one image is selected as the representative image every several images.

[0178] Under the sequence-level resampling strategy, the representative image is the image in at least part of the groups of pictures included in the video sequence to be processed, and the image to be resampled and decided is all the images in the video sequence to be processed. The determination of the group of pictures to which the representative image belongs can be random or according to a preset rule. For example, according to the playback order, several groups of pictures with earlier rankings are selected from the video sequence to be processed as the groups of pictures to which the representative image belongs, and representative images are respectively selected from the selected groups of pictures.

[0179] S53: Determine the representative image and the corresponding image to be resampled and decided from the video sequence to be processed based on the target resampling strategy.

[0180] S54: Make a resampling decision on the image to be resampled and decided based on the resampling decision result of the representative image.

[0181] Under the group-level resampling strategy, if there are multiple representative images, the resampling decision result of the image to be resampled and decided can be obtained by voting on the resampling decision results of the multiple representative images.

[0182] Under the sequence-level resampling strategy, if the representative image is the image in multiple groups of pictures, the resampling decision result of the video sequence to be processed can be obtained by voting on the resampling decision results of the multiple groups of pictures to which the representative image belongs.

[0183] S55: Judge whether the resampling decision result of the image to be resampled and decided is to perform resampling.

[0184] If it is to perform resampling, then execute S56~S57; otherwise, do not execute S56 and directly enter S57.

[0185] S56: Resample the resampling decision image.

[0186] S57: Codec the video sequence to be processed.

[0187] For the resampled image, codec is performed based on the resampling result; otherwise, codec is performed based on the original image.

[0188] By implementing this embodiment, the present application proposes multiple candidate resampling strategies, and each candidate resampling strategy is applicable to different resampling decision methods, which can effectively promote the application of resampling technology in the codec process.

[0189] It should be noted that any embodiments of the present application can be combined with each other. The embodiments related to the determination method of the resampling strategy, the resampling decision method, and the acquisition method of the quantization parameter offset value in the video codec process can be implemented separately or in combination. For the case of combined implementation, in the case of combined implementation of the determination method of the resampling strategy and the resampling decision method, the resampling strategy should be determined based on the determination method of the resampling strategy first, and then, under the determined resampling strategy, the resampling decision is made based on the resampling decision method; in the case of combined implementation of the resampling decision method and the acquisition method of the quantization parameter offset value, the resampling decision should be made based on the resampling decision method first, and then the quantization parameter offset value is obtained based on the acquisition method of the quantization parameter offset value; in the case of combined implementation of the determination method of the resampling strategy, the resampling decision method, and the acquisition method of the quantization parameter offset value, the resampling strategy should be determined based on the determination method of the resampling strategy first, and then, under the determined resampling strategy, the resampling decision is made based on the resampling decision method, and finally the quantization parameter offset value is obtained based on the acquisition method of the quantization parameter offset value.

[0190] Figure 7 It is a schematic structural diagram of an embodiment of an electronic device of the present application. As Figure 7 shown, the electronic device includes a processor 21 and a memory 22 coupled to the processor 21.

[0191] Among them, the memory 22 stores program instructions for implementing the method of any of the above embodiments; the processor 21 is configured to execute the program instructions stored in the memory 22 to implement the steps of the above method embodiments. Among them, the processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip with signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0192] Figure 8 is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. As Figure 8 shown, the computer-readable storage medium 30 of the embodiment of the present application stores program instructions 31, and when the program instructions 31 are executed, the method provided in the above embodiments of the present application is implemented. Among them, the program instructions 31 may form a program file and be stored in the above computer-readable storage medium 30 in the form of a software product, so that a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor can execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned computer-readable storage medium 30 includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, or terminal devices such as computers, servers, mobile phones, and tablets.

[0193] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other may be through some interfaces, and the indirect couplings or communication connections of devices or units may be in electrical, mechanical, or other forms.

[0194] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units. The above is only the implementation manner of the present application and does not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A video encoding and decoding method, characterized in that Including: Obtain a video sequence to be processed, where the video sequence to be processed includes at least one representative image; Obtain the quantization parameter offset value of the video sequence to be processed before and after resampling. Wherein, the quantization parameter offset value enables the rate-quantization balance cost of the representative image before resampling and the rate-quantization balance cost after resampling to satisfy a first constraint relationship. The rate-quantization balance cost before resampling is obtained by fusing the rate before resampling and the quantization parameter, and the rate-quantization balance cost after resampling is obtained by fusing the expected rate after resampling and the quantization parameter; Offset the quantization parameter before resampling based on the quantization parameter offset value to obtain the quantization parameter after resampling; Resample at least one image in the video sequence to be processed; Encode and decode the at least one image based on the quantization parameter after resampling.

2. The method according to claim 1, wherein The obtaining the quantization parameter offset value of the video sequence to be processed before and after resampling includes: Determine the expected rate after resampling according to the rate before resampling and a second constraint relationship between the rate before resampling and the expected rate after resampling; Calculate the quantization parameter offset value according to the rate and quantization parameter before resampling, the expected rate after resampling, and the first constraint relationship.

3. The method according to claim 2, wherein There is a first functional relationship between the rate-quantization balance cost before resampling and the rate and quantization parameter before resampling, there is a second functional relationship between the rate-quantization balance cost after resampling and the expected rate and quantization parameter after resampling, there is a third functional relationship between the rate and quantization parameter before resampling, and there is a fourth functional relationship between the expected rate and quantization parameter after resampling; The calculating the quantization parameter offset value according to the rate and quantization parameter before resampling, the expected rate after resampling, and the first constraint relationship includes: Calculate the first coefficient of the first functional relationship based on the third functional relationship, and calculate the second coefficient of the second functional relationship based on the fourth functional relationship; Calculate the quantization parameter offset value based on the first coefficient, the second coefficient, the rate and quantization parameter before resampling, the expected rate after resampling, and the first constraint relationship.

4. The method according to claim 3, characterized in that, In the first functional relationship, the rate-quantization balance cost before resampling is the linear summation between the product of the rate before resampling and the first coefficient and the quantization parameter before resampling. In the second functional relationship, the rate-quantization balance cost after resampling is the linear summation between the product of the expected rate after resampling and the second coefficient and the quantization parameter after resampling; The calculating the first coefficient of the first functional relationship based on the third functional relationship and calculating the second coefficient of the second functional relationship based on the fourth functional relationship includes: Take the absolute value of the partial derivative of the third functional relationship at the bitrate before resampling as the first coefficient, and take the absolute value of the partial derivative of the fourth functional relationship at the expected bitrate after resampling as the second coefficient.

5. The method according to claim 4, wherein In the third functional relationship, the quantization parameter before resampling is a power function of the bitrate before resampling. In the fourth functional relationship, the quantization parameter after resampling is a power function of the expected bitrate after resampling.

6. The method according to claim 2, wherein The expected bitrate after resampling is a range value; The calculating the quantization parameter offset value according to the bitrate and quantization parameter before resampling, the expected bitrate after resampling, and the first constraint relationship includes: Determine the value range of the quantization parameter offset value according to the value range of the expected bitrate after resampling; Select at least two candidate values from the value range of the quantization parameter offset value and perform weighted summation to be used as the quantization parameter offset value.

7. The method according to claim 2, wherein The first constraint relationship is that there is a proportional relationship between the rate-quantization balance cost before resampling and the rate-quantization balance cost after resampling. The second constraint relationship is that there is a proportional range between the bitrate before resampling and the expected bitrate after resampling.

8. The method according to claim 1, characterized in that, The obtaining the quantization parameter offset value of the video sequence to be processed before and after resampling includes: Obtain the quantization parameter offset value by looking up a table according to the difference characterization value between the bitrate before resampling and the actual bitrate after resampling.

9. The method according to claim 1, wherein Before the obtaining the quantization parameter offset value of the video sequence to be processed before and after resampling, the method further includes: Perform a resampling decision based on the quality characterization value of the representative image after resampling at the resampling ratio; If the resampling decision result of the representative image is to perform resampling, then execute the steps of obtaining the quantization parameter offset value of the video sequence to be processed before and after resampling and subsequent steps.

10. The method according to claim 9, wherein The quality characterization value is the rate-distortion balance cost, and the rate-distortion balance cost is obtained by fusing the actual bitrate and the distortion degree of the representative image after resampling at the resampling ratio.

11. The method according to claim 10, wherein The rate-distortion balance cost is the weighted sum of the actual bitrate after resampling and the distortion degree.

12. The method according to claim 9, wherein The resampling ratio is single. The performing a resampling decision based on the quality characterization value of the representative image after resampling at the resampling ratio includes: Judge whether the quality characterization value corresponding to the single resampling ratio is less than the characterization threshold; if it is less than, then decide to perform resampling; or The resampling ratio is at least two. The performing a resampling decision based on the quality characterization value of the representative image after resampling at the resampling ratio includes: Judge whether the minimum value among the quality characterization values corresponding to at least two resampling ratios is less than the characterization threshold; if it is less than, then perform resampling based on the resampling ratio corresponding to the minimum value.

13. The method according to claim 1, wherein Before resampling at least one image in the video sequence to be processed, it includes: Determine a target resampling strategy from multiple candidate resampling strategies; Determine the representative image and the corresponding image to be resampled and decided from the video sequence to be processed based on the target resampling strategy; Make a resampling decision on the image to be resampled and decided based on the resampling decision result of the representative image; wherein the corresponding relationships between the representative images and the images to be resampled and decided in different candidate resampling strategies are different.

14. The method according to claim 13, wherein The candidate resampling strategies include at least two of a frame-level resampling strategy, a group-level resampling strategy, and a sequence-level resampling strategy. Among them, under the frame-level resampling strategy, the representative image and the image to be resampled and decided are the same image. Under the group-level resampling strategy, the representative image is at least part of the images in the image group included in the video sequence to be processed, and the image to be resampled and decided is all the images in the image group to which the representative image belongs. Under the sequence-level resampling strategy, the representative image is the image in at least part of the image groups included in the video sequence to be processed, and the image to be resampled and decided is all the images in the video sequence to be processed.

15. The method according to claim 14, characterized in that, Under the group-level resampling strategy, if there are multiple representative images, the resampling decision result of the image to be resampled and decided is obtained by voting on the resampling decision results of the multiple representative images; and / or Under the sequence-level resampling strategy, if the representative image is the image in multiple image groups, the resampling decision result of the video sequence to be processed is obtained by voting on the resampling decision results of the multiple image groups to which the representative image belongs.

16. An electronic device, characterized in that, Comprising a processor and a memory connected to the processor, wherein, The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method according to any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor and, when executed, implement the method according to any one of claims 1-15.

Images