Video encoding method, encoder, system and computer storage medium

Abstract

The embodiment of the present application discloses a video encoding method, an encoder, a system and a computer storage medium, the method comprising: determining pre-parameters of a video to be encoded, and determining a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; determining a target distortion value according to the first distortion value and the second distortion value; determining encoding parameters of the video to be encoded using the target Lagrangian multiplier and the target distortion value, and encoding the video to be encoded.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and in particular, to a video coding method, encoder, system and computer storage medium. Background Art

[0002] At present, the International Telecommunication Union (ITU) and the International Organization for Standardization (ISO) have established a Joint Video Experts Team (JVET) to study the latest video coding standard H.266 / Versatile Video Coding (VVC). The performance of H.266 / VVC is about 40% higher than that of H.265 / High Efficiency Video Coding (HEVC), making it the industry's most advanced video compression technology solution.

[0003] Generally speaking, for the same video encoding algorithm, the higher the bit rate, the better the quality of the reconstructed video and the lower the distortion; however, the encoded file will occupy more storage space and the bit rate will be higher. In this case, rate distortion optimization (RDO) technology is needed to find a balance between the distortion and bit rate of the reconstructed video to achieve the best compression effect.

[0004] However, in the current related technologies, the rate-distortion optimization algorithm can either only guarantee the fidelity of the reconstructed video, or can guarantee the subjective quality of the reconstructed video, but the fidelity performance of the video will be greatly reduced. Especially in video coding for machine vision and human-computer vision, the distortion criteria adopted by the existing rate-distortion optimization algorithm are single and incomplete, making the existing rate-distortion optimization algorithm not well adapted to the application scenarios for machine vision and human-computer vision. Summary of the invention

[0005] The embodiments of the present application provide a video encoding method, encoder, system and computer storage medium, which can be well adapted to application scenarios for machine vision and human-machine vision, and can improve the semantic segmentation accuracy of the reconstructed video under a certain bit rate while maintaining good fidelity performance, thereby improving the encoding efficiency.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a video encoding method, applied to an encoder, the method comprising:

[0008] Determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters;

[0009] Determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0010] Determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion comprises a semantic distortion metric criterion;

[0011] Determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion;

[0012] Determining a target distortion value according to the first distortion value and the second distortion value;

[0013] The target Lagrange multiplier and the target distortion value are used to determine the encoding parameters of the video to be encoded, and the video to be encoded is encoded.

[0014] In a second aspect, an embodiment of the present application provides an encoder, the encoder comprising a determination unit, a calculation unit and an encoding unit; wherein,

[0015] The determining unit is configured to determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters;

[0016] The calculation unit is configured to determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0017] The determining unit is further configured to determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion;

[0018] The calculation unit is further configured to determine a target distortion value according to the first distortion value and the second distortion value;

[0019] The encoding unit is configured to determine the encoding parameters of the video to be encoded by using the target Lagrange multiplier and the target distortion value, and encode the video to be encoded.

[0020] In a third aspect, an embodiment of the present application provides an encoder, the encoder comprising a memory and a processor; wherein:

[0021] The memory is used to store a computer program that can be run on the processor;

[0022] The processor is used to execute the method as described in the first aspect when running the computer program.

[0023] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and when the computer program is executed by at least one processor, it implements the method described in the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a video system, the video system comprising an encoder and a decoder; wherein:

[0025] The encoder is configured to determine pre-parameters of a video to be encoded, determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; and determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; and determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; and determine a target distortion value according to the first distortion value and the second distortion value; and determine encoding parameters of the video to be encoded using the target Lagrangian multiplier and the target distortion value, encode the video to be encoded to generate a bitstream, and transmit the bitstream to the decoder;

[0026] The decoder is configured to parse the bit stream to obtain a decoded video.

[0027] The embodiment of the present application provides a video encoding method, an encoder, a system and a computer storage medium, by determining pre-parameters of a video to be encoded, and determining a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; determining a target distortion value according to the first distortion value and the second distortion value; using the target Lagrangian multiplier and the target distortion value, determining the encoding parameters of the video to be encoded, and encoding the video to be encoded. In this way, the first distortion metric based on semantic distortion metric and the second distortion metric based on numerical error metric are comprehensively considered in video coding for rate-distortion optimization, which can be well adapted to application scenarios for machine vision and human-computer vision. Moreover, under a certain bit rate, the semantic segmentation accuracy of the reconstructed video can be improved while maintaining good fidelity performance, thereby improving the coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A structural schematic diagram of an RD curve provided for a related technical solution;

[0029] Figure 2 A schematic diagram of the system composition structure of an encoder provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of a video encoding method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a curve showing a functional relationship between a first distortion value and a bit rate provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of a curve showing a functional relationship between a bit rate and a quantization parameter provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of a curve showing a functional relationship between a first distortion value and MSE provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0035] Figure 8 A schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0036] Fig. 9A schematic diagram of the composition structure of a video system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0038] With the development of the digital media era, transmitting continuous media data through the network has become a general trend. At the same time, more and more users want to use personal computers (PCs) and non-PC devices to perform video communications and services through the Internet and wireless networks. This kind of anytime and anywhere video communication and service poses a greater challenge to the current video coding technology.

[0039] It should be understood that the International Telecommunication Union (ITU) and the International Organization for Standardization (ISO) have established a Joint Video Experts Team (JVET) to study the next-generation video coding standard H.266 / Versatile Video Coding (VVC). In addition, the technology accumulated in the industry has further improved the performance of H.266 / VVC by about 40% compared to H.265 / HEVC, making it the industry's most advanced video compression technology solution.

[0040] Generally speaking, for the same video encoding algorithm, the higher the bit rate, the better the quality of the reconstructed video and the smaller the distortion; however, the larger the storage space occupied by the encoded file, the higher the bit rate generated. Therefore, at this time, it is necessary to find a balance between the distortion and bit rate of the reconstructed video through a rate-distortion optimization algorithm to achieve the best compression effect.

[0041] It should be noted that rate-distortion optimization can be expressed as minimizing the distortion of the video reconstructed by decoding when the encoded file does not exceed a certain bit rate, as shown in the following formula (1).

[0042] min{D}stR<=R max (1)

[0043] Wherein, D and R represent the distortion and bit rate under certain coding parameters respectively.

[0044] Encode the video with given encoding parameters, and calculate the bit rate (R) and the distortion (D) of the reconstructed video. By changing the encoding parameters and repeatedly encoding the video to be encoded, multiple RD points consisting of bit rate and distortion can be obtained, such as Figure 1 Typically, for a preset bit rate, the point with the minimum distortion will appear at Figure 1 For the input video to be encoded, the encoder needs to determine a set of encoding parameters that can make the encoded RD point as close to this convex curve as possible.

[0045] At this time, the constrained problem of equation (1) can be converted into an unconstrained problem by using the Lagrange multiplier method, as shown in the following equation (2).

[0046] min{J=D+λ·R} (2)

[0047] Among them, λ represents the Lagrange multiplier, and J represents the rate-distortion cost function. For each possible λ, its corresponding value is the slope of the tangent line of the RD curve. The encoder can find the optimal encoding parameters by minimizing the rate-distortion cost function.

[0048] In this way, using the rate-distortion optimization algorithm, the encoder can determine the optimal block division method, the optimal intra-frame prediction mode, and the optimal inter-frame prediction motion mode (including motion vectors, reference images, prediction weights, etc.) to achieve optimal coding performance.

[0049] In related technical solutions (such as VVC), the rate-distortion optimization here uses the sum of square errors (SSE) as the distortion criterion, and the corresponding reconstructed video quality can be measured by the peak signal-to-noise ratio (PSNR). SSE distortion can measure the fidelity of the video objectively, and its calculation formula is shown in the following formula (3).

[0050]

[0051] Wherein, M and N represent the horizontal spatial resolution and vertical spatial resolution of the video respectively, f(x, y) represents the original pixel value at the pixel position (x, y), and g(x, y) represents the reconstructed pixel value at the pixel position (x, y).

[0052] Since rate-distortion optimization is a key technology in video coding, it affects the performance of the encoder. Although the rate-distortion optimization algorithm in existing video coding uses SSE distortion, it can measure the fidelity of the video from an objective perspective; however, SSE distortion is not consistent with the perception of the human visual system. For example, for some areas with large SSE distortion, the human eye will not notice the decrease in the quality of the reconstructed video. At this time, when the encoder needs to ensure the subjective quality of the reconstructed video, it is necessary to change the distortion criterion to a distortion metric that can measure the subjective quality. For example, the structural similarity (SSIM) distortion that is consistent with human eye perception can be used. The calculation formula is shown in the following formula (4).

[0053]

[0054] Among them, x and y represent the original image and the reconstructed image respectively, μ x and μ y represent the means of the original image and the reconstructed image respectively, and denote the variance of the original image and the reconstructed image, σ xy Represents the covariance of the original image and the reconstructed image, C1 and C2 are two constants. In order to avoid and Instability occurs when it is close to 0. Here, in order to obtain a robust quality evaluation result, we can take C1 = (K1L) 2 , C2=(K2L) 2 ; Where L = 2 bit_depth -1 (bit_depth indicates the bit depth, for an image with a bit depth of 8 bits, L=255), K1=0.01, K2=0.03.

[0055] In the related art, the rate-distortion optimization algorithm based on SSE distortion can ensure the fidelity of the reconstructed video; however, although the SSIM distortion considering the subjective quality can ensure the subjective quality of the reconstructed video, the fidelity performance of the video will be greatly reduced.

[0056] Since the current rate-distortion optimization algorithms are all aimed at traditional application scenarios of reconstructing videos for people to watch and study, the advent of the fifth generation (5G) era has spawned a large number of machine-oriented applications, such as Internet of Vehicles, unmanned driving, industrial Internet, smart and safe cities, wearables, video surveillance and other machine vision content, and the application scenarios are more extensive. In the 5G era and the post-5G era, most videos will be used by machines, such as intelligent analysis of reconstructed videos such as pedestrian detection, semantic segmentation, and target detection. However, in video coding for machine vision and human-machine vision, the distortion criterion used by the current rate-distortion optimization algorithm only considers fidelity distortion, but not semantic distortion; although the reconstructed video obtained after encoding has good fidelity performance, the semantic accuracy of the reconstructed video cannot be guaranteed, resulting in the current rate-distortion optimization algorithm not being well adapted to many scenarios for machine vision and human-machine vision.

[0057] Based on this, an embodiment of the present application provides a video encoding method, the basic idea of ​​which is: determining pre-parameters of a video to be encoded, and determining a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; determining a target distortion value according to the first distortion value and the second distortion value; using the target Lagrangian multiplier and the target distortion value, determining the encoding parameters of the video to be encoded, and encoding the video to be encoded. In this way, the first distortion metric based on semantic distortion metric and the second distortion metric based on numerical error metric are comprehensively considered in video coding for rate-distortion optimization, which can be well adapted to application scenarios for machine vision and human-computer vision. Moreover, under a certain bit rate, the semantic segmentation accuracy of the reconstructed video can be improved while maintaining good fidelity performance, thereby improving the coding efficiency.

[0058] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0059] See also Figure 2 , which shows an example of a system composition block diagram of an encoder provided in an embodiment of the present application. Figure 2As shown, the encoder 10 may include a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, a coding unit 109 and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement deblocking filtering and sample adaptive offset (Sample Adaptive Offset, SAO) filtering, and the coding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (Context-based Adaptive Binary Arithmatic Coding, CABAC). For the input original video signal, through the coding block (Coding A video coding block can be obtained by dividing the video coding block into a video unit (CU), and then the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain, and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is a process of generating a motion vector, and the motion vector can estimate the motion of the video coding block, and then the motion compensation unit 104 calculates the temporal prediction information based on the motion vector determined by the motion estimation unit 105. Perform motion compensation; after determining the intra-frame prediction mode, the intra-frame prediction unit 103 is also used to provide the selected intra-frame prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated and determined motion vector data to the encoding unit 109; in addition, the inverse transform and inverse quantization unit 106 is used to reconstruct the video coding block, reconstruct the residual block in the pixel domain, and the reconstructed residual block is removed from the block effect artifacts through the filter control analysis unit 107 and the filtering unit 108, and then the reconstructed residual block is added to a predictive block in the frame of the decoded image cache unit 110 to generate a reconstructed video coding block; the encoding unit 109 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode and output the code stream of the video signal; and the decoded image cache unit 110 is used to store the reconstructed video coding block for prediction reference.As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .

[0060] The video encoding method in the embodiment of the present application is mainly applied to the encoding control part in the encoder 10, such as including Figure 2 The coding unit (CU) division, intra-frame prediction unit 103, motion compensation unit 104 and motion estimation unit 105 are shown. That is, the video encoding method of the embodiment of the present application is mainly used to determine the encoding parameters so as to perform encoding according to the determined encoding parameters. The encoding parameters may include the CU division method, the intra-frame prediction mode or the inter-frame prediction mode of the CU.

[0061] Based on this, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Before the detailed description, it should be noted that the "first", "second", "third", etc. mentioned throughout the specification are only for distinguishing different features and do not have the function of limiting priority, sequence, size relationship, etc.

[0062] The embodiment of the present application provides a video encoding method, which is applied to a video encoding device, i.e., an encoder. The function implemented by the method can be implemented by a processor in the encoder calling a computer program, and of course the computer program can be stored in a memory. It can be seen that the encoder includes at least a processor and a memory.

[0063] See also Figure 3 , which shows a schematic flow chart of a video encoding method provided by an embodiment of the present application. Figure 3 As shown, the method may include:

[0064] S301: Determine pre-parameters of a video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters;

[0065] S302: determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0066] S303: determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion;

[0067] S304: determining a target distortion value according to the first distortion value and the second distortion value;

[0068] S305: Determine encoding parameters of the video to be encoded by using the target Lagrange multiplier and the target distortion value, and encode the video to be encoded.

[0069] It should be noted that the video encoding method of the embodiment of the present application can be applicable to the encoder of the H.266 / VVC standard, the encoder of the H.265 / HEVC standard, and even the encoder of other standards, such as the encoder of the first generation video coding standard (Alliance for Open Media Video 1, AV-1) developed by the Open Media Alliance, etc., and the embodiment of the present application does not impose any limitation.

[0070] It should also be noted that the rate-distortion optimization algorithm used in the video encoding method of the embodiment of the present application comprehensively considers the first distortion metric criterion based on the semantic distortion metric and the second distortion metric criterion based on the numerical error metric, so that the rate-distortion optimization can be a multi-distortion criterion rate-distortion optimization algorithm for human-machine vision. That is to say, in video encoding, in addition to the second Lagrange multiplier and the second distortion value derived by the relevant technical solution, for the human-machine vision application scenario of video semantic segmentation, the embodiment of the present application can also define a semantic distortion metric, and then derive the corresponding calculation formula of the first Lagrange multiplier and the first distortion value.

[0071] In this way, the target Lagrangian multiplier can be determined according to the first Lagrangian multiplier and the second Lagrangian multiplier, and the target distortion value can also be determined according to the first distortion value determined by the first distortion metric criterion and the second distortion value determined according to the second distortion metric criterion; in this way, after the encoding parameters of the video to be encoded are determined according to the target Lagrangian multiplier and the target distortion value, the encoding parameters are used to encode the video to be encoded, which can improve the semantic segmentation accuracy of the reconstructed video, improve the fidelity of the reconstructed video, and also reduce the encoding bit rate of the video, thereby shortening the time required for encoding and increasing the encoding speed, so as to improve the encoding efficiency.

[0072] It can be understood that, in a possible implementation manner, the pre-parameters of the video to be encoded may include a quantization parameter (Quantization Parameter, QP).

[0073] At this time, for S301, determining the pre-parameters of the video to be encoded may include:

[0074] Determine a quantization parameter of a coding unit in the video to be encoded; wherein the coding unit may include at least one of the following: an image, a slice, a sub-picture, a tile, and a coding block.

[0075] Here, the quantization parameter may be a quantization step size of a quantizer in the encoder, or an index number value corresponding to the quantization step size of a quantizer in the encoder.

[0076] Regarding the determination of the first Lagrangian multiplier, in some embodiments, for S301, the determining the first Lagrangian multiplier according to the pre-parameters may include:

[0077] Determining a first calculation model parameter, wherein the first calculation model represents a corresponding relationship between the first Lagrange multiplier and a quantization parameter;

[0078] The first Lagrange multiplier is determined according to the quantization parameter and the first calculation model.

[0079] It should be noted that, for the first computing model, determining the first computing model parameters may include:

[0080] In the first calculation model, the first Lagrange multiplier is set to be equal to a weighted value of an exponential power of the quantization parameter;

[0081] The first calculation model parameters include a first exponent parameter indicating the exponential power and a first weighting coefficient indicating the weighting.

[0082] It should be noted that the first Lagrange multiplier is λ miou Indicates that the quantization parameter is represented by QP, then the calculation formula of the first calculation model is as follows:

[0083] λ miou =2.30422*10 -8 *QP 6.3612072 (5)

[0084] Here, equation (5) is the first calculation model, which is used to express the corresponding relationship between the first Lagrange multiplier and the quantization parameter. The first calculation model parameters may include a first exponential parameter (i.e., 6.3612072 in the equation) and a first weighting coefficient (i.e., 2.30422*10 in the equation). -8 ).

[0085] Furthermore, the determination of the first calculation model parameters may be a preset value, or may be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0086] Optionally, in some embodiments, the method may further include:

[0087] The first calculation model parameter is set to a preset value.

[0088] Here, for the first calculation model parameter, the first exponential parameter can be set to 6.3612072, and the first weighting coefficient can be set to 2.30422*10 -8 After the first exponential parameter and the first weighting coefficient are determined, the first calculation model can be obtained so as to determine the first Lagrange multiplier according to the quantization parameter.

[0089] Optionally, in some embodiments, the method may further include:

[0090] Based on the test video, using the first distortion metric criterion, determine a first relationship function between the first distortion value and the bit rate of the test video, perform a derivative operation on the first relationship function, and determine a derivative function of the first relationship function;

[0091] Based on the test video, a second relationship function between the bit rate and the quantization parameter is determined, and the first calculation model parameter is determined according to the derivative function and the second relationship function.

[0092] It should be noted that the test video in the embodiment of the present application may be one or more test videos. For example, the test video may be a plurality of (for example, 59) test video sequences in a large-scale urban scene dataset (Cityscapes).

[0093] In this way, by using the first distortion metric criterion for the test video, a first relationship function between the first distortion value and the bit rate of the test video can be determined. The first relationship function is shown in the following formula:

[0094] D miou =0.2299*R -0.7553 +3.848 (6)

[0095] Among them, D miou represents the first distortion value, and R represents the bit rate.

[0096] It should be noted that in Figure 1 In the RD curve shown, λ miou The value of is the slope of the tangent line of the curve (λ miou >0), which is equal to the negative derivative of the curve. Based on the test video, the average bit rate of the reconstructed video under different quantization parameters can be calculated according to the encoded file. At this time, the first distortion value (D miou ) and the bit rate (R), the fitting curve is as follows Figure 4 shown.

[0097] By performing derivative operation on equation (6), the derivative function of the first relationship function can be obtained, and the derivative function is used to represent the corresponding relationship between the first Lagrange multiplier and the bit rate. Here, the derivative function is shown as follows:

[0098] λ miou =0.17364347*R -1.7553 (7)

[0099] In addition, according to the test video, a large amount of experimental test data can be used to determine the second relationship function between the bit rate (R) and the quantization parameter (QP) by fitting. The fitting curve is as follows: Figure 5 Here, the second relationship function is as follows:

[0100] R=8278*QP -3.624 (8)

[0101] According to equations (7) and (8), by substituting equation (8) into equation (7), we can get λ miou The functional relationship between θ and QP, that is, the corresponding relationship between the first Lagrange multiplier and the quantization parameter, gives the first calculation model shown in formula (5); thereby the first calculation model parameters are determined so that the first Lagrange multiplier can be determined according to the quantization parameter.

[0102] It should be noted that the calculation formula of the first Lagrange multiplier can also be modified to other functional forms. For example, the functional relationship of the above formula (8) can also be fitted to the e-exponential form, then the corresponding formula (5), i.e., the calculation formula of the first Lagrange multiplier, can also be the e-exponential form of QP, and no limitation is made here.

[0103] Regarding the determination of the second Lagrangian multiplier, in some embodiments, for S301, the determining the second Lagrangian multiplier according to the pre-parameters may include:

[0104] The second Lagrangian multiplier is determined according to a preset third calculation model, wherein the third calculation model represents a corresponding relationship between the second Lagrangian multiplier and a quantization parameter.

[0105] It should be noted that, for the third calculation model, it can be constructed using the SSE distortion criterion in the prior art. In some embodiments, determining the third calculation model parameters may include:

[0106] In the third calculation model, the second Lagrange multiplier is set to a weighted value equal to an exponential power of 2;

[0107] The third calculation model parameter includes a third exponential parameter indicating the exponential power and a third weighting coefficient indicating the weighting, and the third exponential parameter is related to the quantization parameter.

[0108] It should be noted that the second Lagrange multiplier is λ SSE Indicates that the quantization parameter is represented by QP, then the calculation formula of the third calculation model is as follows:

[0109] λ SSE =0.57*2 (QP-12) / 3 (9)

[0110] Here, formula (9) is the third calculation model, which is used to express the corresponding relationship between the second Lagrange multiplier and the quantization parameter. The third calculation model parameters may include a third exponential parameter (i.e., (QP-12) / 3 in the formula) and a third weighting coefficient (i.e., 0.57 in the formula), and the value of the third exponential parameter is related to the quantization parameter (QP).

[0111] It should also be noted that the determination of the third calculation model parameters can be preset values; or can be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0112] Furthermore, both the first Lagrangian multiplier and the second Lagrangian multiplier are related to the quantization parameter. Regarding the determination of the quantization parameter, in some embodiments, the determining of the quantization parameter of the coding unit in the to-be-encoded video may include: determining the quantization parameter of the coding unit in the to-be-encoded video using a rate control method.

[0113] Alternatively, in some embodiments, determining a quantization parameter of a coding unit in the to-be-encoded video may include: setting the quantization parameter to a preset value.

[0114] That is to say, for the quantization parameter in the video to be encoded, on the one hand, the quantization parameter can be set to a preset value, such as 22, 27, 32, 37, etc. On the other hand, the quantization parameter can also be determined by using the bit rate control method; specifically, the current bit rate control algorithm mainly controls the bit rate by adjusting the size of the quantization parameter; in this way, by controlling the size of the bit rate, the required quantization parameter can be obtained.

[0115] In another possible implementation manner, the pre-parameters of the video to be encoded may include a quantization parameter and a target bit rate.

[0116] At this time, for S301, determining the pre-parameters of the video to be encoded may include:

[0117] Determine a quantization parameter and a target bit rate of a coding unit in the video to be encoded; wherein the coding unit may include at least one of the following: an image, a slice, a sub-picture, a tile, and a coding block.

[0118] Here, the quantization parameter may be a quantization step size of a quantizer in the encoder, or an index number value corresponding to the quantization step size of a quantizer in the encoder.

[0119] Regarding the determination of the first Lagrangian multiplier, in some embodiments, for S301, the determining the first Lagrangian multiplier according to the pre-parameters may include:

[0120] Determining a second calculation model parameter, where the second calculation model represents a corresponding relationship between the first Lagrange multiplier and a bit rate;

[0121] The first Lagrange multiplier is determined according to the target bit rate and the second calculation model.

[0122] It should be noted that the determining the target bit rate of the coding unit in the video to be encoded may include: determining the target bit rate of the coding unit in the video to be encoded by using a bit allocation method.

[0123] That is, the target bit rate for the coding unit in the video to be encoded can be obtained by bit allocation. Here, the target bit rate can be dynamically adjusted according to the number of bits consumed by the coding unit in the video to be encoded to ensure the real-time and accuracy of bit allocation.

[0124] It should also be noted that, for the second computing model, determining the second computing model parameters may include:

[0125] In the second calculation model, the first Lagrange multiplier is set to a weighted value equal to an exponential power of the target bit rate;

[0126] The second calculation model parameters include a second exponent parameter indicating the exponential power and a second weighting coefficient indicating the weighting.

[0127] It should be noted that the first Lagrange multiplier is λ miou Indicated by , the code rate is represented by R, then the calculation formula of the second calculation model is as follows,

[0128] λ miou =0.17364347*R -1.7553 (10)

[0129] Here, formula (10) is the second calculation model, which is used to express the corresponding relationship between the first Lagrange multiplier and the bit rate. The second calculation model parameters may include a second exponential parameter (ie, -1.7553 in the formula) and a second weighting coefficient (ie, 0.17364347 in the formula).

[0130] Furthermore, the determination of the second calculation model parameters may be preset values, or may be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0131] Optionally, in some embodiments, the method may further include:

[0132] The second calculation model parameter is set to a preset value.

[0133] Here, for the second calculation model parameters, the second exponent parameter can be set to -1.7553, and the second weighting coefficient can be set to 0.17364347. After determining the second exponent parameter and the second weighting coefficient, the second calculation model can be obtained to determine the first Lagrange multiplier according to the target bit rate.

[0134] Optionally, in some embodiments, the method may further include:

[0135] Based on the test video, using the first distortion metric criterion, determining a first relationship function between the first distortion value and the bit rate of the test video;

[0136] Perform a derivative operation on the first relationship function to determine the second calculation model parameters.

[0137] It should be noted that the test video here may also be one or more test videos. For example, the test video may be a plurality of (for example, 59) test video sequences in a large-scale city scene dataset (Cityscapes).

[0138] In this way, the first distortion metric criterion is used for the test video to determine the first relationship function between the first distortion value and the bit rate of the test video, and the first relationship function is shown in the above formula (6). Then, the derivative operation is performed on formula (6) to obtain the derivative function of the first relationship function, and the derivative function is used to express the corresponding relationship between the first Lagrange multiplier and the bit rate, and the second calculation model shown in formula (10) is obtained; thereby, the second calculation model parameters are determined, so that the first Lagrange multiplier can be determined according to the quantization parameter.

[0139] It should be noted that when determining the first relationship function, based on Figure 1 The RD curve shown, λ miou The value of is the slope of the tangent line of the curve (λmiou >0), which is equal to the negative derivative of the curve. At this time, based on the test video, the average bit rate of the reconstructed video under different quantization parameters can be calculated according to the encoded file. At this time, the first distortion value (D miou ) and the bit rate (R), the fitting curve is as follows Figure 4 As shown, to obtain the first relationship function.

[0140] Regarding the determination of the second Lagrangian multiplier, in some embodiments, for S301, the determining the second Lagrangian multiplier according to the pre-parameters may include:

[0141] The second Lagrangian multiplier is determined according to a preset third calculation model, wherein the third calculation model represents a corresponding relationship between the second Lagrangian multiplier and a quantization parameter.

[0142] It should be noted that, for the third calculation model, it can be constructed using the SSE distortion criterion in the prior art. In some embodiments, determining the third calculation model parameters may include:

[0143] In the third calculation model, the second Lagrange multiplier is set to a weighted value equal to an exponential power of 2;

[0144] The third calculation model parameter includes a third exponential parameter indicating the exponential power and a third weighting coefficient indicating the weighting, and the third exponential parameter is related to the quantization parameter.

[0145] It should be noted that the second Lagrange multiplier is λ SSE Indicates that the quantization parameter is represented by QP, then the calculation formula of the third calculation model is as shown in the above formula (9).

[0146] Here, formula (9) is the third calculation model, which is used to express the corresponding relationship between the second Lagrange multiplier and the quantization parameter. The third calculation model parameters may include a third exponential parameter (i.e., (QP-12) / 3 in the formula) and a third weighting coefficient (i.e., 0.57 in the formula), and the value of the third exponential parameter is related to the quantization parameter (QP).

[0147] It should also be noted that the determination of the third calculation model parameters can be preset values; or can be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0148] Furthermore, both the first Lagrangian multiplier and the second Lagrangian multiplier are related to the quantization parameter. Regarding the determination of the quantization parameter, in some embodiments, the determining of the quantization parameter of the coding unit in the to-be-encoded video may include: determining the quantization parameter of the coding unit in the to-be-encoded video using a rate control method.

[0149] Alternatively, in some embodiments, determining a quantization parameter of a coding unit in the to-be-encoded video may include: setting the quantization parameter to a preset value.

[0150] That is to say, for the quantization parameter in the video to be encoded, on the one hand, the quantization parameter can be set to a preset value, such as 22, 27, 32, 37, etc. On the other hand, the quantization parameter can also be determined by using the bit rate control method; specifically, the current bit rate control algorithm mainly controls the bit rate by adjusting the size of the quantization parameter; in this way, by controlling the size of the bit rate, the required quantization parameter can be obtained.

[0151] Thus, for the rate-distortion optimization algorithm of the embodiment of the present application, in addition to determining the first Lagrangian multiplier and the second Lagrangian multiplier, it is also necessary to determine the distortion value. Here, it is assumed that the value obtained based on the first distortion metric is called the first distortion value, and the value obtained based on the second distortion metric is called the second distortion value.

[0152] It should be noted that the first distortion metric criterion can be a semantic distortion metric criterion. Taking the encoder of the H.266 / VVC standard as an example, in order to improve the semantic segmentation accuracy of the reconstructed video, it is first necessary to define a semantic distortion metric. Specifically, multiple quantization parameters can be selected, and then multiple (for example, 59) test video sequences in a large-scale urban scene dataset (Cityscapes) can be VVC encoded under random access (RA) conditions, and video semantic segmentation can be performed on the videos before and after encoding, so that the accuracy of the semantic segmentation results can be calculated based on the corresponding labeled data.

[0153] The accuracy of semantic segmentation can usually be measured using mean Intersection over Union (mIoU), which refers to the average value of the Intersection over Union (IoU) of all categories. Here, IoU is used as a detection evaluation function. Simply put, it is the overlap rate between the generated prediction window and the real window, that is, the ratio of the intersection of the detection result area (Detection Result) and the ground truth area (Ground Truth) to the union of the two, that is, the semantic accuracy (expressed by IoU).

[0154] In a possible implementation manner, for S303, determining the first distortion value according to the first distortion metric criterion may include:

[0155] Based on the test video, performing semantic segmentation on the test video to determine the semantic accuracy of one or more categories;

[0156] Determining a target semantic accuracy according to the semantic accuracy of the one or more categories;

[0157] The fourth computing model is used to perform distortion measurement on the target semantic accuracy to obtain the first distortion value.

[0158] Furthermore, determining the target semantic accuracy according to the semantic accuracy of one or more categories may include:

[0159] A weighted sum of the semantic accuracies of the one or more categories is calculated, and the obtained weighted sum is determined as the target semantic accuracy.

[0160] It should be noted that, when calculating the weighted sum of the semantic accuracy of one or more categories, a specific implementation method is to set the weight to 1. In this case, the average value of the semantic accuracy of the one or more categories is calculated, and the obtained average value is determined as the target semantic accuracy.

[0161] For example, in the semantic segmentation scenario, the two sets can represent the predicted value and the true value respectively, that is, A pred is the predicted segmentation result area, A true is the labeled segmentation result area; the semantic accuracy of each category can be expressed by IoU, and the calculation of IoU is as follows.

[0162]

[0163] After obtaining the IoU of each category, the target semantic accuracy can be obtained by taking the average value, which can be expressed as mIoU. Here, mIoU refers to the average IoU of all categories, and its value range is 0 to 1; the larger the value, the higher the semantic accuracy. For n categories of IoU, the calculation of mIoU is as follows.

[0164]

[0165] Among them, IoU i represents the IoU of the i-th category, i=1,...,n, and n represents the number of all categories.

[0166] It should also be noted that, for the fourth computing model, using the fourth computing model to perform distortion measurement on the target semantic accuracy to obtain the first distortion value includes:

[0167] Determining a fourth computing model parameter, wherein the fourth computing model represents a corresponding relationship between the first distortion value and the target semantic accuracy;

[0168] The first distortion value is obtained according to the target semantic accuracy and the fourth calculation model.

[0169] Further, the determining of the fourth calculation model parameter may include:

[0170] In the fourth computing model, the first distortion value is set to a weighted value equal to the logarithm of the target semantic accuracy;

[0171] The fourth calculation model parameters include a base parameter indicating the logarithm and a fourth weight coefficient parameter indicating the weighting.

[0172] Wherein, the fourth calculation model parameter is set to a preset value.

[0173] Here, for the scene of video semantic segmentation, a semantic distortion metric can be defined (i.e., the first distortion value, expressed as D miou The calculation formula is shown in the following formula (13).

[0174] D miou =-10*ln(mIoU) (13)

[0175] Wherein, formula (13) is the fourth calculation model, which is used to express the first distortion value (D miou ) and the target semantic accuracy (mIoU). The fourth calculation model parameter may include a base parameter (i.e., the base of ln in the formula is 10) and a fourth weighting coefficient parameter (i.e., -10 in the formula), where the fourth weighting coefficient parameter can also be regarded as a preset magnification of ln(mIoU).

[0176] It should also be noted that the determination of the fourth calculation model parameters may be preset values; or may be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0177] In addition, according to formula (13), the natural logarithm function maps the limited mIoU value to an infinite range, and the multiplied coefficient amplifies the obtained value to match the distortion size in the rate-distortion optimization algorithm. In this way, when mIoU approaches 0, D miou tends to infinity; when mIoU tends to 1, D miou Tends to 0.

[0178] In another possible implementation, the first distortion value may also be related to a target mean square error of a coding unit in the video to be encoded. Determining the first distortion value according to the first distortion metric criterion may include:

[0179] Determining a fifth calculation model parameter, wherein the fifth calculation model represents a third relationship function between the first distortion value and the mean square error;

[0180] Determine a target mean square error of a coding unit in the to-be-encoded video, and determine the first distortion value according to the target mean square error and the fifth calculation model.

[0181] It should be noted that the reconstructed video is obtained by decoding and reconstructing the encoded video. After encoding the test video using the quantization parameter to obtain the encoded video under the quantization parameter, video reconstruction can be performed on the encoded video under the quantization parameter to obtain the reconstructed video under the quantization parameter; based on the reconstructed video and the original video, the mean square error (MSE) of the reconstructed video under the quantization parameter can be obtained. Here, the mean square error refers to the expected value of the square of the difference between the predicted value of the parameter and the true value of the parameter. MSE can evaluate the degree of change of the data. The smaller the value of MSE, the better the accuracy of the prediction model in describing the experimental data.

[0182] It should also be noted that, for the fifth computing model, determining the fifth computing model parameters may include:

[0183] In the fifth calculation model, the first distortion value is set equal to the sum of the product of the target mean square error and the first parameter factor and the superposition of the second parameter factor;

[0184] The fifth calculation model parameter includes the first parameter factor and the second parameter factor.

[0185] It should be noted that the first distortion value is expressed as D miou Indicated by MSE, the calculation formula of the fifth calculation model is as follows:

[0186] D miou =0.6276*MSE+3.48 (14)

[0187] Here, formula (14) is the fifth calculation model, which is used to express the corresponding relationship between the first distortion value and the mean square error. The fifth calculation model parameters may include a first parameter factor (ie, 0.6276 in the formula) and a second parameter factor (ie, 3.48 in the formula).

[0188] Furthermore, the determination of the fifth calculation model parameter may be a preset value, or may be obtained by fitting based on a large amount of experimental test data of the test video, and no limitation is made here.

[0189] Optionally, in some embodiments, the method may further include:

[0190] The fifth calculation model parameter is set to a preset value.

[0191] Here, for the fifth calculation model parameters, the first parameter factor can be set to 0.6276, and the second parameter factor can be set to 3.48. After determining the first parameter factor and the second parameter factor, the fifth calculation model can be obtained to determine the first distortion value according to the target mean square error.

[0192] Optionally, in some embodiments, the method may further include:

[0193] Based on the test video, using the first distortion metric criterion, determining a third relationship function between the first distortion value and a mean square error of the test video;

[0194] The fifth calculation model parameter is determined according to the third relationship function.

[0195] It should be noted that the test video here may also be one or more test videos. For example, the test video may be a plurality of (for example, 59) test video sequences in a large-scale city scene dataset (Cityscapes).

[0196] In this way, the first distortion metric criterion is used for the test video, and the average MSE of the reconstructed video under different quantization parameters is calculated according to the encoded file. At this time, a large amount of experimental test data can be used to determine the first distortion value (D miou ) and MSE, the fitting curve is as follows Figure 6 As shown, and the fitting curve is linear, the fifth calculation model can be obtained.

[0197] It should be noted that the determining of the pre-parameters of the video to be encoded may also include: determining a target mean square error of the encoding unit in the video to be encoded. Thus, after the fifth calculation model is obtained, the first distortion value can be determined according to the target mean square error and the fifth calculation model shown in formula (14).

[0198] In addition, other methods can be used to determine the first distortion value, such as the difference between the semantic segmentation results of the video before and after encoding. Here, the first distortion value is determined according to the difference between the first mIoU value and the second mIoU value based on the semantic segmentation result of the video before encoding (the first mIoU value) and the semantic segmentation result of the video after encoding (the second mIoU value); this embodiment of the application does not specifically limit this.

[0199] It should also be noted that the second distortion metric criterion may be a numerical error criterion. In some embodiments, for S303, determining the second distortion value according to the second distortion metric criterion may include:

[0200] Determine a reconstruction value of a coding unit in the video, wherein the coding unit includes at least one of the following: a picture, a slice, a sub-picture, a tile, and a coding block;

[0201] Determining the second distortion value according to the reconstructed value and the original value of the coding unit based on the numerical error criterion;

[0202] The numerical error criterion is one of the following: Sum of Absolute Differences (SAD) criterion, Mean Absolute Deviation (MAD) criterion, Sum of Square Error (SSE) criterion, Mean-square error (MSE) criterion. It should be noted that the numerical error criterion is not limited to these criteria, but may also be other criteria, which are not specifically limited in the embodiments of the present application.

[0203] It should be noted that, taking the numerical error criterion as the SSE criterion as an example, the second distortion value is represented by SSE at this time, and its calculation formula is as follows:

[0204]

[0205] Wherein, M and N represent the horizontal spatial resolution and vertical spatial resolution of the video respectively, f(x, y) represents the original pixel value at the pixel position (x, y), and g(x, y) represents the reconstructed pixel value at the pixel position (x, y).

[0206] It can be understood that after obtaining the first Lagrange multiplier (λ miou ) and the second Lagrange multiplier (λ SSE ), and the first distortion value (D miou ) and the second distortion value (SSE), can be obtained by λ miou and λ SSECalculate the target Lagrange multiplier (expressed by λ) and use D miou The target distortion value (denoted by D) is calculated by SSE.

[0207] In some embodiments, for S302, determining the target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier may include:

[0208] Determine a first preset parameter; wherein the first preset parameter is used to control the weight values ​​corresponding to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0209] The first preset parameter is used to perform weighted calculation on the first Lagrangian multiplier and the second Lagrangian multiplier to obtain the target Lagrangian multiplier.

[0210] It should be noted that the first preset parameter can control the weight values ​​corresponding to the first Lagrangian multiplier and the second Lagrangian multiplier. Specifically, in some embodiments, determining the first preset parameter may include:

[0211] The first preset parameter is set according to the configuration information of the encoder.

[0212] Furthermore, the method may also include:

[0213] When the configuration information of the encoder indicates that the first preset parameter is equal to k, the target Lagrangian multiplier is set equal to the weighted sum of the first Lagrangian multiplier and the second Lagrangian multiplier, wherein k is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first Lagrangian multiplier is set equal to 1-k, and the weighting coefficient of the second Lagrangian multiplier is set equal to k.

[0214] It should be noted that, assuming that the first preset parameter is represented by k, when the weighting coefficient of the second Lagrangian multiplier is set to k, the weighting coefficient of the first Lagrangian multiplier can be set to 1-k; thus, the calculation formula of the target Lagrangian multiplier is as follows:

[0215] λ=k*λ SSE +(1-k)*λ miou (16)

[0216] Where λ represents the target Lagrange multiplier, λ miou represents the first Lagrange multiplier, λ SSE represents the second Lagrange multiplier; 1-k and k represent the weighting coefficients of the first Lagrange multiplier and the second Lagrange multiplier, respectively.

[0217] It should be noted that k can be a constant in the range of 0 to 1. Here, the value of k can be equal to 0.5, or 0.75, or a variable value (for example, obtained by performing some calculation on the current coding unit), which is not specifically limited in the embodiments of the present application. In general, the typical value of k can be equal to 0.75.

[0218] In some embodiments, for S304, determining a target distortion value according to the first distortion value and the second distortion value includes:

[0219] Determining a second preset parameter; wherein the second preset parameter is used to control a weight value corresponding to the first distortion value and the second distortion value;

[0220] The first distortion value and the second distortion value are weightedly calculated using the second preset parameter to obtain the target distortion value.

[0221] It should be noted that the second preset parameter may control the weight values ​​corresponding to the first distortion value and the second distortion value. Specifically, in some embodiments, determining the second preset parameter may include:

[0222] The second preset parameter is set according to the configuration information of the encoder.

[0223] Furthermore, the method may also include:

[0224] When the configuration information of the encoder indicates that the second preset parameter is equal to m, the target distortion value is set to be equal to the weighted sum of the first distortion value and the second distortion value, wherein m is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first distortion value is set to be equal to 1-m, and the weighting coefficient of the second distortion value is set to be equal to m.

[0225] It should be noted that, assuming that the second preset parameter is represented by m, when the weighting coefficient of the second distortion value is set to m, the weighting coefficient of the first distortion value can be set to 1-m; thus, the calculation formula of the target distortion value is as follows:

[0226] D=m*SSE+(1-m)*D miou (17)

[0227] Where D represents the target distortion value, D miou represents the first distortion value, SSE represents the second distortion value; 1-m and m represent the weighting coefficients of the first distortion value and the second distortion value respectively.

[0228] It should be noted that m can be a constant in the range of 0 to 1. Here, the value of m can be equal to 0.5, or 0.75, or a variable value (for example, obtained by performing some calculation on the current coding unit), which is not specifically limited in the embodiments of the present application. In general, the typical value of m can be equal to 0.75.

[0229] It should also be noted that the values ​​of the first preset parameter and the second preset parameter can be set to be the same or different. Generally speaking, the values ​​of the first preset parameter and the second preset parameter are the same, for example, both can be represented by θ. At this time, the calculation formula of the target Lagrange multiplier and the target distortion value can be as follows:

[0230] λ=θ*λ SSE +(1-θ)*λ miou (18)

[0231] D=θ*SSE+(1-θ)*D miou (19)

[0232] Among them, θ is a constant in the range of 0 to 1, which can be used not only to control the weight of the first Lagrangian multiplier and the second Lagrangian multiplier, but also to control the weight of the semantic distortion (i.e., the first distortion value) and the fidelity distortion (i.e., the second distortion value). Optionally, θ is usually set to 0.75.

[0233] In short, the existing video encoding methods are not well suited for machine video applications. miou represents the machine-oriented quality, λ SSE It represents the subjective quality of human eyes, and θ represents the adjustment between the subjective quality of human eyes and the quality for machines. For example, if θ is equal to 1, then the target distortion value at this time is completely the subjective quality of human eyes; if θ is equal to 0, then the target distortion value at this time is completely the quality for machines.

[0234] Here, the value of θ can be set through the configuration information of the encoder. Specifically, one implementation method is to set it directly according to the application requirements, such as the cases of 0 and 1 described above; another implementation method is that the encoder sets different working modes, for example, if it is set to work for the human eye, the encoder sets the value of θ to 1; if it is set to work for the machine, the encoder sets the value of θ to 0; if it is set to human-machine hybrid, the encoder adaptively determines the value of θ, such as using a pre-encoding method in the preprocessing stage to pre-encode the video to be encoded, and then estimating the value of θ from the pre-encoding result.

[0235] It should also be noted that after obtaining the target Lagrangian multiplier and the target distortion value, the encoding parameters of the video to be encoded can be determined according to the target Lagrangian multiplier and the target distortion value to encode the video to be encoded. In some embodiments, for S305, the use of the target Lagrangian multiplier and the target distortion value to determine the encoding parameters of the video to be encoded may include:

[0236] Constructing a rate-distortion cost function based on the target Lagrangian multiplier and the target distortion value;

[0237] Pre-encoding the video to be encoded using one or more candidate encoding parameters to determine rate-distortion cost values ​​corresponding to the one or more candidate encoding parameters;

[0238] A minimum rate-distortion cost value is selected from the determined rate-distortion cost values, and the candidate encoding parameters corresponding to the minimum rate-distortion cost value are determined as the encoding parameters of the video to be encoded.

[0239] Here, the encoding parameters at least include parameters indicating the division method of the video to be encoded and parameters for constructing prediction values ​​of encoding blocks in the video to be encoded.

[0240] Further, in some embodiments, encoding the video to be encoded may include: writing the encoding parameters into a bitstream.

[0241] It should be pointed out that, according to the target Lagrange multiplier and the target distortion value, a rate-distortion cost function can be constructed; then, one or more candidate coding parameters are used to pre-encode the video to be encoded, so as to determine the rate-distortion cost values ​​corresponding to the one or more candidate coding parameters; then, the minimum rate-distortion cost value is selected from the determined rate-distortion cost values, and the candidate coding parameters corresponding to the minimum rate-distortion cost value are determined as the coding parameters of the video to be encoded. At this time, the determined coding parameters are the optimal coding parameters (with the minimum rate-distortion cost), and then encoding is performed based on them; in this process, the coding parameters can also be written into the bitstream so that they can be transmitted from the encoder to the decoder, so as to restore the original video to be encoded on the decoder side.

[0242] In the embodiment of the present application, taking the VVC standard encoder as an example, in order to improve the semantic accuracy of the reconstructed video while ensuring the fidelity of the video and meet the subjective viewing requirements in the human-machine vision scene, the embodiment of the present application modifies the distortion criterion in the VVC rate-distortion optimization process to the semantic distortion D miou and the weighting of the fidelity distortion SSE, as shown in the above equation (17) or equation (19); the corresponding target Lagrange multiplier is modified to λ miou and λ SSE, as shown in the above formula (16) or formula (18), so that according to the rate-distortion optimization algorithm of the multi-distortion criterion for human-computer vision, the rate-distortion process of the VVC standard encoder can be optimized to improve the semantic segmentation accuracy of the reconstructed video under a certain bit rate while maintaining good fidelity performance.

[0243] Exemplarily, based on the VVC reference software test platform (VVC TEST MODE, VTM), assuming that it is implemented on VTM7.1, different QPs are selected at this time, the test video sequences in the large-scale urban scene dataset are encoded under RA conditions, and the reconstructed video is tested for semantic segmentation.

[0244] First, different QPs are selected, and the test video is encoded through the VVC standard encoder to obtain the encoding bit rate and the PSNR of the reconstructed video, and the reconstructed video is semantically segmented and the accuracy of the segmentation is calculated. Then, the rate-distortion process in the VVC is optimized according to the video encoding method of the embodiment of the present application, and then different QPs are selected, and the test video is encoded through the optimized encoder to obtain the encoding bit rate, and the encoded reconstructed video is semantically segmented and the accuracy of the segmentation is calculated. Based on the experimental results in these two cases, the BD-rate and BD-miou of the reconstructed video compared to the VVC standard encoder can be calculated, and the performance of the video encoding method of the embodiment of the present application and the VVC standard encoder in terms of video semantic accuracy at the same bit rate can be measured.

[0245] Specifically, when QP is 22, 27, 32, and 37, the BD-miou and BD-rate of the reconstructed video of the embodiment of the present application compared with the reconstructed video of the VVC standard encoder are calculated according to the experimental results, and the performance of the video encoding method of the embodiment of the present application in terms of semantic accuracy can be measured. The experimental results are shown in Table 1. Here, BD-miou represents the improvement of the semantic accuracy of the reconstructed video under the same bit rate. BD-miou is greater than 0, indicating that the semantic accuracy has been improved; BD-miou is less than 0, indicating that the semantic accuracy has decreased; BD-rate represents the increase in the encoding bit rate under the same semantic accuracy. BD-rate is greater than 0, indicating that the bit rate has increased; BD-rate is less than 0, indicating that the bit rate has decreased, that is, the encoding efficiency has been improved.

[0246] In terms of video fidelity, the PSNR and encoding bit rate of the reconstructed video are counted from the encoded file. According to the experimental results, the BD-rate and BD-PSNR of the reconstructed video compared with the VVC standard encoder are obtained, which can measure the performance of the video encoding method of the embodiment of the present application compared with the VVC standard encoder in terms of video fidelity at the same bit rate.

[0247] Specifically, the BD-PSNR and BD-rate of the reconstructed video of the embodiment of the present application compared with the reconstructed video of the VVC standard encoder are calculated according to the experimental results, and the performance of the video encoding method of the embodiment of the present application in terms of fidelity can be measured. The experimental results are shown in Table 2. Here, BD-PSNR represents the increase in the fidelity of the reconstructed video under the same bit rate. If BD-PSNR is greater than 0, it indicates that the fidelity has increased; if BD-PSNR is less than 0, it indicates that the fidelity has decreased; BD-rate represents the increase in the encoding bit rate under the same fidelity. If BD-rate is greater than 0, it indicates that the bit rate has increased; if BD-rate is less than 0, it indicates that the bit rate has decreased, that is, the encoding efficiency has been improved.

[0248] Table 1

[0249]

[0250] Table 2

[0251]

[0252] Furthermore, according to the above experimental results, the video encoding method according to the embodiment of the present application can achieve the following technical benefits:

[0253] According to Table 1, the BD-miou obtained according to the experimental results is 0.0112, which means that under the same bit rate, the video encoding method of the embodiment of the present application can improve the semantic segmentation accuracy of the reconstructed video. In addition, under the condition of a lower bit rate, the overall semantic effect of the embodiment of the present application is better than that of the VVC standard encoder. In other words, the embodiment of the present application can improve the semantic segmentation accuracy of the reconstructed video under the same bit rate.

[0254] According to Table 1, the BD-rate obtained according to the experimental results is -24.8673, which means that under the same semantic accuracy, the video encoding method of the embodiment of the present application can reduce the encoding bit rate of the video. In other words, the embodiment of the present application can reduce the bit rate under the same semantic accuracy.

[0255] According to Table 2, the BD-PSNR obtained according to the experimental results is 0.0316, which means that under the same bit rate, the video encoding method of the embodiment of the present application can improve the fidelity of the reconstructed video. In other words, the embodiment of the present application can improve the fidelity of the reconstructed video under the same bit rate.

[0256] According to Table 2, the BD-rate obtained according to the experimental results is -1.0836, which means that under the same fidelity, the video encoding method of the embodiment of the present application can reduce the encoding bit rate of the video. In other words, the embodiment of the present application can reduce the bit rate under the same fidelity.

[0257] In addition, after using the video encoding method of the embodiment of the present application, the PSNR performance of the reconstructed video has not substantially decreased compared to the VVC standard encoder. In the case of a lower bit rate, the subjective performance of the embodiment of the present application is better than that of VVC, which shows that the video encoding method of the embodiment of the present application can ensure the fidelity of the reconstructed video while improving the semantic accuracy and meet the subjective viewing requirements of the video. In other words, the embodiment of the present application can ensure the fidelity of the video while improving the semantic effect.

[0258] It should also be noted that the video encoding method of the embodiment of the present application optimizes the rate distortion process in VVC, does not change the video encoding and decoding process and the bitstream structure, and therefore does not increase the complexity of encoding and decoding. Moreover, the video encoding method of the embodiment of the present application can also reduce the encoding bit rate of the video, thereby shortening the time required for encoding and increasing the encoding speed.

[0259] That is to say, for the application scenario of human-computer vision such as video semantic segmentation, the embodiment of the present application defines a semantic distortion metric and derives the corresponding first Lagrangian multiplier. The weights of the semantic distortion and SSE distortion, as well as the weights of the first Lagrangian multiplier and the second Lagrangian multiplier are adjusted by preset parameters (including first preset parameters and second preset parameters), thereby optimizing the rate-distortion process of video encoding, so as to improve the semantic segmentation accuracy of the reconstructed video under a certain bit rate, and also maintain good fidelity performance.

[0260] The present embodiment provides a video encoding method, which is applied to an encoder. Pre-parameters of a video to be encoded are determined, and a first Lagrangian multiplier and a second Lagrangian multiplier are determined according to the pre-parameters; a target Lagrangian multiplier is determined according to the first Lagrangian multiplier and the second Lagrangian multiplier; a first distortion value is determined according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; a second distortion value is determined according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; a target distortion value is determined according to the first distortion value and the second distortion value; the encoding parameters of the video to be encoded are determined using the target Lagrangian multiplier and the target distortion value, and the video to be encoded is encoded. In this way, the first distortion metric based on semantic distortion metric and the second distortion metric based on numerical error metric are comprehensively considered in video coding for rate-distortion optimization, which can be well adapted to application scenarios for machine vision and human-computer vision. Moreover, under a certain bit rate, the semantic segmentation accuracy of the reconstructed video can be improved while maintaining good fidelity performance, thereby improving the coding efficiency.

[0261] Based on the same inventive concept as the above embodiments, see Figure 7 , which shows a schematic diagram of the structure of an encoder 70 provided in an embodiment of the present application. Figure 7 As shown, the encoder 70 may include: a determination unit 701, a calculation unit 702 and an encoding unit 703; wherein,

[0262] The determining unit 701 is configured to determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters;

[0263] A calculation unit 702 is configured to determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0264] The determining unit 701 is further configured to determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion;

[0265] The calculating unit 702 is further configured to determine a target distortion value according to the first distortion value and the second distortion value;

[0266] The encoding unit 703 is configured to determine the encoding parameters of the video to be encoded by using the target Lagrange multiplier and the target distortion value, and encode the video to be encoded.

[0267] In some embodiments, the pre-parameters include quantization parameters, and the determination unit 701 is further configured to determine the quantization parameters of the coding units in the video to be encoded, wherein the coding units include at least one of the following: images, slices, sub-images, tiles, and coding blocks.

[0268] In some embodiments, the determination unit 701 is further configured to determine a first calculation model parameter, where the first calculation model represents the correspondence between the first Lagrangian multiplier and the quantization parameter; and determine the first Lagrangian multiplier according to the quantization parameter and the first calculation model.

[0269] In some embodiments, the determination unit 701 is further configured to set the first Lagrange multiplier to a weighted value equal to the exponential power of the quantization parameter in the first computing model; wherein the first computing model parameters include a first exponential parameter indicating the exponential power and a first weighting coefficient indicating the weighting.

[0270] In some embodiments, the determination unit 701 is further configured to set the first computing model parameter to a preset value.

[0271] In some embodiments, the determination unit 701 is further configured to determine, based on a test video, a first relationship function between the first distortion value and the bit rate of the test video using the first distortion metric criterion, perform a derivative operation on the first relationship function, and determine a derivative function of the first relationship function; and determine, based on the test video, a second relationship function between the bit rate and the quantization parameter, and determine the first calculation model parameter according to the derivative function and the second relationship function.

[0272] In some embodiments, the pre-parameters include quantization parameters and target bit rate, and the determination unit 701 is further configured to determine the quantization parameters and target bit rate of the coding unit in the video to be encoded, wherein the coding unit includes at least one of the following: image, slice, sub-image, tile, coding block.

[0273] In some embodiments, the determination unit 701 is further configured to determine a second calculation model parameter, where the second calculation model represents the corresponding relationship between the first Lagrangian multiplier and the bit rate; and determine the first Lagrangian multiplier according to the target bit rate and the second calculation model.

[0274] In some embodiments, the determination unit 701 is further configured to set the first Lagrange multiplier to a weighted value equal to the exponential power of the target bit rate in the second computing model; wherein the second computing model parameters include a second exponential parameter indicating the exponential power and a second weighting coefficient indicating the weighting.

[0275] In some embodiments, the determination unit 701 is further configured to set the second computing model parameter to a preset value.

[0276] In some embodiments, the determination unit 701 is further configured to determine, based on the test video, a first relationship function between the first distortion value and the bit rate of the test video using the first distortion metric criterion; and perform a derivative operation on the first relationship function to determine the second calculation model parameters.

[0277] In some embodiments, the determination unit 701 is further configured to determine a target bit rate of a coding unit in the to-be-encoded video by using a bit allocation method.

[0278] In some embodiments, the determination unit 701 is further configured to determine the second Lagrangian multiplier according to a preset third calculation model; wherein the third calculation model represents the corresponding relationship between the second Lagrangian multiplier and the quantization parameter.

[0279] In some embodiments, the determination unit 701 is further configured to determine the quantization parameter of the coding unit in the to-be-encoded video by using a rate control method.

[0280] In some embodiments, the determination unit 701 is further configured to set the quantization parameter to a preset value.

[0281] In some embodiments, the determination unit 701 is further configured to perform semantic segmentation on the test video based on the test video to determine the semantic accuracy of one or more categories; and determine the target semantic accuracy according to the semantic accuracy of the one or more categories;

[0282] The calculation unit 702 is further configured to perform distortion measurement on the target semantic accuracy by using a fourth calculation model to obtain the first distortion value.

[0283] In some embodiments, the calculation unit 702 is further configured to calculate a weighted sum of the semantic accuracy of the one or more categories, and determine the obtained weighted sum as the target semantic accuracy.

[0284] In some embodiments, the determination unit 701 is further configured to determine the fourth computing model parameters, wherein the fourth computing model represents the correspondence between the first distortion value and the target semantic accuracy; and obtain the first distortion value according to the target semantic accuracy and the fourth computing model.

[0285] In some embodiments, the determination unit 701 is further configured to set the first distortion value to a weighted value equal to the logarithm of the target semantic accuracy in the fourth computing model; wherein the fourth computing model parameters include a base parameter indicating the logarithm and a fourth weighting coefficient parameter indicating the weighting.

[0286] In some embodiments, the determination unit 701 is further configured to set the fourth calculation model parameter to a preset value.

[0287] In some embodiments, the determination unit 701 is also configured to determine a fifth calculation model parameter, wherein the fifth calculation model represents a third relationship function between the first distortion value and the mean square error; and determine a target mean square error of the coding unit in the video to be encoded, and determine the first distortion value based on the target mean square error and the fifth calculation model.

[0288] In some embodiments, the determination unit 701 is further configured to set the first distortion value to be equal to the product of the target mean square error and the first parameter factor and superimposed with the sum of the second parameter factor in the fifth calculation model; wherein the fifth calculation model parameters include the first parameter factor and the second parameter factor.

[0289] In some embodiments, the determination unit 701 is further configured to set the fifth computing model parameter to a preset value.

[0290] In some embodiments, the determination unit 701 is further configured to determine, based on the test video, a third relationship function between the first distortion value and the mean square error of the test video using the first distortion metric criterion; and determine the fifth calculation model parameters according to the third relationship function.

[0291] In some embodiments, the determining unit 701 is further configured to determine a reconstruction value of a coding unit in the video, wherein the coding unit includes at least one of the following: an image, a slice, a sub-image, a tile, and a coding block;

[0292] The calculation unit 702 is also configured to determine the second distortion value based on the numerical error criterion, according to the reconstructed value and the original value of the coding unit; wherein the numerical error criterion is one of the following: absolute error sum criterion, mean absolute error criterion, error square sum criterion, mean square error criterion.

[0293] In some embodiments, the determination unit 701 is further configured to determine a first preset parameter; wherein the first preset parameter is used to control the weight values ​​corresponding to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0294] The calculation unit 702 is further configured to perform weighted calculation on the first Lagrangian multiplier and the second Lagrangian multiplier using the first preset parameter to obtain the target Lagrangian multiplier.

[0295] In some embodiments, see Figure 7 The encoder 70 may further include a configuration unit 704 configured to set the first preset parameter according to the configuration information of the encoder.

[0296] In some embodiments, the configuration unit 704 is further configured to, when the configuration information of the encoder indicates that the first preset parameter is equal to k, set the target Lagrangian multiplier to be equal to the weighted sum of the first Lagrangian multiplier and the second Lagrangian multiplier, wherein k is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first Lagrangian multiplier is set to be equal to 1-k, and the weighting coefficient of the second Lagrangian multiplier is set to be equal to k.

[0297] In some embodiments, the value of k is equal to 0.75.

[0298] In some embodiments, the determination unit 701 is further configured to determine a second preset parameter; wherein the second preset parameter is used to control a weight value corresponding to the first distortion value and the second distortion value;

[0299] The calculation unit 702 is further configured to perform weighted calculation on the first distortion value and the second distortion value by using the second preset parameter to obtain the target distortion value.

[0300] In some embodiments, the configuration unit 704 is further configured to set the second preset parameter according to the configuration information of the encoder.

[0301] In some embodiments, the configuration unit 704 is further configured to set the target distortion value to be equal to the weighted sum of the first distortion value and the second distortion value when the configuration information of the encoder indicates that the second preset parameter is equal to m, wherein m is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first distortion value is set to be equal to 1-m, and the weighting coefficient of the second distortion value is set to be equal to m.

[0302] In some embodiments, the value of m is equal to 0.75.

[0303] In some embodiments, the determination unit 701 is further configured to construct a rate-distortion cost function based on the target Lagrangian multiplier and the target distortion value; perform pre-coding processing on the video to be encoded using one or more candidate coding parameters to determine rate-distortion cost values ​​corresponding to the one or more candidate coding parameters; and select a minimum rate-distortion cost value from the determined rate-distortion cost values, and determine the candidate coding parameter corresponding to the minimum rate-distortion cost value as the coding parameter of the video to be encoded.

[0304] In some embodiments, the encoding parameters include at least parameters indicating the division method of the video to be encoded and parameters for constructing prediction values ​​of encoding blocks in the video to be encoded.

[0305] In some embodiments, see Figure 7 The encoder 70 may further include a writing unit 705 configured to write the encoding parameters into a bitstream.

[0306] It is understandable that in the embodiments of the present application, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course, it may be a module, or it may be non-modular. Moreover, the components in the present embodiment may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional module.

[0307] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0308] Therefore, an embodiment of the present application provides a computer storage medium, which is applied to an encoder 70, and the computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0309] Based on the composition of the encoder 70 and the computer storage medium, see Figure 8 , which shows a specific hardware structure example of the encoder 70 provided in the embodiment of the present application, which may include: a communication interface 801, a memory 802 and a processor 803; each component is coupled together through a bus system 804. It can be understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 7 Various buses are labeled as bus system 804. Among them, the communication interface 801 is used for receiving and sending signals in the process of sending and receiving information between other external network elements;

[0310] A memory 802, used to store a computer program that can be run on the processor 803;

[0311] The processor 803 is configured to execute, when running the computer program:

[0312] Determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters;

[0313] Determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier;

[0314] Determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion comprises a semantic distortion metric criterion;

[0315] Determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion;

[0316] Determining a target distortion value according to the first distortion value and the second distortion value;

[0317] The target Lagrange multiplier and the target distortion value are used to determine the encoding parameters of the video to be encoded, and the video to be encoded is encoded.

[0318] It can be understood that the memory 802 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 802 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0319] The processor 803 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 803. The above processor 803 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 802, and the processor 803 reads the information in the memory 802 and completes the steps of the above method in combination with its hardware.

[0320] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0321] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0322] Optionally, as another embodiment, the processor 803 is further configured to execute the steps of the method in any one of the aforementioned embodiments when running the computer program.

[0323] This embodiment provides an encoder, which includes a determination unit, a calculation unit and an encoding unit; wherein the determination unit is configured to determine pre-parameters of a video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; the calculation unit is configured to determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; and the determination unit is further configured to determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; the calculation unit is further configured to determine a target distortion value according to the first distortion value and the second distortion value; the encoding unit is configured to determine the encoding parameters of the video to be encoded using the target Lagrangian multiplier and the target distortion value, and encode the video to be encoded. In this way, the first distortion metric based on semantic distortion metric and the second distortion metric based on numerical error metric are comprehensively considered in video coding for rate-distortion optimization, which can be well adapted to application scenarios for machine vision and human-computer vision. Moreover, under a certain bit rate, the semantic segmentation accuracy of the reconstructed video can be improved while maintaining good fidelity performance, thereby improving the coding efficiency.

[0324] See also Fig. 9 , which shows a schematic diagram of the composition structure of a video system provided by an embodiment of the present application. Fig. 9 As shown, the video system 90 may include an encoder 901 and a decoder 902. The encoder 901 may be the encoder 70 described in any one of the above embodiments.

[0325] The encoder 901 is configured to determine pre-parameters of a video to be encoded, determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; and determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; and determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; and determine a target distortion value according to the first distortion value and the second distortion value; and determine encoding parameters of the video to be encoded using the target Lagrangian multiplier and the target distortion value, encode the video to be encoded to generate a bitstream, and transmit the bitstream to the decoder;

[0326] The decoder 902 is configured to parse the bitstream to obtain a decoded video.

[0327] Furthermore, in some embodiments, the decoder 902 is also configured to parse the bitstream, obtain decoding parameters, and obtain the decoded video according to the decoding parameters; wherein the decoding parameters at least include parameters indicating the division method of the video to be decoded and parameters for constructing the predicted value of the decoding block in the video to be decoded.

[0328] In an embodiment of the present application, the video system 90 comprehensively considers a first distortion metric criterion based on a semantic distortion metric and a second distortion metric criterion based on a numerical error metric to perform rate-distortion optimization in video encoding, which can be well adapted to application scenarios for machine vision and human-machine vision. Moreover, under a certain bit rate, it can improve the semantic segmentation accuracy of the reconstructed video while maintaining good fidelity performance, thereby improving the encoding efficiency.

[0329] It should also be noted that in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0330] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0331] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0332] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0333] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0334] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0335] Industrial Applicability

[0336] In the embodiment of the present application, by determining the pre-parameters of the video to be encoded, according to the pre-parameters, the first Lagrangian multiplier and the second Lagrangian multiplier are determined; according to the first Lagrangian multiplier and the second Lagrangian multiplier, the target Lagrangian multiplier is determined; according to the first distortion metric criterion, the first distortion value is determined, wherein the first distortion metric criterion includes the semantic distortion metric criterion; according to the second distortion metric criterion, the second distortion value is determined, wherein the second distortion metric criterion includes the numerical error metric criterion; according to the first distortion value and the second distortion value, the target distortion value is determined; using the target Lagrangian multiplier and the target distortion value, the encoding parameters of the video to be encoded are determined, and the video to be encoded is encoded. In this way, in video encoding, the first distortion metric criterion based on the semantic distortion metric and the second distortion metric criterion based on the numerical error metric are comprehensively considered for rate-distortion optimization, which can be well adapted to application scenarios for machine vision and human-machine vision, and under a certain bit rate, the semantic segmentation accuracy of the reconstructed video can be improved, while maintaining good fidelity performance, thereby also improving the encoding efficiency.

Claims

1. A video encoding method, applied to an encoder, the method comprising: Determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; Determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; Determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion comprises a semantic distortion metric criterion; Determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; Determining a target distortion value according to the first distortion value and the second distortion value; Determine the encoding parameters of the video to be encoded by using the target Lagrange multiplier and the target distortion value, and encode the video to be encoded; The determining the first distortion value according to the first distortion metric criterion includes: Based on the test video, performing semantic segmentation on the test video to determine the semantic accuracy of one or more categories; Determining a target semantic accuracy based on the semantic accuracy of the one or more categories; The target semantic accuracy is distorted by using a fourth computing model to obtain the first distortion value; wherein the semantic accuracy represents the ratio of the intersection and union of the detection result area and the true value area of ​​the corresponding category in the test video, and the fourth computing model represents the corresponding relationship between the first distortion value and the target semantic accuracy.

2. The method according to claim 1, wherein: The pre-parameters include quantization parameters, and the determining of the pre-parameters of the video to be encoded includes: Determine a quantization parameter of a coding unit in the to-be-coded video, wherein the coding unit includes at least one of the following: an image, a slice, a sub-image, a tile, and a coding block.

3. The method according to claim 2, wherein: Determining the first Lagrange multiplier according to the pre-parameters includes: Determining a first calculation model parameter, wherein the first calculation model represents a corresponding relationship between the first Lagrange multiplier and a quantization parameter; The first Lagrange multiplier is determined according to the quantization parameter and the first calculation model.

4. The method according to claim 3, wherein: The determining of the first computing model parameter comprises: In the first calculation model, the first Lagrange multiplier is set to be equal to a weighted value of an exponential power of the quantization parameter; The first calculation model parameters include a first exponent parameter indicating the exponential power and a first weighting coefficient indicating the weighting.

5. The method according to claim 4, wherein: The method further comprises: The first calculation model parameter is set to a preset value.

6. The method according to claim 4, wherein: The method further comprises: Based on the test video, using the first distortion metric criterion, determine a first relationship function between the first distortion value and the bit rate of the test video, perform a derivative operation on the first relationship function, and determine a derivative function of the first relationship function; Based on the test video, a second relationship function between the bit rate and the quantization parameter is determined, and the first calculation model parameter is determined according to the derivative function and the second relationship function.

7. The method according to claim 1, wherein: The pre-parameters include a quantization parameter and a target bit rate, and the pre-parameters of the video to be encoded are determined, including: Determine a quantization parameter and a target bit rate of a coding unit in the to-be-coded video, wherein the coding unit includes at least one of the following: an image, a slice, a sub-image, a tile, and a coding block.

8. The method according to claim 7, wherein: Determining the first Lagrange multiplier according to the pre-parameters includes: Determining a second calculation model parameter, where the second calculation model represents a corresponding relationship between the first Lagrange multiplier and a bit rate; The first Lagrange multiplier is determined according to the target bit rate and the second calculation model.

9. The method according to claim 8, wherein: The determining of the second computing model parameter comprises: In the second calculation model, the first Lagrange multiplier is set to a weighted value equal to an exponential power of the target bit rate; The second calculation model parameters include a second exponent parameter indicating the exponential power and a second weighting coefficient indicating the weighting.

10. The method according to claim 9, wherein: The method further comprises: The second calculation model parameter is set to a preset value.

11. The method according to claim 9, wherein: The method further comprises: Based on the test video, using the first distortion metric criterion, determining a first relationship function between the first distortion value and the bit rate of the test video; Perform a derivative operation on the first relationship function to determine the second calculation model parameters.

12. The method according to claim 7, wherein: The determining a target bit rate of a coding unit in the to-be-encoded video includes: A target bit rate of a coding unit in the video to be encoded is determined by using a bit allocation method.

13. The method according to claim 2 or 7, wherein: Determining the second Lagrange multiplier according to the pre-parameters includes: The second Lagrangian multiplier is determined according to a preset third calculation model, wherein the third calculation model represents a corresponding relationship between the second Lagrangian multiplier and a quantization parameter.

14. The method according to claim 2 or 7, wherein: The determining of the quantization parameter of the coding unit in the to-be-encoded video includes: The quantization parameter of the coding unit in the to-be-encoded video is determined by using a rate control method.

15. The method according to claim 2 or 7, wherein: The determining of the quantization parameter of the coding unit in the to-be-encoded video includes: The quantization parameter is set to a preset value.

16. The method according to claim 1, wherein: The step of determining the target semantic accuracy according to the semantic accuracy of one or more categories includes: A weighted sum of the semantic accuracies of the one or more categories is calculated, and the obtained weighted sum is determined as the target semantic accuracy.

17. The method according to claim 1, wherein: The using the fourth computing model to perform distortion measurement on the target semantic accuracy to obtain the first distortion value includes: Determining the fourth computing model parameters; The first distortion value is obtained according to the target semantic accuracy and the fourth calculation model.

18. The method according to claim 17, wherein: The determining of the fourth calculation model parameter comprises: In the fourth computing model, the first distortion value is set to a weighted value equal to the logarithm of the target semantic accuracy; The fourth calculation model parameters include a base parameter indicating the logarithm and a fourth weight coefficient parameter indicating the weighting.

19. The method according to claim 18, wherein: The method further comprises: The fourth calculation model parameter is set to a preset value.

20. The method according to claim 1, wherein: The determining the first distortion value according to the first distortion metric criterion includes: Determining a fifth calculation model parameter, wherein the fifth calculation model represents a third relationship function between the first distortion value and the mean square error; Determine a target mean square error of a coding unit in the to-be-encoded video, and determine the first distortion value according to the target mean square error and the fifth calculation model.

21. The method according to claim 20, wherein: The determining of the fifth computing model parameter comprises: In the fifth calculation model, the first distortion value is set equal to the sum of the product of the target mean square error and the first parameter factor and the superposition of the second parameter factor; The fifth calculation model parameter includes the first parameter factor and the second parameter factor.

22. The method according to claim 21, wherein: The method further comprises: The fifth calculation model parameter is set to a preset value.

23. The method according to claim 21, wherein: The method further comprises: Based on the test video, using the first distortion metric criterion, determining a third relationship function between the first distortion value and a mean square error of the test video; The fifth calculation model parameter is determined according to the third relationship function.

24. The method according to claim 1, wherein: The determining the second distortion value according to the second distortion metric criterion includes: Determining a reconstruction value of a coding unit in the video, wherein the coding unit includes at least one of the following: an image, a slice, a sub-image, a tile, and a coding block; Determining the second distortion value based on the numerical error metric and the reconstructed value and the original value of the coding unit; The numerical error measurement criterion is one of the following: absolute error sum criterion, mean absolute error criterion, error square sum criterion, and mean square error criterion.

25. The method of claim 1, wherein: The step of determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier comprises: Determine a first preset parameter; wherein the first preset parameter is used to control the weight values ​​corresponding to the first Lagrangian multiplier and the second Lagrangian multiplier; The first preset parameter is used to perform weighted calculation on the first Lagrangian multiplier and the second Lagrangian multiplier to obtain the target Lagrangian multiplier.

26. The method according to claim 25, wherein: The determining of the first preset parameter comprises: The first preset parameter is set according to the configuration information of the encoder.

27. The method according to claim 25, wherein: The method further comprises: When the configuration information of the encoder indicates that the first preset parameter is equal to k, the target Lagrangian multiplier is set to be equal to the weighted sum of the first Lagrangian multiplier and the second Lagrangian multiplier, wherein k is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first Lagrangian multiplier is set to be equal to 1-k, and the weighting coefficient of the second Lagrangian multiplier is set to be equal to k.

28. The method according to claim 27, wherein: The value of k is equal to 0.

75.

29. The method of claim 1, wherein: The determining a target distortion value according to the first distortion value and the second distortion value includes: Determining a second preset parameter; wherein the second preset parameter is used to control a weight value corresponding to the first distortion value and the second distortion value; The first distortion value and the second distortion value are weightedly calculated using the second preset parameter to obtain the target distortion value.

30. The method of claim 29, wherein: The determining of the second preset parameter comprises: The second preset parameter is set according to the configuration information of the encoder.

31. The method according to claim 30, wherein: The method further comprises: When the configuration information of the encoder indicates that the second preset parameter is equal to m, the target distortion value is set to be equal to the weighted sum of the first distortion value and the second distortion value, wherein m is any value greater than or equal to 0 and less than or equal to 1, the weighting coefficient of the first distortion value is set to be equal to 1-m, and the weighting coefficient of the second distortion value is set to be equal to m.

32. The method according to claim 31, wherein: The value of m is equal to 0.

75.

33. The method of claim 1, wherein: The step of determining the encoding parameters of the to-be-encoded video by using the target Lagrange multiplier and the target distortion value includes: Constructing a rate-distortion cost function based on the target Lagrangian multiplier and the target distortion value; Pre-encoding the video to be encoded using one or more candidate encoding parameters to determine rate-distortion cost values ​​corresponding to the one or more candidate encoding parameters; A minimum rate-distortion cost value is selected from the determined rate-distortion cost values, and the candidate encoding parameters corresponding to the minimum rate-distortion cost value are determined as the encoding parameters of the video to be encoded.

34. The method according to claim 1 or 33, characterized in that The encoding parameters at least include parameters indicating the division method of the video to be encoded and parameters for constructing prediction values ​​of encoding blocks in the video to be encoded.

35. The method of claim 34, wherein: The encoding of the video to be encoded comprises: The encoding parameters are written into the bitstream.

36. An encoder, comprising a determination unit, a calculation unit and an encoding unit; wherein: The determining unit is configured to determine pre-parameters of the video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; The calculation unit is configured to determine a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; The determining unit is further configured to determine a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determine a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; The calculation unit is further configured to determine a target distortion value according to the first distortion value and the second distortion value; The encoding unit is configured to determine the encoding parameters of the video to be encoded by using the target Lagrange multiplier and the target distortion value, and encode the video to be encoded; The determination unit is further configured to perform semantic segmentation on the test video based on the test video to determine the semantic accuracy of one or more categories; and determine the target semantic accuracy based on the semantic accuracy of the one or more categories; the calculation unit is further configured to perform distortion measurement on the target semantic accuracy using a fourth calculation model to obtain the first distortion value; wherein the semantic accuracy represents the ratio of the intersection and union of the detection result area and the true value area of ​​the corresponding category in the test video, and the fourth calculation model represents the correspondence between the first distortion value and the target semantic accuracy.

37. An encoder, comprising a memory and a processor; wherein: The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 35 when running the computer program.

38. A computer storage medium, wherein: The computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 35 is implemented.

39. A video system, comprising an encoder and a decoder; wherein: The encoder is configured to determine pre-parameters of a video to be encoded, and determine a first Lagrangian multiplier and a second Lagrangian multiplier according to the pre-parameters; and determining a target Lagrangian multiplier according to the first Lagrangian multiplier and the second Lagrangian multiplier; and determining a first distortion value according to a first distortion metric criterion, wherein the first distortion metric criterion includes a semantic distortion metric criterion; and determining a second distortion value according to a second distortion metric criterion, wherein the second distortion metric criterion includes a numerical error metric criterion; and determining a target distortion value according to the first distortion value and the second distortion value; and determining a coding parameter of the to-be-encoded video using the target Lagrange multiplier and the target distortion value, encoding the to-be-encoded video to generate a bitstream, and transmitting the bitstream to the decoder; The encoder is further configured to perform semantic segmentation on the test video based on the test video to determine the semantic accuracy of one or more categories; determine the target semantic accuracy according to the semantic accuracy of the one or more categories; and perform distortion measurement on the target semantic accuracy using a fourth computing model to obtain the first distortion value; wherein the semantic accuracy represents the ratio of the intersection and union of the detection result area and the true value area of ​​the corresponding category in the test video, and the fourth computing model represents the corresponding relationship between the first distortion value and the target semantic accuracy; The decoder is configured to parse the bit stream to obtain a decoded video.

40. The system of claim 39, wherein: The decoder is further configured to parse the bitstream, obtain decoding parameters, and obtain the decoded video according to the decoding parameters; wherein the decoding parameters at least include parameters indicating the division method of the video to be decoded and parameters for constructing the predicted value of the decoding block in the video to be decoded.

Images