Multi-resolution coding method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a multi-resolution coding method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, when the target resolution of the to-be-coded frame changes relative to the original resolution of the coding reference frame, the coding reference frame is scaled according to the scaling to obtain the scaled reference frame, and the motion estimation is performed on the to-be-coded frame on the scaled reference frame to obtain the optimal motion vector. And determining a matching block starting point coordinate according to the optimal motion vector and the scaling, taking pixel points on the coding reference frame according to the matching block starting point coordinate and the scaling, performing interpolation filtering to generate a coding prediction frame, and generating a target coding frame according to residual information between the frame to be coded and the coding prediction frame. When the resolution is changed, the encoder does not need to be restarted and an immediate refresh frame does not need to be generated, so that the image encoding effect is ensured, the computing resource overhead can be effectively reduced, and the encoding delay is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a multi-resolution encoding method, apparatus, device, storage medium and product. Background Art

[0002] With the development of Internet technology and image processing technology, real-time communication has been widely used in real-time interactive scenarios such as video conferencing, online medical care, and online education. People's requirements for the coding delay of real-time communication have become increasingly stringent.

[0003] In order to improve the video fluency in real-time communication scenarios under weak network conditions, the target bit rate, encoding frame rate and even encoding resolution are usually adjusted to reduce the output bit rate to adapt to the current bandwidth situation. Therefore, the encoding resolution switching in real-time communication scenarios under weak network conditions is relatively frequent. However, when the resolution changes, the encoder must restart and encode an instantaneous refresh frame (IDR frame, Instantaneous Decoding Refresh) of the switched resolution for output. On the one hand, the encoder restart needs to destroy the old encoder first, release the memory, and then re-apply for memory according to the latest size, initialize various encoding parameters, and the computing resource overhead is large, resulting in increased encoding delay; on the other hand, the code control state when the encoder encodes the first instant refresh frame is the initial value. Generally, the first instant refresh frame is large, and the first instant refresh frame encoded is more likely to increase the risk of jamming. At present, in multi-resolution scenarios, when the resolution changes, the encoder needs to be restarted and an instant refresh frame is generated, resulting in large computing resource overhead, increased encoding delay, and poor image encoding effect. Summary of the invention

[0004] The embodiments of the present application provide a multi-resolution encoding method, apparatus, device, storage medium and product to solve the technical problem in the related art that when the resolution changes, the encoder needs to be restarted and an immediate refresh frame needs to be generated, resulting in large computing resource overhead, increased encoding delay and poor image encoding effect. The embodiments of the present application can effectively reduce computing resource overhead, reduce encoding delay and improve image encoding effect.

[0005] In a first aspect, an embodiment of the present application provides a multi-resolution encoding method, including:

[0006] In a case where a target resolution of the frame to be encoded changes relative to an original resolution of the encoded reference frame, scaling the encoded reference frame according to a scaling ratio to obtain a scaled reference frame, wherein the scaling ratio is determined according to the original resolution and the target resolution;

[0007] Performing motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector;

[0008] Determine the coordinates of the starting point of the matching block according to the optimal motion vector and the scaling ratio;

[0009] According to the coordinates of the starting point of the matching block and the scaling ratio, pixel points are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame, and a target coding frame is generated according to the residual information between the frame to be coded and the coding prediction frame. The target coding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target coding frame and the scaling ratio to generate a decoding prediction frame, and a target decoding frame is generated according to the decoding prediction frame and the residual information.

[0010] In a second aspect, an embodiment of the present application provides a multi-resolution encoding device, including a scaling processing module, a motion estimation module, a starting coordinate determination module and an encoding processing module, wherein:

[0011] The scaling processing module is configured to scale the coding reference frame according to a scaling ratio to obtain a scaled reference frame when a target resolution of the frame to be coded changes relative to an original resolution of the coding reference frame, wherein the scaling ratio is determined according to the original resolution and the target resolution;

[0012] The motion estimation module is configured to perform motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector;

[0013] The starting coordinate determination module is configured to determine the starting point coordinates of the matching block according to the optimal motion vector and the scaling ratio;

[0014] The encoding processing module is configured to take pixels on the encoding reference frame for interpolation filtering according to the coordinates of the starting point of the matching block and the scaling ratio to generate an encoding prediction frame, and generate a target encoding frame according to the residual information between the frame to be encoded and the encoding prediction frame. The target encoding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target encoding frame and the scaling ratio to generate a decoding prediction frame, and generate a target decoding frame according to the decoding prediction frame and the residual information.

[0015] In a third aspect, an embodiment of the present application provides a multi-resolution encoding device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-resolution encoding method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the multi-resolution encoding method as described in the first aspect when executed by a computer processor.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and at least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the multi-resolution encoding method as described in the first aspect.

[0020] In an embodiment of the present application, when a target resolution of the frame to be encoded changes relative to an original resolution of the encoding reference frame, the encoding reference frame is scaled according to a scaling ratio to obtain a scaled reference frame, motion estimation is performed on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector, the coordinates of a starting point of a matching block are determined according to the optimal motion vector and the scaling ratio, pixel points are taken on the encoding reference frame to perform interpolation filtering according to the coordinates of the starting point of the matching block and the scaling ratio to generate an encoding prediction frame, and a target encoding frame is generated according to residual information between the frame to be encoded and the encoding prediction frame. The target encoding frame can be used by a decoder to perform interpolation filtering on a decoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to generate a decoding prediction frame, and a target decoding frame is generated according to the decoding prediction frame and the residual information. When a resolution change occurs, there is no need to restart the encoder and generate an immediate refresh frame. While ensuring the image encoding effect, the computing resource overhead and the encoding delay can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a multi-resolution encoding method provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of another multi-resolution encoding method provided by an embodiment of the present application;

[0023] Figure 3 is a structural schematic diagram of a multi-resolution encoding device provided in an embodiment of the present application;

[0024] Figure 4 It is a structural diagram of a multi-resolution encoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The multi-resolution encoding method provided in the present application can be applied to video encoding processing in real-time communication scenarios, and aims to scale the encoding reference frame according to a scaling ratio to obtain a scaled reference frame when the target resolution of the frame to be encoded changes relative to the original resolution of the encoding reference frame, and encode the frame to be encoded on the scaled reference frame to obtain a target encoding frame.

[0027] In the existing video encoding scheme, when the resolution changes, the encoder must be restarted and an immediate refresh frame (IDR frame, Instantaneous Decoding Refresh) of the switched resolution must be encoded and output. Since the encoder restart needs to destroy the old encoder first, release the memory, and then re-apply for memory according to the latest size, and initialize various encoding parameters, there is a large computing resource overhead, which leads to an increase in encoding delay, and the code control parameters when the encoder encodes the first immediate refresh frame are the initial values. At this time, the code control parameters have not converged, the first immediate refresh frame is large, and the first immediate refresh frame encoded is more likely to increase the risk of delay and jamming. Based on this, a multi-resolution encoding method of an embodiment of the present application is provided to solve the technical problem that the existing video encoding scheme needs to restart the encoder and generate an immediate refresh frame when the resolution changes, resulting in a large computing resource overhead, increased encoding delay, and poor image encoding effect.

[0028] Figure 1 A flowchart of a multi-resolution encoding method provided in an embodiment of the present application is given. The multi-resolution encoding method provided in an embodiment of the present application can be executed by a multi-resolution encoding device, which can be implemented by hardware and / or software and integrated in a multi-resolution encoding device.

[0029] The following description is made by taking a multi-resolution encoding device executing a multi-resolution encoding method as an example. Figure 1 , the multi-resolution encoding method comprises:

[0030] S110: When the target resolution of the frame to be encoded changes relative to the original resolution of the encoded reference frame, the encoded reference frame is scaled according to a scaling ratio to obtain a scaled reference frame, where the scaling ratio is determined according to the original resolution and the target resolution.

[0031] The multi-resolution encoding device provided by the scheme can use the encoder to encode the frame to be encoded. The scheme does not need to restart the encoder according to the changed resolution when the resolution changes. When creating the encoder, the encoder can be allocated a cache according to the maximum resolution of the preset multiple resolutions, and the target resolution allowed to be switched during the image encoding process will be limited to the resolution range of the maximum size. Similarly, the decoder also allocates a cache for the encoder according to the maximum resolution of the preset multiple resolutions, and does not need to restart the decoder when the resolution changes during the decoding process, that is, the cache allocated when the codec is created is allocated according to the maximum resolution, and the resolution selected during the encoding and decoding process is within the maximum resolution range, ensuring that the encoder can encode the image normally. For example, when the preset maximum resolution is 1280*720, the width w≤1280 of the resolution allowed to be switched, and the height h≤720, at this time, the target resolution of the frame to be encoded can be cut down from 1280*720 to 960*540, and then cut up to 1000*700, but it is not allowed to switch to 1280*960 (at this time, the height exceeds the height limit).

[0032] In one embodiment, after the encoding of a frame to be encoded is completed, the resolution of the frame to be encoded is recorded, and the recorded resolution is the original resolution. When a new frame to be encoded needs to be encoded, the target resolution for encoding the new frame to be encoded can be determined. When the target resolution is consistent with the recorded original resolution, the frame to be encoded can be encoded according to a preset encoding strategy (at this time, the encoding reference frame does not need to be scaled). When the target resolution is inconsistent with the recorded original resolution, it can be determined that the target resolution of the frame to be encoded has a resolution change relative to the original resolution of the encoding reference frame. At this time, the encoding reference frame needs to be scaled, and then the encoding process is performed based on the scaled reference frame obtained by scaling.

[0033] For example, the variables lastSize and renderSize can be set in the encoder. LastSize is used to record the resolution size of the last encoded image frame (i.e., the original resolution provided by this solution), and renderSize is used to record the target resolution of the current frame to be encoded. LastSize is updated after encoding a frame. Optionally, the resolution may include height and width. LastSize includes the original resolution width lastWidht and the original resolution height lastHeigh, and renderSize includes the target resolution width renderWidth and the target resolution height renderHeight. When lastSize is different from renderSize, it means that the target resolution has changed relative to the original resolution.

[0034] Exemplarily, when the target resolution of the frame to be encoded changes relative to the original resolution of the encoded reference frame, the scaling ratio can be determined based on the original resolution and the target resolution, and the encoded reference frame can be scaled according to the scaling ratio to obtain a scaled reference frame having a size corresponding to the target resolution.

[0035] S120: Perform motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector.

[0036] Exemplarily, after obtaining the scaled reference frame, based on a preset motion search algorithm (such as a block matching algorithm, a pixel recursive algorithm, a gradient algorithm, a frequency domain algorithm, a global motion estimation algorithm, etc.), motion estimation is performed on the frame to be encoded on the scaled reference frame to obtain the optimal motion vector.

[0037] S130: Determine the coordinates of the starting point of the matching block according to the optimal motion vector and the scaling ratio.

[0038] Exemplarily, after performing motion estimation on the frame to be coded on the scaled reference frame to obtain the optimal motion vector, the coordinates of the starting point of the matching block are determined according to the optimal motion vector and the scaling ratio. For example, for each unit to be coded (CU) on the frame to be coded, the coordinates of the starting point of the scaling matching block are obtained according to the sum of the optimal motion vector and the starting coordinates (e.g., the upper left corner coordinates) of the unit to be coded, and the coordinates of the starting point of the scaling matching block are multiplied by the scaling ratio to obtain the coordinates of the starting point of the matching block corresponding to the matching block on the coding reference frame.

[0039] In one embodiment, a large amount of motion estimation is required during the encoding process. When the resolution changes, in order to reuse the previous motion estimation module, the encoded reference frame can be scaled to the same resolution as the frame to be encoded. Therefore, when the frame to be encoded begins to be encoded, additional memory is allocated for the scaled reference frame Scaled_ref, and the Y component of the encoded reference frame Ref_recon is scaled according to the scaling ratio and the scaling result is placed in the scaled reference frame Scaled_ref. The frame to be encoded fenc can perform motion estimation on the scaled reference frame Scaled_ref to calculate the optimal motion vector.

[0040] S140: According to the coordinates of the starting point of the matching block and the scaling ratio, pixel points are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame, and a target coding frame is generated according to the residual information between the frame to be coded and the coding prediction frame. The target coding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target coding frame and the scaling ratio to generate a decoding prediction frame, and a target decoding frame is generated according to the decoding prediction frame and the residual information.

[0041] Among them, when the encoder does inter-frame encoding, motion estimation and prediction frame generation are completed based on reference frames of the same size. When the resolution changes during the encoding process, considering the continuity of the content, prediction frames can still be generated by referring to reference frames of different resolutions.

[0042] Exemplarily, pixel points are taken on the coding reference frame according to the above-determined matching block starting point coordinates and scaling ratio to generate a coding prediction frame by interpolation filtering. For example, a sampling interval is determined according to the above-determined scaling ratio and a preset motion estimation accuracy (for example, the product of the scaling ratio and the preset motion estimation accuracy). Starting from the pixel point corresponding to the matching block starting point coordinates on the coding reference frame, pixel values ​​corresponding to the pixel points are taken on the coding reference frame in sequence according to the above-determined sampling interval. The obtained pixel values ​​are interpolated and filtered to obtain a coding prediction frame. The value of each pixel point on the coding prediction frame is the value obtained by interpolating and filtering the pixel value obtained for the corresponding pixel point.

[0043] After obtaining the coded prediction frame, the frame to be coded can be subtracted from the coded prediction frame to generate residual information, the residual information can be transformed and quantized, and then entropy encoded according to the transform and quantization results to obtain a target coded frame. The target coded frame can be sent to the decoding end so that the decoding end can perform image decoding based on the target coded frame to obtain a target decoded frame.

[0044] The target coding frame provided by the present solution can be used for the decoder to perform interpolation filtering on the decoded reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to generate a decoding prediction frame, and to generate a target decoding frame according to the decoding prediction frame and the residual information. For example, after receiving the target coding frame, the decoder can perform motion estimation on the target coding frame to obtain the optimal motion vector, calculate the coordinates of the starting point of the scaling matching block of the optimal matching block based on the optimal motion vector, multiply the coordinates of the starting point of the scaling matching block by the scaling ratio corresponding to the target coding frame to obtain the coordinates of the starting point of the matching block corresponding to the resolution of the decoding reference frame, and can start from the coordinates of the starting point of the matching block, and take the pixel values ​​corresponding to the pixel points on the decoding reference frame in sequence according to the value interval (the value interval can be determined according to the product of the scaling ratio and the preset motion estimation accuracy), and perform interpolation filtering on the obtained pixel values ​​to obtain a decoding prediction frame, and the value of each pixel point on the decoding prediction frame is the value of the pixel value obtained by the corresponding pixel point after the interpolation filtering. After obtaining the decoding prediction frame, the decoded residual is superimposed and the in-loop filtering is completed to obtain the target decoding frame. Among them, the encoder and decoder can reuse the method of generating predicted images when referring to images of different resolutions, which can effectively ensure the consistency of encoding and decoding and ensure the quality of image encoding and decoding.

[0045] In the above, when the target resolution of the frame to be encoded changes relative to the original resolution of the encoded reference frame, the encoded reference frame is scaled according to the scaling ratio to obtain a scaled reference frame, motion estimation is performed on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector, the coordinates of the starting point of the matching block are determined according to the optimal motion vector and the scaling ratio, pixel points are taken on the encoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to perform interpolation filtering to generate an encoded prediction frame, and a target encoded frame is generated according to the residual information between the frame to be encoded and the encoded prediction frame. The target encoded frame can be used by the decoding end to perform interpolation filtering on the decoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to generate a decoded prediction frame, and a target decoded frame is generated according to the decoded prediction frame and the residual information. When the resolution changes, there is no need to restart the encoder and generate an immediate refresh frame. While ensuring the image encoding effect, the computing resource overhead and the encoding delay can be effectively reduced.

[0046] Based on the above embodiments, Figure 2 A flowchart of another multi-resolution encoding method provided in an embodiment of the present application is given, and the multi-resolution encoding method is a specific embodiment of the multi-resolution encoding method described above. Figure 2 , the multi-resolution encoding method comprises:

[0047] S210: When a target resolution of the frame to be encoded changes relative to an original resolution of the encoded reference frame, the encoded reference frame is scaled according to a scaling ratio to obtain a scaled reference frame, where the scaling ratio is determined according to the original resolution and the target resolution.

[0048] S220: Perform motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector.

[0049] S230: Determine the coordinates of the starting point of the scaling matching block according to the optimal motion vector.

[0050] S240: Scaling the coordinates of the start point of the scaling matching block according to the scaling ratio to obtain the coordinates of the start point of the matching block.

[0051] Exemplarily, after performing motion estimation on the frame to be encoded on the scaled reference frame to obtain the optimal motion vector, for each unit to be encoded on the frame to be encoded, the coordinates of the starting point of the scaled matching block corresponding to each unit to be encoded can be determined according to the optimal motion vector and the starting coordinates (e.g., upper left corner coordinates) of the unit to be encoded. For example, for each unit to be encoded on the frame to be encoded, the sum of the starting coordinates of the unit to be encoded and the optimal motion vector can be used as the coordinates of the starting point of the scaled matching block.

[0052] In one embodiment, after determining the coordinates of the start point of the scaling match block, the scaling ratio is multiplied by the coordinates of the start point of the scaling match block, so as to scale the coordinates of the start point of the scaling match block according to the scaling ratio to obtain the coordinates of the start point of the matching block. In this solution, by determining the coordinates of the start point of the scaling match block according to the optimal motion vector, and scaling the coordinates of the start point of the scaling match block according to the scaling ratio to obtain the coordinates of the start point of the matching block, the pixel points are accurately taken on the coding reference frame for interpolation filtering, thereby improving the generation accuracy of the coding prediction frame.

[0053] In a possible embodiment, the multi-resolution encoding method provided by the present solution determines the coordinates of the start point of the scaling matching block according to the optimal motion vector, which may be to convert the coordinates of the unit start point of the unit to be encoded on the frame to be encoded to a preset motion estimation accuracy to obtain the first start point coordinates, and calculate the sum of the first start point coordinates and the optimal motion vector to obtain the coordinates of the start point of the scaling matching block. Correspondingly, the coordinates of the start point of the scaling matching block are scaled according to the scaling ratio to obtain the coordinates of the start point of the matching block, which may be to scale the coordinates of the start point of the scaling matching block according to the scaling ratio to obtain the second start point coordinates, and convert the second start point coordinates to the integer pixel accuracy to obtain the coordinates of the start point of the matching block.

[0054] Exemplarily, for a unit to be encoded on a frame to be encoded, the unit starting point coordinates of the unit to be encoded on the frame to be encoded are converted to a preset motion estimation accuracy to obtain the first starting point coordinates. For example, in the HEVC coding standard, the preset motion estimation accuracy is 1 / 4 pixel accuracy, and the unit starting point coordinates can be shifted to the left by two bits, so as to convert the unit starting point coordinates to 1 / 4 pixel accuracy to obtain the first starting point coordinates. In one embodiment, after obtaining the first starting point coordinates, the sum of the first starting point coordinates and the optimal motion vector can be calculated to obtain the scaling matching block starting point coordinates corresponding to the unit to be encoded.

[0055] In one embodiment, after obtaining the coordinates of the start point of the scaled matching block, the scaling ratio and the coordinates of the start point of the scaled matching block may be multiplied, so that the coordinates of the start point of the scaled matching block are scaled according to the scaling ratio to obtain the second start point coordinates. After obtaining the second start point coordinates, the second start point coordinates are converted to integer pixel precision to obtain the coordinates of the start point of the matching block. For example, in the HEVC coding standard, the preset motion estimation precision is 1 / 4 pixel precision, and the second start point coordinates may be shifted right by two bits, so that the second start point coordinates are converted to integer pixel precision to obtain the coordinates of the start point of the matching block.

[0056] The coordinates include horizontal axis coordinates and vertical axis coordinates, and the horizontal axis coordinates and vertical axis coordinates of the unit starting point coordinates can be shifted to the left by two positions respectively, so as to convert the unit starting point coordinates to 1 / 4 pixel accuracy to obtain the horizontal axis coordinates and vertical axis coordinates corresponding to the first starting point coordinates, and the horizontal axis coordinates and vertical axis coordinates corresponding to the first starting point coordinates are added to the horizontal axis component and the vertical axis component of the optimal motion vector respectively to obtain the horizontal axis coordinates and vertical axis coordinates corresponding to the scaling matching block starting point coordinates. Correspondingly, the horizontal axis coordinates and vertical axis coordinates corresponding to the scaling matching block starting point coordinates are multiplied by the horizontal scaling ratio and the vertical scaling ratio of the scaling ratio respectively to obtain the horizontal axis coordinates and vertical axis coordinates corresponding to the second starting point coordinates, and the horizontal axis coordinates and vertical axis coordinates corresponding to the second starting point coordinates are shifted to the right by two positions respectively to obtain the horizontal axis coordinates and vertical axis coordinates corresponding to the matching block starting point coordinates.

[0057] For example, for a unit to be encoded on a frame to be encoded, the coordinates of the unit starting point are (cu_x, cu_y), and the coordinates of the scaling matching block starting point (xs, ys) can be determined by the following formula:

[0058] xs=(cu_x<<a)+mv.x

[0059] ys=(cu_y<<a)+mv.y

[0060] Wherein, a is a displacement parameter determined based on a preset motion estimation accuracy. For example, when the preset motion estimation accuracy is 1 / 4 pixel accuracy, a=2, and mv.x and mv.y are the horizontal axis component and the vertical axis component of the optimal motion vector.

[0061] Correspondingly, the coordinates of the starting point of the matching block (xs_o, ys_o) can be determined by the following formula:

[0062] xs_o=(sf x *xs)>>a

[0063] ys_o=(sf y *ys)>>a

[0064]

[0065] Among them, sf x and sf y The horizontal scaling ratio and the vertical scaling ratio of the scaling ratio are obtained by converting the unit starting point coordinates to the preset motion estimation accuracy to obtain the first starting point coordinates, and calculating the sum of the first starting point coordinates and the optimal motion vector to obtain the scaling matching block starting point coordinates, and then scaling the scaling matching block starting point coordinates according to the scaling ratio to obtain the second starting point coordinates, and converting the second starting point coordinates to the integer pixel accuracy to accurately obtain the matching block starting point coordinates, which can effectively ensure that the pixel points on the coding reference frame are accurately taken for interpolation filtering, thereby improving the generation accuracy of the coding prediction frame.

[0066] S250: According to the coordinates of the starting point of the matching block and the scaling ratio, pixels are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame, and a target coding frame is generated according to the residual information between the frame to be coded and the coding prediction frame. The target coding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target coding frame and the scaling ratio to generate a decoding prediction frame, and a target decoding frame is generated according to the decoding prediction frame and the residual information.

[0067] In a possible embodiment, the multi-resolution encoding method provided by the present solution takes pixels on the encoding reference frame for interpolation filtering to generate an encoding prediction frame according to the coordinates of the starting point of the matching block and the scaling ratio, which may be:

[0068] S251: Determine a value interval according to a scaling ratio and a preset motion estimation accuracy.

[0069] S252: According to the coordinates of the starting point of the matching block and the value interval, pixel points are taken on the coding reference frame to perform interpolation filtering to generate a coding prediction frame.

[0070] Exemplarily, after obtaining the coordinates of the starting point of the matching block, the interval base is determined according to the preset motion estimation accuracy (for example, the reciprocal of the preset motion estimation accuracy is used as the interval base), the product of the scaling ratio and the interval base is calculated, and the value interval is determined according to the product result. Optionally, the product result may be rounded down, and the rounded down result is used as the value interval.

[0071] In one embodiment, the value interval includes a horizontal value interval and a vertical value interval. The horizontal value interval and the vertical value interval corresponding to the value interval can be obtained by multiplying the horizontal value interval and the vertical value interval of the scaling ratio by the interval base and rounding down. Optionally, the value interval can be determined by the following formula:

[0072] step x =[sf x *b]

[0073] step y=[sf y *b]

[0074] Among them, step x and step y are the horizontal and vertical value intervals of the value interval, and b is the interval base determined based on the preset motion estimation accuracy. For example, when the preset motion estimation accuracy is 1 / 4 pixel accuracy, b=4.

[0075] In one embodiment, after determining the value interval, pixels can be taken on the coding reference frame according to the coordinates of the starting point of the matching block and the value interval to perform interpolation filtering to generate a coding prediction frame. For example, starting from the pixel corresponding to the coordinates of the starting point of the matching block on the coding reference frame, pixel values ​​corresponding to the pixels are taken on the coding reference frame in sequence according to the value interval determined above, and the obtained pixel values ​​are interpolated and filtered to obtain the target coding frame, and the value of each pixel on the target coding frame is the value of the pixel value obtained by the corresponding pixel point after interpolation filtering. This solution determines the value interval according to the scaling ratio and the preset motion estimation accuracy, and accurately takes pixels on the coding reference frame according to the coordinates of the starting point of the matching block and the value interval to perform interpolation filtering to generate a coding prediction frame, thereby improving the generation accuracy of the coding prediction frame when the resolution changes.

[0076] Optionally, when this solution takes pixel points on the encoded reference frame for interpolation filtering according to the coordinates of the starting point of the matching block and the scaling ratio to generate an encoded prediction frame, an interpolation filter can be used for interpolation filtering. This solution provides a variety of different interpolation filters for interpolation filtering, and different interpolation filters correspond to different interpolation filter indexes. The interpolation filter used for interpolation filtering can be determined based on the interpolation filter index. Among them, the interpolation filter includes a horizontal interpolation filter and a vertical interpolation filter for horizontal interpolation filtering and vertical interpolation filtering, which are respectively configured with corresponding horizontal interpolation filter indexes and vertical interpolation filter indexes.

[0077] In a possible embodiment, the interpolation filter corresponding to the interpolation filtering performed on the pixel points in this solution can be determined based on the following formula:

[0078] filterH idx =(((sf x *xs)&3)+i_x*step x )&3

[0079] filterV idx =(((sf y *ys)&3)+i_y*step y )&3

[0080] Among them, filterH idxis the horizontal interpolation filter index, sf x is the scaling ratio in the horizontal direction, xs is the horizontal coordinate of the first starting point, i_x is the i_xth column of the coding unit pixel, and step x is the horizontal value interval, filterV idx is the vertical interpolation filter index, sf y is the scaling ratio in the vertical direction, ys is the ordinate of the first starting point, i_y is the i_yth row of pixels in the coding unit, and step y is the vertical value interval. In one embodiment, the interpolation filter (including horizontal interpolation filter and vertical interpolation filter) used by the decoding end needs to reuse the encoder to determine the interpolation filter method to ensure the correct decoding of the image. This scheme uses a method similar to HEVC pixel interpolation to generate predicted pixels corresponding to the encoded prediction frame. The difference between this scheme and HEVC pixel interpolation is that the horizontal interpolation filter of HEVC pixel interpolation is obtained by mv.x&3, and the vertical interpolation filter is obtained by mv.y&3. While encoding across resolutions, this scheme adaptively adjusts the index value of the interpolation filter according to the value interval, which can effectively maintain the stability of the difference filter of the entire coding unit during the interpolation process, thereby ensuring the image encoding quality.

[0081] Optionally, after generating the target coding frame, the target coding frame can be sent to the decoding end in the form of a data stream (bit stream). The data stream is configured with slice header information (sliceHeader) and a data packet corresponding to the target coding frame. Whether the target resolution of the frame to be coded has changed relative to the original resolution of the coding reference frame can be recorded in the slice header information of the data stream (i.e., the slice header information corresponding to the target coding frame). Optionally, a resolution change flag can be added to the slice header information. When a resolution change occurs, the resolution change flag is set to a first set value (e.g., 1), and when no resolution change occurs, the resolution change flag is set to a second set value (e.g., 0). The changed target resolution can also be recorded in the slice header information.

[0082] In a possible embodiment, after the target coding frame is generated according to the residual information between the frame to be coded and the coded prediction frame, the multi-resolution coding method provided by the present scheme further includes: when the target resolution is an integer multiple of the minimum coding unit size, recording the target resolution in the fragment header information corresponding to the target coding frame, so that the decoding end determines whether a resolution change occurs according to the fragment header information.

[0083] Exemplarily, when the target resolution is an integer multiple of the minimum coding unit size, the target resolution may be recorded in the slice header information corresponding to the target coding frame, and the slice header information may be used by the decoding end to determine whether a resolution change occurs based on the slice header information.

[0084] For example, in order to reduce the damage to the HEVC header information syntax, the preset extension field in the slice header information (such as the slice_segment_header_extension_data_byte field) can be used, and the resolution change flag slice_segment_header_extension_present_flag on the picture parameter set (PPS, Picture Parameter Set) in the data stream is modified at the same time. When the resolution changes, the value of slice_segment_header_extension_present_flag is set to 1, and when the resolution does not change, the value of slice_segment_header_extension_present_flag is set to 0. For example, when the target resolution is an integer multiple of the minimum coding unit size, 4 is written to the preset extension field length character slice_segment_header_extension_length, and the target resolution (including the target resolution width (2 bytes, low bits first and high bits last) and the target resolution height (2 bytes, low bits first and high bits last)) is continued to be written. This solution records the target resolution in the fragment header information corresponding to the target coding frame when the target resolution is an integer multiple of the minimum coding unit size, so that the decoding end can determine whether the resolution has changed based on the parsed fragment header information, and accurately perform image decoding based on the image decoding strategy after the resolution change, thereby ensuring the image decoding quality.

[0085] In a possible embodiment, after the target coding frame is generated according to the residual information between the frame to be coded and the coded prediction frame, the multi-resolution coding method provided by the present scheme further includes: when the target resolution is a non-integer multiple of the minimum coding unit size, recording the target resolution and size offset information in the slice header information corresponding to the target coding frame, so that the decoding end determines whether a resolution change occurs and the size offset when the resolution change occurs according to the slice header information, and the size offset information is determined according to the padding operation performed on the size of the frame to be coded by the minimum coding unit size.

[0086] Exemplarily, when the target resolution is a non-integer multiple of the minimum coding unit size, the target resolution and size offset information are recorded in the slice header information corresponding to the target coding frame. The target resolution and size offset information recorded in the slice header information can be used by the decoding end to determine whether a resolution change occurs and the size offset when a resolution change occurs according to the slice header information.

[0087] The size offset information provided by this solution can be determined according to the padding operation performed on the size of the frame to be coded according to the minimum coding unit size. For example, the HEVC encoder pads the input frame to be coded according to the integer multiples of the length and width relative to the minimum coding unit size, so that the resolution of the padded frame to be coded is an integer multiple of the minimum coding unit size. The padding of the frame to be coded is generally performed to the right and bottom, and the size offset of the padded frame to be coded relative to the top and left of the frame to be coded is 0.

[0088] Optionally, the size offset information may include size offsets relative to the top, left, right, and bottom of the frame to be encoded, which may be represented by topOffset, leftOffset, bottomOffset, and rightOffset. topOffset, leftOffset, bottomOffset, and rightOffset may be determined by the following formula:

[0089] topOffset = 0;

[0090] leftOffset = 0;

[0091] rightOffset=minCUSize–(renderWidth&(minCUSize–1));

[0092] bottomOffset=minCUSize–(renderHeight&(minCUSize–1))

[0093] Among them, minCUSize is the minimum coding unit size. For example, when the minimum coding unit size is m*m, the minimum coding unit size may be m, renderWidth is the target resolution width, and renderHeight is the target resolution height.

[0094] For example, when the target resolution is a non-integer multiple of the minimum coding unit size, 8 is written to the preset extension field length character slice_segment_header_extension_length, and the target resolution (including the target resolution width (2 bytes, the low bit is in front and the high bit is in the back) and the target resolution height (2 bytes, the low bit is in front and the high bit is in the back)) is continued to be written, and topOffset, leftOffset, bottomOffset and rightOffset are written to the conformance field. This scheme records the target resolution and size offset information in the slice header information corresponding to the target coding frame when the target resolution is a non-integer multiple of the minimum coding unit size, so that the decoding end can determine whether the resolution changes and the size offset when the resolution changes according to the parsed slice header information, and accurately decode the image based on the image decoding strategy after the resolution changes, thereby ensuring the image decoding quality.

[0095] In one embodiment, when the resolution changes, the code control parameters and encoding process parameters corresponding to the encoder can be reset according to the target resolution. Taking the HEVC encoding standard as an example, when the resolution changes, the target bit rate will change. Therefore, when the target resolution changes relative to the original resolution, the encoder variable renderSize can be reset. Considering that the HEVC encoder will fill the input YUV according to the integer multiples of the length and width relative to the minimum coding unit size, it is also necessary to calculate the conformance field (including topOffset, leftOffset, bottomOffset and rightOffset) according to renderSize. This conformance field will be written into the specific syntax of the slice header (sliceHeader) together with renderSize, and the code control state parameters will be cleared, and the target bit rate corresponding to the target resolution will be assigned.

[0096] Since HEVC encoding is block-based encoding, the entire image is divided into coding tree units CTU of equal size (for example, 64*64), and the encoding is traversed CTU by CTU. When the resolution changes, the number of CTU rows per frame, the number of CTUs per row, and the total number of CTUs will change, and the number of CTU rows, the number of CTUs per row, and the total number of CTUs per frame in the encoder and decoder need to be reset. In addition, during encoding, the corresponding pixel starting point of the Y component and the U and V components on the original image will be calculated according to the CTU address. When the resolution changes, the offset addresses m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV on the Y, U, and V components need to be calculated according to the target resolution, and the last calculated m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV are saved to the offset addresses m_cuOffsetY_old, m_cuOffsetU_old, and m_cuOffsetV_old used to record the historical Y, U, and V components. When generating a prediction frame with cross-resolution reference, the pixel value on the coded reference frame before the resolution change can be indexed according to m_cuOffsetY_old, m_cuOffsetU_old, and m_cuOffsetV_old.

[0097] Among them, m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV are three arrays, which respectively store the offset address of each coding tree unit on the three components Y, U, and V of the frame to be encoded. m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV can be determined by the following formula:

[0098] m_cuOffsetY[cuRow*numCuInWidth+cuCol]

[0099] =m_stride*cuRow*maxCUSize+cuCol*maxCUSize

[0100] m_cuOffsetU[cuRow*numCuInWidth+cuCol]

[0101] =m_strideU*cuRow*maxCUSizeChroma+cuCol

[0102] *maxCUSizeChroma

[0103] m_cuOffsetV[cuRow*numCuInWidth+cuCol]

[0104] =m_strideV*cuRow*maxCUSizeChroma+cuCol

[0105] *maxCUSizeChroma

[0106] Among them, cuRow and cuCol are the row and column indexes of the current coding tree unit respectively, that is, when the frame to be encoded is divided into blocks of the coding tree unit size (for example, 64*64), the current coding tree unit is located at the cuRowth row and the cuColth column, numCuInWidth is the number of coding tree units per row calculated based on the Y component, m_stride is the number of pixels per row of the frame to be encoded, maxCUSize is the maximum coding tree unit size specified externally, for example, the 64*64 coding tree unit corresponds to maxCUSize=64, m_strideU is the number of pixels per row of the U component, maxCUSizeChroma is the preset maximum chroma coding tree unit size, for example, when the frame to be encoded is in YUV420 format, assuming that the externally specified coding tree unit size is 64*64, the corresponding maxCUSizeChroma=32, and m_strideU is the number of pixels per row of the V component.

[0107] Among them, m_cuOffsetY_old, m_cuOffsetU_old, and m_cuOffsetV_old retain the values ​​before the resolution changes, that is, when the resolution changes, first copy the data in the arrays m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV to m_cuOffsetY_old, m_cuOffsetU_old, and m_cuOffsetV_old respectively, and then recalculate the values ​​of m_cuOffsetY, m_cuOffsetU, and m_cuOffsetV.

[0108] In a possible embodiment, before scaling the coded reference frame according to the scaling ratio to obtain the scaled reference frame, a combination of one or more of the pixel addressing variables, deblocking filter parameters (Deblock parameters), adaptive compensation parameters (SAO variables), and coding tree unit attribute arrays (m_ctuGeomMap) in the encoder may be reset according to the target resolution. Correspondingly, at the decoding end, a combination of one or more of the pixel addressing variables, deblocking filter parameters, adaptive compensation parameters, and coding tree unit attribute arrays in the decoder may be reset according to the target resolution.

[0109] In the post-coding processing stage, the deblocking filter module and the adaptive compensation module also need to traverse the coding tree unit for filtering, and also reset the number of coding tree unit rows, the number of coding tree units per row, the total number of coding tree units per frame, and the height of the last row of coding tree units of the deblocking filter module and the adaptive compensation module. When encoding each coding tree unit, each coding unit (CU) is traversed according to the coding tree unit attributes. When the coding tree unit is at the right boundary and the bottom boundary, the coding tree unit attributes are different from the coding tree unit in the middle of the image. When the resolution changes, the addresses of the right boundary and the bottom boundary coding tree units will change, so it is necessary to reset the attribute information of each coding tree unit, that is, remap the coding tree unit attribute array.

[0110] The coding tree unit attribute array is an array that stores the attribute information of the coding tree unit scanned by row, which can be used to identify the incomplete coding tree unit at the boundary. When the resolution changes, the total number of coding tree units per frame, the number of coding tree units per row, and the total number of coding tree unit rows will change. For example, when the maximum coding tree unit of the externally specified coding is 64*64 and the resolution of the frame to be encoded is 540*960, there are 15 rows and 9 columns of coding tree units, and the coding tree unit at the right boundary is an incomplete coding tree unit, so the value at m_ctuGeomMap[9*idx_row-1] is a non-zero value, and idx_row is the coding tree unit row number. Similarly, when the resolution is switched to 720*1280, there are 20 rows and 12 columns of coding tree units, and the coding tree unit at the right boundary is an incomplete coding tree unit, so the value at m_ctuGeomMap[12*idx_row-1] is a non-zero value. This solution enables the encoder to adapt to changes in resolution and ensure the normal progress of image encoding by resetting the pixel addressing variables, deblocking filter parameters, adaptive compensation parameters and coding tree unit attribute arrays in the encoder according to the target resolution. When the encoding resolution changes, the code control logic is readjusted and the HEVC pixel interpolation method is referred to to generate a prediction frame based on the encoding reference frames of different resolutions, generate an inter-frame encoding frame (compressed reference value) of different resolutions, and modify the frame-level header information sliceHeader so that the decoder can correctly know the correct encoding resolution and correctly decode the YUV image. Compared with restarting the encoder and encoding an immediate refresh frame, the target encoding frame generated by referring to reference frames of different resolutions with the continuity of the content between images has better quality and lower bit rate, which can effectively solve the jamming problem caused by resolution switching in low-latency network loss scenarios.

[0111] In the above, when the target resolution of the frame to be encoded changes relative to the original resolution of the encoding reference frame, the encoding reference frame is scaled according to the scaling ratio to obtain a scaled reference frame, the frame to be encoded is motion estimated on the scaled reference frame to obtain an optimal motion vector, the coordinates of the starting point of the matching block are determined according to the optimal motion vector and the scaling ratio, the pixel points are taken on the encoding reference frame for interpolation filtering to generate a coding prediction frame according to the coordinates of the starting point of the matching block and the scaling ratio, and the target coding frame is generated according to the residual information between the frame to be encoded and the coding prediction frame. The target coding frame can be used by the decoder to perform interpolation filtering on the decoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to generate a decoding prediction frame, and the target decoding frame is generated according to the decoding prediction frame and the residual information. When the resolution changes, there is no need to restart the encoder and generate an immediate refresh frame. While ensuring the image encoding effect, the computing resource overhead can be effectively reduced and the encoding delay can be reduced. At the same time, by determining the coordinates of the starting point of the scaling matching block according to the optimal motion vector, and scaling the coordinates of the starting point of the scaling matching block according to the scaling ratio to obtain the coordinates of the starting point of the matching block, the pixel points are accurately taken on the coding reference frame for interpolation filtering, and the generation accuracy of the coding prediction frame is improved.

[0112] Figure 3 is a schematic diagram of the structure of a multi-resolution encoding device provided in an embodiment of the present application. Figure 3 The multi-resolution encoding device includes a scaling processing module 31, a motion estimation module 32, a starting coordinate determination module 33 and an encoding processing module 34.

[0113] Among them, the scaling processing module 31 is configured to scale the coding reference frame according to the scaling ratio to obtain a scaled reference frame when the target resolution of the frame to be encoded changes relative to the original resolution of the coding reference frame, and the scaling ratio is determined according to the original resolution and the target resolution; the motion estimation module 32 is configured to perform motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector; the starting coordinate determination module 33 is configured to determine the starting point coordinates of the matching block according to the optimal motion vector and the scaling ratio; the coding processing module 34 is configured to take pixels on the coding reference frame for interpolation filtering to generate a coding prediction frame according to the starting point coordinates of the matching block and the scaling ratio, and generate a target coding frame according to the residual information between the frame to be encoded and the coding prediction frame, the target coding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target coding frame and the scaling ratio to generate a decoding prediction frame, and generate a target decoding frame according to the decoding prediction frame and the residual information.

[0114] In the above, when the target resolution of the frame to be encoded changes relative to the original resolution of the encoded reference frame, the encoded reference frame is scaled according to the scaling ratio to obtain a scaled reference frame, motion estimation is performed on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector, the coordinates of the starting point of the matching block are determined according to the optimal motion vector and the scaling ratio, pixel points are taken on the encoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to perform interpolation filtering to generate an encoded prediction frame, and a target encoded frame is generated according to the residual information between the frame to be encoded and the encoded prediction frame. The target encoded frame can be used by the decoding end to perform interpolation filtering on the decoding reference frame according to the coordinates of the starting point of the matching block and the scaling ratio to generate a decoded prediction frame, and a target decoded frame is generated according to the decoded prediction frame and the residual information. When the resolution changes, there is no need to restart the encoder and generate an immediate refresh frame. While ensuring the image encoding effect, the computing resource overhead and the encoding delay can be effectively reduced.

[0115] In a possible embodiment, the starting coordinate determination module 33 determines the coordinates of the starting point of the matching block according to the optimal motion vector and the scaling ratio, and is configured as follows:

[0116] Determine the coordinates of the starting point of the scaling matching block according to the optimal motion vector;

[0117] The coordinates of the start point of the scaling matching block are scaled according to the scaling ratio to obtain the coordinates of the start point of the matching block.

[0118] In a possible embodiment, the starting coordinate determination module 33 determines the coordinates of the starting point of the scaling matching block according to the optimal motion vector, and is configured as follows:

[0119] The unit starting point coordinates of the unit to be encoded on the frame to be encoded are converted to a preset motion estimation accuracy to obtain the first starting point coordinates, and the sum of the first starting point coordinates and the optimal motion vector is calculated to obtain the scaling matching block starting point coordinates.

[0120] The coordinates of the starting point of the scaled matching block are scaled according to the scaling ratio to obtain the coordinates of the starting point of the matching block, and the configuration is:

[0121] The coordinates of the start point of the scaled matching block are scaled according to the scaling ratio to obtain the coordinates of the second start point, and the coordinates of the second start point are converted to integer pixel accuracy to obtain the coordinates of the start point of the matching block.

[0122] In a possible embodiment, the encoding processing module 34 takes pixels on the encoding reference frame and performs interpolation filtering to generate an encoding prediction frame according to the coordinates of the starting point of the matching block and the scaling ratio, and is configured as follows:

[0123] Determine the value interval according to the scaling ratio and the preset motion estimation accuracy;

[0124] According to the coordinates of the starting point of the matching block and the value interval, pixel points are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame.

[0125] In a possible embodiment, the interpolation filter corresponding to the interpolation filtering is determined based on the following formula:

[0126] filterH idx =(((sf x *xs)&3)+i_x*step x )&3

[0127] filterV idx =(((sf y *ys)&3)+i_y*step y )&3

[0128] Among them, filterH idx is the horizontal interpolation filter index, sf x is the scaling ratio in the horizontal direction, xs is the horizontal coordinate of the first starting point, i_x is the i_xth column of the coding unit pixel, and step x is the horizontal value interval, filterV idx is the vertical interpolation filter index, sf y is the scaling ratio in the vertical direction, ys is the ordinate of the first starting point, i_y is the i_yth row of pixels in the coding unit, and step y The vertical value interval.

[0129] In a possible embodiment, the multi-resolution encoding device also includes a first recording module, which is configured to: when the target resolution is an integer multiple of the minimum coding unit size, record the target resolution in the fragment header information corresponding to the target coding frame, so that the decoding end can determine whether the resolution changes according to the fragment header information.

[0130] In a possible embodiment, the multi-resolution encoding device also includes a second recording module, and the second recording module is configured to: when the target resolution is a non-integer multiple of the minimum coding unit size, record the target resolution and size offset information in the fragment header information corresponding to the target coding frame, so that the decoding end can determine whether the resolution changes and the size offset when the resolution changes according to the fragment header information, and the size offset information is determined according to the padding operation performed on the size of the frame to be encoded by the minimum coding unit size.

[0131] In a possible embodiment, the multi-resolution encoding device also includes a reset processing module, which is configured to: reset a combination of one or more of pixel addressing variables, deblocking filter parameters, adaptive compensation parameters and coding tree unit attribute arrays in the encoder according to the target resolution.

[0132] It is worth noting that in the embodiment of the above-mentioned multi-resolution encoding device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0133] The embodiment of the present application also provides a multi-resolution encoding device, which can integrate the multi-resolution encoding apparatus provided in the embodiment of the present application. Figure 4 is a schematic diagram of the structure of a multi-resolution encoding device provided in an embodiment of the present application. Figure 4 The multi-resolution encoding device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the multi-resolution encoding method provided in the above embodiment. The multi-resolution encoding device, equipment and computer provided above can be used to execute the multi-resolution encoding method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0134] The embodiment of the present application also provides a non-volatile storage medium storing computer executable instructions, which are used to execute the multi-resolution encoding method provided in the above embodiment when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the multi-resolution encoding method provided above, can also execute the related operations in the multi-resolution encoding method provided in any embodiment of the present application. The multi-resolution encoding apparatus, device and storage medium provided in the above embodiments can execute the multi-resolution encoding method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the multi-resolution encoding method provided in any embodiment of the present application.

[0135] On the basis of the above embodiments, the embodiments of the present application further provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the multi-resolution encoding method provided in each embodiment of the present application.

Claims

1. A multi-resolution encoding method, characterized in that: include: In a case where a target resolution of the frame to be encoded changes relative to an original resolution of the encoded reference frame, scaling the encoded reference frame according to a scaling ratio to obtain a scaled reference frame, wherein the scaling ratio is determined according to the original resolution and the target resolution; Performing motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector; Determine the coordinates of the starting point of the matching block according to the optimal motion vector and the scaling ratio; According to the coordinates of the starting point of the matching block and the scaling ratio, pixel points are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame, and a target coding frame is generated according to the residual information between the frame to be coded and the coding prediction frame. The target coding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target coding frame and the scaling ratio to generate a decoding prediction frame, and a target decoding frame is generated according to the decoding prediction frame and the residual information.

2. The multi-resolution encoding method according to claim 1, characterized in that: The step of determining the coordinates of the starting point of the matching block according to the optimal motion vector and the scaling ratio comprises: Determine the coordinates of the starting point of the scaling matching block according to the optimal motion vector; The coordinates of the start point of the scaled matching block are scaled according to the scaling ratio to obtain the coordinates of the start point of the matching block.

3. The multi-resolution encoding method according to claim 2, characterized in that: The step of determining the coordinates of the starting point of the scaling matching block according to the optimal motion vector comprises: The unit starting point coordinates of the unit to be encoded on the frame to be encoded are converted to a preset motion estimation accuracy to obtain a first starting point coordinate, and the sum of the first starting point coordinates and the optimal motion vector is calculated to obtain the scaling matching block starting point coordinates. Scaling the coordinates of the scaling matching block starting point according to the scaling ratio to obtain the coordinates of the matching block starting point includes: The scaling matching block starting point coordinates are scaled according to the scaling ratio to obtain second starting point coordinates, and the second starting point coordinates are converted to integer pixel accuracy to obtain matching block starting point coordinates.

4. The multi-resolution encoding method according to claim 1, characterized in that: The step of taking pixels on the coding reference frame and performing interpolation filtering to generate a coding prediction frame according to the coordinates of the starting point of the matching block and the scaling ratio comprises: Determine a value interval according to the scaling ratio and a preset motion estimation accuracy; According to the coordinates of the starting point of the matching block and the value interval, pixel points are taken on the coding reference frame for interpolation filtering to generate a coding prediction frame.

5. The multi-resolution encoding method according to claim 1, characterized in that: The interpolation filter corresponding to the interpolation filtering is determined based on the following formula: filterH idx =(((sf x *xs)&3)+i_x*step x )&3 filterV idx =(99sf y *ys)&3)+i_y*step y )&3 Among them, filterH idx is the horizontal interpolation filter index, sf x is the scaling ratio in the horizontal direction, xs is the horizontal coordinate of the first starting point, i_x is the i_xth column of the coding unit pixel, and step x is the horizontal value interval, filterV idx is the vertical interpolation filter index, sf y is the scaling ratio in the vertical direction, ys is the ordinate of the first starting point, i_y is the i_yth row of pixels in the coding unit, and step y The vertical value interval.

6. The multi-resolution encoding method according to claim 1, characterized in that: After generating a target coded frame according to residual information between the to-be-coded frame and the coded prediction frame, the method further includes: In the case where the target resolution is an integer multiple of the minimum coding unit size, the target resolution is recorded in the slice header information corresponding to the target coding frame, so that the decoding end determines whether a resolution change occurs according to the slice header information.

7. The multi-resolution encoding method according to claim 1, characterized in that: After generating a target coded frame according to residual information between the to-be-coded frame and the coded prediction frame, the method further includes: In the case where the target resolution is a non-integer multiple of the minimum coding unit size, the target resolution and size offset information are recorded in the slice header information corresponding to the target coding frame, so that the decoding end determines whether a resolution change occurs and a size offset when a resolution change occurs based on the slice header information. The size offset information is determined based on the padding operation performed on the size of the frame to be encoded based on the minimum coding unit size.

8. The multi-resolution encoding method according to claim 1, characterized in that: Before scaling the coded reference frame according to the scaling ratio to obtain the scaled reference frame, the method further includes: One or more combinations of pixel addressing variables, deblocking filter parameters, adaptive compensation parameters, and coding tree unit attribute arrays in the encoder are reset according to the target resolution.

9. A multi-resolution encoding device, characterized in that: It includes a scaling processing module, a motion estimation module, a starting coordinate determination module and an encoding processing module, wherein: The scaling processing module is configured to scale the coding reference frame according to a scaling ratio to obtain a scaled reference frame when a target resolution of the frame to be coded changes relative to an original resolution of the coding reference frame, wherein the scaling ratio is determined according to the original resolution and the target resolution; The motion estimation module is configured to perform motion estimation on the frame to be encoded on the scaled reference frame to obtain an optimal motion vector; The starting coordinate determination module is configured to determine the starting point coordinates of the matching block according to the optimal motion vector and the scaling ratio; The encoding processing module is configured to take pixels on the encoding reference frame for interpolation filtering according to the coordinates of the starting point of the matching block and the scaling ratio to generate an encoding prediction frame, and generate a target encoding frame according to the residual information between the frame to be encoded and the encoding prediction frame. The target encoding frame is used by the decoding end to perform interpolation filtering on the decoding reference frame according to the target encoding frame and the scaling ratio to generate a decoding prediction frame, and generate a target decoding frame according to the decoding prediction frame and the residual information.

10. A multi-resolution encoding device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-resolution encoding method according to any one of claims 1 to 8.

11. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to perform the multi-resolution encoding method according to any one of claims 1 to 8 when executed by a computer processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-resolution encoding method according to any one of claims 1 to 8 is implemented.

Images