Video image processing method and device, codec, code stream, storage medium

Abstract

The application discloses a video image processing method and device, a codec, a code stream and a storage medium, wherein the method comprises the following steps: performing ordering processing on samples in a first image according to the difference values between the samples, to obtain a first list; and obtaining the sample value of a first component with the same position coordinates as the sample in a second image according to the index sequence number of the sample in the first list and the bit depth of the first component.

Description

Technical Field

[0001] This application relates to video technology, including but not limited to video image processing methods and apparatus, codecs, bitstreams, and storage media. Background Technology

[0002] Some depth cameras output images with three channels (e.g., RGB images). When encoding video based on images from such cameras, as... Figure 1 As shown, the three-channel image 101 (color image) output by this type of camera needs to be converted to image 102 (color image) in the image format required by the 2D encoder (e.g., YUV420 format) through color space transformation. After conversion, it can be compressed using a 2D encoder 103, such as VVC. At the decoding end, after decoding the YUV420 format image 105 by the 2D decoder 104, it is converted back to RGB image 106, and then the corresponding depth image is recovered based on the transformation relationship between RGB image 106 and depth image. For the decoding end, the transformation relationship between RGB image and depth image needs to be known in order to recover the depth image from the RGB image. Summary of the Invention

[0003] The video image processing method, apparatus, codec, bitstream, and storage medium provided in this application embodiment can, at the decoding end, restore the first image to the second image (e.g., a depth image) even when the transformation relationship between the first image (e.g., an RGB image) and the second image (e.g., a depth image) is unknown. The video image processing method, apparatus, codec, bitstream, and storage medium provided in this application embodiment are implemented as follows:

[0004] According to one aspect of the embodiments of this application, a video image processing method is provided, the method being applied to a decoder, the method comprising: sorting the samples according to the differences between samples in a first image to obtain a first list; and obtaining the sample value of a first component in a second image having the same position coordinates as the sample according to the index number of the sample in the first list and the bit depth of a first component.

[0005] Thus, for the decoder, even when the transformation relationship between the first image and the second image is unknown, it can still recover the second image from the first image; the encoder does not need to transmit the transformation relationship between the first image and the second image to the decoder, thereby saving transmission resources; the decoder does not need to receive and decode the transformation relationship, thereby saving power consumption.

[0006] According to another aspect of the embodiments of this application, a video image processing method is provided, the method being applied to an encoder, the method comprising: sorting the samples according to the differences between samples in a first image to obtain a first list; and obtaining the sample value of a first component in a second image that has the same position coordinates as the sample according to the index number of the sample in the first list and the bit depth of a first component.

[0007] Thus, for the encoder, even when the transformation relationship between the first and second images is unknown, it can recover the second image from the first image. Therefore, on the one hand, since the first image is transformed into a single-channel second image (i.e., the first component contains content), the transformation efficiency is improved and computational overhead is reduced, while also decreasing the data volume of the second image. This results in reduced power consumption during compression encoding of the second image due to the smaller workload, and reduced transmission overhead due to the smaller data volume after compression encoding. On the other hand, this means that the decoder does not need to transform the second image after decoding the bitstream, compared to... Figure 1 The decoder's processing flow shown eliminates two transformation processes (such as the transformation from RGB image to YUV420 image and the transformation from YUV420 image to depth image), thereby saving decoder power consumption and improving decoder efficiency.

[0008] According to another aspect of the embodiments of this application, a video image processing apparatus is provided, the apparatus being applied to a decoder, the apparatus comprising: a first sorting module and a first mapping module; wherein, the first sorting module is configured to sort the samples according to the differences between samples in a first image to obtain a first list; the first mapping module is configured to obtain the sample value of a first component in a second image having the same position coordinates as the sample according to the index number of the sample in the first list and the bit depth of the first component.

[0009] According to another aspect of the embodiments of this application, a video image processing apparatus is provided. The apparatus is applied to an encoder and includes: a second sorting module and a second mapping module; wherein the second sorting module is configured to sort the samples according to the differences between samples in a first image to obtain a first list; and the second mapping module is configured to obtain the sample value of the first component in the second image that has the same position coordinates as the sample according to the index number of the sample in the first list and the bit depth of the first component.

[0010] According to another aspect of the embodiments of this application, a decoder is provided, including a first memory and a first processor; wherein, the first memory is used to store a computer program that can run on the first processor; the first processor is used to execute the video image processing method applied to the decoder as described in the embodiments of this application when running the computer program.

[0011] According to another aspect of the embodiments of this application, an encoder is provided, including a second memory and a second processor; wherein the second memory is used to store a computer program that can run on the second processor; and the second processor is used to execute the video image processing method applied to the encoder as described in the embodiments of this application when running the computer program.

[0012] According to another aspect of the embodiments of this application, a bitstream is provided, the bitstream being generated by bit encoding a second image; wherein the second image is obtained through steps in a video image processing method applied to the encoder.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, which, when executed, implements the video image processing method applied to the decoder as described in the embodiments of this application, or implements the video image processing method applied to the encoder as described in the embodiments of this application.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0017] Figure 1 This is a schematic diagram of the processing flow of a three-channel image 101;

[0018] Figure 2This is a schematic diagram of the encoder 20 of the encoding / decoding system 1 provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the structure of the decoder 30 of the encoding / decoding system 1 provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram of the encoder 40 of the encoding / decoding system 2 provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the structure of the decoder 50 of the encoding / decoding system 2 provided in the embodiments of this application;

[0022] Figure 6 A schematic diagram illustrating the implementation flow of the video image processing method provided in this application embodiment;

[0023] Figure 7 This is a schematic diagram of a colormap;

[0024] Figure 8 A schematic diagram illustrating the implementation flow of the sorting processing method provided in the embodiments of this application;

[0025] Figure 9 A schematic diagram illustrating the implementation flow of another sorting method provided in an embodiment of this application;

[0026] Figure 10 A schematic diagram illustrating the implementation flow of the video image processing method provided in this application embodiment;

[0027] Figure 11 This is a schematic diagram illustrating the principle of pseudocolor transformation;

[0028] Figure 12 This is a schematic diagram of pseudo-color transformation applied to depth map 1201;

[0029] Figure 13 This is a schematic diagram of the processing flow for RGB24 sequence 1202;

[0030] Figure 14 A schematic diagram of the processing flow of the RGB24 sequence 1202 provided in the embodiments of this application;

[0031] Figure 15 This is a schematic diagram of the structure of the video image processing apparatus provided in the embodiments of this application;

[0032] Figure 16 This is a schematic diagram of the structure of another video image processing apparatus provided in an embodiment of this application;

[0033] Figure 17 This is a schematic diagram of the decoder provided in an embodiment of this application;

[0034] Figure 18 This is a schematic diagram of the encoder provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0037] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It is understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] The encoder and decoder framework and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will understand that, with the evolution of encoders and decoders and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0040] An encoding / decoding system that may be applicable to embodiments of this application, such as... Figure 2 and 3 As shown, the encoding / decoding system 1 includes Figure 2 The encoder 20 shown and Figure 3 The decoder 30 shown.

[0041] like Figure 2As shown, the encoder (specifically, the "video encoder") 20 may include a color space transformation module 200 and an encoding module 201; wherein, the encoding module 201 includes: a transformation and quantization unit 2011, an intra-frame estimation unit 2012, an intra-frame prediction unit 2013, a motion compensation unit 2014, a motion estimation unit 2015, an inverse transformation and inverse quantization unit 2016, a filter control and analysis unit 2017, a filtering unit 2018, an encoding unit 2019, and a decoded image buffer unit 2010, etc.; wherein, the filtering unit 2018 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 2019 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).

[0042] First, the color space transformation module 200 sorts the samples in the input first image according to the difference between the samples in the first image to obtain a first list; according to the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component with the same position coordinate as the sample in the second image is obtained; then, the obtained second image is output to the encoding module 201.

[0043] Encoding module 201, for the input second image, uses coding tree blocks (Coding Tree) The partitioning of a video coding unit (CTU) yields a video coding block. The residual pixel information obtained after intra-frame or inter-frame prediction is then transformed by the transform and quantization unit 2011. This transformation includes converting the residual information from the pixel domain to the transform domain and quantizing the resulting transform coefficients to further reduce the bit rate. The intra-frame estimation unit 2012 and intra-frame prediction unit 2013 perform intra-frame prediction on the video coding block. Specifically, the intra-frame estimation unit 2012 and intra-frame prediction unit 2013 determine the intra-frame prediction mode to be used to encode the video coding block. The motion compensation unit 2014 and motion estimation unit 2015 perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information. The motion estimation performed by the motion estimation unit 2015 is a process of generating motion vectors, which can estimate the motion of the video coding block. The motion compensation unit 2014 then uses the motion vectors determined by the motion estimation unit 2015 to generate motion vectors. The process involves motion compensation; after determining the intra-prediction mode, the intra-prediction unit 2013 provides the selected intra-prediction data to the coding unit 2019, and the motion estimation unit 2015 also sends the calculated motion vector data to the coding unit 2019; furthermore, the inverse transform and inverse quantization unit 2016 is used to reconstruct the video coding block, reconstructing the residual block in the pixel domain. This reconstructed residual block is then processed by the filter control analysis unit 2017 and the filtering unit 2018 to remove block artifacts, and then the reconstructed residual block is added to a predictive block in the frame of the decoding image buffer unit 2010 to generate the reconstructed video coding block; the coding unit 2019 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-prediction mode, outputting the bitstream of the second image; while the decoding image buffer unit 2010 is used to store the reconstructed video coding block for prediction reference. As video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoding image buffer unit 2010.

[0044] like Figure 3 As shown, the decoder (specifically, the "video decoder") 30 includes a decoding unit 301, an inverse transform and inverse quantization unit 302, an intra-frame prediction unit 303, a motion compensation unit 304, a filtering unit 305, and a decoded image buffer unit 306. The decoding unit 301 can perform header information decoding and CABAC decoding, while the filtering unit 305 can perform deblocking filtering and SAO filtering. The second image is processed... Figure 2After encoding processing by the encoding module 201, the bitstream of the second image is output. This bitstream is input to the decoder 30, first passing through the decoding unit 301 to obtain the decoded transform coefficients. These transform coefficients are then processed by the inverse transform and inverse quantization unit 302 to generate residual blocks in the pixel domain. The intra-frame prediction unit 303 can generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and data from previously decoded blocks in the current frame or image. The motion compensation unit 304 determines the prediction information for the video decoding block by analyzing motion vectors and other associated syntax elements, and uses... The prediction information is used to generate a predictive block of the video block being decoded; the decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 302 with the corresponding predictive block generated by the intra-frame prediction unit 303 or the motion compensation unit 304; the decoded video signal is passed through the filtering unit 305 to remove block artifacts, which can improve video quality; then the decoded video block is stored in the decoding image buffer unit 306, which stores reference images for subsequent intra-frame prediction or motion compensation, and is also used for the output of the video signal, thus obtaining the recovered second image.

[0045] Another encoding / decoding system that may be applicable to the embodiments of this application, such as Figure 4 and 5 As shown, the encoding / decoding system 2 includes Figure 4 The encoder 40 shown and Figure 5 The decoder 50 shown.

[0046] like Figure 4 As shown, the encoder (specifically, a "video encoder") 40 includes: a color space conversion module 400 and an encoding module 401; wherein, the structure of the encoding module 401 is similar to... Figure 2 The structure of the encoding module 201 shown is the same, so it will not be described again here. The color space transformation module 400 has a different function than the color space transformation module 200, and it uses a different color space transformation algorithm.

[0047] First, the color space transformation module 400 transforms the input first image into a third image with the same number of components according to a predefined pseudo-color transformation relationship (e.g., colormap); wherein the third image has a different number of channels than the second image. For example, the third image is a YUV420 format image, the second image is a depth image, and the first image is an RGB image (which can be any format of RGB image); then, the resulting third image is output to the encoding module 401.

[0048] Encoding module 401, for the input third image, uses coding tree blocks (Coding Tree) The partitioning of a video coding unit (CTU) yields a video coding block. The residual pixel information obtained after intra-frame or inter-frame prediction is then transformed by the transform and quantization unit 4011. This transformation includes converting the residual information from the pixel domain to the transform domain and quantizing the resulting transform coefficients to further reduce the bit rate. Intra-frame estimation unit 4012 and intra-frame prediction unit 4013 perform intra-frame prediction on the video coding block. Specifically, intra-frame estimation unit 4012 and intra-frame prediction unit 4013 determine the intra-frame prediction mode to be used to encode the video coding block. Motion compensation unit 4014 and motion estimation unit 4015 perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information. The motion estimation performed by motion estimation unit 4015 is a process of generating motion vectors, which can estimate the motion of the video coding block. Then, motion compensation unit 4014 uses the motion vectors determined by motion estimation unit 4015 to generate motion vectors. The motion compensation is performed. After determining the intra-prediction mode, the intra-prediction unit 4013 provides the selected intra-prediction data to the coding unit 4019, and the motion estimation unit 4015 also sends the calculated motion vector data to the coding unit 4019. In addition, the inverse transform and inverse quantization unit 4016 is used to reconstruct the video coding block, reconstructing the residual block in the pixel domain. The reconstructed residual block removes block artifacts through the filter control analysis unit 4017 and the filtering unit 4018, and then adds the reconstructed residual block to a predictive block in the frame of the decoding image buffer unit 4010 to generate the reconstructed video coding block. The coding unit 4019 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-prediction mode, outputting the bitstream of the third image. The decoding image buffer unit 4010 is used to store the reconstructed video coding block for prediction reference. As video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoding image buffer unit 4010.

[0049] like Figure 5 As shown, the decoder (specifically, the "video decoder") 50 includes a color space transformation module 501 and a decoding module 502; wherein, the structure of the decoding module 502 is similar to... Figure 3 The decoder 30 shown has the same structure, so it will not be described again here.

[0050] The input third image is processed Figure 4After encoding by the encoding module 401, the bitstream of the third image is output. This bitstream is input to the decoding module 502 of the decoder 50, and first passes through the decoding unit 5021 to obtain the decoded transform coefficients. The transform coefficients are then processed by the inverse transform and inverse quantization unit 5022 to generate residual blocks in the pixel domain. The intra-frame prediction unit 5023 can generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and data from the previously decoded blocks in the current frame or image. The motion compensation unit 5024 determines the prediction information for the video decoding block by analyzing motion vectors and other associated syntax elements, and uses this prediction information to generate predictive blocks for the video decoding block being decoded. The decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 5022 with the corresponding predictive block generated by the intra-frame prediction unit 5023 or the motion compensation unit 5024. The decoded video signal is then filtered. Unit 5025 is used to remove block artifacts, which can improve video quality; then the decoded video block is stored in the decoding image buffer unit 5026, which stores reference images for subsequent intra-frame prediction or motion compensation, and is also used for the output of the video signal, i.e., outputting a third image; and the third image is input to the color space transformation module 501, which is configured to: inversely transform the third image into a first image using a transformation relationship opposite to that of the encoding end color space transformation module 400, i.e., based on a predefined pseudo-color transformation relationship (e.g., colormap), or a predefined pseudo-color inverse transformation relationship; and sort the samples according to the differences between the samples in the first image to obtain a first list; and obtain the sample value of the first component with the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component, i.e., recover the second image.

[0051] Thus, the color space transformation module 501 can recover the second image without knowing the transformation relationship between the second and first images. Therefore, the encoder 40 does not need to transmit the transformation relationship between the first and second images to the decoder 50, thereby saving the resource overhead of transmission; the decoder also does not need to receive and decode the transformation relationship, thereby saving power consumption.

[0052] For the encoding / decoding system 1, since the color space transformation module 200 in the encoder 20 directly transforms the first image into the second image when the transformation relationship between the first and second images is unknown, and then outputs it to the encoding module 201; thus, on the one hand, since the first image is transformed into a single-channel second image, i.e., the first component has content, the number of transformation samples is reduced, thereby improving transformation efficiency and saving computational overhead, while also reducing the data volume of the second image. This results in less power consumption during the compression encoding of the second image due to the smaller workload, and also saves transmission overhead due to the smaller data volume after compression encoding; on the other hand, this means that after the decoder 30 obtains the second image from the decoded bitstream, it does not need to transform it again, compared to... Figure 5 After obtaining the third image through the decoding module 502, the decoder 50 shown in the figure still needs to use the color space transformation module 501 to transform the third image twice to recover the second image. The decoder 30 saves two transformation processes, thereby saving power consumption and improving the working efficiency of the decoder 30.

[0053] It should be noted that in video images, three color components are generally used to represent a coding block (CB); these three color components are a luminance component, a blue chrominance component, and a red chrominance component. For example, the luminance component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb or U, and the red chrominance component is usually represented by the symbol Cr or V; thus, video images can be represented in YCbCr format or YUV format. In addition, video images can also be in RGB format or YCgCo format, etc., and this application embodiment does not impose any limitations.

[0054] This application provides a video image processing method, which can be applied to... Figure 2 The encoder 20 shown, Figure 6 This is a schematic diagram illustrating the implementation flow of the video image processing method provided in the embodiments of this application, such as... Figure 6 As shown, the method includes the following steps 601 and 603:

[0055] Step 601: Sort the samples according to the differences between the samples in the first image to obtain a first list.

[0056] When sorting the samples, three samples with the same location coordinates are sorted as a group or as a whole.

[0057] In some embodiments, this can be achieved through steps 6011 to 6019 of the following embodiments, which will not be described here.

[0058] In some embodiments, the first image is a three-channel image, for example, an RGB image. Accordingly, the samples in the first image refer to samples of the three components. For example, the samples include samples of the R component, G component, and B component.

[0059] In this embodiment, the format of the first image is not limited; it can be any format of three-channel image. In short, the first image is a three-channel image transformed from a single-channel image based on a pseudo-color transformation relationship. Essentially, it represents the information of the single-channel image using a three-channel image. For this scenario, the video image processing methods provided in this embodiment are applicable.

[0060] For example, based on Figure 7 The colormap shown transforms a depth image, and the resulting transformation is an example of the first image. The colormap is predefined and only for human visual perception; it has no specific physical meaning. In this colormap, the R, G, and B components change with the depth value—for example, increasing or decreasing as the depth value increases.

[0061] Step 602: Based on the index number of the sample in the first list and the bit depth of the first component, obtain the sample value of the first component in the second image that has the same position coordinates as the sample.

[0062] In this embodiment, the format of the second image is not limited. Whether the second image has three components or one component, the sample value of a certain component only needs to represent the actual information. For example, at the encoding end, if the first image is transformed from a depth image, the sample value of the first component of the second image only needs to represent the depth information, while the sample values ​​of the remaining components of the second image have no practical meaning. The sample values ​​of the remaining components at each position coordinate can be set to a first value.

[0063] In some embodiments, the format of the second image can be YUV400 or YUV420. When the format of the second image is YUV420, the first component is the luminance component, and the second and third components are chrominance components; the sample values ​​of the second and third components of the second image are set to a first value.

[0064] The first value can be any value. For example, the first value can be 0 or... N is the bit depth of the first component.

[0065] Step 603: Encode the second image and write the obtained encoded bits into the bitstream.

[0066] Understandably, in this embodiment, the encoder 20 can transform the first image into the second image through steps 601 and 602 without needing to know the transformation relationship between the first and second images. Since the first image is transformed into a single-channel second image (i.e., the first component contains content), the number of transformation samples is reduced. This improves transformation efficiency and saves computational overhead, while also reducing the data volume of the second image. Consequently, power consumption is saved during the compression encoding of the second image due to the smaller workload, and transmission overhead is saved due to the smaller data volume after compression encoding. Furthermore, after the decoder 30 obtains the second image from the decoded bitstream, it does not need to transform it again, compared to… Figure 5 After obtaining the third image through the decoding module 502, the decoder 50 shown in the figure still needs to use the color space transformation module 501 to transform the third image twice to recover the second image. The decoder 30 saves two transformation processes, thereby saving power consumption and improving the working efficiency of the decoder 30.

[0067] The following sections will describe further optional implementation methods for each of the above steps, as well as related terms.

[0068] In step 601, the samples are sorted according to the differences between the samples in the first image to obtain a first list.

[0069] In some embodiments, such as Figure 8 As shown, step 601 can be achieved through the following steps 6011 to 6019:

[0070] Step 6011: Set the index number of the first sample in the first image in the first list to the second value.

[0071] In some embodiments, when the first image is a three-component image, the first sample includes samples of the three components at the same location coordinates in the first image. The samples of the three components at the same location coordinates can be treated as a whole (i.e., a combination or vector), and an index number can be assigned to this whole.

[0072] For example, in some embodiments, the samples in the first image are sorted using a linked list, so that samples of the three components at the same position coordinate can be stored in the same node.

[0073] In this embodiment of the application, the first sample can be a sample at any position coordinate in the first image, for example, it can be a sample at position coordinate (0, 0).

[0074] In this embodiment, the second value can be any value, as long as it can accurately represent the position of the first sample in the first list and obtain an accurate second image. For example, the second value is 1.

[0075] Step 6012: Determine the difference between the first sample and the adjacent second sample.

[0076] In some embodiments, "adjacent" refers to adjacent position coordinates. The second sample is a sample whose position coordinates are adjacent to those of the first sample. For example, if the position coordinates of the first sample are (i, j), the position coordinates of the second sample are (i+1, j) or (i, j+1).

[0077] In the case that the first image is a three-component image, the first sample includes samples of the three components at the same location coordinates in the first image, and the second sample is the same. Accordingly, the difference between the first sample and the second sample refers to the sum of the corresponding differences of the three components.

[0078] In this embodiment, the method for calculating the difference is not limited; that is, the parameter characterizing the difference is not limited. The Manhattan distance between the first sample and the second sample can be used as the difference, as can the cosine similarity or Euclidean distance between them. In short, any parameter that can measure the difference in sample values ​​between two samples is acceptable.

[0079] The method for calculating the difference between the other two samples is the same as the method for calculating the difference between the first sample and the second sample, and will not be repeated below.

[0080] Step 6013: Determine whether the absolute value of the difference between the first sample and the second sample is equal to the second threshold; if yes, proceed to step 6014; otherwise, proceed to step 6015.

[0081] In some embodiments, the second threshold is equal to 0; that is, the sample value of the second sample is the same as the sample value of the first sample.

[0082] For example, in the first sample, the sample value of the R component is R1, the sample value of the G component is G1, and the sample value of the B component is B1; in the second sample, the sample value of the R component is also R1, the sample value of the G component is also G1, and the sample value of the B component is also B1; then the absolute value of the difference between the first sample and the second sample is equal to 0, and their sample values ​​are the same; where R1, G1, and B1 are constants.

[0083] In this embodiment, the value of the second threshold is not limited; it can be equal to 0 or a value greater than 0. When the second threshold is greater than 0, the index numbers of two sample values ​​that are close to each other are set to the same value.

[0084] Step 6014: The index number of the second sample is the same as the index number of the first sample; proceed to step 6018;

[0085] Step 6015: Determine whether the absolute value of the difference between the first sample and the second sample is less than or equal to the first threshold; if yes, proceed to step 6017; otherwise, proceed to step 6016; wherein the first threshold is greater than the second threshold.

[0086] In some embodiments, where the difference is represented by Manhattan distance, the first threshold is 1, and accordingly, step 6015 is to determine whether the Manhattan distance between the first sample and the second sample is equal to 1; if yes, proceed to step 6017; otherwise, proceed to step 6016.

[0087] In this embodiment, the size of the first threshold is not limited. In short, it is sufficient to determine the order of each sample (here, the combination of samples with the same position coordinate) in the first image according to their size relationship.

[0088] Step 6016: Add the second sample to the first list and do not sort the second sample; that is, do not set an index number for the second sample; proceed to step 6018;

[0089] Step 6017: Set the index number of the second sample in the first list to the third value; proceed to step 6018;

[0090] In some embodiments, the third value is greater than the second value; in other embodiments, the third value is less than the second value.

[0091] In some embodiments, the absolute value of the difference between the third value and the second value is 1. For example, the second value is 1 and the third value is 2.

[0092] Of course, there is no limit to the size of the third value. In short, it is sufficient to accurately represent the position of the second sample in the first list and to obtain an accurate second image.

[0093] Step 6018: Perform the first operation on the untraversed third sample in the first image that is adjacent to the second sample;

[0094] The first operation includes: finding a first target sample in the first list whose absolute difference with the third sample is equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, finding a second target sample in the first list whose absolute difference with the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to the index number of the second target sample plus a fourth value; if the second target sample of the third sample is not found, adding the third sample to the first list and not sorting the third sample; then, continuing to perform the first operation on the untraversed samples adjacent to the third sample in the first image until the last sample of the first image is traversed to obtain the first list; then, proceeding to step 6019.

[0095] It should be noted that "untraversed samples" refers to samples that have not yet been added to the first list. Samples with index numbers are already in the first list.

[0096] In some embodiments, if the first target sample is not found, but a second target sample is found, the index number of the third sample in the first list is set to the index number of the second target sample plus a fourth value; then, the first operation is performed on the untraversed samples adjacent to the third sample in the first image until the last sample of the first image is traversed to obtain the first list; then, proceed to step 6019.

[0097] In this embodiment, the fourth value can be any value, for example, the fourth value is 1. In short, the goal is to accurately represent the position of the sample in the first list and obtain an accurate second image.

[0098] Step 6019: Perform the second operation on the unsorted samples in the first list;

[0099] The second operation includes: selecting any unsorted sample in the first list as the current sample; determining a third target sample with the smallest absolute value of the difference between the current sample and the other unsorted samples that are different from the current sample; and setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0100] Then, return to continue performing the second operation until the cutoff condition is met, resulting in the first list. Here, the cutoff condition is that there are two unsorted samples in the first list; one of these two samples is taken as the start of the list, and the other as the end of the list.

[0101] In the embodiments of this application, steps 6011 to 6019 can be implemented in any way. The data structure of the first list is not limited; it can be a linked list structure or other types of data structures. In short, it can represent the sequential relationship between the samples.

[0102] Taking the first list as a doubly linked list as an example, such as Figure 9 As shown, a doubly linked list can be obtained in the following way (i.e., steps 901 to 908):

[0103] Step 901: Select a certain coordinate in the first image as the current position, and use the samples of the three components at the current position as nodes of a doubly linked list, with the pointer of the node pointing to NULL; then proceed to step 902.

[0104] Step 902: Determine if the current position is the last untraversed position in the first image; if yes, proceed to step 907; otherwise, proceed to step 903.

[0105] Step 903: Select the next position coordinate as the current position, and determine whether the samples of the three components at the current position are already in the linked list; if yes, return to step 903; otherwise, proceed to step 904.

[0106] Step 904: Take the samples of the three components at the current position as a linked list node, and set the pointer of the node to NULL, then proceed to step 905.

[0107] Step 905: Traverse the nodes in the linked list and calculate the Manhattan distance between the sample of the three components at the current position and other nodes in the linked list; determine whether the distance is equal to 1; if yes, proceed to step 906; otherwise, continue traversing the next node in the linked list until the traversal ends, and then return to execute step 903.

[0108] Step 906: Point the pointer of the linked list node corresponding to the current position to the other node, and point the pointer of the other node to the linked list node corresponding to the current position; then, return to execute step 903.

[0109] Step 907: Determine if the number of linked list endpoints is greater than 2; if yes, proceed to step 908; otherwise, end; wherein, the linked list endpoints refer to nodes in the linked list whose pointers point to NULL;

[0110] Step 908: Select any endpoint of the linked list as the current node, find the target endpoint with the smallest Manhattan distance from the current node in the linked list, point the pointer of the current node to the target endpoint, and point the pointer of the target endpoint to the current node, and then return to execute step 907.

[0111] In step 602, based on the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

[0112] In some embodiments, after obtaining the first list by the above method, the second image is obtained through the following steps 6021 to 6022, i.e., step 602 is achieved:

[0113] Step 6021: Determine the maximum sample value representing the first component based on the bit depth of the first component.

[0114] In some embodiments, the maximum sample value is , where N is the bit depth of the first component.

[0115] Step 6022: Based on the index number of the sample in the first list and the maximum sample value, obtain the sample value of the first component in the second image that has the same position coordinates as the sample.

[0116] Specifically, in some embodiments, a first ratio is determined based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and the product of the first ratio and the maximum sample value is used as the sample value of the first component in the second image that has the same position coordinates as the sample.

[0117] For example, the sample value of the first component in the second image that has the same position coordinates as the sample can be obtained by the following formula (1). :

[0118] (1);

[0119] In equation (1), M is the total number of index numbers, and k is the index number of the sample in the first list.

[0120] Thus, based on formula (1), the correspondence between the first image and the second image is established, and the first image is transformed into the second image.

[0121] In step 603, the second image is encoded, and the obtained encoded bits are written into the bitstream.

[0122] Specifically, in some embodiments, the second image can be sent to the encoding module 201 to encode the second image.

[0123] In some embodiments, the method further includes setting the sample values ​​of the second component and the third component of the second image to a first value.

[0124] In the embodiments of this application, the first numerical value can be any value. For example, in some embodiments, the first numerical value is 0 or 2 to the power of (N-1), where N is the bit depth.

[0125] In some embodiments, the samples of the first image represent depth information.

[0126] In some embodiments, the samples of the first component of the second image represent depth information.

[0127] It should be noted that the difference between the first and second images lies in their formats. For example, the first image is in RGB format, while the second image is in YUV format.

[0128] This application embodiment further provides a video image processing method, which can be applied to... Figure 5 The decoder 50 shown is... Figure 10 This is a schematic diagram illustrating the implementation flow of the video image processing method provided in the embodiments of this application, such as... Figure 10 As shown, the method includes the following steps 1001 to 1004:

[0129] Step 1001: Decode the received bitstream to obtain the third image;

[0130] Step 1002: Perform color space transformation on the third image to obtain the first image;

[0131] Step 1003: Sort the samples according to the differences between the samples in the first image to obtain a first list;

[0132] Step 1004: Based on the index number of the sample in the first list and the bit depth of the first component, obtain the sample value of the first component in the second image that has the same position coordinates as the sample.

[0133] In some embodiments, the sorting process of the samples based on the differences between samples in the first image includes: setting the index number of the first sample in the first image in the first list to a second value, determining the difference between the first sample and an adjacent second sample; and if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, setting the index number of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample; performing a first operation on an untraversed third sample adjacent to the second sample in the first image, the first operation including: finding a first target sample in the first list whose absolute value of the difference between it and the third sample is equal to the second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, finding a second target sample in the first list whose absolute value of the difference between it and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to the index number of the second target sample plus a fourth value.

[0134] In some embodiments, the method further includes: adding the second sample to the first list and not sorting the second sample if the absolute value of the difference between the first sample and the second sample is greater than a first threshold; the first operation further includes: adding the third sample to the first list and not sorting the third sample if no second target sample of the third sample is found; correspondingly, after traversing to the last sample of the first image, the method further includes: performing a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, determining the third target sample with the smallest absolute value of the difference between it and the current sample from the remaining unsorted samples different from the current sample; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0135] In some embodiments, obtaining the sample value of the first component in the second image that has the same position coordinates as the sample based on the index number of the sample in the first list and the bit depth of the first component includes: determining the maximum sample value representing the first component based on the bit depth; and obtaining the sample value of the first component in the second image that has the same position coordinates as the sample based on the index number of the sample in the first list and the maximum sample value.

[0136] Further, in some embodiments, obtaining the sample value of the first component in the second image that has the same positional coordinates as the sample based on the index number of the sample in the first list and the maximum sample value includes: determining a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and the product of the first ratio and the maximum sample value is taken as the sample value of the first component in the second image that has the same positional coordinates as the sample.

[0137] In some embodiments, the method further includes setting the sample values ​​of the second component and the third component of the second image to a first value.

[0138] Furthermore, in some embodiments, the first value is 0 or 2 to the power of (N-1), where N is the bit depth.

[0139] In some embodiments, the samples of the first image represent depth information.

[0140] In some embodiments, the samples of the first component of the second image represent depth information.

[0141] It should be noted that the description of the above-described embodiment of the decoder 50 side is similar to the description of the above-described embodiment of the encoder 20 side. For undisclosed technical details of embodiments or implementation steps with the same technical features, please refer to the description of the above-described embodiment of the encoder 20 side for understanding.

[0142] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0143] Some cameras used to acquire depth information output images in RGB24 format, meaning they output RGB depth sequences. Unlike the RGB texture map sequences obtained from visible light imaging, RGB depth sequences are derived from pseudo-color transformations of depth values. The principle of pseudo-color transformation is as follows... Figure 11 As shown, the depth value is transformed into RGB components through three transformation relationships: red transformation, green transformation, and blue transformation.

[0144] Figure 11The pseudocolor transformation relationships in colormaps are predefined and only exist for human visual perception; they have no specific physical meaning. Commonly used pseudocolor transformation relationships are continuous functions, mapping continuously changing depth values ​​to continuous red, green, or blue components. That is, the sample values ​​of the R, G, or B components change with the depth value. The transformation relationship of these three components defines a colormap. There is a one-to-one correspondence between normalized depth values ​​and the colormap. For example... Figure 7 As shown, this is a common HSV colormap. Based on this HSV colormap, such as... Figure 12 As shown, a pseudo-color transformation is performed on the depth map 1201 to obtain the RGB24 sequence 1202.

[0145] When using this type of camera to obtain an RGB24 sequence for video encoding, such as... Figure 13 As shown, the RGB24 sequence 1202 needs to be converted to the YUV420 sequence 1301 first through color space transformation. After the conversion, it can be compressed using a 2D encoder, such as VVC. At the decoding end, after decoding to obtain the YUV420 sequence 1302, it is then converted back to the RGB24 sequence 1303.

[0146] However, Figure 13 The processing of the RGB24 sequence 1202 shown ignores its depth characteristics. In RGB image sequences obtained from visible light imaging, pixel values ​​are discretely distributed in the RGB color space without a specific pattern. However, in the RGB24 sequence 1202 obtained by pseudo-color transformation of the depth map 1201, pixel values ​​are only distributed on the colormap used for pseudo-color transformation, that is, only on a specific curve in the RGB color space.

[0147] To address the unique characteristics of the RGB sequence obtained by pseudo-color transformation of depth map 1201, this application provides a universal algorithm that fits an approximate pseudo-color transformation relationship when the pseudo-color transformation relationship (i.e., colormap) is unknown, thereby recovering the normalized single-channel grayscale value. The normalized single-channel grayscale value, after scaling, serves as the Y component in the YUV420 format. The overall framework of this algorithm (i.e., an example of the overall framework of encoding / decoding system 1) is as follows: Figure 14As shown, this algorithm is used to restore the grayscale values ​​of the RGB24 sequence 1202 (an example of the first image described in the video image processing method on the encoder 20 side), resulting in a YUV420 sequence 1401 (an example of the second image described in the video image processing method on the encoder 20 side) with only the Y component having content; then, it is compressed by a two-dimensional encoder (such as VVC, an example of the encoding module 201). At the decoding end, the YUV420 sequence 1402 with only the Y component having content is obtained.

[0148] Thus, on the one hand, for the encoding end, the YUV420 sequence 1401 input to the 2D encoder only has content in the Y component, which facilitates compression and transmission, reducing the consumption of computing resources and transmission bandwidth; on the other hand, for the decoding end, the decoded YUV240 sequence 1402 does not need to be converted to an RGB24 sequence, that is, it can recover grayscale values ​​close to the depth values ​​even when the colormap is unknown.

[0149] For the input RGB24 sequence 1202 of the depth map, the processing flow of this algorithm includes the following steps 1601 to 1606:

[0150] Step 1601: Select the first pixel (r1, g1, b1) of RGB24 sequence 1202 as the current pixel, as a node of the doubly linked list, and point the pointer to NULL.

[0151] Here, samples of three components at the same coordinate position constitute a pixel.

[0152] Step 1602: If the current pixel is the last pixel, proceed to step 1605. If the current pixel is not the last pixel, select the next pixel as the current pixel.

[0153] Step 1603, if the current pixel (r) j g j b j If the current pixel (r) is already in the list, proceed to step 1602; if the current pixel (r) is already in the list, proceed to step 1602. j g j b j If the node is not in the linked list, treat it as a linked list node, set its pointer to NULL, and then proceed to step 1604.

[0154] Step 1604, traverse the nodes in the linked list: calculate the Manhattan distance d(r, g, b) between the current pixel and each node in the linked list. If d(r, g, b) = 1, point the current pixel's node to the corresponding node, and point the corresponding node to the current pixel's node. If d(r, g, b) ≠ 1, continue traversing the next node until all nodes in the linked list have been traversed. Go to step 1602.

[0155] Step 1605: If the number of endpoints in the linked list (i.e., at least one node whose pointer points to NULL) is greater than 2, select any endpoint of the linked list as the current node and go to 1606; if the number of endpoints in the linked list is 2, then end.

[0156] Step 1606: Find the linked list endpoint with the smallest Manhattan distance to the current node. Set the current node to point to this endpoint, and the endpoint to point to the current node. Proceed to step 1605.

[0157] After the above processing, a linked list is established. One end of the linked list is selected as the starting point, corresponding to a grayscale value of 0, and the other end is selected as the ending point, corresponding to a grayscale value of 2. N -1, where N is the bit depth. In a linked list with M nodes, the gray value corresponding to the k-th node is... .

[0158] This establishes a correspondence between RGB24 image 1202 and grayscale image, mapping it to a grayscale image as the Y component of YUV420 format, with the UV components set to 0 or... .

[0159] Compared to Figure 13 The processing flow shown demonstrates that, on the one hand, this algorithm can recover grayscale values ​​close to the depth values ​​even when the colormap is unknown; on the other hand, in this algorithm, only the Y component of the obtained YUV420 sequence contains content. Compared to... Figure 13 The YUV420 sequence 1301 obtained by this algorithm is easy to compress and transmit.

[0160] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0161] Based on the foregoing embodiments, this application provides a video image processing apparatus applied to a decoder (e.g., Figure 5 The decoder shown is 50). Figure 15 This is a schematic diagram of the structure of the video image processing apparatus provided in the embodiments of this application, such as... Figure 15 As shown, the video image processing device 150 includes: a first sorting module 1501 and a first mapping module 1502; wherein,

[0162] The first sorting module 1501 is configured to sort the samples according to the differences between the samples in the first image to obtain a first list;

[0163] The first mapping module 1502 is configured to obtain the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component.

[0164] In some embodiments, the first sorting module 1501 and the first mapping module 1502 are configured in the color space transformation module 501.

[0165] In some embodiments, the video image processing apparatus 150 further includes a decoding module configured to: decode the received bitstream to obtain a third image; and perform color space transformation on the third image to obtain the first image.

[0166] In some embodiments, the first sorting module 1501 is configured to: set the index number of the first sample in the first image in the first list to a second value; determine the difference between the first sample and the adjacent second sample; and, if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than the second threshold, set the index number of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample; and perform a first operation on an untraversed third sample adjacent to the second sample in the first image, the first operation including: finding a first target sample in the first list whose absolute value of the difference between it and the third sample is equal to the second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, finding a second target sample in the first list whose absolute value of the difference between it and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to the index number of the second target sample plus a fourth value.

[0167] In some embodiments, the first sorting module 1501 is further configured to: add the second sample to the first list and not sort the second sample if the absolute value of the difference between the first sample and the second sample is greater than a first threshold; the first operation further includes: adding the third sample to the first list and not sorting the third sample if no second target sample of the third sample is found; correspondingly, after traversing to the last sample of the first image, a second operation is performed on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, determining the third target sample with the smallest absolute value of the difference between it and the current sample from the remaining unsorted samples different from the current sample; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0168] In some embodiments, the first mapping module 1502 is configured to: determine the maximum sample value representing the first component based on the bit depth; and obtain the sample value of the first component in the second image that has the same position coordinates as the sample based on the index number of the sample in the first list and the maximum sample value.

[0169] Further, in some embodiments, the first mapping module 1502 is configured to: determine a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and multiply the first ratio by the maximum sample value as the sample value of the first component in the second image that has the same position coordinates as the sample.

[0170] This application provides a video image processing apparatus, applied to an encoder (e.g., Figure 2 The encoder 20 shown) Figure 16 This is a schematic diagram of the structure of the video image processing apparatus provided in the embodiments of this application, such as... Figure 16 As shown, the video image processing device 160 includes: a second sorting module 1601 and a second mapping module 1602; wherein,

[0171] The second sorting module 1601 is configured to sort the samples according to the differences between the samples in the first image to obtain a first list;

[0172] The second mapping module 1602 is configured to obtain the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component.

[0173] In some embodiments, the second sorting module 1601 and the second mapping module 1602 are configured in the color space transformation module 200.

[0174] In some embodiments, the video image processing apparatus 160 further includes an encoding module configured to: encode the second image and write the obtained encoded bits into a bitstream.

[0175] In some embodiments, the second sorting module 1601 is configured to: set the index number of the first sample in the first image in the first list to a second value; determine the difference between the first sample and the adjacent second sample; and, if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, set the index number of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample; and perform a first operation on an untraversed third sample adjacent to the second sample in the first image, the first operation including: finding a first target sample in the first list whose absolute value of the difference between it and the third sample is equal to the second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, finding a second target sample in the first list whose absolute value of the difference between it and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to the index number of the second target sample plus a fourth value.

[0176] In some embodiments, the second sorting module 1601 is further configured to: add the second sample to the first list and not sort the second sample if the absolute value of the difference between the first sample and the second sample is greater than a first threshold; the first operation further includes: adding the third sample to the first list and not sorting the third sample if no second target sample of the third sample is found; correspondingly, after traversing to the last sample of the first image, a second operation is performed on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, determining the third target sample with the smallest absolute value of the difference between it and the current sample from the remaining unsorted samples different from the current sample; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0177] In some embodiments, the second mapping module 1602 is configured to: determine the maximum sample value representing the first component based on the bit depth; and obtain the sample value of the first component in the second image that has the same position coordinates as the sample based on the index number of the sample in the first list and the maximum sample value.

[0178] Further, in some embodiments, the second mapping module 1602 is configured to: determine a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and multiply the first ratio by the maximum sample value as the sample value of the first component in the second image that has the same position coordinates as the sample.

[0179] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0180] It should be noted that the module division of the apparatus described in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0181] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0182] This application provides a computer-readable storage medium storing a computer program that, when executed, implements a video image processing method such as that on the encoder side or the decoder side.

[0183] This application provides a decoder, such as... Figure 17 As shown, the decoder 170 includes a first communication interface 1701, a first memory 1702, and a first processor 1703; the various components are coupled together via a first bus system 1704. It is understood that the first bus system 1704 is used to implement communication between these components. In addition to a data bus, the first bus system 1704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 17 The general designated all buses as the first bus system 1704. Among them,

[0184] The first communication interface 1701 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0185] The first memory 1702 is used to store computer programs that can run on the first processor 1703;

[0186] The first processor 1703 is configured to, when running the computer program, execute:

[0187] The samples are sorted according to the differences between the samples in the first image to obtain a first list;

[0188] Based on the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

[0189] It is understood that the first memory 1702 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The first memory 1702 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0190] The first processor 1703 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the first processor 1703 or by instructions in software form. The first processor 1703 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the first memory 1702. The first processor 1703 reads the information in the first memory 1702 and completes the steps of the above method in conjunction with its hardware.

[0191] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), DSP devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof. For software implementation, the technology described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0192] Alternatively, as another embodiment, the first processor 1703 is also configured to execute any of the aforementioned method embodiments on the decoder 50 side when running the computer program.

[0193] This application provides an encoder, such as Figure 18 As shown, encoder 180 includes: a second communication interface 1801, a second memory 1802, and a second processor 1803; the various components are coupled together via a second bus system 1804. It is understood that the second bus system 1804 is used to implement communication between these components. In addition to a data bus, the second bus system 1804 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 18 The lieutenant general labeled all buses as the second bus system 1804. Among them,

[0194] The second communication interface 1801 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0195] The second memory 1802 is used to store computer programs that can run on the second processor 1803;

[0196] The second processor 1803 is configured to, when running the computer program, perform:

[0197] The samples are sorted according to the differences between the samples in the first image to obtain a first list;

[0198] Based on the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

[0199] Alternatively, as another embodiment, the second processor 1803 is also configured to execute the aforementioned method embodiment on the encoder 20 side when running the computer program.

[0200] It is understood that the second memory 1802 has similar hardware functions to the first memory 1702, and the second processor 1803 has similar hardware functions to the first processor 1703; these will not be described in detail here.

[0201] This application provides a method for transmitting a bitstream, comprising: generating a bitstream by executing the method described in the encoding end of this application, and transmitting the bitstream.

[0202] This application provides a method for receiving a bitstream, including: receiving the bitstream, and executing the method described in the decoding end of this application to decode the bitstream and generate a video or image.

[0203] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0204] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0205] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0206] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0208] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0209] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0210] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0211] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0212] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0213] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0214] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0215] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video image processing method, the method being applied to a decoder, the method comprising: The samples are sorted according to the differences between the samples in the first image to obtain a first list; Based on the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

2. The method according to claim 1, wherein, The method further includes: Set the sample values ​​of the second and third components of the second image to the first value.

3. The method according to claim 1, wherein, The process of sorting the samples based on the differences between samples in the first image includes: Set the index number of the first sample in the first image in the first list to the second value, and determine the difference between the first sample and the adjacent second sample; and If the absolute value of the difference between the first sample and the second sample is less than or equal to the first threshold and greater than the second threshold, the index number of the second sample in the first list is set to the third value. When the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample. Perform a first operation on an untraversed third sample in the first image that is adjacent to the second sample. The first operation includes: finding a first target sample in the first list whose absolute value of the difference between it and the third sample is equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, find a second target sample in the first list whose absolute value of the difference between it and the third sample is less than or equal to the first threshold and greater than the second threshold, and set the index number of the third sample in the first list to the index number of the second target sample plus a fourth value.

4. The method according to claim 3, wherein, The method further includes: If the absolute value of the difference between the first sample and the second sample is greater than a first threshold, the second sample is added to the first list and the second sample is not sorted. The first operation further includes: adding the third sample to the first list if no second target sample of the third sample is found, and not sorting the third sample; Accordingly, after traversing to the last sample of the first image, the method further includes: Perform a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, determining a third target sample from the remaining unsorted samples different from the current sample whose absolute value of the difference between the current sample and the current sample is the smallest; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

5. The method according to any one of claims 1 to 4, wherein, The step of obtaining the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component, includes: Based on the bit depth, determine the maximum sample value representing the first component; Based on the index number of the sample in the first list and the maximum sample value, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

6. The method according to claim 5, wherein, The step of obtaining the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the maximum sample value, includes: A first ratio is determined based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein, the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; The product of the first ratio and the maximum sample value is used as the sample value of the first component in the second image that has the same position coordinates as the sample.

7. The method according to claim 2, wherein, The first value is 0 or 2 to the power of (N-1), where N is the bit depth.

8. The method according to any one of claims 1 to 4, wherein, The samples in the first image represent depth information.

9. The method according to any one of claims 1 to 4, wherein, The samples of the first component of the second image represent depth information.

10. The method according to any one of claims 1 to 4, wherein, The method further includes: The received bitstream is decoded to obtain the third image; The third image is transformed using a color space transformation to obtain the first image.

11. A video image processing method, the method being applied to an encoder, the method comprising: The samples are sorted according to the differences between the samples in the first image to obtain a first list; Based on the index number of the sample in the first list and the bit depth of the first component, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

12. The method according to claim 11, wherein, The method further includes: Set the sample values ​​of the second and third components of the second image to the first value.

13. The method according to claim 11, wherein, The process of sorting the samples based on the differences between samples in the first image includes: Set the index number of the first sample in the first image in the first list to the second value, and determine the difference between the first sample and the adjacent second sample; and If the absolute value of the difference between the first sample and the second sample is less than or equal to the first threshold and greater than the second threshold, the index number of the second sample in the first list is set to the third value. When the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample. Perform a first operation on an untraversed third sample in the first image that is adjacent to the second sample. The first operation includes: finding a first target sample in the first list whose absolute value of the difference between it and the third sample is equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, find a second target sample in the first list whose absolute value of the difference between it and the third sample is less than or equal to the first threshold and greater than the second threshold, and set the index number of the third sample in the first list to the index number of the second target sample plus a fourth value.

14. The method according to claim 13, wherein, The method further includes: If the absolute value of the difference between the first sample and the second sample is greater than a first threshold, the second sample is added to the first list and the second sample is not sorted. The first operation further includes: adding the third sample to the first list if no second target sample of the third sample is found, and not sorting the third sample; Accordingly, after traversing to the last sample of the first image, the method further includes: Perform a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, determining a third target sample from the remaining unsorted samples different from the current sample whose absolute value of the difference between the current sample and the current sample is the smallest; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

15. The method according to any one of claims 11 to 14, wherein, The step of obtaining the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component, includes: Based on the bit depth, determine the maximum sample value representing the first component; Based on the index number of the sample in the first list and the maximum sample value, the sample value of the first component in the second image that has the same position coordinates as the sample is obtained.

16. The method according to claim 15, wherein, The step of obtaining the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the maximum sample value, includes: A first ratio is determined based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein, the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; The product of the first ratio and the maximum sample value is used as the sample value of the first component in the second image that has the same position coordinates as the sample.

17. The method according to claim 12, wherein, The first value is 0 or 2 to the power of (N-1), where N is the bit depth.

18. The method according to any one of claims 11 to 14, wherein, The samples in the first image represent depth information.

19. The method according to any one of claims 11 to 14, wherein, The samples of the first component of the second image represent depth information.

20. The method according to any one of claims 11 to 14, wherein, The method further includes: The second image is encoded, and the resulting encoded bits are written into the bitstream.

21. A video image processing apparatus applied to a decoder, the apparatus comprising: The first sorting module and the first mapping module; wherein, The first sorting module is configured to sort the samples according to the differences between samples in the first image to obtain a first list; The first mapping module is configured to obtain the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component.

22. A decoder, comprising a first memory and a first processor; wherein, The first memory is used to store computer programs that can run on the first processor; The first processor is configured to perform the method as described in any one of claims 1 to 10 when running the computer program.

23. A video image processing apparatus applied to an encoder, the apparatus comprising: The second sorting module and the second mapping module; wherein, The second sorting module is configured to sort the samples according to the differences between samples in the first image to obtain a first list; The second mapping module is configured to obtain the sample value of the first component in the second image that has the same position coordinates as the sample, based on the index number of the sample in the first list and the bit depth of the first component.

24. An encoder, comprising a second memory and a second processor; wherein, The second memory is used to store computer programs that can run on the second processor; The second processor is configured to perform the method as described in any one of claims 11 to 20 when running the computer program.

25. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed, implements the method as claimed in any one of claims 1 to 10, or implements the method as claimed in any one of claims 11 to 20.

26. A method for transmitting a code stream, comprising: The method described in any one of claims 11 to 20 is used to generate a bitstream and to transmit the bitstream.

27. A method for receiving a bitstream, comprising: Receive a bitstream and decode the bitstream to generate a video or image by performing the method according to any one of claims 1 to 10.

Images