Video image processing method and device, codec, code stream and storage medium

Abstract

The invention discloses a video image processing method and device, a codec, a code stream and a storage medium, and the method comprises the steps: carrying out the sorting processing of samples according to the difference value between the samples in a first image, and obtaining a first list; and according to the index sequence number of the sample in the first list and the bit depth of the first component, obtaining a sample value of the first component having the same position coordinate as the sample in the second image.

Description

Video image processing method and device, codec, code stream, and storage medium Technical Field

[0001] The embodiments of the present application relate to video technology, including but not limited to video image processing methods and devices, codecs, bit streams, and storage media. Background Art

[0002] The images output by some depth cameras are images with three channels (such as RGB images). When performing video encoding based on the images output by such cameras, as shown in Figure 1, it is necessary to first convert the three-channel image 101 (color image) output by such cameras into an image 102 (color image) in the image format required by the two-dimensional encoder (such as YUV420 format) through color space transformation. After the conversion is completed, it can be compressed using a two-dimensional encoder 103, such as VVC. At the decoding end, after the image 105 in YUV420 format is decoded by the two-dimensional decoder 104, it is converted back to an RGB image 106, and then based on the transformation relationship between the RGB image 106 and the depth image, the corresponding depth image is restored. For the decoding end, the transformation relationship between the RGB image and the depth image needs to be known before the RGB image can be restored to a depth image.

[0003] Summary of the Invention

[0004] The video image processing method and apparatus, codec, bitstream, and storage medium provided in the embodiments of the present application can, at the decoding end, restore a first image (e.g., an RGB image) to a second image (e.g., a depth image) when the transformation relationship between the first image (e.g., an RGB image) and the second image (e.g., a depth image) is unknown. The video image processing method and apparatus, codec, bitstream, and storage medium provided in the embodiments of the present application are implemented as follows:

[0005] According to one aspect of an embodiment of the present application, a video image processing method is provided, which is applied to a decoder, and the method includes: sorting the samples in a first image according to the difference between the samples to obtain a first list; and obtaining the sample value of the first component in the second image having the same position coordinates as the sample according to the index sequence number of the sample in the first list and the bit depth of the first component.

[0006] In this way, for the decoder, even when the transformation relationship between the first image and the second image is unknown, the first image can be restored to the second image; the encoder does not need to transmit the transformation relationship between the first image and the second image to the decoder, thereby saving transmission resources; the decoder does not need to receive and decode the transformation relationship, thereby saving power consumption.

[0007] According to another aspect of an embodiment of the present application, a video image processing method is provided, which is applied to an encoder and includes: sorting the samples in a first image according to the difference between the samples to obtain a first list; and obtaining the sample value of the first component in the second image having the same position coordinates as the sample according to the index number of the sample in the first list and the bit depth of the first component.

[0008] In this way, for the encoder, when the transformation relationship between the first image and the second image is unknown, the first image can be restored to the second image; thus, on the one hand, since the first image is transformed into a single-channel second image, that is, the first component has content, while improving the transformation efficiency and saving the computational overhead, the data volume of the second image is also reduced, thereby saving power consumption due to the smaller workload when compressing and encoding the second image, and saving the transmission overhead due to the smaller amount of data after compression and encoding; on the other hand, after the decoder decodes the code stream to obtain the second image, there is no need to transform it again. Compared with the processing process of the decoder shown in Figure 1, two transformation processes are saved (for example, the transformation of RGB image to YUV420 image and the transformation of YUV420 image to depth image), thereby saving the power consumption of the decoder and improving the working efficiency of the decoder.

[0009] According to another aspect of an embodiment of the present application, a video image processing device is provided, which is applied to a decoder, and the device includes: a first sorting module and a first mapping module; wherein the first sorting module is configured to sort the samples according to the difference between the samples in the first image to obtain a first list; the first mapping module is configured to obtain the sample value of the first component in the second image having the same position coordinates as the sample based on the index number of the sample in the first list and the bit depth of the first component.

[0010] According to another aspect of an embodiment of the present application, a video image processing device is provided, which is applied to an encoder, and the device includes: a second sorting module and a second mapping module; wherein the second sorting module is configured to sort the samples according to the difference between the samples in the first image to obtain a first list; the second mapping module is configured to obtain the sample value of the first component in the second image having the same position coordinates as the sample based on the index number of the sample in the first list and the bit depth of the first component.

[0011] According to another aspect of an embodiment of the present application, a decoder is provided, comprising a first memory and a first processor; wherein the first memory is used to store a computer program that can be run on the first processor; and the first processor is used to execute the video image processing method applied to the decoder as described in an embodiment of the present application when running the computer program.

[0012] According to another aspect of an embodiment of the present application, an encoder is provided, comprising a second memory and a second processor; wherein the second memory is used to store a computer program that can be run on the second processor; and the second processor is used to execute the video image processing method applied to the encoder as described in an embodiment of the present application when running the computer program.

[0013] According to another aspect of the embodiments of the present application, a code stream is provided, which is generated by bit encoding a second image; wherein the second image is obtained by the steps in the video image processing method applied to the encoder.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the video image processing method applied to the decoder described in the embodiment of the present application is implemented, or the video image processing method applied to the encoder described in the embodiment of the present application is implemented.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings herein are incorporated into and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, serve to illustrate the technical solutions of the present application. Obviously, the drawings described below are merely some embodiments of the present application. Those skilled in the art can, without inventive effort, derive other drawings from these drawings.

[0017] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0018] FIG1 is a schematic diagram of a processing flow of a three-channel image 101;

[0019] FIG2 is a schematic diagram of the structure of an encoder 20 of the encoding and decoding system 1 provided in an embodiment of the present application;

[0020] FIG3 is a schematic structural diagram of a decoder 30 of the encoding and decoding system 1 provided in an embodiment of the present application;

[0021] FIG4 is a schematic diagram of the structure of an encoder 40 of the encoding and decoding system 2 provided in an embodiment of the present application;

[0022] FIG5 is a schematic structural diagram of a decoder 50 of the encoding and decoding system 2 provided in an embodiment of the present application;

[0023] FIG6 is a schematic diagram of an implementation flow of a video image processing method provided in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of a colormap;

[0025] FIG8 is a schematic diagram of an implementation flow of a sorting processing method provided in an embodiment of the present application;

[0026] FIG9 is a schematic diagram of an implementation flow of another sorting processing method provided in an embodiment of the present application;

[0027] FIG10 is a schematic diagram of an implementation flow of a video image processing method provided in an embodiment of the present application;

[0028] FIG11 is a schematic diagram showing the principle of pseudo-color conversion;

[0029] FIG12 is a schematic diagram of performing pseudo-color conversion on a depth map 1201;

[0030] FIG13 is a schematic diagram of the processing flow of the RGB24 sequence 1202;

[0031] FIG14 is a schematic diagram of a processing flow of an RGB24 sequence 1202 provided in an embodiment of the present application;

[0032] FIG15 is a schematic structural diagram of a video image processing device provided in an embodiment of the present application;

[0033] FIG16 is a schematic structural diagram of another video image processing device provided in an embodiment of the present application;

[0034] FIG17 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0035] FIG18 is a schematic diagram of the structure of the encoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0038] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are intended to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0040] The encoder and decoder frameworks and business scenarios described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. It will be appreciated by those skilled in the art that with the evolution of encoders and decoders and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0041] An encoding and decoding system to which an embodiment of the present application may be applicable is shown in FIG2 and FIG3 . The encoding and decoding system 1 includes an encoder 20 shown in FIG2 and a decoder 30 shown in FIG3 .

[0042] As shown in Figure 2, the encoder (specifically, a "video encoder") 20 may include a color space transformation module 200 and an encoding module 201; wherein the encoding module 201 includes: a transformation and quantization unit 2011, an intra-frame estimation unit 2012, an intra-frame prediction unit 2013, a motion compensation unit 2014, a motion estimation unit 2015, an inverse transformation and inverse quantization unit 2016, a filter control analysis unit 2017, a filtering unit 2018, an encoding unit 2019 and a decoded image cache unit 2010, etc.; wherein the filtering unit 2018 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 2019 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).

[0043] First, the color space conversion module 200 sorts the samples of the input first image according to the differences between the samples in the first image to obtain a first list; then, based on the index numbers of the samples in the first list and the bit depths of the first components, obtains the sample values ​​of the first components of the second image that have the same position coordinates as the samples; and then, outputs the obtained second image to the encoding module 201;

[0044] The coding module 201, for the second input image, uses the coding tree block (Coding Tree A video coding block can be obtained by dividing the video coding block into a plurality of frames (CTUs). Then, residual pixel information obtained after intra-frame or inter-frame prediction is transformed by a transform and quantization unit 2011, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate. An intra-frame estimation unit 2012 and an intra-frame prediction unit 2013 are used to perform intra-frame prediction on the video coding block. Specifically, the intra-frame estimation unit 2012 and the intra-frame prediction unit 2013 are used to determine an intra-frame prediction mode to be used to encode the video coding block. A motion compensation unit 2014 and a motion estimation unit 2015 are used to perform inter-frame prediction coding on the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information. The motion estimation performed by the motion estimation unit 2015 is a process of generating a motion vector, which can estimate the motion of the video coding block. The motion compensation unit 2014 then calculates the motion vector based on the motion vector determined by the motion estimation unit 2015. After determining the intra-frame prediction mode, the intra-frame prediction unit 2013 is further configured to provide the selected intra-frame prediction data to the encoding unit 2019, and the motion estimation unit 2015 also sends the calculated motion vector data to the encoding unit 2019. In addition, the inverse transform and inverse quantization unit 2016 is configured to reconstruct the video coding block and reconstruct a residual block in the pixel domain. The reconstructed residual block is subjected to the filter control analysis unit 2017 and the filtering unit 2018 to remove blocking artifacts. The reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 2010 to generate a reconstructed video coding block. The encoding unit 2019 is configured to encode various coding parameters and quantized transform coefficients. In a CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, thereby outputting a code stream of the second image. The decoded image buffer unit 2010 is configured to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image cache unit 2010 .

[0045] As shown in FIG3 , the decoder (specifically, a “video decoder”) 30 includes a decoding unit 301, an inverse transform and inverse quantization unit 302, an intra-frame prediction unit 303, a motion compensation unit 304, a filtering unit 305, and a decoded image cache unit 306, among others. The decoding unit 301 can implement header information decoding and CABAC decoding, and the filtering unit 305 can implement deblocking filtering and SAO filtering. After the second image is encoded by the encoding module 201 shown in FIG2 , a code stream of the second image is output; the code stream is input to the decoder 30, and first passes through the decoding unit 301 to obtain decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 302 to generate residual blocks in the pixel domain; the intra-frame prediction unit 303 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and data from the previously decoded blocks of the current frame or picture; the motion compensation unit 304 determines prediction information for the video decoding block by analyzing motion vectors and other associated syntax elements. , and uses the prediction information to generate a predictive block of the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 302 and the corresponding predictive block generated by the intra-frame prediction unit 303 or the motion compensation unit 304; the decoded video signal passes through the filtering unit 305 to remove blocking artifacts, which can improve the video quality; the decoded video block is then stored in the decoded image buffer unit 306, which stores the reference image used for subsequent intra-frame prediction or motion compensation, and is also used for outputting the video signal, that is, the restored second image is obtained.

[0046] Another encoding and decoding system to which the embodiments of the present application may be applicable is shown in FIG. 4 and FIG. 5 . The encoding and decoding system 2 includes an encoder 40 shown in FIG. 4 and a decoder 50 shown in FIG. 5 .

[0047] As shown in FIG4 , encoder (specifically, "video encoder") 40 includes a color space conversion module 400 and an encoding module 401. The structure of encoding module 401 is identical to that of encoding module 201 shown in FIG2 , and therefore will not be described again here. The function of color space conversion module 400 differs from that of color space conversion module 200 in that it employs a different color space conversion algorithm.

[0048] First, the color space conversion module 400 converts the input first image into a third image with the same number of components according to a predefined pseudo-color conversion relationship (e.g., colormap); wherein the third image has a different number of channels than the second image. For example, the third image is an image in YUV420 format, the second image is a depth image, and the first image is an RGB image (which can be an RGB image in any format). Then, the obtained third image is output to the encoding module 401.

[0049] The coding module 401, for the input third image, uses the coding tree block (Coding Tree A video coding block can be obtained by dividing the video coding block into a plurality of frames (CTUs) and then the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform and quantization unit 4011, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 4012 and the intra-frame prediction unit 4013 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 4012 and the intra-frame prediction unit 4013 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 4014 and the motion estimation unit 4015 are used to perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 4015 is a process of generating a motion vector, which can estimate the motion of the video coding block, and then the motion compensation unit 4014 calculates the motion vector based on the motion vector determined by the motion estimation unit 4015. After determining the intra-frame prediction mode, the intra-frame prediction unit 4013 is further configured to provide the selected intra-frame prediction data to the encoding unit 4019, and the motion estimation unit 4015 also sends the calculated motion vector data to the encoding unit 4019. In addition, the inverse transform and inverse quantization unit 4016 is configured to reconstruct the video coding block and reconstruct a residual block in the pixel domain. The reconstructed residual block is subjected to the filter control analysis unit 4017 and the filtering unit 4018 to remove the blocking artifacts. The reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 4010 to generate a reconstructed video coding block. The encoding unit 4019 is configured to encode various coding parameters and quantized transform coefficients. In a CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, thereby outputting a code stream of the third image. The decoded image buffer unit 4010 is configured to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image cache unit 4010 .

[0050] As shown in FIG5 , the decoder (specifically a “video decoder”) 50 includes a color space conversion module 501 and a decoding module 502 ; wherein the structure of the decoding module 502 is the same as that of the decoder 30 shown in FIG3 , and therefore will not be described again here.

[0051] After the input third image is encoded by the encoding module 401 shown in Figure 4, a code stream of the third image is output; the code stream is input to the decoding module 502 of the decoder 50, and first passes through the decoding unit 5021 to obtain decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 5022 to generate a residual block in the pixel domain; the intra-frame prediction unit 5023 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and data from the previously decoded block of the current frame or picture; the motion compensation unit 5024 determines prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses the prediction information to generate a predictive block for the video decoding block being decoded; the decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 5022 with the corresponding predictive block generated by the intra-frame prediction unit 5023 or the motion compensation unit 5024; the decoded video signal The signal passes through the filtering unit 5025 to remove block effect artifacts, which can improve video quality; the decoded video block is then stored in the decoded image cache unit 5026, which stores the reference image for subsequent intra-frame prediction or motion compensation, and is also used to output the video signal, that is, output the third image, and input the third image into the color space conversion module 501. The module 501 is configured to: use the opposite conversion relationship of the encoding end color space conversion module 400, that is, based on a predefined pseudo-color conversion relationship (such as colormap), or a predefined pseudo-color inverse conversion relationship to inversely transform the third image into the first image; and, according to the difference between the samples in the first image, sort the samples to obtain a first list; and, according to the index number of the sample in the first list and the bit depth of the first component, obtain the sample value of the first component with the same position coordinate as the sample in the second image, that is, restore the second image.

[0052] Thus, the color space conversion module 501 can recover the second image without knowing the transformation relationship between the second image and the first image. Consequently, the encoder 40 does not need to transmit the transformation relationship between the first image and the second image to the decoder 50, thereby saving resource overhead associated with transmission. The decoder also does not need to receive and decode the transformation relationship, thereby saving power consumption.

[0053] As for the codec system 1, the color space conversion module 200 in the encoder 20 directly converts the input first image into the second image when the conversion relationship between the first image and the second image is unknown, and then outputs it to the encoding module 201. In this way, on the one hand, since the first image is converted into a single-channel second image, that is, the first component has content, the number of converted samples is reduced, thereby improving the conversion efficiency and saving the computational overhead, and also reducing the data volume of the second image. In addition, when compressing and encoding the second image, power consumption is saved due to the smaller workload, and the transmission overhead is saved due to the smaller data volume after compression encoding. On the other hand, after decoding the code stream to obtain the second image, the decoder 30 does not need to perform further transformation. Compared with the decoder 50 shown in Figure 5, after obtaining the third image through the decoding module 502, it needs to restore the third image to the second image through two transformations through the color space conversion module 501. The decoder 30 saves two transformation processes, thereby saving power consumption and improving the working efficiency of the decoder 30.

[0054] It should be noted that in a video image, three color components are generally used to represent a coding block (CB); wherein the three color components are a luminance component, a blue chrominance component, and a red chrominance component. For example, the luminance component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb or U, and the red chrominance component is usually represented by the symbol Cr or V; thus, the video image can be represented in YCbCr format or in YUV format. In addition, the video image can also be in RGB format or YCgCo format, etc., which is not limited in the embodiments of the present application.

[0055] The embodiment of the present application provides a video image processing method, which can be applied to the encoder 20 shown in FIG2 . FIG6 is a schematic diagram of an implementation flow of the video image processing method provided in the embodiment of the present application. As shown in FIG6 , the method includes the following steps 601 and 603:

[0056] Step 601 : sorting the samples in a first image according to the differences between the samples to obtain a first list.

[0057] When the samples are sorted, three samples at the same position coordinate are sorted as a combination or a whole.

[0058] In some embodiments, this can be achieved through steps 6011 to 6019 of the following embodiments, which will not be described again here.

[0059] In some embodiments, the first image is an image having three channels, for example, an RGB image. Accordingly, the samples in the first image are samples of three components. For example, the samples include samples of an R component, a G component, and a B component.

[0060] In the embodiments of the present application, the format of the first image is not limited and can be a three-channel image in any format. In short, the first image is a three-channel image transformed from a single-channel image based on a pseudo-color transformation relationship. Essentially, the information of the single-channel image is represented by a three-channel image. For this scenario, the video image processing methods provided in the embodiments of the present application are applicable.

[0061] For example, based on the colormap shown in Figure 7, the depth image is transformed, and the resulting transformation is an example of the first image. The colormap is predefined for ease of visual interpretation and has no specific physical meaning. In this colormap, the R, G, and B components change with depth values, for example, increasing or decreasing with increasing depth values.

[0062] Step 602: Obtain a sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component.

[0063] In the embodiments of the present application, there is no restriction on the format of the second image. Regardless of whether the second image has three components or one component, as long as the sample value of a component represents actual information, it is sufficient. For example, at the encoding end, if the first image is converted from a depth image, the sample value of the first component of the second image only represents depth information, while the sample values ​​of the remaining components of the second image are meaningless. The sample values ​​of the remaining components at each position coordinate can be set to the first value.

[0064] In some embodiments, the format of the second image can be YUV400 or YUV420. When the format of the second image is YUV420, the first component is the luminance component, and the second and third components are chrominance components; the sample values ​​of the second and third components of the second image are set to the first value.

[0065] The first value can be any value. For example, the first value is 0 or 2. N-1 , N is the bit depth of the first component.

[0066] Step 603: Encode the second image and write the obtained coded bits into a bitstream.

[0067] It can be understood that in the embodiment of the present application, the encoder 20 can transform the first image into the second image through the above-mentioned steps 601 and 602 without knowing the transformation relationship between the first image and the second image; since the first image is transformed into a single-channel second image, that is, the first component has content, the number of transformed samples is reduced, thereby improving the transformation efficiency and saving computing overhead, while also reducing the data volume of the second image, thereby saving power consumption due to the smaller workload when compressing and encoding the second image, and saving the transmission overhead due to the smaller amount of data after compression encoding; on the other hand, the decoder 30 does not need to transform the second image after decoding the code stream. Compared with the decoder 50 shown in Figure 5, after obtaining the third image through the decoding module 502, it is necessary to restore the third image to the second image through two transformations through the color space transformation module 501. The decoder 30 saves two transformation processes, thereby saving power consumption and improving the working efficiency of the decoder 30.

[0068] The following describes further optional implementations and related terms of each of the above steps.

[0069] In step 601, samples in a first image are sorted according to differences between the samples to obtain a first list.

[0070] In some embodiments, as shown in FIG8 , step 601 may be implemented through steps 6011 to 6019 as follows:

[0071] Step 6011: Set the index number of the first sample in the first image in the first list to a second value.

[0072] In some embodiments, when the first image is a three-component image, the first sample includes samples of the three components at the same position coordinate in the first image. The samples of the three components at the same position coordinate can be treated as a whole (i.e., a combination or vector), and an index number is assigned to the whole.

[0073] For example, in some embodiments, the samples in the first image are sorted by a linked list, and samples of three components of the same position coordinate can be stored in the same node.

[0074] In the embodiment of the present application, the first sample may be a sample at any position coordinate in the first image, for example, it may be a sample at the position coordinate (0, 0).

[0075] In the embodiment of the present application, the second value can be any value as long as it can accurately represent the ranking of the first sample in the first list and can obtain an accurate second image. For example, the second value is 1.

[0076] Step 6012: Determine the difference between the first sample and the adjacent second sample.

[0077] In some embodiments, the term "adjacent" refers to adjacent position coordinates. The second sample is a sample adjacent to the position coordinates of the first sample. For example, the position coordinates of the first sample are (i, j), and the position coordinates of the second sample are (i+1, j) or (i, j+1).

[0078] In the case where the first image is an image with three components, the first sample includes samples of the three components at the same position coordinate in the first image, and the same is true for the second sample. Accordingly, the difference between the first sample and the second sample refers to the sum of the corresponding differences of the three components.

[0079] In the embodiments of the present application, there is no limitation on the method for calculating the difference, that is, there is no limitation on the parameter representing the difference. The Manhattan distance between the first sample and the second sample can be used as the difference, or the cosine similarity or Euclidean distance between the first sample and the second sample can be used as the difference. In short, any method that can measure the difference in sample values ​​between the two samples is sufficient.

[0080] The method for calculating the difference between the other two samples is the same as the method for calculating the difference between the first sample and the second sample, and will not be repeated below.

[0081] Step 6013 , determining whether the absolute value of the difference between the first sample and the second sample is equal to a second threshold; if so, executing step 6014 ; otherwise, executing step 6015 .

[0082] In some embodiments, the second threshold is equal to 0; that is, the sample value of the second sample is the same as the sample value of the first sample.

[0083] For example, the sample value of the R component in the first sample is R1, the sample value of the G component is G1, and the sample value of the B component is B1; the sample value of the R component in the second sample is also R1, the sample value of the G component is also G1, and the sample value of the B component is also B1; then the absolute value of the difference between the first sample and the second sample is equal to 0, and the sample values ​​of the two are the same; where R1, G1, and B1 are constants.

[0084] In the embodiment of the present application, the second threshold is not limited and can be equal to 0 or a value greater than 0. When the second threshold is greater than 0, the index numbers of two sample values ​​that are close are set to the same value.

[0085] In step 6014, the index number of the second sample is the same as the index number of the first sample; then proceed to step 6018;

[0086] Step 6015, determine whether the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold; if so, execute step 6017; otherwise, execute step 6016; wherein the first threshold is greater than the second threshold.

[0087] In some embodiments, when the difference is represented by Manhattan distance, the first threshold is 1. Accordingly, step 6015 is to determine whether the Manhattan distance between the first sample and the second sample is equal to 1; if so, execute step 6017; otherwise, execute step 6016.

[0088] In the embodiment of the present application, there is no restriction on the size of the first threshold. In short, the samples in the first image can be finally sorted according to their size to determine the order of the samples (here refers to the sample combination with the same position coordinates).

[0089] Step 6016: Add the second sample to the first list, and do not sort the second sample; that is, do not set an index number for the second sample; then proceed to step 6018;

[0090] Step 6017: Set the index number of the second sample in the first list to a third value; then proceed to step 6018;

[0091] In some embodiments, the third value is greater than the second value; in other embodiments, the third value is less than the second value.

[0092] In some embodiments, the absolute value of the difference between the third value and the second value is 1. For example, the second value is 1 and the third value is 2.

[0093] Of course, there is no limitation on the size of the third value. In short, it only needs to be able to accurately represent the ranking of the second sample in the first list and to obtain an accurate second image.

[0094] Step 6018: Perform a first operation on a third sample in the first image that is adjacent to the second sample and has not been traversed.

[0095] The first operation includes: searching for a first target sample in the first list whose absolute value of the difference between the first target sample and the third sample is equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, searching for a second target sample in the first list whose absolute value of the difference between the first target sample and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to the index number of the second target sample plus a fourth value; if the second target sample of the third sample is not found, adding the third sample to the first list and not sorting the third sample; then, continuing to perform the first operation on samples in the first image that are adjacent to the third sample but have not been traversed until the last sample of the first image is traversed to obtain the first list; then, entering step 6019.

[0096] It should be noted that the so-called untraversed samples refer to samples that have not yet been added to the first list. The samples with set index numbers are already in the first list.

[0097] In some embodiments, if a second target sample is found when the first target sample is not found, the index number of the third sample in the first list is set to the index number of the second target sample plus a fourth value; then, the first operation is continued to be performed on the untraversed samples adjacent to the third sample in the first image until the last sample of the first image is traversed, and the first list is obtained; then, step 6019 is entered.

[0098] In the embodiment of the present application, the fourth value may be any value, for example, the fourth value is 1. In short, it is sufficient as long as the position of the sample in the first list can be accurately represented and an accurate second image can be obtained.

[0099] Step 6019: Perform a second operation on the unsorted samples in the first list;

[0100] The second operation includes: selecting any unsorted sample in the first list as the current sample, determining a third target sample whose absolute difference with the current sample is the smallest from the remaining unsorted samples different from the current sample; and setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0101] Then, the second operation is continued until the cutoff condition is met to obtain the first list. Here, the cutoff condition is that there are two unsorted samples in the first list, and one of the two samples is used as the starting point of the list and the other as the end point of the list.

[0102] In the embodiment of the present application, steps 6011 to 6019 can be implemented in any manner. The data structure of the first list is not limited and can be a linked list structure or other types of data structures. In short, any method that can represent the order of each sample is sufficient.

[0103] Taking the first list as a doubly linked list as an example, as shown in FIG9 , the doubly linked list can be obtained as follows (i.e., steps 901 to 908):

[0104] Step 901: Select a position coordinate in the first image as the current position, and use the samples of the three components at the current position as nodes of a bidirectional linked list, with the pointer of the node pointing to NULL; then proceed to step 902;

[0105] Step 902, determining whether the current position is the last untraversed position in the first image; if so, executing step 907; otherwise, executing step 903;

[0106] Step 903: Select the next position coordinate as the current position and determine whether the samples of the three components at the current position are already in the linked list; if yes, return to step 903; otherwise, execute step 904;

[0107] Step 904: Set the samples of the three components at the current position as a linked list node, with the pointer of the node pointing to NULL, and then proceed to step 905;

[0108] Step 905: traverse the nodes in the linked list and calculate the Manhattan distance between the samples of the three components at the current position and other nodes in the linked list; determine whether the distance is equal to 1; if so, execute step 906; otherwise, continue traversing the next node in the linked list until the traversal is completed, and then return to execute step 903;

[0109] Step 906: Point the pointer of the linked list node corresponding to the current position to the other node, and point the pointer of the other node to the linked list node corresponding to the current position; then, return to step 903;

[0110] Step 907, determine whether the number of linked list endpoints is greater than 2; if so, execute step 908; otherwise, end; wherein the linked list endpoint refers to the node in the linked list whose pointer points to NULL;

[0111] Step 908: Select any linked list endpoint as the current node, find the target endpoint with the smallest Manhattan distance to the current node from the linked list, point the pointer of the current node to the target endpoint, and point the pointer of the target endpoint to the current node, and then return to execute step 907.

[0112] In step 602, a sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the bit depth of the first component.

[0113] In some embodiments, after obtaining the first list through the above method, the second image is obtained through the following steps 6021 to 6022, that is, step 602 is implemented:

[0114] Step 6021: Determine a maximum sample value representing the first component according to the bit depth of the first component.

[0115] In some embodiments, the maximum sample value is (2 N -1), where N is the bit depth of the first component.

[0116] Step 6022: Obtain the sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the maximum sample value.

[0117] Specifically, in some embodiments, a first ratio is determined based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and the product of the first ratio and the maximum sample value is used as the sample value of the first component in the second image having the same position coordinates as the sample.

[0118] For example, the sample value y(k) of the first component having the same position coordinates as the sample in the second image is obtained by the following formula (1):

[0119]

[0120] In formula (1), M is the total number of index numbers, and k is the index number of the sample in the first list.

[0121] In this way, based on formula (1), the correspondence between the first image and the second image is established, and the first image is transformed into the second image.

[0122] In step 603, the second image is encoded, and the obtained encoded bits are written into a bitstream.

[0123] Specifically, in some embodiments, the second image may be sent to the encoding module 201 to implement encoding of the second image.

[0124] In some embodiments, the method further comprises: setting sample values ​​of the second component and the third component of the second image to a first numerical value.

[0125] In the embodiments of the present application, the first value may be any value. For example, in some embodiments, the first value is 0 or 2 to the power of (N-1), where N is the bit depth.

[0126] In some embodiments, the samples of the first image represent depth information.

[0127] In some embodiments, the samples of the first component of the second image represent depth information.

[0128] It should be noted that the difference between the first image and the second image is that they are in different formats. For example, the first image is an image in RGB format, and the second image is an image in YUV format.

[0129] The embodiment of the present application further provides a video image processing method, which can be applied to the decoder 50 shown in FIG5 . FIG10 is a schematic diagram of an implementation flow of the video image processing method provided in the embodiment of the present application. As shown in FIG10 , the method includes the following steps 1001 to 1004:

[0130] Step 1001: decode the received code stream to obtain a third image;

[0131] Step 1002: performing color space transformation on the third image to obtain a first image;

[0132] Step 1003: sorting the samples in the first image according to the differences between the samples to obtain a first list;

[0133] Step 1004 : Obtain, according to the index number of the sample in the first list and the bit depth of the first component, a sample value of the first component having the same position coordinates as the sample in the second image.

[0134] In some embodiments, the sorting of samples in the first image according to the difference between samples includes: setting an index number of a first sample in the first image in the first list to a second value, determining a difference between the first sample and an adjacent second sample; and, if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, setting the index number of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index number of the second sample is the same as the index number of the first sample; performing a first operation on a third sample in the first image that is adjacent to the second sample and has not been traversed, the first operation including: searching for a first target sample in the first list whose absolute value of the difference between the first sample and the third sample is equal to the second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, searching for a second target sample in the first list whose absolute value of the difference between the first sample and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index number of the third sample in the first list to be the index number of the second target sample plus the fourth value.

[0135] In some embodiments, the method further includes: when the absolute value of the difference between the first sample and the second sample is greater than a first threshold, adding the second sample to the first list and not sorting the second sample; the first operation further includes: when the second target sample of the third sample is not found, adding the third sample to the first list and not sorting the third sample; accordingly, after traversing to the last sample of the first image, the method further includes: performing a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, and determining, from the remaining unsorted samples different from the current sample, a third target sample with the smallest absolute value of the difference with the current sample; and setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0136] In some embodiments, obtaining the sample value of the first component in the second image having the same position coordinates as the sample based on the index number of the sample in the first list and the bit depth of the first component includes: determining the maximum sample value representing the first component based on the bit depth; and obtaining the sample value of the first component in the second image having the same position coordinates as the sample based on the index number of the sample in the first list and the maximum sample value.

[0137] Further, in some embodiments, obtaining the sample value of the first component in the second image having the same position coordinates as the sample based on the index number of the sample in the first list and the maximum sample value includes: determining a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and taking the product of the first ratio and the maximum sample value as the sample value of the first component in the second image having the same position coordinates as the sample.

[0138] In some embodiments, the method further comprises: setting sample values ​​of the second component and the third component of the second image to a first numerical value.

[0139] Further, in some embodiments, the first value is 0 or 2 to the power of (N-1), where N is the bit depth.

[0140] In some embodiments, the samples of the first image represent depth information.

[0141] In some embodiments, the samples of the first component of the second image represent depth information.

[0142] It should be noted that the description of the embodiment on the decoder 50 side is similar to the description of the embodiment on the encoder 20 side. For undisclosed technical details of embodiments or implementation steps with the same technical features, please refer to the description of the embodiment on the encoder 20 side for understanding.

[0143] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0144] Some cameras used to capture depth information output images in RGB24 format, meaning they produce an RGB depth sequence. Unlike the RGB texture sequence obtained from visible light imaging, the RGB depth sequence is derived from a pseudo-color transformation of the depth values. The principle of pseudo-color transformation is shown in Figure 11. Depth values ​​are converted to RGB components using three transformation relationships: red, green, and blue.

[0145] The pseudo-color transformation relationship in Figure 11 is pre-defined only for the convenience of human eye discrimination and has no specific physical meaning. The commonly used pseudo-color transformation relationship is a continuous function that maps continuously changing depth values ​​to continuous red, green or blue components. In other words, the sample values ​​of the R, G or B components change with the change of the depth value. The transformation relationship of the three components determines a colormap. The normalized depth value and the colormap have a one-to-one correspondence. For example, as shown in Figure 7, it is a common HSV colormap. Based on this HSV colormap, as shown in Figure 12, the depth map 1201 is pseudo-colored to obtain an RGB24 sequence 1202.

[0146] When encoding the RGB24 sequence obtained using this type of camera, as shown in Figure 13, the RGB24 sequence 1202 must first be converted to a YUV420 sequence 1301 through color space conversion. Once the conversion is complete, it can be compressed using a two-dimensional encoder such as VVC. At the decoding end, the YUV420 sequence 1302 is decoded and then converted back to an RGB24 sequence 1303.

[0147] However, the processing of the encoding-side RGB24 sequence 1202 shown in FIG13 ignores the depth characteristics of the RGB24 sequence 1202. In the RGB image sequence obtained by visible light imaging, pixel values ​​are discretely distributed in the RGB color space without a specific pattern. In contrast, in the RGB24 sequence 1202 obtained by pseudo-color transformation of the depth map 1201, pixel values ​​are distributed only on the colormap used for pseudo-color transformation, that is, only on a specific curve in the RGB color space.

[0148] In view of the particularity of the RGB sequence obtained by pseudo-color transformation of the depth map 1201, in an embodiment of the present application, a universal algorithm is provided. Under the premise that the pseudo-color transformation relationship (i.e., colormap) is unknown, a transformation relationship similar to the pseudo-color transformation is fitted to restore the normalized single-channel grayscale value. The normalized single-channel grayscale value is scaled and used as the Y component in the YUV420 format. As shown in FIG14 , the overall framework of the algorithm (i.e., an example of the overall framework of the encoding and decoding system 1) is used to restore the grayscale value of the RGB24 sequence 1202 (i.e., an example of the first image described in the video image processing method on the encoder 20 side) using the algorithm to obtain a YUV420 sequence 1401 (i.e., an example of the second image described in the video image processing method on the encoder 20 side) with only the Y component having content; and then it is compressed by a two-dimensional encoder (such as VVC, i.e., an example of the encoding module 201). At the decoding end, the YUV420 sequence 1402 with only the Y component having content is obtained by decoding;

[0149] In this way, on the one hand, for the encoding end, the YUV420 sequence 1401 input to the two-dimensional encoder only has content in the Y component, which is convenient for compression and transmission, reducing the consumption of computing resources and transmission bandwidth; on the other hand, for the decoding end, the decoded YUV240 sequence 1402 does not need to be converted into an RGB24 sequence, that is, when the colormap is unknown, the grayscale value close to the depth value can be restored.

[0150] For the input RGB24 sequence 1202 of the depth map, the processing flow of this algorithm includes the following steps 1601 to 1606:

[0151] Step 1601: Select the first pixel (r1, g1, b1) of the RGB24 sequence 1202 as the current pixel, and use it as a node of a bidirectional linked list, with the pointer pointing to NULL.

[0152] Here, samples of three components at the same position coordinates constitute one pixel.

[0153] Step 1602: If the current pixel is the last pixel, go to step 1605. If the current pixel is not the last pixel, select the next pixel as the current pixel;

[0154] Step 1603: If the current pixel (r j , g j , b j ) is already in the linked list, go to step 1602; if the current pixel (r j , g j , b j ) is not in the linked list, treat it as a linked list node, point the pointer to NULL, and then go to step 1604;

[0155] Step 1604, traverse the nodes in the linked list: calculate the Manhattan distance d(r, g, b) between the current pixel and each node in the linked list. If d(r, g, b) = 1, point the node of the current pixel to the corresponding node, and point the corresponding node to the node of the current pixel. If d(r, g, b) ≠ 1, continue traversing the next node until all nodes in the linked list are traversed. Go to step 1602.

[0156] Step 1605: If the number of linked list endpoints (i.e., at least one node with a pointer pointing to NULL) is greater than 2, select any linked list endpoint as the current node and go to step 1606; if the number of linked list endpoints is 2, then end;

[0157] Step 1606: Find the linked list endpoint with the smallest Manhattan distance to the current node. Point the current node to this endpoint, and the endpoint points to the current node, and then go to step 1605.

[0158] After the above processing, a linked list is established. One end of the linked list is selected as the starting point, corresponding to the grayscale value 0, and the other end of the linked list is selected as the end point, corresponding to the grayscale value 2 N -1, where N is the bit depth. In a linked list with M nodes, the grayscale value corresponding to the kth node is

[0159] Thus, the correspondence between the RGB24 image 1202 and the grayscale image is established, and it is mapped to a grayscale image as the Y component of the YUV420 format, and the UV component is set to 0 or 2. N-1 .

[0160] Compared to the processing flow shown in Figure 13, this algorithm can, on the one hand, recover grayscale values ​​close to the depth value even when the colormap is unknown. On the other hand, in this algorithm, only the Y component of the resulting YUV420 sequence has content. Compared to the YUV420 sequence 1301 in Figure 13, the YUV420 sequence generated by this algorithm is easier to compress and transmit.

[0161] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps; or steps in different embodiments may be combined to form a new technical solution.

[0162] Based on the aforementioned embodiments, an embodiment of the present application provides a video image processing device, which is applied to a decoder (e.g., the decoder 50 shown in FIG5 ). FIG15 is a schematic structural diagram of the video image processing device provided by the embodiment of the present application. As shown in FIG15 , the video image processing device 150 includes: a first sorting module 1501 and a first mapping module 1502; wherein,

[0163] A first sorting module 1501 is configured to sort the samples in the first image according to the differences between the samples to obtain a first list;

[0164] The first mapping module 1502 is configured to obtain a sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component.

[0165] In some embodiments, the first sorting module 1501 and the first mapping module 1502 are configured in the color space conversion module 501 .

[0166] In some embodiments, the video image processing device 150 further includes a decoding module configured to: decode the received code stream to obtain a third image; and perform color space transformation on the third image to obtain the first image.

[0167] In some embodiments, the first sorting module 1501 is configured to: set the index of a first sample in the first image in the first list to a second value, determine a difference between the first sample and an adjacent second sample; and if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, set the index of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index of the second sample is the same as the index of the first sample; perform a first operation on a third sample in the first image that is adjacent to the second sample and has not been traversed, the first operation comprising: searching for a first target sample in the first list whose absolute value of the difference between the first sample and the third sample is equal to the second threshold; if the first target sample is found, the index of the third sample is the same as the index of the first target sample; if the first target sample is not found, searching for a second target sample in the first list whose absolute value of the difference between the first sample and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index of the third sample in the first list to be the index of the second target sample plus the fourth value.

[0168] In some embodiments, the first sorting module 1501 is further configured to: when the absolute value of the difference between the first sample and the second sample is greater than a first threshold, add the second sample to the first list and do not sort the second sample; the first operation also includes: when the second target sample of the third sample is not found, add the third sample to the first list and do not sort the third sample; accordingly, after traversing to the last sample of the first image, perform a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, and determining the third target sample with the smallest absolute value of the difference between the sample and the current sample from the remaining unsorted samples different from the current sample; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0169] In some embodiments, the first mapping module 1502 is configured to: determine the maximum sample value representing the first component based on the bit depth; and obtain the sample value of the first component with the same position coordinates as the sample in the second image based on the index number of the sample in the first list and the maximum sample value.

[0170] Further, in some embodiments, the first mapping module 1502 is configured to: determine a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and use the product of the first ratio and the maximum sample value as the sample value of the first component in the second image having the same position coordinates as the sample.

[0171] The embodiment of the present application provides a video image processing device, which is applied to an encoder (e.g., the encoder 20 shown in FIG2 ). FIG16 is a schematic structural diagram of the video image processing device provided by the embodiment of the present application. As shown in FIG16 , the video image processing device 160 includes: a second sorting module 1601 and a second mapping module 1602; wherein,

[0172] A second sorting module 1601 is configured to sort the samples in the first image according to the differences between the samples to obtain a first list;

[0173] The second mapping module 1602 is configured to obtain a sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component.

[0174] In some embodiments, the second sorting module 1601 and the second mapping module 1602 are configured in the color space conversion module 200 .

[0175] In some embodiments, the video image processing device 160 further includes an encoding module configured to: encode the second image and write the obtained encoding bits into a bit stream.

[0176] In some embodiments, the second sorting module 1601 is configured to: set the index of a first sample in the first image in the first list to a second value, determine the difference between the first sample and an adjacent second sample; and if the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, set the index of the second sample in the first list to a third value; if the absolute value of the difference between the first sample and the second sample is equal to the second threshold, the index of the second sample is the same as the index of the first sample; perform a first operation on an untraversed third sample adjacent to the second sample in the first image, the first operation comprising: searching for a first target sample in the first list whose absolute value of the difference between the first sample and the third sample is equal to the second threshold; if the first target sample is found, the index of the third sample is the same as the index of the first target sample; if the first target sample is not found, searching for a second target sample in the first list whose absolute value of the difference between the first sample and the third sample is less than or equal to the first threshold and greater than the second threshold, and setting the index of the third sample in the first list to be the index of the second target sample plus the fourth value.

[0177] In some embodiments, the second sorting module 1601 is further configured to: when the absolute value of the difference between the first sample and the second sample is greater than a first threshold, add the second sample to the first list and do not sort the second sample; the first operation also includes: when the second target sample of the third sample is not found, add the third sample to the first list and do not sort the third sample; accordingly, after traversing to the last sample of the first image, perform the second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as the current sample, and determining the third target sample with the smallest absolute value of the difference between the sample and the current sample from the remaining unsorted samples different from the current sample; setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

[0178] In some embodiments, the second mapping module 1602 is configured to: determine the maximum sample value representing the first component based on the bit depth; and obtain the sample value of the first component with the same position coordinates as the sample in the second image based on the index number of the sample in the first list and the maximum sample value.

[0179] Further, in some embodiments, the second mapping module 1602 is configured to: determine a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is the ratio of (k-1) to (M-1); M is the total number of index numbers, and k is the index number of the sample in the first list; and use the product of the first ratio and the maximum sample value as the sample value of the first component in the second image having the same position coordinates as the sample.

[0180] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0181] It should be noted that the division of modules in the device described in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or they can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It can also be implemented in the form of a combination of software and hardware.

[0182] It should be noted that, in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0183] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, it implements a video image processing method such as an encoder side, or implements a video image processing method such as a decoder side.

[0184] The embodiment of the present application provides a decoder, as shown in FIG17 , the decoder 170 includes: a first communication interface 1701, a first memory 1702, and a first processor 1703; each component is coupled together via a first bus system 1704. It is understood that the first bus system 1704 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 1704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG17 , various buses are labeled as the first bus system 1704. Among them,

[0185] The first communication interface 1701 is used to receive and send signals when sending and receiving information with other external network elements;

[0186] A first memory 1702 is used to store computer programs that can be run on the first processor 1703;

[0187] The first processor 1703 is configured to, when running the computer program, execute:

[0188] Sorting the samples in the first image according to differences between the samples to obtain a first list;

[0189] A sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the bit depth of the first component.

[0190] It is understood that the first memory 1702 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 1702 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0191] The first processor 1703 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 1703. The above-mentioned first processor 1703 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 1702 , and the first processor 1703 reads the information in the first memory 1702 and completes the steps of the above method in combination with its hardware.

[0192] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0193] Optionally, as another embodiment, the first processor 1703 is further configured to execute any one of the method embodiments on the decoder 50 side when running the computer program.

[0194] The present application implements an encoder, as shown in FIG18 , the encoder 180 includes: a second communication interface 1801, a second memory 1802, and a second processor 1803; each component is coupled together via a second bus system 1804. It is understood that the second bus system 1804 is used to implement connection and communication between these components. In addition to the data bus, the second bus system 1804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in FIG18 , various buses are labeled as the second bus system 1804. Among them,

[0195] The second communication interface 1801 is used to receive and send signals when sending and receiving information with other external network elements;

[0196] The second memory 1802 is used to store computer programs that can be run on the second processor 1803;

[0197] The second processor 1803 is configured to, when running the computer program, execute:

[0198] Sorting the samples in the first image according to differences between the samples to obtain a first list;

[0199] A sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the bit depth of the first component.

[0200] Optionally, as another embodiment, the second processor 1803 is further configured to execute the aforementioned method embodiment on the encoder 20 side when running the computer program.

[0201] It can be understood that the hardware functions of the second memory 1802 are similar to those of the first memory 1702, and the hardware functions of the second processor 1803 are similar to those of the first processor 1703; they will not be described in detail here.

[0202] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0203] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0204] The term "and / or" in this article is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, object A and / or object B can mean: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0205] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0207] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed across multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

[0208] In addition, all functional modules in the embodiments of the present application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the above-mentioned integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0209] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0210] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0211] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0212] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0213] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0214] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video image processing method, applied to a decoder, comprising: Sorting the samples in the first image according to differences between the samples to obtain a first list; A sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the bit depth of the first component.

2. The method according to claim 1, wherein The method further comprises: The sample values ​​of the second component and the third component of the second image are set to the first value.

3. The method according to claim 1, wherein The sorting of the samples according to the differences between the samples in the first image includes: Setting the index number of a first sample in the first list in the first image to a second value, and determining a difference between the first sample and an adjacent second sample; and When the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, setting the index number of the second sample in the first list to a third value; When the absolute value of the difference between the first sample and the second sample is equal to a second threshold, the index number of the second sample is the same as the index number of the first sample; A first operation is performed on an untraversed third sample adjacent to the second sample in the first image, the first operation comprising: searching for a first target sample in the first list, the absolute value of a difference between the first target sample and the third sample being equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, searching for a second target sample in the first list, the absolute value of a difference between the first target sample and the third sample being less than or equal to the first threshold and greater than a second threshold, and setting the index number of the third sample in the first list to be the index number of the second target sample plus a fourth value.

4. The method according to claim 3, wherein: The method further comprises: When the absolute value of the difference between the first sample and the second sample is greater than a first threshold, adding the second sample to the first list and not sorting the second sample; The first operation further includes: if the second target sample of the third sample is not found, adding the third sample to the first list, and not sorting the third sample; Accordingly, after traversing to the last sample of the first image, the method further includes: Performing a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as a current sample, determining a third target sample whose absolute value of difference with the current sample is the smallest from the remaining unsorted samples different from the current sample; and setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

5. The method according to any one of claims 1 to 4, wherein: Obtaining, according to the index number of the sample in the first list and the bit depth of the first component, a sample value of the first component having the same position coordinates as the sample in the second image, includes: determining, according to the bit depth, a maximum sample value representing the first component; The sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the maximum sample value.

6. The method according to claim 5, wherein: Obtaining, according to the index number of the sample in the first list and the maximum sample value, a sample value of a first component having the same position coordinates as the sample in the second image, includes: Determining a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is a ratio of (k-1) to (M-1), where M is the total number of index numbers and k is the index number of the sample in the first list; The product of the first ratio and the maximum sample value is used as the sample value of the first component having the same position coordinates as the sample in the second image.

7. The method according to claim 2, wherein: The first value is 0 or 2 to the power of (N-1), where N is the bit depth.

8. The method according to any one of claims 1 to 4, wherein: The samples of the first image represent depth information.

9. The method according to any one of claims 1 to 4, wherein: The samples of the first component of the second image represent depth information.

10. The method according to any one of claims 1 to 4, wherein: The method further comprises: Decoding the received code stream to obtain a third image; Performing color space transformation on the third image to obtain the first image.

11. A video image processing method, the method being applied to an encoder, the method comprising: Sorting the samples in the first image according to differences between the samples to obtain a first list; A sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the bit depth of the first component.

12. The method according to claim 11, wherein The method further comprises: The sample values ​​of the second component and the third component of the second image are set to the first value.

13. The method according to claim 11, wherein The sorting of the samples according to the differences between the samples in the first image includes: Setting the index number of a first sample in the first list in the first image to a second value, and determining a difference between the first sample and an adjacent second sample; and When the absolute value of the difference between the first sample and the second sample is less than or equal to a first threshold and greater than a second threshold, setting the index number of the second sample in the first list to a third value; When the absolute value of the difference between the first sample and the second sample is equal to a second threshold, the index number of the second sample is the same as the index number of the first sample; A first operation is performed on an untraversed third sample adjacent to the second sample in the first image, the first operation comprising: searching for a first target sample in the first list, the absolute value of a difference between the first target sample and the third sample being equal to a second threshold; if the first target sample is found, the index number of the third sample is the same as the index number of the first target sample; if the first target sample is not found, searching for a second target sample in the first list, the absolute value of a difference between the first target sample and the third sample being less than or equal to the first threshold and greater than a second threshold, and setting the index number of the third sample in the first list to be the index number of the second target sample plus a fourth value.

14. The method according to claim 13, wherein The method further comprises: When the absolute value of the difference between the first sample and the second sample is greater than a first threshold, adding the second sample to the first list and not sorting the second sample; The first operation further includes: if the second target sample of the third sample is not found, adding the third sample to the first list, and not sorting the third sample; Accordingly, after traversing to the last sample of the first image, the method further includes: Performing a second operation on the unsorted samples in the first list; the second operation includes: selecting any unsorted sample in the first list as a current sample, determining a third target sample whose absolute value of difference with the current sample is the smallest from the remaining unsorted samples different from the current sample; and setting the index number of the third target sample in the first list to the index number of the current sample plus a fourth value.

15. The method according to any one of claims 11 to 14, wherein: Obtaining, according to the index number of the sample in the first list and the bit depth of the first component, a sample value of the first component having the same position coordinates as the sample in the second image, includes: determining, according to the bit depth, a maximum sample value representing the first component; The sample value of the first component having the same position coordinates as the sample in the second image is obtained according to the index number of the sample in the first list and the maximum sample value.

16. The method according to claim 15, wherein Obtaining, according to the index number of the sample in the first list and the maximum sample value, a sample value of a first component having the same position coordinates as the sample in the second image, includes: Determining a first ratio based on the index number of the sample in the first list and the total number of index numbers in the first list; wherein the first ratio is a ratio of (k-1) to (M-1), where M is the total number of index numbers and k is the index number of the sample in the first list; The product of the first ratio and the maximum sample value is used as the sample value of the first component having the same position coordinates as the sample in the second image.

17. The method according to claim 12, wherein: The first value is 0 or 2 to the power of (N-1), where N is the bit depth.

18. The method according to any one of claims 11 to 14, wherein: The samples of the first image represent depth information.

19. The method according to any one of claims 11 to 14, wherein: The samples of the first component of the second image represent depth information.

20. The method according to any one of claims 11 to 14, wherein: The method further comprises: The second image is encoded, and the obtained encoded bits are written into a bitstream.

21. A video image processing device, applied to a decoder, comprising: A first sorting module and a first mapping module; wherein, The first sorting module is configured to sort the samples in the first image according to the differences between the samples to obtain a first list; The first mapping module is configured to obtain a sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component.

22. A decoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 10 when running the computer program.

23. A video image processing device, applied to an encoder, comprising: A second sorting module and a second mapping module; wherein, The second sorting module is configured to sort the samples according to the differences between the samples in the first image to obtain a first list; The second mapping module is configured to obtain a sample value of the first component having the same position coordinates as the sample in the second image according to the index number of the sample in the first list and the bit depth of the first component.

24. An encoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 11 to 20 when running the computer program.

25. A code stream, wherein the code stream is generated by bit-coding a second image; The second image is obtained by any one of claims 11 to 20.

26. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 10 is implemented, or the method according to any one of claims 11 to 20 is implemented.

Images