Image coding method, image decoding method and related equipment

Abstract

The invention relates to the field of coding and decoding, and discloses an image coding method, a decoding method and related equipment, and the method comprises the steps: obtaining first information according to a first code stream; size transformation is carried out on the first information to obtain second information, and the second information comprises information of chrominance components of the first image format; reconstructing the first image according to the second information; third information is acquired according to the second code stream, and the sizes of the third information in the spatial dimension and the channel dimension are consistent with those of the first information; performing size transformation on the third information to obtain fourth information; the fourth information comprises information of chrominance components of the second image format; and reconstructing the second image according to the fourth information. According to the method and the device, the information of the chrominance components of different image formats can be obtained by performing size transformation on the information of the same size in the decoding process, so that the computing power overhead and the time overhead of the decoding network in the decoding process of the different image formats are consistent, and the stability of information processing is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an image encoding method, a decoding method, and related devices. Background Art

[0002] Image / video compression refers to a technology that uses data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / videos losslessly or losslessly with as few bits as possible. This technology can achieve effective transmission and storage of data information, and plays an important role in the current media era where the types of transmitted data are becoming increasingly diverse and the amount of data is increasing. Data compression is divided into lossy compression and lossless compression. Lossy compression achieves a higher compression ratio at the cost of a certain degree of image / video quality reduction, while lossless compression does not cause image / video quality loss.

[0003] Currently, JPEG AI can encode images of different formats through different downsampling multiples, such as YUV420, YUV444, YUV422, etc. However, different formats have a greater impact on the computing power of the current JPEG AI network. Summary of the invention

[0004] The present application provides an image encoding method and an image decoding method, so that the computing power overhead and time overhead of the encoding network or the decoding network when processing information of different (or at least two) image formats are consistent (or nearly consistent, or at least smaller than the difference in overhead of processing original information of different formats), thereby improving the stability of information processing.

[0005] In a first aspect, the present application provides an image encoding method, the method comprising: obtaining first information, the first information including information of a chromaticity component of a first image format; resizing the first information to obtain second information; encoding the second information through a coding network or a part of a coding network; obtaining third information, the third information including information of a chromaticity component of a second image format; the second image format is different from the first image format; resizing the third information to obtain fourth information; the size of the fourth information in spatial dimension and channel dimension is consistent with the second information; encoding the fourth information through a coding network or a part of a coding network.

[0006] In the existing implementation, the information of the chrominance component is directly input into the encoding network, and the encoding network encodes the information of the chrominance component. Since the sizes of information in different image formats are different, that is, the sizes of information input into the encoding network are different, the computing power and time costs of the encoding network when processing information in different image formats are different, resulting in poor stability of information processing. In an embodiment of the present application, the information of the chrominance components of different image formats is converted into information of the same size through size transformation and input into the encoding network, so that the computing power and time costs of the encoding network when processing information in different (or at least two) image formats are consistent (or nearly consistent, at least smaller than the difference in costs of processing original information in different formats), thereby improving the stability of information processing, which is better in terms of both machine and user experience.

[0007] In addition, in the prior art, if a model is trained for a low-resolution image format (such as YUV420), the encoding performance of this model for a high-resolution format (such as YUV444) will be very poor. (If a normal 444 encoding model is required, another model needs to be trained separately). In the embodiment of the present application, a single trained model can be used to encode multiple formats, and all of them show good and stable encoding performance.

[0008] Among them, the encoding network can also be called an analysis transformation network.

[0009] In a possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0010] In a possible implementation, the amount of non-redundant information included in the second information is consistent with the amount of non-redundant information included in the first information; the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0011] In a possible implementation, the information of the chrominance components of different formats has different amounts of information due to the different sizes (for low-resolution image formats, the amount of information carried is less than that of high-resolution image formats due to downsampling). The amount of information here refers to the amount of effective information, that is, the amount of non-redundant information. For example, if the same information is copied multiple times, although the amount of information is increased, the amount of non-redundant information remains unchanged. The size transformation of the embodiment of the present application can make the size of the information of the chrominance components of different (or at least two) image formats consistent, which is to maintain the stability of the computational overhead (including computing power overhead or time overhead). However, in a better case, even if the information of the chrominance component is resized, the code stream obtained after the entire encoding process is consistent with the code stream obtained by encoding before the size transformation. If it is inconsistent, the meaning of using different image formats is lost. In order to ensure that the result of the encoding process (that is, the code stream) is consistent (or nearly consistent) with the existing scheme, it is necessary to ensure that the effective amount of information before and after the size transformation is unchanged.

[0012] In a possible implementation, the size transformation includes at least one of the following: copying information and using the copied information as information on a new channel; splitting information on at least one channel into information on multiple different channels; or performing a convolution operation.

[0013] In a possible implementation, the operation of splitting the information on at least one channel into information on a plurality of different channels is implemented by an inverse pixel recombination layer.

[0014] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0015] In a possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0016] In a possible implementation, the method also includes: obtaining fifth information, the fifth information being information of a brightness component Y; performing a size transformation on the fifth information to obtain sixth information; the size of the sixth information in a spatial dimension is consistent with the second information; encoding the second information through a coding network or part of a coding network includes: encoding a fusion result of the second information and the fourth information through a coding network.

[0017] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0018] In a second aspect, the present application provides an image decoding method, the method comprising: obtaining first information according to a first code stream; resizing the first information to obtain second information, the second information including information about a chroma component of a first image format; reconstructing the first image according to the second information; obtaining third information according to the second code stream, the size of the third information in a spatial dimension and a channel dimension being consistent with the first information; resizing the third information to obtain fourth information; the fourth information including information about a chroma component of a second image format, the second image format being different from the first image format; and reconstructing the second image according to the fourth information.

[0019] In the existing implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or, at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network in the decoding process of different (or, at least two) image formats are consistent (or nearly consistent), thereby improving the stability of information processing, which is better in terms of both the machine and user experience.

[0020] Among them, the decoding network can also be called a synthetic transformation network.

[0021] In addition, there is no binding relationship between input and output, and YUV444 input can simultaneously generate 444, 420, or 422 output. And the complexity of the embodiment of the present application when performing 444, 422 decoding is almost the same as 420, and is lower than the complexity of the original JPEG AI when performing 444, 422 decoding.

[0022] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is greater than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0023] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0024] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0025] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0026] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0027] In a possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0028] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0029] In a third aspect, the present application provides an image encoding device, the device comprising:

[0030] an acquisition module, configured to acquire first information, wherein the first information includes information of a chrominance component in a first image format; and acquire third information, wherein the third information includes information of a chrominance component in a second image format; wherein the second image format is different from the first image format;

[0031] A processing module is used to resize the first information to obtain second information; encode the second information through a coding network or a part of the coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through the coding network.

[0032] In a possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0033] In a possible implementation, the amount of non-redundant information included in the second information is consistent with the amount of non-redundant information included in the first information; the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0034] In a possible implementation, the size transformation includes at least one of the following:

[0035] Copy the information and use the copied information as the information on the new channel;

[0036] Splitting the information on at least one channel into information on a plurality of different channels; or,

[0037] Perform convolution operation.

[0038] In a possible implementation, the operation of splitting the information on at least one channel into information on a plurality of different channels is implemented by an inverse pixel recombination layer.

[0039] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or,

[0040] The first image format is YUV422 format, and the second image format is YUV420 format.

[0041] In a possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0042] In a possible implementation, the acquisition module is further used to:

[0043] Acquire fifth information, where the fifth information is information of a brightness component Y;

[0044] The processing module is further used to perform a size transformation on the fifth information to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with that of the second information;

[0045] The processing module is specifically used to encode the fusion result of the second information and the fourth information through a coding network.

[0046] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0047] In a fourth aspect, the present application provides an image decoding device, the device comprising:

[0048] An acquisition module, configured to acquire first information according to the first bit stream; and acquire third information according to the second bit stream, wherein the size of the third information in the spatial dimension and the channel dimension is consistent with that of the first information;

[0049] A processing module is used to resize the first information to obtain second information, wherein the second information includes information of chromaticity components in a first image format; reconstruct the first image based on the second information; resize the third information to obtain fourth information; the fourth information includes information of chromaticity components in a second image format, and the second image format is different from the first image format; and reconstruct the second image based on the fourth information.

[0050] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is greater than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0051] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0052] In a possible implementation, the size transformation includes at least one of the following methods:

[0053] Delete the information of some channels;

[0054] Combine different data from multiple channels onto the same channel; or,

[0055] Perform convolution operation.

[0056] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0057] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or,

[0058] The first image format is YUV422 format, and the second image format is YUV420 format.

[0059] In a possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0060] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0061] In a fifth aspect, the present application provides an image decoding method, the method comprising: obtaining first information according to a first code stream; resizing the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of the chrominance component of the first image format, the third information is information of the chrominance component of the second image format, the fourth information is information of the chrominance component of the third image format, and the first image format, the second image format and the third image format are different; and, performing one or more of the following steps: reconstructing the first image according to the second information; reconstructing the second image according to the third information; or reconstructing the third image according to the fourth information.

[0062] In the existing implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or, at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network in the decoding process of different (or, at least two) image formats are consistent (or nearly consistent), thereby improving the stability of information processing, which is better in terms of both the machine and user experience.

[0063] In addition, for the information obtained from the same code stream (such as the first information), chrominance information of multiple image formats can be generated by resizing the first information. There is no binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. And the complexity of the embodiment of the present application when performing 444, 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI for 444, 422 decoding.

[0064] In a possible implementation, the sizes of the second information, the third information, and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information, and the fourth information in the channel dimension are smaller than the first information.

[0065] In a possible implementation, the amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

[0066] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0067] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0068] In a possible implementation, the first image format is a YUV444 format, the second image format is a YUV422 format, and the third image format is a YUV420 format.

[0069] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0070] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0071] In a sixth aspect, the present application provides an image decoding device, the device comprising:

[0072] An acquisition module, used for acquiring first information according to the first code stream;

[0073] a processing module, configured to perform a size conversion on the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of a chroma component in a first image format, the third information is information of a chroma component in a second image format, the fourth information is information of a chroma component in a third image format, and the first image format, the second image format and the third image format are different; and performing one or more of the following steps:

[0074] reconstructing the first image according to the second information;

[0075] reconstructing the second image according to the third information; or,

[0076] The third image is reconstructed according to the fourth information.

[0077] It should be understood that the processing module introduced in the sixth aspect can have the ability to convert the information obtained from the same code stream into chromaticity information in multiple image formats. When running the processing model, one or more of the multiple image formats can be selected for generation based on demand (such as user specification or device selection).

[0078] In a possible implementation, the first information is an output of a decoding network or an output of an intermediate layer of a decoding network.

[0079] In a possible implementation, the sizes of the second information, the third information, and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information, and the fourth information in the channel dimension are smaller than the first information.

[0080] In a possible implementation, the amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

[0081] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0082] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0083] In a possible implementation, the first image format is a YUV444 format, the second image format is a YUV422 format, and the third image format is a YUV420 format.

[0084] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0085] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0086] In a seventh aspect, the present application provides an image decoding method, the method comprising: obtaining first information according to a first code stream; performing a first size transformation on the first information to obtain second information, the second information including information of a chroma component of a first image format; reconstructing a first image according to the second information; performing a second size transformation on the first information to obtain third information; the third information including information of a chroma component of a second image format, the second image format being different from the first image format; and reconstructing a second image according to the third information.

[0087] In the existing implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or, at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network in the decoding process of different (or, at least two) image formats are consistent (or nearly consistent), thereby improving the stability of information processing, which is better in terms of both the machine and user experience.

[0088] In addition, for the information obtained from the same code stream (such as the first information), chrominance information of multiple image formats can be generated by resizing the first information. There is no binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. And the complexity of the embodiment of the present application when performing 444, 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI for 444, 422 decoding.

[0089] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to that of the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0090] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0091] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0092] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0093] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0094] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0095] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0096] In an eighth aspect, the present application provides an image decoding device, the device comprising:

[0097] An acquisition module, used for acquiring first information according to the first code stream;

[0098] A processing module is used to perform a first size transformation on the first information to obtain second information, where the second information includes information about the chrominance components of the first image format, and reconstruct the first image based on the second information; or to perform a second size transformation on the first information to obtain third information, where the third information includes information about the chrominance components of the second image format, where the second image format is different from the first image format, and reconstruct the second image based on the third information.

[0099] It should be understood that the processing module introduced in the eighth aspect can have the ability to convert the information obtained from the same code stream into chromaticity information in multiple image formats. When running the processing model, one or more of the multiple image formats can be selected for generation based on demand (such as user specification or device selection).

[0100] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to that of the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0101] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0102] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; and performing a convolution operation.

[0103] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0104] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0105] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0106] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0107] A ninth aspect of an embodiment of the present application provides a coding device, comprising: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the coding device implements the method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0108] The tenth aspect of an embodiment of the present application provides a decoding device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the decoding device implements the method in the second aspect or any possible implementation of the second aspect, or the fifth aspect or any possible implementation of the fifth aspect, or the seventh aspect or any possible implementation of the seventh aspect.

[0109] An eleventh aspect of an embodiment of the present application provides a coding and decoding system, comprising: the coding device of the ninth aspect above, and / or the decoding device of the tenth aspect above.

[0110] A twelfth aspect of an embodiment of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction is executed on a computer, the computer executes the method in the first aspect or any possible implementation of the first aspect, or the computer executes the method in the second aspect or any possible implementation of the second aspect, or the method in the fifth aspect or any possible implementation of the fifth aspect, or the method in the seventh aspect or any possible implementation of the seventh aspect.

[0111] A thirteenth aspect of an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, it enables the computer to execute the method in the first aspect or any possible implementation of the first aspect, or enables the computer to execute the method in the second aspect or any possible implementation of the second aspect, or the method in the fifth aspect or any possible implementation of the fifth aspect, or the method in the seventh aspect or any possible implementation of the seventh aspect.

[0112] A fourteenth aspect of an embodiment of the present application provides a readable storage medium, on which is stored a code stream encoded according to the method in the aforementioned first aspect or any possible implementation manner of the first aspect.

[0113] A fifteenth aspect of an embodiment of the present application provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0114] A sixteenth aspect of an embodiment of the present application provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0115] A seventeenth aspect of an embodiment of the present application provides a system for distributing code streams, the system comprising at least one storage medium and a video stream device; at least one storage medium is used to store the code stream, the code stream is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect; the video stream device is used to respond to a request from a decoder so that the target code stream in the at least one storage medium can be sent to the decoder.

[0116] An eighteenth aspect of an embodiment of the present application provides a method for distributing a code stream, the method comprising: receiving a first request; selecting a target code stream from at least one storage medium in response to the first request; sending the target code stream to a destination device; at least one storage medium is used to store the code stream, the code stream being encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation manner of the first aspect.

[0117] In a nineteenth aspect of an embodiment of the present application, a system for processing a code stream is provided, the system comprising an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain image data of the image source device through an interface, and encode the image data to obtain one or more code streams, the code streams being encoded by the encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0118] In the twentieth aspect, the present application provides a chip system, which includes a processor for supporting the device to implement some or all of the functions involved in the above aspects, such as sending or processing the data involved in the above method; or, information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the device or the device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1a A schematic diagram of an application scenario of an embodiment of the present application;

[0120] Figure 1b A schematic diagram of an application scenario of an embodiment of the present application;

[0121] Figure 2a This is an illustration of the application scenario of the embodiment of the present application;

[0122] Figure 2b This is an illustration of the application scenario of the embodiment of the present application;

[0123] Figure 3 An example of an image processing method provided in an embodiment of the present application is shown;

[0124] Figure 4 An example of an image processing method provided in an embodiment of the present application is shown;

[0125] Figure 5 An example of an image processing method provided in an embodiment of the present application is shown;

[0126] Figure 6 An example of an image processing method provided in an embodiment of the present application is shown;

[0127] Figure 7 An example of an image processing method provided in an embodiment of the present application is shown;

[0128] Figure 8a An example of an image processing method provided in an embodiment of the present application is shown;

[0129] Figure 8b An example of an image processing method provided in an embodiment of the present application is shown;

[0130] Fig. 9 An example of an image processing method provided in an embodiment of the present application is shown;

[0131] Fig.10 An example of an image processing method provided in an embodiment of the present application is shown;

[0132] Fig.11 An example of an image processing method provided in an embodiment of the present application is shown;

[0133] Fig.12 A schematic diagram of the structure of an image encoding device provided in an embodiment of the present application;

[0134] Fig.13 A schematic diagram of the structure of an image decoding device provided in an embodiment of the present application;

[0135] Fig.14 A schematic diagram of the structure of a device provided in an embodiment of the present application;

[0136] Fig.15 A schematic diagram of the structure of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0137] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0138] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0139] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0140] The following describes the application scenarios to which the method provided in the embodiments of the present application is applicable.

[0141] The method provided in the embodiments of the present application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live broadcast technology, storage, and transmission.

[0142] Figure 1a An example of an application scenario provided in an embodiment of the present application can also be understood as a coding and decoding application scenario. The application scenario includes: an AI coding unit, a storage / transmission unit, and an AI decoding unit. The method provided in an embodiment of the present application can be applied in an AI coding unit / AI decoding unit.

[0143] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application.

[0144] The storage / transmission unit is used to store (terminal, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live broadcast technology, etc.) the encoded input data.

[0145] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0146] There are many application scenarios in the above examples, which are described in detail below:

[0147] The first one is the storage application scenario.

[0148] The storage application scenarios include: terminal photo album, video surveillance, etc. The schematic diagram of this scenario is as follows Figure 1b As shown, the scene includes: an AI encoding unit, a file saving unit, a file loading unit, and an AI decoding unit.

[0149] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application to obtain a binary file.

[0150] The file saving unit is used to save the binary file output by the AI ​​encoding unit.

[0151] The loader file unit is used to load binary files.

[0152] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0153] The second is the terminal-cloud interaction scenario.

[0154] The terminal-cloud interaction scenarios include: live broadcast scenarios, public clouds, etc. The schematic diagram of this scenario is as follows Figure 2a As shown, the scenario includes: an encoding unit, a decoding unit, an AI encoding unit, an AI decoding unit, an encoding unit, and a decoding unit.

[0155] Among them, the terminal side includes an encoding unit and a decoding unit, and the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0156] The terminal obtains input data, and encodes the input data through the encoding unit to obtain encoding features. The terminal sends the encoding features to the cloud. After the cloud receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data. Then the AI ​​encoding unit encodes the reconstructed data according to the encoding method provided in the embodiment of the present application to obtain a code stream / file. The cloud decodes the code stream / file through the AI ​​decoding unit according to the decoding method provided in the embodiment of the present application to obtain reconstructed data. The cloud encodes the reconstructed data through the encoding unit to obtain the encoding features. And send the encoding features to the terminal. After the terminal receives the encoding features, it decodes the encoding features through the decoding unit to obtain the reconstructed data.

[0157] Understandably, Figure 2aThe encoding methods used by the encoding unit and the AI ​​encoding unit may be inconsistent (or it can be understood that the operations of the decoding unit and the AI ​​encoding unit in the cloud can be called transcoding operations, and the operations of the AI ​​decoding unit and the encoding unit in the cloud can be called transcoding operations). For example, the cloud usually has more computing power resources than the terminal, so an algorithm with high computing power requirements and higher compression efficiency (AI encoding) can be used to re-compress the input data (transcoding) to obtain a higher compression multiple and save cloud storage space. The process of transcoding: decode first, then encode in another way.

[0158] It is understandable that the above-mentioned scenarios are just examples, and there may be other scenarios in actual applications, which are not specifically limited here.

[0159] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0160] Figure 2b A chip hardware structure is provided for an embodiment of the present invention, and the chip includes a neural network processor 40.

[0161] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU) and any other processor suitable for large-scale XOR processing. Take NPU as an example: the neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0162] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.

[0163] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 408.

[0164] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0165] In some implementations, the vector calculation unit 407 stores the vector of processed outputs to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 403, such as for use in a subsequent layer in a neural network.

[0166] The unified memory 406 is used to store input data and output data.

[0167] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0168] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0169] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0170] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0171] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0172] The method provided by the embodiment of the present application is described below. The method can be performed by a coding device, or by a component of the coding device (such as a processor, a chip, or a chip system, etc.). The coding device can be a cloud device or a terminal device. Of course, the method can also be performed by a system consisting of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, GPU in the coding device, and this application does not limit it.

[0173] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0174] (1) Neural Network

[0175] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0176]

[0177] Where s=1, 2, ..., n, n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0178] (2) Deep Neural Networks

[0179] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0180] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0181] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0182] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0183] (3) Convolutional Neural Network

[0184] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0185] (4) Concat

[0186] The concatenation operation refers to concatenating multiple tensors in a certain dimension, and all tensors must have the same shape (except for the concatenation dimension). This patent refers to concatenation on channels.

[0187] (5) Bit rate: In image compression tasks, the average coding length required to encode a unit pixel.

[0188] (6) Rate-distortion performance: An indicator used to measure the performance of a compression algorithm, which comprehensively considers both bit rate and decoded image distortion.

[0189] (7) YUV: A color format that can be converted to and from RGB format. It is often used in various media processing algorithms to reduce the bandwidth of color. There are three commonly used YUV formats:

[0190] YUV4:4:4 (YUV444): Fully sample the chrominance component (UV) to obtain Y[1,H,W], UV[2,H,W];

[0191] YUV4:2:2 (YUV422): 2:1 horizontal sampling of the chrominance component (UV) and full vertical sampling are performed to obtain Y[1,H,W],UV[2,H,W / 2];

[0192] YUV4:2:0 (YUV420): 2:1 horizontal sampling of chrominance components (UV), 2:1 vertical sampling, Y[1,H,W], UV[2,H / 2,W / 2];

[0193] In addition to the default format of YUV422, the present invention also supports another possible YUV422 format: the chrominance component (UV) is fully sampled horizontally and 2:1 vertically to obtain Y[1,H,W], UV[2,H / 2,W];

[0194] (8)Pixel Shuffle:

[0195] Pixel reorganization, the main function is to reorganize low-resolution feature maps into high-resolution feature maps through multi-channel reorganization. In this paper, it is written as PS(s1,s2), where S1 and S2 refer to the upsampling factors, which are usually integers greater than one. Input [C in ,h in ,w in ] Output [C out ,h out ,w out ]. out =s1·h in ;w out =s2·w in ; C out =C in / (s1·s2); To round down. The formula is as follows:

[0196] For c=0,…,C out -1,i=0,...,h out -1and j=0,…,w out -1

[0197] for i′=0..s1-1 and j′=0..s2-1

[0198]

[0199] (9) Pixel Unshuffle:

[0200] The inverse operation of Pixelshuffle is to obtain a low-resolution feature map by reorganizing the high-resolution feature map between multiple channels. In the embodiment of the present application, UPS(s1,s2) is written as UPS(s1,s2), where s1 and s2 refer to the downsampling factors, which are usually integers greater than one. Input [C in ,h in ,w in ] Output [C out ,h out ,w out ].

[0201] in C out =C in*s1*s2, the formula is described as follows:

[0202] For c=0..C in -1,i=0..h out -1and j=0..w out -1

[0203] for i′=0..s1-1 and j′=0..s2-1

[0204] output[c*s1*s2+s2*i′+j′,i,j]=input[c,s1*i+i′,s2*j+j′]

[0205] (10)s UV and Y :

[0206] Up and down sampling factors: The encoder is the down sampling factor, and the decoder is the up sampling factor. UV =[s1,s2] means that the vertical sampling multiple of UV is s1, and the horizontal sampling multiple is s2, and both s1 and s2 are integers greater than or equal to 1. In JPEG AI, upsampling uses bilinear interpolation (bicubic) and downsampling uses (sample) interval sampling. This application also supports other upsampling and downsampling methods.

[0207] (11) Color transformation (ColorTr):

[0208] The function of color transformation is to convert the input image into primary and secondary components. The color transformation used in JPEG AI to convert RGB images into YUV format is in the international standard ITU-R BT.709. The input is three components of R[H,W], G[H,W], and B[H,W]. The spatial shape of input and output is H×W. The specific description is as follows:

[0209] Y[i,j]=0.2126·R[i,j]+0.7152·G[i,j]+0.0722·B[i,j]

[0210] U[i,j]=(-0.2126·R[i,j]-0.7152·G[i,j]+0.9278·B[i,j]) / 1.8556

[0211] V[i,j]=(0.7874·R[i,j]-0.7152·G[i,j]-0.0722·B[i,j]) / 1.5748

[0212] Among them, the subscript [i,j], 0≤i <H,0≤j<W。

[0213] (12) Inverse color transformation (invColorTr): the inverse transformation of ColorTr.

[0214] Image / video compression refers to a technology that uses data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / videos losslessly or losslessly with as few bits as possible. This technology can achieve effective transmission and storage of data information, and plays an important role in the current media era where the types of transmitted data are becoming increasingly diverse and the amount of data is increasing. Data compression is divided into lossy compression and lossless compression. Lossy compression achieves a higher compression ratio at the cost of a certain degree of image / video quality reduction, while lossless compression does not cause image / video quality loss.

[0215] Currently, JPEG AI can encode images of different formats through different downsampling multiples, such as YUV420, YUV444, YUV422, etc. However, different formats have a greater impact on the computing power of the current JPEG AI network.

[0216] The embodiment of the present application is first described by taking the application scenario as a terminal device as an example.

[0217] As an example, the terminal device may be a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and the terminal device may compress the acquired image. As another example, the terminal device may be a virtual reality (VR) device. As another example, the embodiment of the present application may also be applied to intelligent monitoring, and a camera may be configured in the intelligent monitoring, so that the intelligent monitoring may obtain the image to be compressed through the camera, etc. It should be understood that the embodiment of the present application may also be applied to other scenarios where image compression is required, and other application scenarios are not listed one by one here.

[0218] Reference Figure 3 , Figure 3 An example of an image encoding method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, an image encoding method provided by an embodiment of the present application includes:

[0219] 301. Obtain first information, where the first information includes information about a chrominance component of a first image format.

[0220] In the embodiment of the present application, the first information may include information of a chrominance component of the first image in a first image format.

[0221] About the first image:

[0222] In a possible implementation, the first image may be an image to be encoded, wherein the first image may be an image captured by a camera of a terminal device, or the first image may be an image obtained from inside the terminal device (for example, an image stored in an album of the terminal device, or a picture obtained by the terminal device from the cloud). It should be understood that the first image may be an image with image encoding requirements, and this application does not impose any limitation on the source of the first image.

[0223] Information about chroma components:

[0224] In a possible implementation, the input image (such as the first image in the embodiment of the present application) can be preprocessed to convert the image into a primary component and a secondary component, wherein the primary component can be a brightness component, and the secondary component can be a chrominance component. The primary component (i.e., the brightness component) can be, but not limited to, a Y component, and the secondary component (i.e., the chrominance component) can be, but not limited to, a UV component. For example, it can be divided into a Y component and a UV component by ColorTr, wherein YUV is a type of true-color color space (color space) compilation, and Y'UV, YUV, YCbCr, YPbPr, etc. can all be called YUV, and there are overlaps with each other. "Y" represents brightness (Luminance or Luma), that is, a grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0225] In the embodiment of the present application, a first image can be acquired, and the chromaticity component and the luminance component can be obtained by preprocessing the first image, or the chromaticity component and the luminance component corresponding to the first image can be directly acquired without acquiring the first image.

[0226] In a possible implementation, information about the chrominance component in a certain image format and information about the luminance component in a certain image format may be obtained. The image format may be a YUV444 format, a YUV422 format, or a YUV420 format.

[0227] For the YUV444 format, the secondary component of the first image may be directly used as the information of the chrominance component in the YUV444 format (for example, the size is [2, H, W], H is the height of the first image, and W is the width of the first image).

[0228] For the YUV422 format, the secondary components of the first image can be resampled (e.g., downsampled) to obtain information of the chrominance components in the YUV422 format (e.g., the size is [2, H, W / 2], H is the height of the first image, and W is the width of the first image).

[0229] For the YUV420 format, the secondary components of the first image can be resampled (e.g., downsampled) to obtain the chrominance component information in the YUV420 format (e.g., the size is [2, H / 2, W / 2], H is the height of the first image, and W is the width of the first image).

[0230] In an embodiment of the present application, the first information is the information of the chrominance component of the first image format, the third information is the information of the chrominance component of the second image format, and the resolution of the first image format is greater than that of the second image format. For example, the first image format is the YUV444 format, and the second image format is the YUV422 format or the YUV420 format; or, the first image format is the YUV422 format, and the second image format is the YUV420 format.

[0231] 302. Perform size transformation on the first information to obtain second information.

[0232] In one possible implementation, for information on chromaticity components of different (or at least two) image formats, an embodiment of the present application can resize it so that its sizes in the spatial dimension and the channel dimension remain consistent. For example, for first information in a first image format and third information in a second image format, the sizes of the transformed first information (that is, the second information) and the transformed third information (that is, the fourth information) can be made consistent (that is, the channel size and the spatial size are consistent) through resize transformation.

[0233] In the existing implementation, the information of the chrominance component is directly input into the encoding network, and the encoding network encodes the information of the chrominance component. Since the sizes of information in different image formats are different, that is, the sizes of information input into the encoding network are different, the computing power and time costs of the encoding network when processing information in different image formats are different, resulting in poor stability of information processing. In an embodiment of the present application, the information of the chrominance components of different image formats is converted into information of the same size through size transformation and input into the encoding network, so that the computing power and time costs of the encoding network when processing information in different (or at least two) image formats are consistent (or nearly consistent, at least smaller than the difference in costs of processing original information in different formats), thereby improving the stability of information processing, which is better in terms of both machine and user experience.

[0234] Among them, the coding network can also be called an analysis transformation network or a signal encoder. The encoder is implemented based on a neural network and is used to encode the input information (or it can be called feature extraction) to obtain a feature map. The feature map can be used to obtain a bit stream through a quantization and entropy coding module. Since it is an encoder implemented based on a neural network, it can also be called an AI encoder.

[0235] In the embodiment of the present application, the size transformation performed on the first information can reduce the size of the information in the spatial dimension and increase the size of the information in the channel dimension. The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information.

[0236] Among them, the spatial dimension can further include a height dimension and a width dimension. The fact that the size of the second information in the spatial dimension is smaller than that of the first information can be understood as the size of at least one dimension of the height dimension and the width dimension of the second information is smaller than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is larger than that of the first information).

[0237] For example, the height dimension of the second information in the spatial dimension is smaller than that of the first information, the width dimension of the second information in the spatial dimension is smaller than that of the first information, and the dimension of the second information in the channel dimension is larger than that of the first information.

[0238] For example, the height dimension of the second information in the spatial dimension is smaller than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is larger than that of the first information.

[0239] The above two illustrated cases can be understood as the size of the second information in the spatial dimension is smaller than the first information.

[0240] Next, the size transformation of the first information in the embodiment of the present application is described in detail:

[0241] In a possible implementation, the information of chrominance components of different formats has different amounts of information due to their different sizes (for low-resolution image formats, due to downsampling, the amount of information carried is less than that of high-resolution image formats). The amount of information here refers to the amount of effective information, that is, the amount of non-redundant information. For example, if the same information is copied multiple times, although the amount of information increases, the amount of non-redundant information remains unchanged. The amount of non-redundant information is the amount of information that will have a substantial impact on the encoding or decoding results.

[0242] The resizing in the embodiment of the present application can make the size of the information of the chrominance components of different (or at least two) image formats consistent, which is to maintain the stability of the computational overhead (including computing power overhead or time overhead). However, a better situation is that even if the chrominance component information is resized, the code stream obtained after the entire encoding process is consistent with the code stream obtained by encoding before the resizing. If they are inconsistent, the meaning of using different image formats is lost. In order to ensure that the result of the encoding process (that is, the code stream) is consistent (or nearly consistent) with the existing scheme, it is necessary to ensure that the effective information amount of the information is unchanged before and after the resizing.

[0243] Specifically, for the second information obtained by resizing the first information, the amount of non-redundant information contained therein is consistent with the amount of non-redundant information contained in the first information. The amount of information here refers to the amount of effective information, that is, the amount of non-redundant information. For example, if the same information is copied multiple times, although the amount of information increases, the amount of non-redundant information remains unchanged.

[0244] Resizing can include operations such as information splitting, information replication, and convolution. The following are introduced separately:

[0245] 1. Information splitting

[0246] In a possible implementation, resizing the first information may include: splitting information on at least one channel of the first information into information on multiple different channels. In other words, part of the first information may be placed on an extended channel.

[0247] That is, in order to increase the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be expanded, and the information of the expanded channels needs to come from the existing data in the first information. Then, a part of the information in the first information can be taken out and used as the information of the expanded channel. This part of the information is deleted on the original channel, which is equivalent to splitting the original information of the first information to obtain more channels, thereby reducing the size of the first information in the spatial dimension.

[0248] Since the characteristic of chrominance component information is that the size of information on each channel is consistent, the information of each channel can be taken out to obtain information of the same size (since the size of the taken out information is the same, the size of information on each channel of the second information obtained after deleting this part of the information is consistent).

[0249] In a possible implementation, different data of the same channel in the first information may be placed in different channels of the new extension.

[0250] Exemplarily, channel 1 of the first information (including channel 1 and channel 2) may include element A1, element A2, and element A3.

[0251] The elements A1, A2 and A3 may be deleted from the channel 1, and the elements A1, A2 and A3 may be used as data of the channel 3 (a channel not originally included in the first information).

[0252] Alternatively, element A1, element A2 and element A3 can be deleted from channel 1, and element A1 can be used as data for channel 3 (a channel not originally included in the first information), element A2 can be used as data for channel 4 (a channel not originally included in the first information), and element A3 can be used as data for channel 5 (a channel not originally included in the first information).

[0253] In a possible implementation, data on different channels in the first information can be taken out and combined and placed in different channels of the new extension. In other words, data of the same channel of the new extension can come from different channels.

[0254] Exemplarily, the first information (including channel 1 and channel 2) may include element A1, element A2 and element A3 on channel 1, and may include element B1, element B2 and element B3 on channel 2.

[0255] Elements A1, A2 and A3 can be deleted from channel 1, elements B1, B2 and B3 can be deleted from channel 2, and elements A1, A2 and A3 can be used as data of channel 3 (a channel not originally included in the first information), and elements B1, B2 and B3 can be used as data of channel 4 (a channel not originally included in the first information).

[0256] Alternatively, element A1, element A2 and element A3 can be deleted from channel 1, element B1, element B2 and element B3 can be deleted from channel 2, and elements A1 and B1 can be used as data for channel 3 (a channel not originally included in the first information), elements A2 and B2 can be used as data for channel 4 (a channel not originally included in the first information), and elements A3 and B3 can be used as data for channel 5 (a channel not originally included in the first information).

[0257] In a possible implementation, the size transformation method may include inverse pixel unshuffle (UPS), that is, UPS may be performed on the first information.

[0258] Reference Figure 4 , Figure 4 This is a schematic diagram of the processing results of a UPS. Figure 4The example of expanding the information of each channel of the C channels in the first information to data of four channels is used for illustration, wherein the data of an information block with a size of 2*2 (height is 2, width is 2) is expanded to data of four channels, so that the number of channels is also changed from C to 4C. If the spatial size of the first information is H*W, the number of channels of the second information obtained after the size transformation is H / 2*W / 2.

[0259] In a possible implementation, the first information is information of the chrominance component of YUV444, the size of the channel dimension of the first information is 2, the height dimension of the spatial dimension of the first information is H, the width dimension of the spatial dimension of the first information is W, the number of channel dimensions of the second information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2. The size conversion method in this illustration can be as follows Figure 4 As shown, no further description is given here.

[0260] 2. Information Copying

[0261] In a possible implementation, in addition to increasing the number of channels by splitting the information, information can also be copied when performing the size transformation. This method is more necessary when the first information is an image format with a relatively low resolution (at least not the highest resolution, such as YUV422 and YUV420), because: the amount of information (that is, the information size) of the chrominance component information of the low-resolution image format is less than that of the chrominance component information of the high-resolution image format. Under the condition that the amount of non-redundant information before and after the size transformation is guaranteed to remain unchanged, if only the information is split and the amount of data is not increased (it should be understood that the amount of data and the amount of non-redundant information introduced above are not the same concept, the former is the total amount of information included in the information, and both redundant and non-redundant information are counted), the size of the chrominance component information of the low-resolution (at least not the highest resolution) image format will definitely be lower than the size of the chrominance component information of the high-resolution image format. In this case, the chrominance component information of the low-resolution (at least not the highest resolution) image format can be copied to increase the amount of data. Compared with the data before the copy, the amount of non-redundant information of the information after the copy operation remains unchanged, but the amount of data included is increased (that is, the size is increased).

[0262] In a possible implementation, the information (all or part of the information) of the first information may be copied, and the copied information may be used as the information on the new channel. Using the copied information as the information on the new channel may be understood as: compared with before the copy, the number of channels will increase, and the data included in the newly added channel is composed of the copied data.

[0263] In a possible implementation, the first information can be copied and the copied information can be used as information on a new channel, and the information on at least one channel can be split into information on multiple different channels, and there is no time sequence limitation between the two operations. The number of each operation is also not limited, for example, the information is copied first, then the information is split, and then the information is copied again. As long as the non-redundant information amount of the information remains unchanged and can be transformed to the required size, the embodiment of the present application does not limit the type, number, combination method and sequence of operations.

[0264] This application does not limit the order in which the copies are placed on the channel.

[0265] For example, the information before copying is in the order of channels: channel 1, channel 2, channel 3, channel 4. Each of channel 1, channel 2, channel 3 and channel 4 can be copied once and used as the data of the new channel. If each copied channel is used as the channel adjacent to the channel before copying, the following information can be obtained: channel 1, channel 1, channel 2, channel 2, channel 3, channel 3, channel 4, channel 4. Alternatively, if each copied channel is placed in the channel after the first information according to the sequential relationship between the channels before copying, the following information can be obtained: channel 1, channel 2, channel 3, channel 4, channel 1, channel 2, channel 3, channel 4. The channel order between the new channel information and the original channel information is not limited in this application.

[0266] For example, in one possible implementation, the first information is information of the chrominance component of YUV422, the size of the channel dimension of the first information is 2, the height dimension of the spatial dimension of the first information is H, and the width dimension of the spatial dimension of the first information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the second information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2.

[0267] In this illustration, the first information (size is (2, H, W / 2)) can be firstly obtained by splitting the information to obtain information of size (4, H / 2, W / 2), and then the information is copied to obtain the second information (8, H / 2, W / 2). Alternatively, in this illustration, the first information (size is (2, H, W / 2)) can be firstly obtained by copying the information to obtain information of size (4, H, W / 2), and then the information is split to obtain the second information (8, H / 2, W / 2).

[0268] In a possible implementation, the above-mentioned information duplication or information splitting operation may be replaced by convolution.

[0269] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant information is guaranteed to remain unchanged and the size can be converted to a required size, and this application is not limited thereto.

[0270] It should be understood that when the first information is information of the chrominance component of YUV420, operations such as copying, splitting, and convolution may also be performed thereon, which will be introduced in subsequent embodiments.

[0271] 303. Encode the second information by using a coding network or a part of a coding network.

[0272] It should be understood that step 302 itself can be an operation independent of the coding network, or can be implemented by the coding network itself. This can be done, for example, by a network layer located inside the input of the encoding network. introduce:

[0273] 1. Step 302 itself is an operation independent of the encoding network

[0274] In a possible implementation, when step 302 itself is an operation independent of the encoding network, the second information may be encoded through the encoding network. Compared with the prior art, step 302 is a step added before the encoding network.

[0275] 2. Step 302 is performed by a network layer in the encoding network close to the input of the encoding network

[0276] In a possible implementation, when step 302 is performed through a network layer in the encoding network close to the input of the encoding network, the network layer in the existing encoding network close to the input of the encoding network can be improved, for example, the operation value of the UPS layer or the convolution layer is changed, and then the second information is to analyze the output of the network layer inside the conversion network.

[0277] In a possible implementation, when step 302 is performed by a network layer in the coding network close to the input of the coding network, the second information may be encoded by a portion of the coding network (ie, a network connected after the improved network layer).

[0278] It should be understood that there is also a certain amount of information interaction between the main component information (such as the information of the Y component) and the secondary component information (such as the information of the UV component), which can further improve the compression effect.

[0279] In a possible implementation, information exchange can be performed before or after the size change. After, and before inputting into the encoding network, they are introduced below:

[0280] 1. Before size change

[0281] In a possible implementation, the first information may also include information of a luminance component, that is, the first information needs to include at least information of a chrominance component, but may also be fused with information of a luminance component. For example, the first information may include information of a luminance component and information of a chrominance component and a fusion result (e.g., a splicing result).

[0282] For example, the first information may include a fusion result (eg, a splicing result) of information of a luminance component and information of a chrominance component in a YUV444 format.

[0283] For example, the first information may include a fusion result (eg, a splicing result) of information of a luminance component and information of a chrominance component in a YUV422 format.

[0284] 2. After resizing and before inputting into the encoding network

[0285] In a possible implementation, the second information and the information of the brightness component may be fused (eg, concatenated), and the fusion result may be input into the encoding network.

[0286] In a possible implementation, the fusion is obtained by splicing in the channel dimension. Since the size of the second information is changed compared to the first information, in order to be able to fuse the second information with the information of the brightness component, the size of the brightness component information (that is, the fifth information in the embodiment of the present application) can also be transformed to obtain information that is at least the same size as the second information in the spatial dimension (that is, the sixth information in the embodiment of the present application), so that the second information and the sixth information can be spliced ​​in the channel dimension.

[0287] Specifically, in a possible implementation, fifth information can be obtained, and the fifth information is information of a brightness component; the fifth information can be resized to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with the second information, and then, the fusion result (for example, a splicing result) of the second information and the fourth information can be encoded through a coding network.

[0288] 304. Obtain third information, where the third information includes information about a chrominance component of a second image format; the second image format is different from the first image format.

[0289] In the embodiment of the present application, the third information may include information about the chrominance component of the second image in the second image format, and the second image may be different from or the same as the first image.

[0290] The resolution of the second image format may be lower than that of the first image format. The introduction of step 304 may refer to the introduction of step 301 in the above embodiment, and the similarities will not be repeated here.

[0291] 305. Perform a size transformation on the third information to obtain fourth information; the sizes of the fourth information in the spatial dimension and the channel dimension are consistent with those of the second information;

[0292] In a possible implementation, the first image format is YUV444, the second image format is YUV422, the size of the fourth information in a spatial dimension is smaller than that of the third information, and the size of the fourth information in a channel dimension is larger than that of the third information.

[0293] For example, in one possible implementation, the third information is the information of the chrominance component of YUV422, the size of the channel dimension of the third information is 2, the height dimension of the spatial dimension of the third information is H, and the width dimension of the spatial dimension of the third information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the fourth information is 8, the height dimension of the spatial dimension of the fourth information is H / 2, and the width dimension of the spatial dimension of the fourth information is W / 2.

[0294] In this illustration, the third information (size is (2, H, W / 2)) can first be copied to obtain information of size (4, H / 2, W / 2), and then the information is copied to obtain fourth information (8, H / 2, W / 2). Alternatively, in this illustration, the third information (size is (2, H, W / 2)) can first be copied to obtain information of size (4, H, W / 2), and then the information is copied to obtain fourth information (8, H / 2, W / 2).

[0295] In a possible implementation, the first image format is YUV422, the second image format is YUV420, the size of the fourth information in a spatial dimension is equal to the third information, and the size of the fourth information in a channel dimension is greater than the third information.

[0296] For example, in one possible implementation, the third information is information of the chrominance component of YUV420, the size of the channel dimension of the third information is 2, the height dimension of the spatial dimension of the third information is H / 2, and the width dimension of the spatial dimension of the third information is W / 2, which can be described as (2, H / 2, W / 2), the number of channel dimensions of the fourth information is 8, the height dimension of the spatial dimension of the fourth information is H / 2, and the width dimension of the spatial dimension of the fourth information is W / 2.

[0297] In this illustration, the third information (with a size of (2, H / 2, W / 2)) can be copied to obtain information with a size of (8, H / 2, W / 2).

[0298] In a possible implementation, the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0299] 306. Encode the fourth information by using a coding network or a part of a coding network.

[0300] The introduction of step 306 may refer to the introduction of step 303 in the above embodiment, and the similarities are not repeated here.

[0301] It should be understood that the above steps 301 to 303 are the encoding process of the first information, and steps 304 to 306 are the encoding process of the third information. The embodiment of the present application does not limit the timing relationship between the encoding process of the first information and the encoding process of the second information. For example, steps 301 to 303 can be performed before steps 304 to 306, or, steps 301 to 303 can be performed after steps 304 to 306, or, steps 301 to 303 can be performed simultaneously with steps 304 to 306.

[0302] Next, combine with Figure 5 , introduces an image encoding method in an embodiment of the present application, which takes the encoding process of three image formats, YUV444, YUV422 and YUV420, as examples. Figure 5 In the image, the interaction between the chrominance component information and the luminance information occurs after the size transformation.

[0303] Reference Figure 5 , the method comprising:

[0304] Input image in RGB format, the size is [3,H,W], 3 is the size of the channel dimension, H is the height dimension of the spatial dimension, and W is the width dimension of the spatial dimension.

[0305] The RGB image is converted into YUV444 format information through the color transformation ColorTr: YUV[3,H,W]. YUV[3,H,W] can be divided into the chrominance component information UV[2,H,W] and the brightness component information Y[1,H,W].

[0306] Information processing for the Y component:

[0307] Y[1,H,W] skips downsampling to generate x Y [1,H,W] and sent to the encoding network (Signal Encoder_Y) for encoding.

[0308] Regarding the processing of UV components:

[0309] During a process, one of three formats can be selected:

[0310] About YUV444 encoding: The input UV component UV[2,H,W] (the number of channels is 2, the height and width are H,W respectively) is reassembled by UPS(2,2) to generate x UV [8,H / 2,W / 2]. Generate auxiliary information by UPS(2,2) for Y[1,H,W]. (and x UV Keep the size of the spatial dimension consistent), and x UV After combining (concat) by channel dimension, they are sent to Signal Encoder_UV together.

[0311] The input UV component UV[2,H,W] may be the first information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the second information introduced in the above embodiment, Y[1, H, W] may be the fifth information introduced in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to obtain the UV component UV[2,H,W], which can refer to the description of step 301 in the above embodiment, and generate x by UPS(2,2) inverse pixel reconstruction. UV [8, H / 2, W / 2] can refer to the description of step 302 in the above embodiment, and generate auxiliary information after Y[1, H, W] passes UPS(2, 2) (and x UV The size of the spatial dimension remains the same) can refer to the relevant description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The signals are combined (concat) according to the channel dimension and sent to Signal Encoder_UV together, which can refer to the introduction of step 303 in the above embodiment.

[0312] About YUV420 encoding: Downsample the input UV component UV[2,H,W] UV = [2,2] to generate 420 format UV [2,H / 2,W / 2]. After 4 times the channel dimension, repeat(4) (cin i is the channel before copying, i is the channel number, and the channel after copying is cout 4i+j =cin i , i=0~1, j=0~3) generate x UV [8,H / 2,W / 2](because x UVThe unified shape [8,H / 2,W / 2] requires 4 times the number of pixels of the current shape [2,H / 2,W / 2]. Y[1,H,W] generates auxiliary information through UPS(2,2) (and x UV Keep the shape consistent), and x UV After being combined by channel dimension, they are sent to the Signal Encoder together.

[0313] Wherein, UV[2,H / 2,W / 2] may be the third information introduced in the above embodiment, and x UV [8, H / 2, W / 2] may be the fourth information introduced in the above embodiment, Y[1, H, W] may be the fifth information introduced in the above embodiment, and auxiliary information It can be the sixth information introduced in the above embodiment. To generate the 420 format UV[2,H / 2,E / 2], refer to the introduction of step 304 in the above embodiment, and generate x by 4 times the channel dimension repeat(4) UV [8, H / 2, W / 2] can refer to the description of step 305 in the above embodiment, and generate auxiliary information after Y[1, H, W] passes UPS(2, 2) (and x UV The size of the spatial dimension remains the same) can refer to the relevant description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The combination according to the channel dimension and the sending to the Signal Encoder together can refer to the introduction of step 306 in the above embodiment.

[0314] About the YUV422 encoding process: UV[2,H,W] downsampling UV = [1,2] to generate 422 format UV [2, H, W / 2]. After UPS (2,1) to generate UV '[4, H / 2, W / 2], and then through the channel dimension twice the copy repeat (2) (cin i is the channel before copying, i is the channel number, and the channel after copying is cout 2i+j =cin i , i = 0 ~ 3, j = 0 ~ 1) to generate x UV [8,H / 2,W / 2](because x UV The unified shape [8,H / 2,W / 2] requires twice as many pixels as the current shape [4,H / 2,W / 2]. Y[1,H,W] generates auxiliary information after UPS(2,2) (and x UV Keep the shape consistent), and xUV After combining by channel dimension, they are sent to Signal Encoder_UV together.

[0315] The input UV component UV[2,H,W / 2] may be the first information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the second information introduced in the above embodiment, Y[1, H, W] may be the fifth information introduced in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to obtain the UV component UV[2,H,W / 2], which can refer to the description of step 301 in the above embodiment, and generate UV′[4,H / 2,W / 2] through UPS(2,1), and then generate x through repeat(2) with a channel dimension doubled. UV [8, H / 2, W / 2] can refer to the description of step 302 in the above embodiment, and generate auxiliary information after Y[1, H, W] passes UPS(2, 2) (and x UV The size of the spatial dimension remains the same) can refer to the relevant description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The signals are combined (concat) according to the channel dimension and sent to Signal Encoder_UV together, which can refer to the introduction of step 303 in the above embodiment.

[0316] Wherein, UV[2,H,W / 2] may be the third information introduced in the above embodiment, and x UV [8, H / 2, W / 2] may be the fourth information introduced in the above embodiment, Y[1, H, W] may be the fifth information introduced in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to generate the 422 format UV[2,H,W / 2], which can refer to the description of step 304 in the above embodiment, and generate UV′[4,H / 2,W / 2] by UPS(2,1), and generate x by repeat(4) with a channel dimension of 2 times. UV [8, H / 2, W / 2] can refer to the description of step 305 in the above embodiment, and generate auxiliary information after Y[1, H, W] passes UPS(2, 2) (and x UV The size of the spatial dimension remains the same) can refer to the relevant description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UVThe combination according to the channel dimension and the sending to the Signal Encoder together can refer to the introduction of step 306 in the above embodiment.

[0317] Next, combine with Figure 6 , introduces an image encoding method in an embodiment of the present application, which is introduced by taking the encoding process of the YUV444 image format as an example, and Figure 5 The difference is that in Figure 6 In the image, the interaction between the chrominance component information and the luminance information occurs before the size transformation.

[0318] Reference Figure 6 , the method comprising:

[0319] The information of the Y component and the UV component can be combined by channel first, and then UPS is performed. Input image RGB[3,H,W], RGB is converted to 444 format YUV[3,H,W] by color transformation ColorTr. If the input is in YUV444 format, skip this step.

[0320] Processing of information about the Y component:

[0321] Y[1,H,W] skips downsampling to generate x Y [1,H,W], sent to Signal Encoder_Y.

[0322] Processing of UV component information:

[0323] Y[1,H,W] and UV[2,H,W] are combined into YUV[3,H,W] according to the channel dimension and then UPS(2,2) is used to generate x UV [12,H / 2,W / 2], sent to Signal Encoder_UV.

[0324] Figure 6 The information about the Y component and the UV component can be combined by channel first, and then the UPS step can refer to the introduction of the information interaction between the main component information and the slave component information before the size transformation in the above embodiment.

[0325] Reference Figure 7 , Figure 7 An example of an image decoding method provided in an embodiment of the present application is shown as follows: Figure 7 As shown, an image decoding method provided by an embodiment of the present application includes:

[0326] 701. Obtain first information according to a first code stream.

[0327] The first code stream may be an encoding result obtained by encoding the first image.

[0328] The first information may be the output of a decoding network or the output of an intermediate layer of a decoding network.

[0329] On the decoding side, when the decoding network is decoding different image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), wherein the output of the decoding network includes information of the chrominance component, and the output of the intermediate layer of the decoding network is an intermediate quantity, and the chrominance component information or the intermediate quantity is the first information in the embodiment of the present application.

[0330] In one possible implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, it can be resized to obtain the chrominance component information that meets the size requirements specified by the corresponding image format.

[0331] In the existing implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or, at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network in the decoding process of different (or, at least two) image formats are consistent (or nearly consistent), thereby improving the stability of information processing, which is better in terms of both the machine and user experience.

[0332] In addition, there is no binding relationship between input and output, and YUV444 input can simultaneously generate 444, 420, or 422 output. And the complexity of the embodiment of the present application when performing 444, 422 decoding is almost the same as 420, and is lower than the complexity of the original JPEG AI when performing 444, 422 decoding.

[0333] Among them, the decoding network can also be called a signal decoder. The decoder is implemented based on a neural network and is used to decode (or reconstruct features) the input information (for example, entropy decoding results) to obtain YUV information. Since it is a decoder implemented based on a neural network, it can also be called an AI decoder.

[0334] 702. Perform a size conversion on the first information to obtain second information, where the second information includes information of a chroma component in a first image format;

[0335] In the present application embodiment, and Figure 3 In contrast to the corresponding embodiment, the size transformation performed on the first information may increase the size of the information in the spatial dimension (optionally, for information in the YUV420 format, the size of the spatial dimension may remain unchanged) and reduce the size of the information in the channel dimension. The size of the second information in the spatial dimension is greater than or equal to the first information, and the size of the second information in the channel dimension is smaller than the first information.

[0336] Among them, the spatial dimension can further include a height dimension and a width dimension. The fact that the size of the second information in the spatial dimension is larger than that of the first information can be understood as the size of at least one dimension of the height dimension and the width dimension of the second information is larger than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is smaller than that of the first information).

[0337] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is greater than that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0338] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0339] The above two illustrated cases can be understood as the second information having a size in the spatial dimension being larger than the first information.

[0340] Next, the size transformation of the first information in the embodiment of the present application is described in detail:

[0341] The size conversion in the embodiment of the present application can convert information of the same size into information of a size specified by the corresponding image format.

[0342] Resizing can include operations such as information merging, information selection, and convolution. The following are introduced separately:

[0343] 1. Information Merger

[0344] In a possible implementation, performing size conversion on the first information may include: combining different data of multiple channels into the same channel.

[0345] That is, in order to reduce the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be reduced, and the information of the reduced channels needs to be merged into the existing channels in the first information. Then, the information of some channels in the first information can be taken out and merged with the information of other channels, thereby reducing the size of the first information in the channel dimension.

[0346] In a possible implementation, the size transformation method may include pixel shuffle (PS), that is, UPS may be performed on the first information.

[0347] Reference Figure 8a , Figure 8a This is a schematic diagram of the processing result of PS. Figure 8a An example is given of merging the information of every 4 channels of 4C channels in the first information into the data of one channel, wherein 4 different data belonging to the same position on the 4 channels are merged into one channel, so that the number of channels is changed from 4C to C. If the spatial size of the first information is H / 2*W / 2, the number of channels of the second information obtained after the size transformation is H*W.

[0348] In a possible implementation, the second information is information of the chrominance component of YUV444, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, the width dimension of the spatial dimension of the second information is W, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2. The size conversion method in this illustration can be as follows Figure 8a As shown, no further description is given here.

[0349] 2. Information Selection

[0350] The so-called selection is to select information of some channels in the first information and then merge them into new data with the same spatial dimension and lower channel dimension, which is equivalent to deleting the information of the unselected channels and retaining only the information of the selected channels.

[0351] In a possible implementation, in addition to increasing the number of channels by merging information, information selection can also be performed when performing size transformation. This method is more necessary when the second information is an image format with a relatively low resolution (at least not the highest resolution, such as YUV422 and YUV420), because: the amount of information (that is, the information size) of the chrominance component information of the low-resolution image format is less than that of the chrominance component information of the high-resolution image format. If only information merging and other operations are performed without reducing the amount of data (it should be understood that the amount of data and the amount of non-redundant information introduced above are not the same concept, the former is the total amount of information included in the information, and both redundant and non-redundant information are counted), the size of the chrominance component information of the low-resolution (at least not the highest resolution) image format will definitely be lower than the size of the chrominance component information of the high-resolution image format, and it will not meet the size requirements specified by the image format. In this case, when reconstructing the chrominance component information of the low-resolution (at least not the highest resolution) image format, the first information can be selected to reduce the amount of data, and the non-redundant information amount of the information after the selection operation may also be smaller than the data before the selection.

[0352] In a possible implementation, the first information can be selected, and different data from multiple channels can be combined into the same channel, and there is no time limit between the two operations. The number of each operation is also not limited. For example, information selection is performed first, then information merging is performed, and then selection is performed. As long as the information can be transformed to the required size, the embodiment of the present application does not limit the type, number, combination method and sequence of operations.

[0353] For example, in one possible implementation, the second information is information of the chrominance component of YUV422, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2.

[0354] In this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H / 2, W / 2) through information selection, and then the information is merged to obtain the second information (2, H, W / 2). Alternatively, in this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H, W / 2) through information selection, and then the information is selected to obtain the second information (2, H, W / 2).

[0355] In a possible implementation, the above-mentioned information selection or information merging operation may be replaced by convolution.

[0356] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant information is guaranteed to remain unchanged and the size can be converted to a required size, and this application is not limited thereto.

[0357] It should be understood that step 702 itself can be an operation independent of the decoding network, or can be implemented by the decoding network. This can be achieved, for example, by an internal network layer located close to the output of the decoding network (e.g. The network layer before the last convolutional layer of the decoding network or the network layer before the last pixel reconstruction layer ) is carried out. Next, introduce:

[0358] 1. Step 702 itself is an operation independent of the decoding network

[0359] In a possible implementation, when step 702 itself is an operation independent of the decoding network, the decoding network may output the first information. Compared with the prior art, step 702 is a step added after the decoding network.

[0360] 2. Step 702 is performed by a network layer in the decoding network close to the output of the decoding network

[0361] In one possible implementation, when step 702 is performed by a network layer in the decoding network close to the output of the decoding network, the network layer in the existing decoding network close to the output of the decoding network may be improved, for example, by changing the operation value of the UPS layer or the convolution layer, so that the first information is the output of the internal network layer of the synthetic conversion network.

[0362] 703. Reconstruct the first image according to the second information;

[0363] To convert to RGB, upsampling, combination of chrominance component information and luminance component information, ICCI filtering, and generation of an RGB image through a color conversion module invColorTr may be performed but is not limited thereto.

[0364] 704. Obtain third information according to the second bit stream, where sizes of the third information in a spatial dimension and a channel dimension are consistent with those of the first information;

[0365] The second code stream may be an encoding result obtained by encoding the second image.

[0366] The third information may be the output of the decoding network or the output of an intermediate layer of the decoding network.

[0367] 705. Perform a size conversion on the third information to obtain fourth information; the fourth information includes information of a chrominance component in a second image format, and the second image format is different from the first image format;

[0368] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0369] The relationship and introduction between the first image format and the second image format may refer to the description in the above embodiment, which will not be repeated here.

[0370] In a possible implementation, a size of the fourth information in a spatial dimension is greater than or equal to that of the third information, and a size of the fourth information in a channel dimension is smaller than that of the third information.

[0371] In a possible implementation, the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0372] 706. Reconstruct the second image according to the fourth information.

[0373] It should be understood that the above steps 701 to 703 are a decoding process for the first code stream, and steps 704 to 706 are a decoding process for the second code stream. The embodiment of the present application does not limit the timing relationship between the decoding process for the first code stream and the decoding process for the second code stream. For example, steps 701 to 703 may be performed before steps 704 to 706, or steps 701 to 703 may be performed after steps 704 to 706, or steps 701 to 703 may be performed simultaneously with steps 704 to 306.

[0374] Reference Figure 8b , Figure 8b An example of an image decoding method provided in an embodiment of the present application is shown as follows: Figure 8b As shown, an image decoding method provided by an embodiment of the present application includes:

[0375] 801. Obtain first information according to a first code stream.

[0376] The first information may be the output of a decoding network or the output of an intermediate layer of a decoding network.

[0377] On the decoding side, when the decoding network is decoding different image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), wherein the output of the decoding network includes information of the chrominance component, and the output of the intermediate layer of the decoding network is an intermediate quantity, and the chrominance component information or the intermediate quantity is the first information in the embodiment of the present application.

[0378] In one possible implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, it can be resized to obtain the chrominance component information that meets the size requirements specified by the corresponding image format.

[0379] In the existing implementation, when the decoding network is decoding different (or, at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or, at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network in the decoding process of different (or, at least two) image formats are consistent (or nearly consistent), thereby improving the stability of information processing, which is better in terms of both the machine and user experience.

[0380] In addition, for the information obtained from the same code stream (such as the first information), chrominance information of multiple image formats can be generated by resizing the first information. There is no binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. And the complexity of the embodiment of the present application when performing 444, 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI for 444, 422 decoding.

[0381] Among them, the coding synthesis transformation network or signal decoder (signal decoder), which is implemented based on a neural network, is used to decode (or can be called feature reconstruction) the input information (for example, the entropy decoding result) to obtain YUV information. Since it is a decoder implemented based on a neural network, it can also be called an AI decoder.

[0382] 802. Resize the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of chromaticity components in a first image format, the third information is information of chromaticity components in a second image format, the fourth information is information of chromaticity components in a third image format, and the first image format, the second image format and the third image format are different.

[0383] For example, when performing a scale transformation, at least one of the following actions may be selected to obtain chromaticity information of one or more image formats. That is, in the embodiment of the present application, for the same information, chromaticity information of multiple image formats may be obtained through different scale transformation methods, and then image reconstruction of the corresponding image format may be performed.

[0384] Performing a first size transformation on the first information to obtain second information, where the second information includes information of a chrominance component in a first image format;

[0385] Performing a second size transformation on the first information to obtain second information, where the second information includes information of a chrominance component in the first image format;

[0386] Perform a third size transformation on the first information to obtain second information, where the second information includes information of the chrominance component of the first image format.

[0387] Taking the second information as an example, in the embodiment of the present application, and Figure 3 In contrast to the corresponding embodiment, the size transformation performed on the first information may increase the size of the information in the spatial dimension (optionally, for information in the YUV420 format, the size of the spatial dimension may remain unchanged) and reduce the size of the information in the channel dimension. The size of the second information in the spatial dimension is greater than or equal to the first information, and the size of the second information in the channel dimension is smaller than the first information.

[0388] Among them, the spatial dimension can further include a height dimension and a width dimension. The fact that the size of the second information in the spatial dimension is larger than that of the first information can be understood as the size of at least one dimension of the height dimension and the width dimension of the second information is larger than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is smaller than that of the first information).

[0389] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is greater than that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0390] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0391] The above two illustrated cases can be understood as the second information having a size in the spatial dimension being larger than the first information.

[0392] Next, the size transformation of the first information in the embodiment of the present application is described in detail:

[0393] The size conversion in the embodiment of the present application can convert information of the same size into information of a size specified by the corresponding image format.

[0394] Resizing can include operations such as information merging, information selection, and convolution. The following are introduced separately:

[0395] 1. Information Merger

[0396] In a possible implementation, performing size conversion on the first information may include: combining different data of multiple channels into the same channel.

[0397] That is, in order to reduce the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be reduced, and the information of the reduced channels needs to be merged into the existing channels in the first information. Then, the information of some channels in the first information can be taken out and merged with the information of other channels, thereby reducing the size of the first information in the channel dimension.

[0398] In a possible implementation, the size transformation method may include pixel shuffle (PS), that is, UPS may be performed on the first information.

[0399] Reference Figure 8a , Figure 8a This is a schematic diagram of the processing result of PS. Figure 8a The example of merging the information of four channels in the first information into the data of one channel is used for illustration, wherein the four data at the same position on the four channels are merged into the data block of one channel, so that the number of channels is changed from four to one. If the number of channels of the first information is 4C, the number of channels of the second information obtained after the size transformation is C. If the spatial size of the first information is H / 2*W / 2, the number of channels of the second information obtained after the size transformation is H*W.

[0400] In a possible implementation, the second information is information of the chrominance component of YUV444, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, the width dimension of the spatial dimension of the second information is W, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2. The size conversion method in this illustration can be as follows Figure 8a As shown, no further description is given here.

[0401] 2. Information Selection

[0402] The so-called selection is to select information of some channels in the first information and then merge them into new data with the same spatial dimension and lower channel dimension, which is equivalent to deleting the information of the unselected channels and retaining only the information of the selected channels.

[0403] In a possible implementation, in addition to increasing the number of channels by merging information, information selection can also be performed when performing size transformation. This method is more necessary when the second information is an image format with a relatively low resolution (at least not the highest resolution, such as YUV422 and YUV420), because: the amount of information (that is, the information size) of the chrominance component information of the low-resolution image format is less than that of the chrominance component information of the high-resolution image format. If only information merging and other operations are performed without reducing the amount of data (it should be understood that the amount of data and the amount of non-redundant data introduced above are not the same concept, the former is the total amount of information included in the information, and both redundant and non-redundant information are counted), the size of the chrominance component information of the low-resolution (at least not the highest resolution) image format will definitely be lower than the size of the chrominance component information of the high-resolution image format, and it cannot meet the size requirements specified by the image format. In this case, when reconstructing the chrominance component information of the low-resolution (at least not the highest resolution) image format, the first information can be selected to reduce the amount of data, and the non-redundant data amount of the information after the selection operation may be smaller than the data before the selection.

[0404] In a possible implementation, the first information can be selected, and different data from multiple channels can be combined into the same channel, and there is no time limit between the two operations. The number of each operation is also not limited. For example, information selection is performed first, then information merging is performed, and then selection is performed. As long as it can be transformed to the required size, the embodiment of the application does not limit the type, number, combination method and sequence of operations.

[0405] For example, in one possible implementation, the second information is information of the chrominance component of YUV422, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2.

[0406] In this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H / 2, W / 2) through information selection, and then the information is merged to obtain the second information (2, H, W / 2). Alternatively, in this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H, W / 2) through information selection, and then the information is selected to obtain the second information (2, H, W / 2).

[0407] In a possible implementation, the above-mentioned information selection or information merging operation may be replaced by convolution.

[0408] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant data of the information is guaranteed to remain unchanged and it can be converted to the required size, and this application is not limited thereto.

[0409] It should be understood that step 802 itself can be an operation independent of the decoding network, or can be implemented by the decoding network. This can be achieved, for example, by an internal network layer located close to the output of the decoding network (e.g. The network layer before the last convolutional layer of the decoding network or the network layer before the last pixel reconstruction layer ) is carried out. Next, introduce:

[0410] 1. Step 802 itself is an operation independent of the decoding network

[0411] In a possible implementation, when step 802 itself is an operation independent of the decoding network, the decoding network may output the first information. Compared with the prior art, step 802 is a step added after the decoding network.

[0412] 2. Step 802 is performed by a network layer in the decoding network close to the output of the decoding network

[0413] In one possible implementation, when step 802 is performed by a network layer in the decoding network close to the output of the decoding network, the network layer in the existing decoding network close to the output of the decoding network can be improved, for example, the operation value of the UPS layer or the convolution layer is changed, and the first information is the output of the internal network layer of the synthetic conversion network.

[0414] 803. Perform one or more of the following steps: reconstruct the first image according to the second information; reconstruct the second image according to the third information; or reconstruct the third image according to the fourth information.

[0415] In a possible implementation, the image may be reconstructed according to at least one of the second information, the third information, or the fourth information.

[0416] To convert to RGB, upsampling, combination of chrominance component information and luminance component information, ICCI filtering, and generation of an RGB image through a color conversion module invColorTr may be performed but is not limited thereto.

[0417] Next, combine with Fig. 9 , introduces an image decoding method in an embodiment of the present application, which takes the decoding process of three image formats, YUV444, YUV422 and YUV420, as examples. Fig. 9 In , the size transformation is implemented by other modules independent of the decoding network.

[0418] The decoding network (Signal Decoder) reconstructs Y and UV (that is, Figure 7 The reconstructed UV features have a uniform shape of [8, H / 2, W / 2], and then the UV features of the uniform shape are converted into formats of different shapes (YUV444, YUV420, YUV422) through pixel shuffle (PS).

[0419] Reference Fig. 9 , the method comprising:

[0420] Get

[0421] Information processing for the Y component:

[0422] Output through Signal Decoder Skip upsampling and generate

[0423] Regarding the processing of UV components:

[0424] During a process, one of three formats can be selected:

[0425] About YUV420 decoding: Signal Decoder output (The reconstructed UV component has a shape of [8, H / 2, W / 2], where 8 represents the number of channels, and H / 2 and W / 2 represent the height and width respectively.) Select a quarter of the channels (because the number of UV pixels required by the 420 format is 1 / 4 of that of the 444 format), that is, 2 channels (for example, select the channel with a remainder of 0 when divided by 4 from channels 0 to 7), and generate (Shape is [2,H / 2,W / 2]). Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0426] in, Can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information described in the embodiment of the present invention obtains the Signal Decoder output The description of steps 701 and 704 in the above embodiment can be referred to. Select one quarter of the channels to generate The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0427] About YUV444 decoding: Signal Decoder output Generated by PS(2,2) At this time, the The merging can output images in YUV444 format.

[0428] in, Can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information described in the embodiment of the present invention obtains the Signal Decoder output Refer to the description of steps 701 and 704 in the above embodiment. Generated by PS(2,2) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0429] About YUV422 decoding: Signal Decoder output from Select half of the channels (because the number of UV pixels required by the 422 format is 1 / 2 of that of the 444 format), that is, 4 channels (such as selecting channels with a remainder of 0 when divided by 2 from channels 0 to 7) to generate And generated by PS(2,1) At this time, the The merging can output images in YUV422 format.

[0430] in, Can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information described in the embodiment of the present invention obtains the Signal Decoder output The description of steps 701 and 704 in the above embodiment can be referred to. Select one-half of the channels and generate them through PS(2,1) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0431] Optional, by and Generate by channel combination Optionally, when the JPEG AI configuration enables ICCI filtering, and To generate Generated by the color conversion module invColorTr

[0432] Next, combine with Fig.10 , introduces an image decoding method in an embodiment of the present application, and Fig. 9 The difference is that in Fig.10 In , the resizing itself is implemented by the pixel reassembly PS layer of the decoding network.

[0433] In the existing implementation, the last layer of the decoding network signal decoder is a pixel reorganization layer PS (4, 4), which can be combined with the resizing PS in the embodiment of the present application. In this embodiment, the resizing operation and the last layer PS in the signalDecoder can be combined into one layer PS.

[0434] Reference Fig.10 , the method comprising:

[0435] Information processing for the Y component:

[0436] Signal Decoder Output Skip upsampling and generate

[0437] Regarding the processing of UV components:

[0438] During a process, one of three formats can be selected:

[0439] About YUV420 decoding: part1 in Signal Decoder (all layers except the last pixel shuffle layer) outputs feature L[128,H / 8,W / 8]. Select 32 channels from L (select channels with a remainder of 0 when divided by 4 from channels 0 to 127) to generate L′[32,H / 8,W / 8] (the number of UV pixels required for the 420 format is one-fourth of that for the 444 format), and generate it through PS(4,4) At this time, the Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0440] Among them, the feature L can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information described in the embodiment of the present invention, the output feature L of part 1 in the Signal Decoder can be obtained by referring to the introduction of steps 701 and 704 in the above embodiment, 32 channels are selected from L and generated by PS (4,4) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0441] About YUV444 decoding: part1 in Signal Decoder (all layers except the last pixel shuffle layer) outputs L[128,H / 8,W / 8]. L is generated by PS(8,8) At this time, the The merging can output images in YUV444 format.

[0442] Among them, the feature L can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7The second information described in the embodiment of the present invention can be obtained by referring to the description of steps 701 and 704 in the above embodiment. Generated by PS(8,8) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0443] About YUV422 decoding: part1 in Signal Decoder (all layers except the last pixel shuffle layer) outputs L[128,H / 8,W / 8]. Select 64 channels from L (select channels with a remainder of 0 when divided by 2 from channels 0 to 127) to generate (The number of UV pixels required for the 422 format is half of that for the 444 format) and is generated by PS(8,4) At this time, the Merge to output YUV422 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[1,2] times generated

[0444] Among them, the feature L can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information described in the embodiment of the present invention can be used to obtain the Signal Decoder output L, and the description of steps 701 and 704 in the above embodiment can be referred to. 64 channels are selected from L and generated by PS (8, 4) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0445] Next, combine with Fig.11 , introduces an image decoding method in an embodiment of the present application, and Fig.10 The difference is that in Fig.11 In , the resizing itself is implemented by the convolutional layers of the decoding network.

[0446] This embodiment describes a possible modification of the decoding end structure. In this embodiment, the feature L reconstructed by the first part of the decoder no longer has a uniform number of channels, but the decoder Signal Decoder reconstructs L with different channels according to the required different output formats. The first part of the decoder includes all layers except the last pixel shuffle (PS) layer and the last convolution layer.

[0447] Reference Fig. 9 , the method comprising:

[0448] Information processing for the Y component:

[0449] Signal Decoder Output Skip upsampling and generate

[0450] Regarding the processing of UV components:

[0451] During a process, one of three formats can be selected:

[0452] Regarding YUV420 decoding: part 1 in Signal Decoder outputs the feature L′[Ch,H / 8,W / 8], and passes through the convolution layer CONV(3×3,32) with an output channel of 32 to generate L′[32,H / 8,W / 8]. Then it is generated by PS(4,4) At this time, the Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0453] Wherein, L′ can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information introduced in the embodiment of the present invention can be used to obtain the signal decoder output L', which can refer to the introduction of step 701 in the above embodiment, and pass through the convolution layer CONV (3 × 3, 32) with an output channel of 32 to generate L', and then pass through PS (4, 4) to generate The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0454] Regarding YUV444 decoding: part1 in Signal Decoder outputs feature L′[Ch,H / 8,W / 8], and passes through the convolution layer CONV(3×3,128) with an output channel of 128 to generate L′[128,H / 8,W / 8]. L is generated by PS(8,8) At this time, the The merging can output images in YUV444 format.

[0455] Wherein, L′ can be the above Figure 7The first information introduced in the embodiments, Can be the above Figure 7 The second information introduced in the embodiment of the present invention can be used to obtain the Signal Decoder output L′, which can refer to the introduction of step 701 in the above embodiment, and is generated by the convolution layer CONV with an output channel of 128 and PS (8, 8) The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0456] Regarding YUV422 decoding: part1 in Signal Decoder outputs feature L′[Ch,H / 8,W / 8], and passes through the convolution layer CONV(3×3,64) with an output channel of 64 to generate L′[64,H / 8,W / 8]. And then passes through PS(8,4) to generate At this time, the Merge to output YUV422 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[1,2] times generated

[0457] Wherein, L′ can be the above Figure 7 The first information introduced in the embodiments, Can be the above Figure 7 The second information introduced in the embodiment of the present invention can be used to obtain the Signal Decoder output L′, which can refer to the introduction of step 701 in the above embodiment, and is generated by the convolution layer CONV with an output channel of 64 and PS (8, 4). The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0458] The encoding and decoding method in the embodiment of the present application is described above. The device in the embodiment of the present application is described below. Fig.12 , an embodiment of the encoding device in the embodiment of the present application includes:

[0459] The acquisition module 1201 is configured to acquire first information, wherein the first information includes information about a chrominance component in a first image format; and acquire third information, wherein the third information includes information about a chrominance component in a second image format; wherein the second image format is different from the first image format;

[0460] Among them, the introduction of the acquisition module 1201 can refer to the introduction of steps 301 and 304 in the above embodiment, and will not be repeated here.

[0461] The processing module 1202 is used to resize the first information to obtain second information; encode the second information through a coding network or part of a coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through a coding network.

[0462] Among them, the introduction of the processing module 1202 can refer to the introduction of steps 302, 303, 305 and 306 in the above embodiment, which will not be repeated here.

[0463] In a possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0464] In a possible implementation, the amount of non-redundant information included in the second information is consistent with the amount of non-redundant information included in the first information; the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0465] In a possible implementation, the size transformation includes at least one of the following:

[0466] Copy the information and use the copied information as the information on the new channel;

[0467] Splitting the information on at least one channel into information on a plurality of different channels; or,

[0468] Perform convolution operation.

[0469] In a possible implementation, the operation of splitting the information on at least one channel into information on a plurality of different channels is implemented by an inverse pixel recombination layer.

[0470] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or,

[0471] The first image format is YUV422 format, and the second image format is YUV420 format.

[0472] In a possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0473] In a possible implementation, the acquisition module 1201 is further configured to:

[0474] Acquire fifth information, where the fifth information is information of a brightness component Y;

[0475] The processing module is further used to perform a size transformation on the fifth information to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with that of the second information;

[0476] The processing module is specifically used to encode the fusion result of the second information and the fourth information through a coding network.

[0477] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0478] See also Fig.13 , an embodiment of the decoding device in the embodiment of the present application includes:

[0479] The acquisition module 1301 is used to acquire first information according to the first bit stream; acquire third information according to the second bit stream, wherein the size of the third information in the spatial dimension and the channel dimension is consistent with the first information;

[0480] Among them, the introduction of the acquisition module 1301 can refer to the introduction of steps 701 and 704 in the above embodiment, and will not be repeated here.

[0481] The processing module 1302 is used to resize the first information to obtain second information, where the second information includes information about the chrominance component of the first image format; reconstruct the first image based on the second information; obtain third information based on the second code stream, where the size of the third information in the spatial dimension and the channel dimension is consistent with that of the first information; resize the third information to obtain fourth information; the fourth information includes information about the chrominance component of the second image format, and the second image format is different from the first image format; and reconstruct the second image based on the fourth information.

[0482] Among them, the introduction of the processing module 1302 can refer to the introduction of steps 702, 703, 705 and 706 in the above embodiment, which will not be repeated here.

[0483] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is greater than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0484] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0485] In a possible implementation, the size transformation includes at least one of the following methods:

[0486] Delete the information of some channels;

[0487] Combine different data from multiple channels onto the same channel; or,

[0488] Perform convolution operation.

[0489] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0490] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or,

[0491] The first image format is YUV422 format, and the second image format is YUV420 format.

[0492] In a possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0493] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0494] In addition, an embodiment of the present application further provides an image decoding device, the device comprising:

[0495] An acquisition module, used for acquiring first information according to the first code stream;

[0496] A processing module is used to resize the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of chromaticity components in a first image format, the third information is information of chromaticity components in a second image format, the fourth information is information of chromaticity components in a third image format, and the first image format, the second image format and the third image format are different.

[0497] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to that of the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0498] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0499] In a possible implementation, the processing module is further configured to:

[0500] The first image is reconstructed according to at least one of the second information, the third information or the fourth information.

[0501] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; and performing a convolution operation.

[0502] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reorganization layer.

[0503] In a possible implementation, the first image format is a YUV444 format, and the second image format is a YUV422 format or a YUV420 format; or, the first image format is a YUV422 format, and the second image format is a YUV420 format.

[0504] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

[0505] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0506] Next, a device provided in an embodiment of the present application is introduced. The device may be an encoding device or a decoding device. Fig.14 , Fig.14 A schematic diagram of a structure of a device provided in an embodiment of the present application, the device 1400 may be a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Among them, the device 1400 may be deployed with Fig.12 or Fig.13 Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403 and a memory 1404 (wherein the number of the processor 1403 in the device 1400 may be one or more, Fig.14 In the example of FIG. 1403 , the processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403 and the memory 1404 may be connected via a bus or other means.

[0507] The memory 1404 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1403. A portion of the memory 1404 may also include a non-volatile random access memory (NVRAM). The memory 1404 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0508] The processor 1403 controls the operation of the device. In a specific application, the various components of the device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0509] The method disclosed in the above embodiment of the present application can be applied to the processor 1403, or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1403. The above processor 1403 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1403 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1404, and the processor 1403 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0510] The receiver 1401 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the device. The transmitter 1402 can be used to output digital or character information through the first interface; the transmitter 1402 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1402 can also include a display device such as a display screen.

[0511] The present application also provides a device, which may be a coding device or a decoding device. Fig.15 , Fig.15 This is a schematic diagram of a structure of a device provided in an embodiment of the present application. The device 1500 may be deployed with Fig.12 or Fig.13Corresponding to the apparatus described in the embodiment, specifically, the device 1500 is implemented by one or more servers, and the device 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or persistent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 to execute a series of instruction operations in the storage medium 1530 on the device 1500.

[0512] The device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0513] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation mode of the above-mentioned image processing device.

[0514] The embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned image processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data for the at least one processor. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the terminal device. The chip system may be composed of chips, or may include chips and other discrete devices.

[0515] The present application also provides a readable storage medium, which stores the Figures 1a to 6 The code stream is obtained by encoding using the method in the illustrated embodiment.

[0516] The embodiment of the present application also provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send a code stream; at least one storage medium is used to store a code stream; the code stream is a code stream generated by an encoder according to the aforementioned Figures 1a to 6 The method in the illustrated embodiment is used for encoding.

[0517] The embodiment of the present application also provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, the code stream being the code stream of the encoder according to the above Figures 1a to 6 The method in the illustrated embodiment is used for encoding.

[0518] The embodiment of the present application also provides a system for distributing code streams, the system comprising at least one storage medium and a video stream device; at least one storage medium is used to store the code stream, the code stream is the code stream of the encoder according to the above Figures 1a to 6 The method in the illustrated embodiment is used to encode the video stream obtained by the method; the video stream device is used to respond to the request of the decoder so that the target code stream in at least one storage medium can be sent to the decoder.

[0519] The embodiment of the present application also provides a method for distributing a bitstream, the method comprising: receiving a first request; selecting a target bitstream from at least one storage medium in response to the first request; sending the target bitstream to a destination device; at least one storage medium is used to store the bitstream, the bitstream being the codestream generated by the encoder according to the aforementioned Figures 1a to 6 The method in the illustrated embodiment is used for encoding.

[0520] The embodiment of the present application also provides a system for processing a code stream, the system comprising an image source device, an encoder device, one or more storage media and a destination device; the image source device is used to provide image data; the encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, the code stream is the encoder according to the above Figures 1a to 13 The method in the illustrated embodiment is used to encode; the encoder device is used to store one or more code streams in one or more storage media; or the encoder device is used to encapsulate one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to a destination device through a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; the destination device is used to decode the one or more code streams to obtain decoded data.

[0521] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0522] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0523] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0524] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0525] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, device, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0526] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0527] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, device or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, device or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a device or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image coding method, characterized in that: The method comprises: Acquire first information, where the first information includes information of a chrominance component of a first image format; Performing a size transformation on the first information to obtain second information; encoding the second information by means of a coding network or a portion of a coding network; Acquire third information, where the third information includes information of a chrominance component of a second image format; the second image format is different from the first image format; Performing a size transformation on the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with that of the second information; The fourth information is encoded by the encoding network or a part of the encoding network.

2. The method according to claim 1, characterized in that The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to that of the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

3. The method according to claim 1 or 2, characterized in that: The amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

4. The method according to any one of claims 1 to 3, characterized in that: The size transformation includes at least one of the following: Copy the information and use the copied information as the information on the new channel; Splitting information on at least one channel into information on a plurality of different channels; or, Perform convolution operation.

5. The method according to claim 4, characterized in that The operation of splitting the information on at least one channel into information on a plurality of different channels is achieved through an inverse pixel recombination layer.

6. The method according to any one of claims 1 to 5, characterized in that: The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

7. The method according to any one of claims 1 to 6, characterized in that: The number of channel dimensions of the second information and the fourth information is 8, the size of the spatial dimension of the second information includes height and width, the height of the second information is H / 2, and the width of the second information is W / 2, wherein H is the height of the first image, and W is the width of the first image.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Acquire fifth information, where the fifth information is information of a brightness component Y; Performing a size transformation on the fifth information to obtain sixth information; the size of the sixth information in a spatial dimension is consistent with that of the second information; The encoding of the second information by means of a coding network or a portion of a coding network comprises: The fusion result of the second information and the fourth information is encoded through a coding network.

9. The method according to any one of claims 1 to 8, characterized in that: The first information or the third information further includes information of a brightness component.

10. An image decoding method, characterized in that: The method comprises: Acquire first information according to the first code stream; Performing a size conversion on the first information to obtain second information, where the second information includes information of a chrominance component in a first image format; reconstructing the first image according to the second information; Acquire third information according to the second bitstream, wherein the sizes of the third information in the spatial dimension and the channel dimension are consistent with those of the first information; Performing a size conversion on the third information to obtain fourth information; the fourth information includes information of a chrominance component in a second image format, and the second image format is different from the first image format; The second image is reconstructed according to the fourth information.

11. The method according to claim 10, characterized in that The first information or the third information is an output of a decoding network or an output of an intermediate layer of a decoding network.

12. The method according to claim 10 or 11, characterized in that: The size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information. The size of the fourth information in the spatial dimension is larger than or equal to that of the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

13. The method according to any one of claims 10 to 12, characterized in that: The size transformation includes at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

14. The method according to claim 13, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

15. The method according to any one of claims 10 to 14, characterized in that: The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

16. The method according to any one of claims 10 to 15, characterized in that: The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

17. The method according to any one of claims 10 to 16, characterized in that: The intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

18. An image encoding device, characterized in that: The device comprises: an acquisition module, configured to acquire first information, wherein the first information includes information of a chrominance component in a first image format; and acquire third information, wherein the third information includes information of a chrominance component in a second image format; wherein the second image format is different from the first image format; A processing module is used to resize the first information to obtain second information; encode the second information through a coding network or a part of the coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through the coding network.

19. The device according to claim 18, characterized in that The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to that of the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

20. The device according to claim 18 or 19, characterized in that The amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

21. The device according to any one of claims 18 to 20, characterized in that The size transformation includes at least one of the following: Copy the information and use the copied information as the information on the new channel; Splitting information on at least one channel into information on a plurality of different channels; or, Perform convolution operation.

22. The device according to claim 21, characterized in that The operation of splitting the information on at least one channel into information on a plurality of different channels is achieved through an inverse pixel recombination layer.

23. The device according to any one of claims 18 to 22, characterized in that The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

24. The device according to any one of claims 18 to 23, characterized in that The number of channel dimensions of the second information and the fourth information is 8, the size of the spatial dimension of the second information includes height and width, the height of the second information is H / 2, and the width of the second information is W / 2, wherein H is the height of the first image, and W is the width of the first image.

25. The device according to any one of claims 18 to 24, characterized in that The acquisition module is further used for: Acquire fifth information, where the fifth information is information of a brightness component Y; The processing module is further used to perform a size transformation on the fifth information to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with that of the second information; The processing module is specifically used to encode the fusion result of the second information and the fourth information through a coding network.

26. The device according to any one of claims 18 to 25, characterized in that The first information or the third information further includes information of a brightness component.

27. An image decoding device, characterized in that: The device comprises: An acquisition module, configured to acquire first information according to the first bit stream; and acquire third information according to the second bit stream, wherein the size of the third information in the spatial dimension and the channel dimension is consistent with that of the first information; A processing module is used to resize the first information to obtain second information, wherein the second information includes information of chromaticity components in a first image format; reconstruct the first image based on the second information; resize the third information to obtain fourth information; the fourth information includes information of chromaticity components in a second image format, and the second image format is different from the first image format; and reconstruct the second image based on the fourth information.

28. The device according to claim 27, characterized in that The first information or the third information is an output of a decoding network or an output of an intermediate layer of a decoding network.

29. The device according to claim 27 or 28, characterized in that The size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information. The size of the fourth information in the spatial dimension is larger than or equal to that of the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

30. The device according to any one of claims 27 to 29, characterized in that The size transformation includes at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

31. The device according to claim 30, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

32. The device according to any one of claims 27 to 31, characterized in that The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

33. The device according to any one of claims 27 to 32, characterized in that The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

34. The device according to any one of claims 27 to 33, characterized in that The intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

35. An image decoding method, characterized in that: The method comprises: Acquire first information according to the first code stream; resizing the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of chrominance components in a first image format, the third information is information of chrominance components in a second image format, the fourth information is information of chrominance components in a third image format, and the first image format, the second image format and the third image format are different; and Perform one or more of the following steps: reconstructing the first image according to the second information; reconstructing the second image according to the third information; or, The third image is reconstructed according to the fourth information.

36. The method according to claim 35, characterized in that The first information is the output of the decoding network or the output of an intermediate layer of the decoding network.

37. The method according to claim 35 or 36, characterized in that The sizes of the second information, the third information and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information and the fourth information in the channel dimension are smaller than the first information.

38. The method according to any one of claims 35 to 37, characterized in that: The amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

39. The method according to any one of claims 35 to 38, characterized in that The first size transformation and the second size transformation include at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

40. The method according to claim 39, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

41. An image decoding device, characterized in that: The device comprises: An acquisition module, used for acquiring first information according to the first code stream; a processing module, configured to perform a size conversion on the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of a chroma component in a first image format, the third information is information of a chroma component in a second image format, the fourth information is information of a chroma component in a third image format, and the first image format, the second image format and the third image format are different; and performing one or more of the following steps: reconstructing the first image according to the second information; reconstructing the second image according to the third information; or, The third image is reconstructed according to the fourth information.

42. A decoding device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the decoding device performs the method as described in any one of claims 10 to 17, and 35 to 40.

43. A coding device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding device executes the method as described in any one of claims 1 to 9.

44. A coding and decoding system, characterized in that: The encoding and decoding system includes the decoding device as claimed in claim 42 and / or the encoding device as claimed in claim 43.

45. A computer-readable storage medium, characterized in that The medium stores instructions, and when the instructions are executed by a computer, the method of any one of claims 1 to 17, and 35 to 40 is implemented.

46. ​​A computer program product, characterized in that The invention comprises instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 17, and 35 to 40.

47. A readable storage medium, characterized in that The readable medium stores a code stream encoded according to any one of the encoding methods of claims 1 to 9.

48. A device for storing a code stream, characterized in that: comprising at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the code stream; The code stream is obtained by encoding by an encoder according to any encoding method of claims 1 to 9.

49. A method for storing a code stream, characterized in that: include: Receiving a code stream via a communication interface; The code stream is stored in one or more storage media, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9.

50. A system for distributing code streams, characterized in that: including at least one storage medium and a video streaming device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9; The video stream device is used to respond to a request of a decoder so that the code stream in the at least one storage medium can be sent to the decoder.

51. A method for distributing code streams, characterized in that: include: receiving a first request; In response to the first request, selecting a code stream from at least one storage medium; Sending the code stream to a destination device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9.

52. A system for processing a code stream, characterized in that: It includes an image source device, an encoder device, one or more storage media and a destination device; The image source device is used to provide image data; The encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to any one of the encoding methods of claims 1 to 9; The encoder device is used to store the one or more code streams in one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; The destination device is used to decapsulate the transport code stream to obtain the one or more code streams; The destination device is used to decode the one or more code streams to obtain decoded data.

Images