Image encoding methods, image decoding methods, image encoders and image decoders.

Abstract

The invention relates to the field of encoding and decoding and reveals a method for encoding images, a method for decoding images, an image encoder, and an image decoder.The image decoding method includes: the first information acquisition step based on the first bit stream; the step of performing a size transformation on the first information to obtain the second information, which includes information about the chromatic composition in the first image format; the step of reconstructing the first image based on the second information; the third information acquisition step based on the second bit stream, where the dimensions of the third information in spatial dimensions and channel dimensions are consistent with the dimensions of the first information; the step of performing a size transformation on the third information to obtain the fourth information, which includes information about the chromatic composition in the second image format, and the second image format is different from the first image format; and the step of reconstructing the second image based on the fourth information.In this invention, size transformation can be performed on information of the same size in the decoding process to obtain information about the tonal components in different image formats, so that computational power and time costs are consistent in the process in which the decoding network performs decoding of different image formats, thereby improving the stability of the information processing.

Description

Image encoding method, decoding method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 18, 2023, with application number 202311361171.2 and application name “An image encoding method, decoding method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to an image encoding method, a decoding method, and related devices. Background Art

[0003] Image / video compression refers to a technology that exploits data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / video with the fewest bits possible, either losslessly or losslessly. This technology enables efficient transmission and storage of data information, playing a crucial role in the current media age, where data types and volumes are increasing. Data compression can be categorized as lossy or lossless. Lossy compression achieves a higher compression ratio at the expense of some image / video quality, while lossless compression does not cause any loss in image / video quality.

[0004] Currently, JPEG AI can encode images in different formats, such as YUV420, YUV444, and YUV422, by using different downsampling multiples. However, different formats have a significant impact on the computing power of the current JPEG AI network.

[0005] Summary of the Invention

[0006] The present application provides an image encoding method and an image decoding method, so that the computing power and time costs of the encoding network or decoding network when processing information of different (or at least two) image formats are consistent (or nearly consistent, or at least smaller than the cost difference of processing original information of different formats), thereby improving the stability of information processing.

[0007] In a first aspect, the present application provides an image encoding method, the method comprising: obtaining first information, the first information including information of a chrominance component of a first image format; performing a size transformation on the first information to obtain second information; encoding the second information through a coding network or a part of a coding network; obtaining third information, the third information including information of a chrominance component of a second image format; the second image format is different from the first image format; performing a size transformation on the third information to obtain fourth information; the size of the fourth information in spatial dimension and channel dimension is consistent with the second information; encoding the fourth information through a coding network or a part of a coding network.

[0008] In the existing implementation, the information of the chrominance component is directly input into the encoding network, and the encoding network encodes the information of the chrominance component. Since the sizes of information in different image formats are different, that is, the sizes of the information input into the encoding network are different, the computing power overhead and time overhead of the encoding network when processing information in different image formats are different, resulting in poor stability of information processing. In an embodiment of the present application, the information of the chrominance components of different image formats is converted into information of the same size through size transformation and input into the encoding network, so that the computing power overhead and time overhead of the encoding network when processing information in different (or at least two) image formats are consistent (or, close to consistent, at least smaller than the difference in overhead of processing original information in different formats), thereby improving the stability of information processing, which is better from the perspective of both machine and user experience.

[0009] Furthermore, in the prior art, if a model is trained for a low-resolution image format (e.g., YUV420), then the model's encoding performance for a high-resolution format (e.g., YUV444) will be very poor. (If a normal 444 encoding model is required, a separate model must be trained.) In the embodiments of this application, a single trained model can be used to encode multiple formats, and all of them exhibit good and stable encoding performance.

[0010] Among them, the encoding network can also be called an analysis transformation network.

[0011] In one possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0012] In a possible implementation, the amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

[0013] In one possible implementation, the information of chroma components in different formats carries different amounts of information due to their different sizes (for low-resolution image formats, the amount of information carried is less than that of high-resolution image formats due to downsampling). The amount of information here refers to the amount of effective information, that is, the amount of non-redundant information. For example, if the same information is copied multiple times, although the amount of information increases, the amount of non-redundant information remains unchanged. The resizing of the embodiments of the present application can make the size of the chroma component information of different (or at least two) image formats consistent. This is to maintain the stability of computational overhead (including computing power overhead or time overhead). However, even if the chroma component information is resized, the codestream obtained after the entire encoding process is consistent with the codestream obtained before the resizing. If it is inconsistent, the point of using different image formats is lost. In order to ensure that the result of the encoding process (i.e., the codestream) is consistent (or nearly consistent) with the existing scheme, it is necessary to ensure that the effective information content of the information is unchanged before and after the resizing.

[0014] In one possible implementation, the size transformation includes at least one of the following: copying information and using the copied information as information on a new channel; splitting information on at least one channel into information on multiple different channels; or performing a convolution operation.

[0015] In a possible implementation, the operation of splitting the information on at least one channel into information on multiple different channels is achieved through an inverse pixel reconstruction layer.

[0016] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0017] In one possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0018] In a possible implementation, the method also includes: obtaining fifth information, the fifth information being information of the brightness component Y; performing a size transformation on the fifth information to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with the second information; encoding the second information through the coding network or part of the coding network includes: encoding the fusion result of the second information and the fourth information through the coding network.

[0019] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0020] In a second aspect, the present application provides an image decoding method, the method comprising: obtaining first information according to a first code stream; performing a size transformation on the first information to obtain second information, wherein the second information includes information on the chroma component of the first image format; reconstructing the first image according to the second information; obtaining third information according to the second code stream, wherein the size of the third information in the spatial dimension and the channel dimension is consistent with the first information; performing a size transformation on the third information to obtain fourth information; the fourth information includes information on the chroma component of the second image format, and the second image format is different from the first image format; and reconstructing the second image according to the fourth information.

[0021] In the existing implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension and the size of the spatial dimension are the same), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network are consistent (or nearly consistent) in the decoding process of different (or at least two) image formats, thereby improving the stability of information processing, which is better from the perspective of both the machine and user experience.

[0022] Among them, the decoding network can also be called a synthetic transformation network.

[0023] In addition, there is no binding relationship between input and output. YUV444 input can simultaneously generate 444, 420, or 422 output. Moreover, the complexity of the embodiment of the present application when performing 444 and 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI when performing 444 and 422 decoding.

[0024] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is larger than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0025] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0026] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0027] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0028] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0029] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0030] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0031] In a third aspect, the present application provides an image encoding device, the device comprising:

[0032] an acquisition module, configured to acquire first information, the first information including information of a chrominance component in a first image format; and acquire third information, the third information including information of a chrominance component in a second image format, the second image format being different from the first image format;

[0033] A processing module is used to resize the first information to obtain second information; encode the second information through a coding network or a part of the coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through the coding network.

[0034] In one possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0035] In a possible implementation, the amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

[0036] In a possible implementation, the size transformation includes at least one of the following:

[0037] Copy the information and use the copied information as the information on the new channel;

[0038] Splitting the information on at least one channel into information on a plurality of different channels; or,

[0039] Perform convolution operation.

[0040] In a possible implementation, the operation of splitting the information on at least one channel into information on multiple different channels is achieved through an inverse pixel reconstruction layer.

[0041] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or,

[0042] The first image format is YUV422 format, and the second image format is YUV420 format.

[0043] In one possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0044] In a possible implementation, the acquisition module is further configured to:

[0045] Acquire fifth information, where the fifth information is information of a brightness component Y;

[0046] The processing module is further configured to perform size transformation on the fifth information to obtain sixth information; the size of the sixth information in a spatial dimension is consistent with that of the second information;

[0047] The processing module is specifically configured to encode a fusion result of the second information and the fourth information through a coding network.

[0048] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0049] In a fourth aspect, the present application provides an image decoding device, the device comprising:

[0050] an acquisition module, configured to acquire first information according to the first code stream; and acquire third information according to the second code stream, wherein the dimensions of the third information in the spatial dimension and the channel dimension are consistent with those of the first information;

[0051] A processing module is configured to resize the first information to obtain second information, where the second information includes information about the chrominance components in a first image format; reconstruct the first image based on the second information; resize the third information to obtain fourth information, where the fourth information includes information about the chrominance components in a second image format that is different from the first image format; and reconstruct the second image based on the fourth information.

[0052] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is larger than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0053] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0054] In a possible implementation, the size conversion includes at least one of the following methods:

[0055] Delete information of some channels;

[0056] Combine different data from multiple channels onto the same channel; or,

[0057] Perform convolution operation.

[0058] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0059] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or,

[0060] The first image format is YUV422 format, and the second image format is YUV420 format.

[0061] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0062] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0063] In a fifth aspect, the present application provides an image decoding method, the method comprising: obtaining first information based on a first code stream; performing a size transformation on the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of the chrominance component of the first image format, the third information is information of the chrominance component of the second image format, the fourth information is information of the chrominance component of the third image format, and the first image format, the second image format and the third image format are different; and performing one or more of the following steps: reconstructing the first image based on the second information; reconstructing the second image based on the third information; or reconstructing the third image based on the fourth information.

[0064] In the existing implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension and the size of the spatial dimension are the same), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network are consistent (or nearly consistent) in the decoding process of different (or at least two) image formats, thereby improving the stability of information processing, which is better from the perspective of both the machine and user experience.

[0065] In addition, for the information obtained from the same code stream (such as the first information), chroma information in multiple image formats can be generated by resizing the first information. This eliminates the need for a binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. Furthermore, the complexity of the embodiment of the present application when performing 444 and 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI when performing 444 and 422 decoding.

[0066] In one possible implementation, the sizes of the second information, the third information, and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information, and the fourth information in the channel dimension are smaller than the first information.

[0067] In a possible implementation, the amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

[0068] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0069] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0070] In a possible implementation, the first image format is a YUV444 format, the second image format is a YUV422 format, and the third image format is a YUV420 format.

[0071] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0072] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0073] In a sixth aspect, the present application provides an image decoding device, the device comprising:

[0074] An acquisition module, configured to acquire first information according to the first code stream;

[0075] a processing module, configured to perform size conversion on the first information to obtain one or more of second information, third information, or fourth information; wherein the second information is information of chrominance components in a first image format, the third information is information of chrominance components in a second image format, and the fourth information is information of chrominance components in a third image format, and the first image format, the second image format, and the third image format are different; and performing one or more of the following steps:

[0076] reconstructing the first image according to the second information;

[0077] reconstructing the second image according to the third information; or,

[0078] The third image is reconstructed according to the fourth information.

[0079] It should be understood that the processing module introduced in the sixth aspect can have the ability to convert the information obtained from the same code stream into chromaticity information in multiple image formats. When running the processing model, one or more of the multiple image formats can be selected for generation based on demand (such as user specification or device selection).

[0080] In a possible implementation, the first information is the output of a decoding network or the output of an intermediate layer of a decoding network.

[0081] In one possible implementation, the sizes of the second information, the third information, and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information, and the fourth information in the channel dimension are smaller than the first information.

[0082] In a possible implementation, the amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

[0083] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0084] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0085] In a possible implementation, the first image format is a YUV444 format, the second image format is a YUV422 format, and the third image format is a YUV420 format.

[0086] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0087] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0088] In a seventh aspect, the present application provides an image decoding method, the method comprising: obtaining first information according to a first code stream; performing a first size transformation on the first information to obtain second information, the second information including information of a chroma component of a first image format; reconstructing a first image according to the second information; performing a second size transformation on the first information to obtain third information; the third information including information of a chroma component of a second image format, the second image format being different from the first image format; and reconstructing a second image according to the third information.

[0089] In the existing implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension and the size of the spatial dimension are the same), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network are consistent (or nearly consistent) in the decoding process of different (or at least two) image formats, thereby improving the stability of information processing, which is better from the perspective of both the machine and user experience.

[0090] In addition, for the information obtained from the same code stream (such as the first information), chroma information in multiple image formats can be generated by resizing the first information. This eliminates the need for a binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. Furthermore, the complexity of the embodiment of the present application when performing 444 and 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI when performing 444 and 422 decoding.

[0091] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0092] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0093] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; or performing a convolution operation.

[0094] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0095] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0096] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0097] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0098] In an eighth aspect, the present application provides an image decoding device, the device comprising:

[0099] An acquisition module, configured to acquire first information according to the first code stream;

[0100] a processing module, configured to perform a first size transformation on the first information to obtain second information, where the second information includes information about the chrominance components in the first image format, and reconstruct the first image based on the second information; or to perform a second size transformation on the first information to obtain third information, where the third information includes information about the chrominance components in a second image format that is different from the first image format, and reconstruct the second image based on the third information.

[0101] It should be understood that the processing module introduced in the eighth aspect can have the ability to convert the information obtained from the same code stream into chromaticity information in multiple image formats. When running the processing model, one or more of the multiple image formats can be selected for generation based on demand (such as user specification or device selection).

[0102] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0103] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0104] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; and performing a convolution operation.

[0105] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0106] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0107] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0108] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0109] A ninth aspect of an embodiment of the present application provides a coding device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, the coding device implements the method in the first aspect or any possible implementation of the first aspect.

[0110] The tenth aspect of an embodiment of the present application provides a decoding device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the decoding device implements the method in the above-mentioned second aspect or any possible implementation of the second aspect, or the above-mentioned fifth aspect or any possible implementation of the fifth aspect, or the above-mentioned seventh aspect or any possible implementation of the seventh aspect.

[0111] An eleventh aspect of an embodiment of the present application provides a coding and decoding system, comprising: the coding device of the ninth aspect above, and / or the decoding device of the tenth aspect above.

[0112] A twelfth aspect of the embodiments of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction runs on a computer, the computer is caused to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, or the computer is caused to execute the method in the aforementioned second aspect or any possible implementation of the second aspect, or the method in the aforementioned fifth aspect or any possible implementation of the fifth aspect, or the method in the aforementioned seventh aspect or any possible implementation of the seventh aspect.

[0113] A thirteenth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, or enables the computer to execute the method in the aforementioned second aspect or any possible implementation of the second aspect, or the method in the aforementioned fifth aspect or any possible implementation of the fifth aspect, or the method in the aforementioned seventh aspect or any possible implementation of the seventh aspect.

[0114] A fourteenth aspect of the embodiments of the present application provides a readable storage medium, on which is stored a code stream encoded according to the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0115] A fifteenth aspect of an embodiment of the present application provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0116] A sixteenth aspect of an embodiment of the present application provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0117] A seventeenth aspect of an embodiment of the present application provides a system for distributing code streams, the system comprising at least one storage medium and a video streaming device; the at least one storage medium is used to store the code stream, which is encoded by an encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect; the video streaming device is used to respond to a request from a decoder so that the target code stream in the at least one storage medium can be sent to the decoder.

[0118] An eighteenth aspect of an embodiment of the present application provides a method for distributing a code stream, the method comprising: receiving a first request; selecting a target code stream from at least one storage medium in response to the first request; and sending the target code stream to a destination device; the at least one storage medium being used to store the code stream, where the code stream is encoded by an encoder according to the method of the aforementioned first aspect or any possible implementation of the first aspect.

[0119] A nineteenth aspect of an embodiment of the present application provides a system for processing a code stream, the system comprising an image source device, an encoder device, one or more storage media, and a destination device; the image source device is used to provide image data; the encoder device is used to obtain image data from the image source device through an interface, and encode the image data to obtain one or more code streams, where the code streams are encoded by the encoder according to the method in the aforementioned first aspect or any possible implementation of the first aspect; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device via a communication link or a communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0120] In a twentieth aspect, the present application provides a chip system, which includes a processor for supporting a device in implementing some or all of the functions involved in the above aspects, such as sending or processing data involved in the above methods; or information. In one possible design, the chip system also includes a memory for storing program instructions and data necessary for the device or the device. The chip system can be composed of a chip or can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] FIG1a is a schematic diagram of an application scenario of an embodiment of the present application;

[0122] FIG1b is a schematic diagram of an application scenario of an embodiment of the present application;

[0123] FIG2a is a schematic diagram of an application scenario of an embodiment of the present application;

[0124] FIG2 b is a schematic diagram of an application scenario of an embodiment of the present application;

[0125] FIG3 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0126] FIG4 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0127] FIG5 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0128] FIG6 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0129] FIG7 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0130] FIG8a is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0131] FIG8 b is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0132] FIG9 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0133] FIG10 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0134] FIG11 is a schematic diagram of an embodiment of an image processing method provided in an embodiment of the present application;

[0135] FIG12 is a schematic structural diagram of an image encoding device provided in an embodiment of the present application;

[0136] FIG13 is a schematic structural diagram of an image decoding device provided in an embodiment of the present application;

[0137] FIG14 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0138] FIG15 is a schematic diagram of a structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0139] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0140] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0141] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0142] The following describes the application scenarios to which the method provided in the embodiments of the present application is applicable.

[0143] The method provided in the embodiments of the present application can be applied to business scenarios such as public cloud, private cloud, video surveillance, live broadcast technology, storage, and transmission.

[0144] Figure 1a illustrates an example application scenario provided by an embodiment of the present application. This application scenario can also be understood as a codec application scenario. This application scenario includes an AI encoding unit, a storage / transmission unit, and an AI decoding unit. The method provided by an embodiment of the present application can be applied in the AI ​​encoding unit / AI decoding unit.

[0145] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application.

[0146] The storage / transmission unit is used to store (terminal, video surveillance, public cloud, private cloud, etc.) or transmit (public cloud, private cloud, live broadcast technology, etc.) the encoded input data.

[0147] The AI ​​decoding unit is used to decode the encoded input data according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0148] The application scenarios mentioned above have multiple situations, which are described in detail below:

[0149] The first one is the storage application scenario.

[0150] This storage application scenario includes: terminal photo album, video surveillance, etc. The schematic diagram of this scenario is shown in Figure 1b, which includes: AI encoding unit, file saving unit, file loading unit and AI decoding unit.

[0151] The AI ​​encoding unit is used to encode input data (such as images, videos, etc.) according to the encoding method provided in the embodiment of the present application to obtain a binary file.

[0152] The file saving unit is used to save the binary file output by the AI ​​encoding unit.

[0153] The loader file unit is used to load binary files.

[0154] The AI ​​decoding unit is used to decode the loaded binary file according to the decoding method provided in the embodiment of the present application to obtain reconstructed data (which can also be understood as data after reconstructing the encoding features of the input data).

[0155] The second type is the terminal-cloud interaction scenario.

[0156] The terminal-cloud interaction scenarios include: live broadcast scenarios, public cloud, etc. The schematic diagram of this scenario is shown in Figure 2a, which includes: encoding unit, decoding unit, AI encoding unit, AI decoding unit, encoding unit, and decoding unit.

[0157] Among them, the terminal side includes an encoding unit and a decoding unit, and the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0158] The terminal obtains input data and encodes the input data through the encoding unit to obtain encoding features. The terminal sends the encoding features to the cloud. After the cloud receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data. The reconstructed data is then encoded by the AI ​​encoding unit according to the encoding method provided in the embodiment of the present application to obtain a code stream / file. The cloud decodes the code stream / file through the AI ​​decoding unit according to the decoding method provided in the embodiment of the present application to obtain reconstructed data. The cloud encodes the reconstructed data through the encoding unit to obtain encoding features. And sends the encoding features to the terminal. After the terminal receives the encoding features, it decodes the encoding features through the decoding unit to obtain reconstructed data.

[0159] It is understandable that the encoding methods used by the encoding unit and the AI ​​encoding unit in Figure 2a may be inconsistent (or it can be understood that the operations of the decoding unit and the AI ​​encoding unit in the cloud can be called transcoding operations, and the operations of the AI ​​decoding unit and the encoding unit in the cloud can be called transcoding operations). For example, the cloud usually has more computing power resources than the terminal, so an algorithm (AI encoding) with high computing power requirements and higher compression efficiency can be used to re-compress the input data (transcoding) to obtain a higher compression multiple and save cloud storage space. The process of transcoding: decode first, then encode in another way.

[0160] It is understandable that the above scenarios are just examples, and there may be other scenarios in actual applications, which are not specifically limited here.

[0161] The following describes a chip hardware structure provided by an embodiment of the present application.

[0162] FIG2 b shows a chip hardware structure provided by an embodiment of the present invention, wherein the chip includes a neural network processor 40 .

[0163] The neural network processor 40 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), any processor suitable for large-scale XOR operation processing. Taking the NPU as an example: the neural network processor 40 is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.

[0164] In some implementations, arithmetic circuit 403 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 403 is a two-dimensional systolic array. Arithmetic circuit 403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 403 is a general-purpose matrix processor.

[0165] For example, assume there are input matrix A, weight matrix B, and output matrix C. Operation circuit 403 retrieves the corresponding data of matrix B from weight memory 402 and caches it on each PE in the operation circuit. The operation circuit retrieves the data of matrix A from input memory 401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 408.

[0166] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0167] In some implementations, the vector calculation unit 407 stores the processed output vector to the unified memory 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, for example, for use in a subsequent layer in a neural network.

[0168] The unified memory 406 is used to store input data and output data.

[0169] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller (DMAC) 405, the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.

[0170] The bus interface unit (BIU) 410 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

[0171] An instruction fetch buffer 409 connected to the controller 404 is used to store instructions used by the controller 404 .

[0172] The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

[0173] Generally, the unified memory 406, the input memory 401, the weight memory 402 and the instruction fetch memory 409 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0174] The method provided by the embodiment of the present application is described below. The method can be performed by an encoding device, or by a component of the encoding device (such as a processor, a chip, or a chip system, etc.). The encoding device can be a cloud device or a terminal device. Of course, the method can also be performed by a system consisting of a cloud device and a terminal device. Optionally, the method can be processed by a processor such as a CPU, NPU, GPU in the encoding device, and this application does not limit it.

[0175] Since the embodiments of the present application involve the application of a large number of neural networks, for ease of understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0176] (1) Neural Network

[0177] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0178] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0179] (2) Deep Neural Networks

[0180] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0181] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0182] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0183] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0184] (3) Convolutional Neural Networks

[0185] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0186] (4) Concat

[0187] The concatenation operation refers to concatenating multiple tensors in a certain dimension. All tensors must have the same shape (except for the concatenation dimension). This patent refers to concatenation on channels.

[0188] (5) Bit rate: In image compression tasks, the average coding length required to encode a unit pixel.

[0189] (6) Rate-distortion performance: An indicator used to measure the performance of a compression algorithm, which comprehensively considers both the bit rate and the distortion of the decoded image.

[0190] (7) YUV: A color format that can be converted to and from RGB format. It is commonly used in various media processing algorithms to reduce the bandwidth of color. There are three common YUV formats:

[0191] YUV4:4:4 (YUV444): Fully sample the chrominance component (UV) to obtain Y[1,H,W],UV[2,H,W];

[0192] YUV4:2:2 (YUV422): The chrominance component (UV) is horizontally sampled at a ratio of 2:1 and fully sampled vertically to obtain Y[1,H,W],UV[2,H,W / 2];

[0193] YUV4:2:0 (YUV420): 2:1 horizontal sampling of the chrominance component (UV) and 2:1 vertical sampling, Y[1,H,W],UV[2,H / 2,W / 2];

[0194] In addition to the default format YUV422, the present invention also supports another possible YUV422 format: the chrominance component (UV) is fully sampled horizontally and 2:1 vertically to obtain Y[1,H,W],UV[2,H / 2,W];

[0195] (8)Pixel Shuffle:

[0196] Pixel recombining is mainly used to recombine low-resolution feature maps into high-resolution feature maps through multi-channel recombining. In this paper, it is written as PS(s1,s2), where S1 and S2 refer to upsampling factors, which are usually integers greater than one. Input [C in ,h in ,w in ] Output [C out ,h out ,w out ]. where h out =s1·h in ;w out =s2·win ; C out =C in / (s1·s2); The formula is described as follows:

[0197] For c=0,…,C out -1,i=0,...,h out -1and j=0,…,w out -1

[0198] for i′=0..s1-1 and j′=0..s2-1

[0199] (9) Pixel Unshuffle:

[0200] The inverse operation of Pixelshuffle is to obtain a low-resolution feature map by recombining high-resolution feature maps between multiple channels. In the embodiment of this application, it is written as UPS(s1,s2), where s1 and s2 refer to the downsampling factors, which are usually integers greater than one. Input [C in ,h in ,w in ] Output [C out ,h out ,w out ].

[0201] in C out =C in *s1*s2, the formula is described as follows:

[0202] For c=0..C in -1,i=0..h out -1and j=0..w out -1

[0203] for i′=0..s1-1 and j′=0..s2-1 output[c*s1*s2+s2*i ′ +j′,i,j]=input[c,s1*i+i ′ ,s2*j+j′]

[0204] (10)s UV and s Y :

[0205] The factors of upsampling and downsampling: The encoder is the downsampling factor, and the decoder is the upsampling factor. UV=[s1,s2] indicates that the vertical subsampling factor for UV is s1, and the horizontal subsampling factor is s2, where both s1 and s2 are integers greater than or equal to 1. In JPEG AI, upsampling uses bilinear interpolation, and downsampling uses sample interval sampling. This application also supports other upsampling and downsampling methods.

[0206] (11) Color transformation (ColorTr):

[0207] The function of color transformation is to convert the input image into primary and secondary components. The color transformation used in JPEG AI to convert RGB images into YUV format is in the international standard ITU-R BT.709. The input is three components R[H,W], G[H,W], and B[H,W]. The input and output spatial shapes are H×W. The specific description is as follows: Y[i,j]=0.2126·R[i,j]+0.7152·G[i,j]+0.0722·B[i,j] U[i,j]=(-0.2126·R[i,j]-0.7152·G[i,j]+0.9278·B[i,j]) / 1.8556 V[i,j]=(0.7874·R[i,j]-0.7152·G[i,j]-0.0722·B[i,j]) / 1.5748

[0208] Where, the subscript [i,j], 0≤i <H,0≤j<W。

[0209] (12) Inverse color transformation (invColorTr): inverse transformation of ColorTr.

[0210] Image / video compression refers to a technology that exploits data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / video with the fewest bits possible, either losslessly or losslessly. This technology enables efficient transmission and storage of data information, playing a crucial role in the current media age, where data types and volumes are increasing. Data compression can be categorized as lossy or lossless. Lossy compression achieves a higher compression ratio at the expense of some image / video quality, while lossless compression does not cause any loss in image / video quality.

[0211] Currently, JPEG AI can encode images in different formats, such as YUV420, YUV444, and YUV422, by using different downsampling multiples. However, different formats have a significant impact on the computing power of the current JPEG AI network.

[0212] The embodiment of the present application is first described using a terminal device as an example of an application scenario.

[0213] As an example, the terminal device may be a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and the terminal device may compress the acquired image. As another example, the terminal device may be a virtual reality (VR) device. As another example, the embodiments of the present application may also be applied to intelligent monitoring, where a camera may be configured in the intelligent monitoring, and the intelligent monitoring may obtain images to be compressed through the camera. It should be understood that the embodiments of the present application may also be applied to other scenarios requiring image compression, and other application scenarios are not listed here one by one.

[0214] 3 , which is a schematic diagram of an embodiment of an image encoding method provided by an embodiment of the present application, as shown in FIG3 , an image encoding method provided by an embodiment of the present application includes:

[0215] 301. Obtain first information, where the first information includes information about a chrominance component in a first image format.

[0216] In the embodiment of the present application, the first information may include information about the chrominance component of the first image in the first image format.

[0217] Regarding the first image:

[0218] In one possible implementation, the first image may be an image to be encoded, wherein the first image may be an image captured by a camera of a terminal device, or the first image may be an image obtained from within the terminal device (for example, an image stored in an album of the terminal device, or a picture obtained by the terminal device from the cloud). It should be understood that the first image may be an image requiring image encoding, and this application does not impose any limitation on the source of the first image.

[0219] Information about chroma components:

[0220] In one possible implementation, the input image (such as the first image in the embodiment of the present application) can be preprocessed to convert the image into a primary component and a secondary component, wherein the primary component can be a luminance component and the secondary component can be a chrominance component. The primary component (i.e., the luminance component) can be, but is not limited to, a Y component, and the secondary component (i.e., the chrominance component) can be, but is not limited to, a UV component. For example, it can be divided into a Y component and a UV component through ColorTr, wherein YUV is a type of true-color color space (color space) to be compiled. Y'UV, YUV, YCbCr, YPbPr, etc. can all be called YUV, and there is overlap with each other. "Y" represents luminance (Luminance or Luma), which is a grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0221] In an embodiment of the present application, a first image may be acquired, and the chromaticity component and the luminance component may be obtained by preprocessing the first image, or the chromaticity component and the luminance component corresponding to the first image may be directly acquired without acquiring the first image.

[0222] In a possible implementation, information about the chrominance component in a certain image format and information about the luminance component in a certain image format can be obtained. The image format can be YUV444 format, YUV422 format, or YUV420 format.

[0223] For the YUV444 format, the secondary component of the first image can be directly used as the chrominance component information in the YUV444 format (for example, the size is [2, H, W], H is the height of the first image, and W is the width of the first image).

[0224] For the YUV422 format, the secondary component of the first image can be resampled (for example, downsampled) to obtain the chrominance component information in the YUV422 format (for example, the size is [2, H, W / 2], H is the height of the first image, and W is the width of the first image).

[0225] For the YUV420 format, the secondary component of the first image can be resampled (for example, downsampled) to obtain the chrominance component information in the YUV420 format (for example, the size is [2, H / 2, W / 2], H is the height of the first image, and W is the width of the first image).

[0226] In an embodiment of the present application, the first information is the information of the chrominance component of the first image format, the third information is the information of the chrominance component of the second image format, and the resolution of the first image format is greater than that of the second image format. For example, the first image format is the YUV444 format, and the second image format is the YUV422 format or the YUV420 format; or, the first image format is the YUV422 format, and the second image format is the YUV420 format.

[0227] 302. Perform size transformation on the first information to obtain second information.

[0228] In one possible implementation, for the information of the chrominance components of different (or at least two) image formats, the embodiment of the present application can resize it so that its sizes in the spatial dimension and the channel dimension are consistent. For example, for the first information in the first image format and the third information in the second image format, the sizes of the transformed first information (that is, the second information) and the transformed third information (that is, the fourth information) can be made consistent (that is, the channel size and the spatial size are consistent) through size transformation respectively.

[0229] In the existing implementation, the information of the chrominance component is directly input into the encoding network, and the encoding network encodes the information of the chrominance component. Since the sizes of information in different image formats are different, that is, the sizes of the information input into the encoding network are different, the computing power overhead and time overhead of the encoding network when processing information in different image formats are different, resulting in poor stability of information processing. In an embodiment of the present application, the information of the chrominance components of different image formats is converted into information of the same size through size transformation and input into the encoding network, so that the computing power overhead and time overhead of the encoding network when processing information in different (or at least two) image formats are consistent (or, close to consistent, at least smaller than the difference in overhead of processing original information in different formats), thereby improving the stability of information processing, which is better from the perspective of both machine and user experience.

[0230] Among them, the coding network can also be called an analysis transformation network or a signal encoder. The encoder is implemented based on a neural network and is used to encode the input information (or it can be called feature extraction) to obtain a feature map. The feature map can be used to obtain a bit stream through a quantization and entropy coding module. Since it is an encoder implemented based on a neural network, it can also be called an AI encoder.

[0231] In this embodiment of the present application, the size transformation performed on the first information can reduce the size of the information in the spatial dimension and increase the size of the information in the channel dimension. The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information.

[0232] Among them, the spatial dimension can further include height dimension and width dimension. The fact that the size of the second information in the spatial dimension is smaller than that of the first information can be understood as that the size of at least one dimension of the height dimension and the width dimension of the second information is smaller than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is larger than that of the first information).

[0233] For example, the height dimension of the second information in the spatial dimension is smaller than that of the first information, the width dimension of the second information in the spatial dimension is smaller than that of the first information, and the dimension of the second information in the channel dimension is larger than that of the first information.

[0234] For example, the height dimension of the second information in the spatial dimension is smaller than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is larger than that of the first information.

[0235] The above two illustrated cases can be understood as the second information having a smaller size in the spatial dimension than the first information.

[0236] The following describes in detail the size conversion of the first information in the embodiment of the present application:

[0237] In one possible implementation, the amount of information carried by chroma components in different formats varies due to their size (low-resolution image formats carry less information than high-resolution image formats due to downsampling). Information volume refers to the amount of effective information, or non-redundant information. For example, if the same information is copied multiple times, the amount of information increases, but the amount of non-redundant information remains the same. Non-redundant information is the amount of information that substantially affects the encoding or decoding results.

[0238] The resizing in the embodiments of the present application can make the size of the chrominance component information of different (or at least two) image formats consistent. This is to maintain the stability of computational overhead (including computing power overhead or time overhead). However, a better situation is that even if the chrominance component information is resized, the bitstream obtained after the entire encoding process is consistent with the bitstream obtained by encoding before the resizing. If they are inconsistent, the point of using different image formats is lost. In order to ensure that the result of the encoding process (that is, the bitstream) is consistent (or nearly consistent) with the existing solution, it is necessary to ensure that the effective information content of the information remains unchanged before and after the resizing.

[0239] Specifically, the amount of non-redundant information contained in the second information obtained by resizing the first information is consistent with the amount of non-redundant information contained in the first information. Information volume here refers to the amount of valid information, that is, the amount of non-redundant information. For example, if the same information is copied multiple times, although the amount of information increases, the amount of non-redundant information remains unchanged.

[0240] Resizing can include operations such as information splitting, information replication, and convolution. The following are introduced separately:

[0241] 1. Information Splitting

[0242] In a possible implementation, resizing the first information may include splitting information on at least one channel of the first information into information on multiple different channels. In other words, part of the first information may be placed on an expanded channel.

[0243] That is, in order to increase the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be expanded. The information of the expanded channels needs to come from the existing data in the first information. Then, a part of the information in the first information can be taken out and used as the information of the expanded channel. This part of the information is deleted on the original channel, which is equivalent to splitting the original information of the first information to obtain more channels, thereby reducing the size of the first information in the spatial dimension.

[0244] Since the characteristic of the chrominance component information is that the size of the information on each channel is consistent, the information of each channel can be taken out to obtain information of the same size (since the size of the taken out information is the same, the size of the information on each channel of the second information obtained after deleting this part of the information is consistent).

[0245] In a possible implementation, different data in the same channel in the first information may be placed in newly expanded different channels.

[0246] Exemplarily, channel 1 of the first information (including channel 1 and channel 2) may include element A1, element A2, and element A3.

[0247] Element A1, element A2 and element A3 may be deleted from channel 1, and element A1, element A2 and element A3 may be used as data of channel 3 (a channel not originally included in the first information).

[0248] Alternatively, element A1, element A2 and element A3 can be deleted from channel 1, and element A1 can be used as data for channel 3 (a channel not originally included in the first information), element A2 can be used as data for channel 4 (a channel not originally included in the first information), and element A3 can be used as data for channel 5 (a channel not originally included in the first information).

[0249] In a possible implementation, data on different channels in the first information can be taken out and combined and placed in different channels of the new expansion. In other words, data of the same channel of the new expansion can come from different channels.

[0250] Exemplarily, the first information (including channel 1 and channel 2) may include element A1, element A2, and element A3 on channel 1, and may include element B1, element B2, and element B3 on channel 2.

[0251] Elements A1, A2 and A3 can be deleted from channel 1, elements B1, B2 and B3 can be deleted from channel 2, and elements A1, A2 and A3 can be used as data for channel 3 (a channel not originally included in the first information), and elements B1, B2 and B3 can be used as data for channel 4 (a channel not originally included in the first information).

[0252] Alternatively, elements A1, A2, and A3 can be deleted from channel 1, elements B1, B2, and B3 can be deleted from channel 2, and elements A1 and B1 can be used as data for channel 3 (a channel not originally included in the first information), elements A2 and B2 can be used as data for channel 4 (a channel not originally included in the first information), and elements A3 and B3 can be used as data for channel 5 (a channel not originally included in the first information).

[0253] In a possible implementation, the size conversion method may include inverse pixel unshuffle (UPS), that is, UPS may be performed on the first information.

[0254] Refer to Figure 4, which is a schematic diagram of the processing results of a UPS. Figure 4 illustrates the example of expanding the information of each of the C channels in the first information to data of four channels, wherein the data of an information block with a size of 2*2 (height is 2, width is 2) is expanded to data of four channels, so that the number of channels is changed from C to 4C. If the spatial size of the first information is H*W, the number of channels of the second information obtained after the size transformation is H / 2*W / 2.

[0255] In one possible implementation, the first information is information about the chrominance components of YUV444, the channel dimension of the first information is 2, the height dimension of the spatial dimension of the first information is H, and the width dimension of the spatial dimension of the first information is W. The number of channel dimensions of the second information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2. The size conversion method in this diagram can be as shown in Figure 4 and is not further described here.

[0256] 2. Information Copying

[0257] In one possible implementation, in addition to increasing the number of channels by splitting information, information duplication can also be performed during resizing. This approach is particularly necessary when the first information is in a relatively low-resolution (at least not the highest resolution, such as YUV422 and YUV420) image format. This is because the amount of information (i.e., information size) of the chroma components in a low-resolution image format is smaller than that of the chroma components in a high-resolution image format. While ensuring that the amount of non-redundant information before and after resizing remains unchanged, if only splitting the information without increasing the amount of data (it should be understood that the amount of data and the amount of non-redundant information described above are not the same concept; the former refers to the total amount of information included in the information, including both redundant and non-redundant information), the size of the chroma components in the low-resolution (at least not the highest resolution) image format will inevitably be smaller than that of the chroma components in the high-resolution image format. In this case, the chroma components in the low-resolution (at least not the highest resolution) image format can be duplicated to increase the amount of data. The amount of non-redundant information in the duplicated information remains unchanged compared to the data before resizing, but the amount of data included is larger (i.e., the size is larger).

[0258] In one possible implementation, the information (all or part) of the first information may be copied and used as information on a new channel. Using the copied information as information on a new channel can be understood as: compared to before the copy, the number of channels will increase, and the data included in the newly added channel is composed of the copied data.

[0259] In one possible implementation, the first information can be copied and used as the information on a new channel, and the information on at least one channel can be split into information on multiple different channels, with no time sequence restrictions between the two operations. The number of operations is also not limited. For example, the information can be copied first, then split, and then copied again. As long as the amount of non-redundant information remains unchanged and the information can be transformed to the required size, the embodiments of the present application do not limit the type, number, combination, or order of the operations.

[0260] This application does not limit the order in which the copied items are placed on the channel.

[0261] For example, the information before copying is in the order of channels: channel 1, channel 2, channel 3, channel 4. Each of channel 1, channel 2, channel 3 and channel 4 can be copied once and used as the data of the new channel. If each copied channel is used as the channel adjacent to the channel before copying, the information of channel 1, channel 1, channel 2, channel 2, channel 3, channel 3, channel 4, and channel 4 can be obtained. Alternatively, if each copied channel is placed in the channel after the first information according to the sequential relationship between the channels before copying, the information of channel 1, channel 2, channel 3, channel 4, and channel 1, channel 2, channel 3, and channel 4 can be obtained. The channel order between the new channel information and the original channel information is not limited in this application.

[0262] For example, in one possible implementation, the first information is information of the chrominance component of YUV422, the size of the channel dimension of the first information is 2, the height dimension of the spatial dimension of the first information is H, and the width dimension of the spatial dimension of the first information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the second information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2.

[0263] In this illustration, the first information (size is (2, H, W / 2)) can be firstly split to obtain information of size (4, H / 2, W / 2), and then the information is copied to obtain the second information (8, H / 2, W / 2). Alternatively, in this illustration, the first information (size is (2, H, W / 2)) can be firstly copied to obtain information of size (4, H, W / 2), and then the information is split to obtain the second information (8, H / 2, W / 2).

[0264] In a possible implementation, the above-mentioned information duplication or information splitting operations can be replaced by convolution.

[0265] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant information is guaranteed to remain unchanged and the size can be converted to the required size. This application does not limit this.

[0266] It should be understood that when the first information is information of the chrominance component of YUV420, operations such as copying, splitting, and convolution may also be performed thereon, which will be introduced in subsequent embodiments.

[0267] 303. Encode the second information using a coding network or a part of a coding network.

[0268] It should be understood that step 302 itself can be an operation independent of the encoding network, or it can be implemented within the encoding network, for example, it can be performed by a network layer located near the input of the encoding network. The following are introduced respectively:

[0269] 1. Step 302 itself is an operation independent of the encoding network;

[0270] In a possible implementation, when step 302 itself is an operation independent of the encoding network, the second information can be encoded by the encoding network. Compared with the prior art, step 302 is a step added before the encoding network.

[0271] 2. Step 302 is performed by a network layer in the encoding network that is close to the input of the encoding network;

[0272] In one possible implementation, when step 302 is performed through a network layer in the encoding network close to the input of the encoding network, the network layer in the existing encoding network close to the input of the encoding network can be improved, for example, the operation value of the UPS layer or the convolution layer is changed, and the second information is to analyze the output of the network layer inside the conversion network.

[0273] In one possible implementation, when step 302 is performed by a network layer in the encoding network close to the input of the encoding network, the second information can be encoded by a part of the encoding network (that is, the network connected after the improved network layer).

[0274] It should be understood that there is also a certain amount of information interaction between the main component information (such as the Y component information) and the secondary component information (such as the UV component information), which can further improve the compression effect.

[0275] In one possible implementation, information exchange can be performed before resizing, or after resizing and before inputting into the encoding network. The following describes each of these two methods:

[0276] 1. Before size change;

[0277] In one possible implementation, the first information may further include information about a luminance component. That is, the first information needs to include at least information about chrominance components, but may also include information about luminance components. For example, the first information may include a fusion result (e.g., a splicing result) of the luminance component information and the chrominance component information.

[0278] For example, the first information may include a fusion result (eg, a splicing result) of information of a luminance component and information of a chrominance component in a YUV444 format.

[0279] For example, the first information may include a fusion result (eg, a splicing result) of information of a luminance component and information of a chrominance component in a YUV422 format.

[0280] 2. After resizing and before inputting into the encoding network;

[0281] In a possible implementation, the second information and the information of the brightness component may be fused (eg, concatenated), and the fusion result may be input into the encoding network.

[0282] In one possible implementation, fusion is achieved by splicing in the channel dimension. Since the size of the second information is transformed compared to the first information, in order to be able to fuse the second information with the information of the luminance component, the size of the luminance component information (that is, the fifth information in the embodiment of the present application) can also be transformed to obtain information that has the same size as the second information in at least the spatial dimension (that is, the sixth information in the embodiment of the present application), so that the second information and the sixth information can be spliced ​​in the channel dimension.

[0283] Specifically, in a possible implementation, fifth information can be obtained, where the fifth information is information of a brightness component; the fifth information can be resized to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with the second information, and then, the fusion result (e.g., a splicing result) of the second information and the fourth information can be encoded through a coding network.

[0284] 304. Obtain third information, where the third information includes information about a chrominance component in a second image format; the second image format is different from the first image format.

[0285] In the embodiment of the present application, the third information may include information about the chrominance component of the second image in the second image format, and the second image may be different from or the same as the first image.

[0286] The resolution of the second image format may be lower than that of the first image format. The introduction of step 304 may refer to the introduction of step 301 in the above embodiment, and the similarities are not repeated here.

[0287] 305. Perform size transformation on the third information to obtain fourth information; the sizes of the fourth information in the spatial dimension and the channel dimension are consistent with those of the second information;

[0288] In a possible implementation, the first image format is YUV444, the second image format is YUV422, the size of the fourth information in the spatial dimension is smaller than the third information, and the size of the fourth information in the channel dimension is larger than the third information.

[0289] For example, in one possible implementation, the third information is the information of the chrominance component of YUV422, the size of the channel dimension of the third information is 2, the height dimension of the spatial dimension of the third information is H, and the width dimension of the spatial dimension of the third information is W / 2, which can be described as (2, H, W / 2). The number of channel dimensions of the fourth information is 8, the height dimension of the spatial dimension of the fourth information is H / 2, and the width dimension of the spatial dimension of the fourth information is W / 2.

[0290] In this illustration, the third information (size is (2, H, W / 2)) can first be split into information with a size of (4, H / 2, W / 2), and then the information is copied to obtain the fourth information (8, H / 2, W / 2). Alternatively, in this illustration, the third information (size is (2, H, W / 2)) can first be copied into information with a size of (4, H, W / 2), and then the information is split to obtain the fourth information (8, H / 2, W / 2).

[0291] In a possible implementation, the first image format is YUV422, the second image format is YUV420, the size of the fourth information in the spatial dimension is equal to the third information, and the size of the fourth information in the channel dimension is larger than the third information.

[0292] For example, in one possible implementation, the third information is the information of the chrominance component of YUV420, the size of the channel dimension of the third information is 2, the height dimension of the spatial dimension of the third information is H / 2, and the width dimension of the spatial dimension of the third information is W / 2, which can be described as (2, H / 2, W / 2). The number of channel dimensions of the fourth information is 8, the height dimension of the spatial dimension of the fourth information is H / 2, and the width dimension of the spatial dimension of the fourth information is W / 2.

[0293] In this illustration, the third information (with a size of (2, H / 2, W / 2)) can be copied to obtain information with a size of (8, H / 2, W / 2).

[0294] In a possible implementation, the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0295] 306. Encode the fourth information using a coding network or a part of a coding network.

[0296] The introduction of step 306 can refer to the introduction of step 303 in the above embodiment, and the similarities are not repeated here.

[0297] It should be understood that the above steps 301 to 303 are the encoding processing process for the first information, and steps 304 to 306 are the encoding processing process for the third information. The embodiment of the present application does not limit the timing relationship between the encoding processing process for the first information and the encoding processing process for the second information. For example, steps 301 to 303 can be performed before steps 304 to 306, or, steps 301 to 303 can be performed after steps 304 to 306, or, steps 301 to 303 can be performed simultaneously with steps 304 to 306.

[0298] Next, in conjunction with Figure 5, an image encoding method in an embodiment of the present application is introduced. The method is introduced by taking the encoding process of three image formats, YUV444, YUV422 and YUV420, as examples. In Figure 5, the interaction between the chrominance component information and the brightness information occurs after the size transformation.

[0299] 5 , the method includes:

[0300] Input image in RGB format, the size is [3,H,W], 3 is the size of the channel dimension, H is the height size of the spatial dimension, and W is the width size of the spatial dimension.

[0301] The RGB image is converted into YUV444 format information through the color transformation ColorTr: YUV[3,H,W]. YUV[3,H,W] can be divided into the chrominance component information UV[2,H,W] and the brightness component information Y[1,H,W].

[0302] Information processing for the Y component:

[0303] Y[1,H,W] skips downsampling to generate x Y [1,H,W] and sent to the encoding network (Signal Encoder_Y) for encoding.

[0304] Regarding the processing of UV components:

[0305] During a process, one of three formats can be selected:

[0306] About YUV444 encoding: The input UV component UV[2,H,W] (the number of channels is 2, the height and width are H,W respectively) is reassembled by UPS(2,2) to generate x UV [8,H / 2,W / 2]. Generate auxiliary information after UPS(2,2) on Y[1,H,W]. (and x UA Keep the size of the spatial dimension consistent), and x UA After combining (concat) by channel dimension, they are sent to Signal Encoder_UV together.

[0307] The input UV component UV[2,H,W] may be the first information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the second information described in the above embodiment, Y[1, H, W] may be the fifth information described in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to obtain the UV component UV[2,H,W], which can be generated by UPS(2,2) inverse pixel reconstruction. UV [8,H / 2,W / 2] can refer to the introduction of step 302 in the above embodiment, and generate auxiliary information after Y[1,H,W] passes UPS(2,2) (and x UV The size of the spatial dimension remains the same) can refer to the description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The signals are combined (concat) according to the channel dimension and sent to Signal Encoder_UV together. The description of step 303 in the above embodiment can be referred to.

[0308] About YUV420 encoding: Downsample the input UV component UV[2,H,W] UV = [2,2] to generate 420 format UV [2,H / 2,W / 2]. After 4 times the channel dimension, repeat(4)(cin i is the channel before copying, i is the channel number, and the channel after copying is cout 4i+j =cin i , i=0~1, j=0~3) generate x UV[8,H / 2,W / 2](because x UV The unified shape [8, H / 2, W / 2] requires 4 times the number of pixels of the current shape [2, H / 2, W / 2]. Y[1, H, W] generates auxiliary information through UPS(2, 2) (and x UV Keep the shape consistent). and x UV After combining by channel dimension, they are sent to Signal Encoder together.

[0309] Wherein, UV[2,H / 2,W / 2] can be the third information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the fourth information described in the above embodiment, Y[1, H, W] may be the fifth information described in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to generate the 420 format UV[2,H / 2,W / 2]. Refer to the description of step 304 in the above embodiment, and generate x by 4 times the channel dimension through repeat (4). UV [8,H / 2,W / 2] can refer to the introduction of step 305 in the above embodiment, and generate auxiliary information after Y[1,H,W] passes UPS(2,2) (and x UV The size of the spatial dimension remains the same) can refer to the description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The signals are combined according to the channel dimension and sent to the Signal Encoder together, as described in step 306 in the above embodiment.

[0310] About the YUV422 encoding process: UV[2,H,W] downsampling UV = [1,2] to generate 422 format UV [2, H, W / 2]. After UPS (2, 1) to generate UV '[4, H / 2, W / 2], and then through the channel dimension twice the copy repeat (2) (cin i is the channel before copying, i is the channel number, and the channel after copying is cout 2i+j =cin i , i=0~3, j=0~1) generates x UV [8,H / 2,W / 2](because x UV The number of pixels required for the unified shape [8, H / 2, W / 2] is twice that of the current shape [4, H / 2, W / 2]. Y[1, H, W] generates auxiliary information after UPS(2, 2) (and xUV Keep the shape consistent). and x UV After combining by channel dimension, they are sent to Signal Encoder_UV.

[0311] The input UV component UV[2,H,W / 2] may be the first information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the second information described in the above embodiment, Y[1, H, W] may be the fifth information described in the above embodiment, and auxiliary information 2] can be the sixth information described in the above embodiment. To obtain the UV component UV[2,H,W / 2], refer to the description of step 301 in the above embodiment, generate UV′[4,H / 2,W / 2] through UPS(2,1), and then generate x through 2 times the channel dimension repeat(2). UV [8,H / 2,W / 2] can refer to the introduction of step 302 in the above embodiment, and generate auxiliary information after Y[1,H,W] passes UPS(2,2) (and x UV The size of the spatial dimension remains the same) can refer to the description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and x UV The signals are combined (concat) according to the channel dimension and sent to Signal Encoder_UV together. The description of step 303 in the above embodiment can be referred to.

[0312] Wherein, UV[2,H,W / 2] can be the third information introduced in the above embodiment, x UV [8, H / 2, W / 2] may be the fourth information described in the above embodiment, Y[1, H, W] may be the fifth information described in the above embodiment, and auxiliary information The sixth information described in the above embodiment can be used to generate the 422 format UV[2,H,W / 2]. The description of step 304 in the above embodiment can be referred to. UV′[4,H / 2,W / 2] is generated by UPS(2,1), and x is generated by repeat(4) with the channel dimension doubled. UV [8,H / 2,W / 2] can refer to the introduction of step 305 in the above embodiment, and generate auxiliary information after Y[1,H,W] passes UPS(2,2) (and x UV The size of the spatial dimension remains the same) can refer to the description of the step of performing size transformation on the fifth information to obtain the sixth information in the above embodiment. and xUV The signals are combined according to the channel dimension and sent to the Signal Encoder together, as described in step 306 in the above embodiment.

[0313] Next, in conjunction with Figure 6, an image encoding method in an embodiment of the present application is introduced. This method is introduced by taking the encoding process of the YUV444 image format as an example. Unlike Figure 5, in Figure 6, the interaction between the chrominance component information and the luminance information occurs before the size transformation.

[0314] 6 , the method includes:

[0315] The Y and UV information can be combined by channel before performing a UPS. For an input image in RGB[3,H,W] format, the RGB is converted to YUV[3,H,W] in 444 format using the ColorTr color transform. If the input is in YUV444 format, skip this step.

[0316] Processing of information about the Y component:

[0317] Y[1,H,W] skips downsampling to generate x Y [1,H,W], sent to Signal Encoder_Y.

[0318] Regarding the processing of UV component information:

[0319] Y[1,H,W] and UV[2,H,W] are combined into YUV[3,H,W] according to the channel dimension, and then UPS(2,2) is used to generate x UV [12,H / 2,W / 2], sent to Signal Encoder_UV.

[0320] In FIG6 , the information about the Y component and the UV component can be combined by channel first, and then the UPS step can refer to the introduction of the information interaction between the main component information and the slave component information before the size conversion in the above embodiment.

[0321] 7 , which is a schematic diagram of an embodiment of an image decoding method provided by an embodiment of the present application. As shown in FIG7 , an image decoding method provided by an embodiment of the present application includes:

[0322] 701. Obtain first information according to a first code stream.

[0323] The first code stream may be an encoding result obtained by encoding the first image.

[0324] The first information may be the output of a decoding network or the output of an intermediate layer of a decoding network.

[0325] On the decoding side, when the decoding network is decoding different image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), wherein the output of the decoding network includes information of the chrominance component, and the output of the intermediate layer of the decoding network is an intermediate quantity, and the information of the chrominance component or the intermediate quantity is the first information in the embodiment of the present application.

[0326] In one possible implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, it can be resized to obtain the information of the chrominance components that meets the size requirements specified by the corresponding image format.

[0327] In the existing implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension and the size of the spatial dimension are the same), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network are consistent (or nearly consistent) in the decoding process of different (or at least two) image formats, thereby improving the stability of information processing, which is better from the perspective of both the machine and user experience.

[0328] In addition, there is no binding relationship between input and output. YUV444 input can simultaneously generate 444, 420, or 422 output. Moreover, the complexity of the embodiment of the present application when performing 444 and 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI when performing 444 and 422 decoding.

[0329] Among them, the decoding network can also be called a signal decoder. The decoder is implemented based on a neural network and is used to decode (or reconstruct features) the input information (for example, the entropy decoding result) to obtain YUV information. Since it is a decoder implemented based on a neural network, it can also be called an AI decoder.

[0330] 702. Perform size conversion on the first information to obtain second information, where the second information includes information of chrominance components in the first image format.

[0331] In this embodiment of the present application, contrary to the embodiment corresponding to FIG3 , the resizing performed on the first information may increase the size of the information in the spatial dimension (optionally, for information in the YUV420 format, the size of the spatial dimension may remain unchanged) and reduce the size of the information in the channel dimension. The size of the second information in the spatial dimension is greater than or equal to the first information, and the size of the second information in the channel dimension is smaller than the first information.

[0332] Among them, the spatial dimension can further include a height dimension and a width dimension. The fact that the size of the second information in the spatial dimension is larger than that of the first information can be understood as that the size of at least one dimension of the height dimension and the width dimension of the second information is larger than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is smaller than that of the first information).

[0333] For example, the height dimension of the second information in the spatial dimension is larger than that of the first information, the width dimension of the second information in the spatial dimension is larger than that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0334] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0335] The above two illustrated situations can be understood as the second information having a larger size in the spatial dimension than the first information.

[0336] The following describes in detail the size conversion of the first information in the embodiment of the present application:

[0337] The size conversion in the embodiment of the present application can convert information of the same size into information of a size specified by the corresponding image format.

[0338] Resizing can include operations such as information merging, information selection, and convolution. The following are introduced separately:

[0339] 1. Information merging

[0340] In a possible implementation, performing size conversion on the first information may include: combining different data of multiple channels into the same channel.

[0341] That is, in order to reduce the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be reduced, and the information of the reduced channels needs to be merged into the existing channels in the first information. Then, the information of some channels in the first information can be taken out and merged with the information of other channels, thereby reducing the size of the first information in the channel dimension.

[0342] In a possible implementation, the size conversion method may include pixel shuffle (PS), that is, UPS may be performed on the first information.

[0343] Refer to Figure 8a, which is a schematic diagram of the processing results of a PS. Figure 8a takes the example of merging the information of each of the 4C channels in the first information into the data of one channel, wherein four different data belonging to the same position on the four channels are merged into one channel, so that the number of channels is changed from 4C to C. If the spatial size of the first information is H / 2*W / 2, the number of channels of the second information obtained after the size transformation is H*W.

[0344] In one possible implementation, the second information is information about the chrominance components of YUV444. The channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W. The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2. The size conversion method in this diagram can be as shown in Figure 8a, and will not be repeated here.

[0345] 2. Information Selection

[0346] The so-called selection is to select the information of some channels in the first information and then merge them into new data with the same spatial dimension but with a smaller channel dimension. This is equivalent to deleting the information of the unselected channels and retaining only the information of the selected channels.

[0347] In one possible implementation, in addition to increasing the number of channels through information merging, information selection can also be performed during resizing. This approach is particularly necessary when the second information is in a relatively low-resolution (at least not the highest resolution, such as YUV422 and YUV420) image format. This is because the amount of information (i.e., the size) of the chrominance components in a low-resolution image format is smaller than that in a high-resolution image format. If only information merging is performed without reducing the amount of data (it should be understood that the amount of data is not the same as the amount of non-redundant information described above; the former refers to the total amount of information included in the information, including both redundant and non-redundant information), the size of the chrominance components in the low-resolution (at least not the highest resolution) image format will inevitably be smaller than that in the high-resolution image format, thus failing to meet the size requirements specified by the image format. In this case, when reconstructing the chrominance components in the low-resolution (at least not the highest resolution) image format, a selection operation can be performed on the first information to reduce the amount of data. The amount of non-redundant information in the information after the selection operation may also be smaller than that in the data before the selection operation.

[0348] In one possible implementation, the first information can be selected and the data from multiple channels can be combined into a single channel, with no time sequence restrictions between the two operations. The number of operations is also not limited. For example, information selection can be performed first, followed by information merging, and then selection. As long as the desired size can be achieved, the present embodiment does not limit the type, number, combination, or order of operations.

[0349] For example, in one possible implementation, the second information is the information of the chrominance component of YUV422, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2.

[0350] In this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H / 2, W / 2) through information selection, and then the information is merged to obtain the second information (2, H, W / 2). Alternatively, in this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H, W / 2) through information selection, and then the second information (2, H, W / 2) is obtained.

[0351] In a possible implementation, the above-mentioned information selection or information merging operation may be replaced by convolution.

[0352] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant information is guaranteed to remain unchanged and the size can be converted to the required size. This application does not limit this.

[0353] It should be understood that step 702 itself can be an operation independent of the decoding network, or it can be implemented within the decoding network, for example, it can be performed by a network layer located close to the output of the decoding network (for example, the network layer before the last convolutional layer of the decoding network or the network layer before the last pixel reconstruction layer). The following are introduced respectively:

[0354] 1. Step 702 itself is an operation independent of the decoding network;

[0355] In a possible implementation, when step 702 itself is an operation independent of the decoding network, the decoding network may output the first information. Compared to the prior art, step 702 is a step added after the decoding network.

[0356] 2. Step 702 is performed by a layer in the decoding network close to the output of the decoding network;

[0357] In one possible implementation, when step 702 is performed through a network layer in the decoding network close to the output of the decoding network, the network layer in the existing decoding network close to the output of the decoding network can be improved, for example, the operation value of the UPS layer or the convolution layer is changed, and the first information is the output of the internal network layer of the synthetic conversion network.

[0358] 703. Reconstruct the first image according to the second information;

[0359] To convert to RGB, upsampling, combining the information of the chrominance component and the information of the luminance component, ICCI filtering, and generating an RGB image through the color conversion module invColorTr can be performed, but is not limited to.

[0360] 704. Obtain third information according to the second code stream, where the sizes of the third information in the spatial dimension and the channel dimension are consistent with those of the first information.

[0361] The second code stream may be an encoding result obtained by encoding the second image.

[0362] The third information may be the output of the decoding network or the output of an intermediate layer of the decoding network.

[0363] 705. Perform size conversion on the third information to obtain fourth information; the fourth information includes information of a chrominance component in a second image format, where the second image format is different from the first image format;

[0364] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0365] The relationship and introduction between the first image format and the second image format can be referred to the description in the above embodiment, which will not be repeated here.

[0366] In a possible implementation, a size of the fourth information in a spatial dimension is greater than or equal to that of the third information, and a size of the fourth information in a channel dimension is smaller than that of the third information.

[0367] In a possible implementation, the amount of non-redundant information included in the fourth information is consistent with the amount of non-redundant information included in the third information.

[0368] 706. Reconstruct the second image according to the fourth information.

[0369] It should be understood that steps 701 to 703 are the decoding process for the first code stream, and steps 704 to 706 are the decoding process for the second code stream. The embodiment of the present application does not limit the timing relationship between the decoding process for the first code stream and the decoding process for the second code stream. For example, steps 701 to 703 may be performed before steps 704 to 706, or steps 701 to 703 may be performed after steps 704 to 706, or steps 701 to 703 may be performed simultaneously with steps 704 to 706.

[0370] 8b , which is a schematic diagram of an embodiment of an image decoding method provided by an embodiment of the present application, as shown in FIG8b , an image decoding method provided by an embodiment of the present application includes:

[0371] 801. Obtain first information according to a first code stream.

[0372] The first information may be the output of a decoding network or the output of an intermediate layer of a decoding network.

[0373] On the decoding side, when the decoding network is decoding different image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), wherein the output of the decoding network includes information of the chrominance component, and the output of the intermediate layer of the decoding network is an intermediate quantity, and the information of the chrominance component or the intermediate quantity is the first information in the embodiment of the present application.

[0374] In one possible implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension is the same as the size of the spatial dimension), and in order to obtain the information of the chrominance components of different image formats, it can be resized to obtain the information of the chrominance components that meets the size requirements specified by the corresponding image format.

[0375] In the existing implementation, when the decoding network is decoding different (or at least two) image formats, the decoding network or the intermediate layer of the decoding network outputs information of different sizes, which makes the computing power and time overhead of the decoding network different in the decoding process of different (or at least two) image formats, resulting in poor stability of information processing. In an embodiment of the present application, the decoding network or the intermediate layer of the decoding network can output information of the same size (the size of the channel dimension and the size of the spatial dimension are the same), and in order to obtain the information of the chrominance components of different image formats, the computing power and time overhead of the decoding network are consistent (or nearly consistent) in the decoding process of different (or at least two) image formats, thereby improving the stability of information processing, which is better from the perspective of both the machine and user experience.

[0376] In addition, for the information obtained from the same code stream (such as the first information), chroma information in multiple image formats can be generated by resizing the first information. This eliminates the need for a binding relationship between input and output. For example, YUV444 input can simultaneously generate 444, 420, or 422 output. Furthermore, the complexity of the embodiment of the present application when performing 444 and 422 decoding is almost the same as that of 420, and is lower than the complexity of the original JPEG AI when performing 444 and 422 decoding.

[0377] Among them, the coding synthesis transformation network or signal decoder is implemented based on a neural network and is used to decode (or reconstruct features) the input information (for example, the entropy decoding result) to obtain YUV information. Since it is a decoder implemented based on a neural network, it can also be called an AI decoder.

[0378] 802. Perform a size transformation on the first information to obtain one or more of second information, third information, or fourth information; wherein the second information is information of the chrominance component of the first image format, the third information is information of the chrominance component of the second image format, and the fourth information is information of the chrominance component of the third image format, and the first image format, the second image format, and the third image format are different.

[0379] For example, when performing a resizing operation, at least one of the following operations may be selected to obtain chromaticity information in one or more image formats. In other words, in embodiments of the present application, for the same information, different resizing methods may be used to obtain chromaticity information in multiple image formats, thereby reconstructing the corresponding image formats.

[0380] Performing a first size transformation on the first information to obtain second information, where the second information includes information of a chrominance component in a first image format;

[0381] Performing a second size transformation on the first information to obtain second information, where the second information includes information of a chrominance component in the first image format;

[0382] Perform a third size transformation on the first information to obtain second information, where the second information includes information of the chrominance component in the first image format.

[0383] Taking the second information as an example, in the embodiment of the present application, contrary to the embodiment corresponding to FIG3 , the size conversion performed on the first information can increase the size of the information in the spatial dimension (optionally, for information in the YUV420 format, the size of the spatial dimension can remain unchanged) and reduce the size of the information in the channel dimension. The size of the second information in the spatial dimension is greater than or equal to the first information, and the size of the second information in the channel dimension is smaller than the first information.

[0384] Among them, the spatial dimension can further include a height dimension and a width dimension. The fact that the size of the second information in the spatial dimension is larger than that of the first information can be understood as that the size of at least one dimension of the height dimension and the width dimension of the second information is larger than that of the first information (of course, it cannot happen that the size of one dimension of the second information in the spatial dimension is smaller than that of the first information).

[0385] For example, the height dimension of the second information in the spatial dimension is larger than that of the first information, the width dimension of the second information in the spatial dimension is larger than that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0386] For example, the height dimension of the second information in the spatial dimension is greater than that of the first information, the width dimension of the second information in the spatial dimension is equal to that of the first information, and the dimension of the second information in the channel dimension is smaller than that of the first information.

[0387] The above two illustrated situations can be understood as the second information having a larger size in the spatial dimension than the first information.

[0388] The following describes in detail the size conversion of the first information in the embodiment of the present application:

[0389] The size conversion in the embodiment of the present application can convert information of the same size into information of a size specified by the corresponding image format.

[0390] Resizing can include operations such as information merging, information selection, and convolution. The following are introduced separately:

[0391] 1. Information merging

[0392] In a possible implementation, performing size conversion on the first information may include: combining different data of multiple channels into the same channel.

[0393] That is, in order to reduce the number of channels of the first information and ensure that the overall amount of information remains unchanged, the channels of the first information can be reduced, and the information of the reduced channels needs to be merged into the existing channels in the first information. Then, the information of some channels in the first information can be taken out and merged with the information of other channels, thereby reducing the size of the first information in the channel dimension.

[0394] In a possible implementation, the size conversion method may include pixel shuffle (PS), that is, UPS may be performed on the first information.

[0395] Refer to Figure 8a, which is a schematic diagram of the processing results of a PS. Figure 8a illustrates the example of merging the information of four channels in the first information into the data of one channel, wherein the four data at the same position on the four channels are merged into the data block of one channel, so that the number of channels is changed from four to one. If the number of channels of the first information is 4C, the number of channels of the second information obtained after the size transformation is C. If the spatial size of the first information is H / 2*W / 2, the number of channels of the second information obtained after the size transformation is H*W.

[0396] In one possible implementation, the second information is information about the chrominance components of YUV444. The channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W. The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2. The size conversion method in this diagram can be as shown in Figure 8a, and will not be repeated here.

[0397] 2. Information Selection

[0398] The so-called selection is to select the information of some channels in the first information and then merge them into new data with the same spatial dimension but with a smaller channel dimension. This is equivalent to deleting the information of the unselected channels and retaining only the information of the selected channels.

[0399] In one possible implementation, in addition to increasing the number of channels through information merging, information selection can also be performed during resizing. This approach is particularly necessary when the second information is in a relatively low-resolution (at least not the highest resolution, such as YUV422 and YUV420) image format. This is because the amount of information (i.e., the size) of the chrominance component information in a low-resolution image format is smaller than that of the chrominance component information in a high-resolution image format. If only information merging is performed without reducing the amount of data (it should be understood that the amount of data is not the same as the amount of non-redundant data described above; the former refers to the total amount of information included in the information, including both redundant and non-redundant data), the size of the chrominance component information in the low-resolution (at least not the highest resolution) image format will inevitably be smaller than that of the chrominance component information in the high-resolution image format, thus failing to meet the size requirements specified by the image format. In this case, when reconstructing the chrominance component information in the low-resolution (at least not the highest resolution) image format, a selection operation can be performed on the first information to reduce the amount of data. The amount of non-redundant data in the information after the selection operation may be smaller than that of the data before the selection operation.

[0400] In one possible implementation, the first information can be selected and the data from multiple channels can be combined into the same channel, with no time sequence restrictions between the two operations. The number of operations is also not limited. For example, information selection can be performed first, followed by information merging, and then selection. As long as the desired size can be achieved, the present embodiment does not limit the type, number, combination, or order of operations.

[0401] For example, in one possible implementation, the second information is the information of the chrominance component of YUV422, the size of the channel dimension of the second information is 2, the height dimension of the spatial dimension of the second information is H, and the width dimension of the spatial dimension of the second information is W / 2, which can be described as (2, H, W / 2), the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2.

[0402] In this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H / 2, W / 2) through information selection, and then the information is merged to obtain the second information (2, H, W / 2). Alternatively, in this illustration, the first information (size is (8, H / 2, W / 2)) can first be merged to obtain information of size (4, H, W / 2) through information selection, and then the second information (2, H, W / 2) is obtained.

[0403] In a possible implementation, the above-mentioned information selection or information merging operation may be replaced by convolution.

[0404] In a possible implementation, the size conversion can also be performed in other ways, as long as the amount of non-redundant data of the information remains unchanged and can be converted to the required size. This application does not limit this.

[0405] It should be understood that step 802 itself can be an operation independent of the decoding network, or it can be implemented within the decoding network, for example, it can be performed by a network layer located close to the output of the decoding network (for example, the network layer before the last convolutional layer of the decoding network or the network layer before the last pixel reconstruction layer). The following are introduced respectively:

[0406] 1. Step 802 itself is an operation independent of the decoding network;

[0407] In a possible implementation, when step 802 itself is an operation independent of the decoding network, the decoding network may output the first information. Compared to the prior art, step 802 is a step added after the decoding network.

[0408] 2. Step 802 is performed by a network layer in the decoding network close to the output of the decoding network;

[0409] In one possible implementation, when step 802 is performed through a network layer in the decoding network close to the output of the decoding network, the network layer in the existing decoding network close to the output of the decoding network can be improved, for example, the operation value of the UPS layer or the convolution layer is changed, and the first information is the output of the internal network layer of the synthetic conversion network.

[0410] 803. Perform one or more of the following steps: reconstruct the first image based on the second information; reconstruct the second image based on the third information; or reconstruct the third image based on the fourth information.

[0411] In a possible implementation, the image may be reconstructed according to at least one of the second information, the third information, or the fourth information.

[0412] To convert to RGB, upsampling, combining the information of the chrominance component and the information of the luminance component, ICCI filtering, and generating an RGB image through the color conversion module invColorTr can be performed, but is not limited to.

[0413] Next, in conjunction with Figure 9, an image decoding method in an embodiment of the present application is introduced. The method is introduced by taking the decoding process of three image formats, YUV444, YUV422 and YUV420, as examples. In Figure 9, the size transformation is implemented by other modules independent of the decoding network.

[0414] The decoding network (Signal Decoder) reconstructs Y and UV (i.e., the first information in the embodiment corresponding to FIG7 ). The reconstructed UV features have a uniform shape of [8, H / 2, W / 2]. Pixel shuffle (PS) then converts these uniform UV features into different formats (YUV444, YUV420, YUV422).

[0415] 9 , the method includes:

[0416] Get

[0417] Information processing for the Y component:

[0418] Output through Signal Decoder Skip upsampling and generate

[0419] Regarding the processing of UV components:

[0420] During a process, one of three formats can be selected:

[0421] About YUV420 decoding: Signal Decoder output (The reconstructed UV component has a shape of [8, H / 2, W / 2], where 8 represents the number of channels, and H / 2 and W / 2 represent the height and width respectively.) Select a quarter of the channels (because the number of UV pixels required by the 420 format is 1 / 4 of the 444 format), that is, 2 channels (for example, select the channel with a remainder of 0 when divided by 4 from channels 0 to 7), and generate (Shape is [2,H / 2,W / 2]). Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0422] in, It can be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be obtained by obtaining the Signal Decoder output Refer to the description of steps 701 and 704 in the above embodiment. One quarter of the channels are selected to generate The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0423] About YUV444 decoding: Signal Decoder output Generated by PS(2,2) At this time, the first step The merging can output images in YUV444 format.

[0424] in, It can be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be obtained by obtaining the Signal Decoder output Please refer to the description of steps 701 and 704 in the above embodiment. Generated by PS(2,2) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0425] About YUV422 decoding: Signal Decoder output from Select half of the channels (because the number of UV pixels required by the 422 format is 1 / 2 of the 444 format), that is, 4 channels (such as selecting the channels with a remainder of 0 when divided by 2 from channels 0 to 7) to generate And generated by PS(2,1) At this time, the first step The merging can output images in YUV422 format.

[0426] in, It can be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be obtained by obtaining the Signal Decoder output Refer to the description of steps 701 and 704 in the above embodiment. Select half of the channels and generate them through PS(2,1) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0427] Optional, by and Generate by channel combination Optionally, when the ICCI filter function is enabled in the JPEG AI configuration, and To generate Generated by the color conversion module invColorTr

[0428] Next, an image decoding method in an embodiment of the present application is introduced with reference to FIG10 . Different from FIG9 , in FIG10 , the size transformation itself is implemented by the pixel reconstruction PS layer of the decoding network.

[0429] In existing implementations, the last layer of the decoding network signal decoder is a pixel reassembly layer PS(4,4), which can be combined with the resizing PS in the embodiment of the present application. In this embodiment, the resizing operation and the last PS layer within the signal decoder can be combined into a single PS layer.

[0430] 10 , the method includes:

[0431] Information processing for the Y component:

[0432] Signal Decoder Output Skip upsampling and generate

[0433] Regarding the processing of UV components:

[0434] During a process, one of three formats can be selected:

[0435] Regarding YUV420 decoding: Part 1 of the Signal Decoder (all layers except the last pixel shuffle layer) outputs the feature L[128,H / 8,W / 8]. 32 channels are selected from L (channels with a remainder of 0 when divided by 4 from channels 0 to 127) to generate L′[32,H / 8,W / 8] (the number of UV pixels required for the 420 format is one-fourth of that for the 444 format), and then generated by PS(4,4). At this time, the first step Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0436] The feature L may be the first information described in the embodiment of FIG. 7 . The second information described in the embodiment of FIG. 7 can be used to obtain the output feature L of part 1 in the Signal Decoder. The output feature L can be obtained by referring to the description of steps 701 and 704 in the above embodiment. 32 channels are selected from L and generated by PS (4, 4). The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0437] About YUV444 decoding: Part 1 in Signal Decoder (all layers except the last pixel shuffle layer) outputs L[128,H / 8,W / 8]. L is generated by PS(8,8) At this time, the first step The merging can output images in YUV444 format.

[0438] The feature L may be the first information described in the embodiment of FIG. 7 . The second information described in the embodiment of FIG. 7 may be used. To obtain the Signal Decoder output L, reference may be made to the description of steps 701 and 704 in the embodiment above. Generated by PS(8,8) The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0439] Regarding YUV422 decoding: Part 1 of the Signal Decoder (all layers except the last pixel shuffle layer) outputs L[128,H / 8,W / 8]. 64 channels are selected from L (channels with a remainder of 0 when divided by 2 from channels 0 to 127) to generate L' (The number of UV pixels required for the 422 format is half of that for the 444 format) and is generated by PS(8,4) At this time, the first step Merge to output YUV422 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[1,2] times generated

[0440] The feature L may be the first information described in the embodiment of FIG. 7 . The second information described in the embodiment of FIG. 7 can be used to obtain the Signal Decoder output L. The description of steps 701 and 704 in the above embodiment can be referred to. 64 channels are selected from L and generated by PS (8, 4). The operations of converting to RGB may refer to the description of steps 703 and 706 in the above embodiment.

[0441] Next, an image decoding method in an embodiment of the present application is introduced with reference to FIG11 . Unlike FIG10 , in FIG11 , the size transformation itself is implemented by the convolutional layer of the decoding network.

[0442] This embodiment describes a possible modification to the decoder architecture. In this embodiment, the feature L reconstructed by the first part of the decoder no longer has a uniform number of channels. Instead, the signal decoder reconstructs L with different channels based on the required output formats. The first part of the decoder includes all layers except the last pixel shuffle (PS) layer and the last convolutional layer.

[0443] 9 , the method includes:

[0444] Information processing for the Y component:

[0445] Signal Decoder Output Skip upsampling and generate

[0446] Regarding the processing of UV components:

[0447] During a process, one of three formats can be selected:

[0448] Regarding YUV420 decoding: Part 1 in Signal Decoder outputs the feature L′[Ch,H / 8,W / 8], which is then passed through the convolutional layer CONV(3×3,32) with an output channel of 32 to generate L′[32,H / 8,W / 8]. Then, it is passed through PS(4,4) to generate At this time, the first step Merge to output YUV420 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[2,2] times generated

[0449] Wherein, L′ may be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be used to obtain the signal decoder output L′. The signal decoder output L′ can refer to the introduction of step 701 in the embodiment above, and L′ is generated through the convolution layer CONV(3×3,32) with an output channel of 32. ′ , and then generated by PS(4,4) The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0450] Regarding YUV444 decoding: Part 1 in Signal Decoder outputs feature L′[Ch,H / 8,W / 8], which is passed through a convolutional layer CONV(3×3,128) with an output channel of 128 to generate L′[128,H / 8,W / 8]. L is generated by PS(8,8) At this time, the first step The merging can output images in YUV444 format.

[0451] Wherein, L′ may be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be used to obtain the Signal Decoder output L′, which can refer to the introduction of step 701 in the above embodiment, and is generated by the convolution layer CONV with an output channel of 128 and PS (8, 8). The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0452] Regarding YUV422 decoding: Part 1 in Signal Decoder outputs feature L′[Ch,H / 8,W / 8], which is then passed through a convolutional layer CONV(3×3,64) with an output channel of 64 to generate L′[64,H / 8,W / 8]. This is then passed through PS(8,4) to generate At this time, the first step Merge to output YUV422 format images. Further, if you want to convert to RGB, you also need to Upsampling UV =[1,2] times generated

[0453] Wherein, L′ may be the first information introduced in the embodiment of FIG. 7 above, The second information described in the embodiment of FIG. 7 can be used to obtain the Signal Decoder output L′, which can be generated by passing through the convolution layer CONV with an output channel of 64 and PS (8, 4) The operations of converting to RGB may refer to the description of steps 703 and 705 in the above embodiment.

[0454] The encoding and decoding method in the embodiment of the present application is described above. The device 1200 in the embodiment of the present application is described below. Please refer to Figure 12. An embodiment of the encoding device in the embodiment of the present application includes:

[0455] An acquisition module 1201 is configured to acquire first information, the first information including information about chrominance components in a first image format; and acquire third information, the third information including information about chrominance components in a second image format, where the second image format is different from the first image format.

[0456] The introduction of the acquisition module 1201 can refer to the introduction of steps 301 and 304 in the above embodiment, and will not be repeated here.

[0457] Processing module 1202 is used to resize the first information to obtain second information; encode the second information through a coding network or part of a coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through a coding network.

[0458] The introduction of the processing module 1202 can refer to the introduction of steps 302, 303, 305 and 306 in the above embodiment, which will not be repeated here.

[0459] In one possible implementation, the size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

[0460] In a possible implementation, the amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

[0461] In a possible implementation, the size transformation includes at least one of the following:

[0462] Copy the information and use the copied information as the information on the new channel;

[0463] Splitting the information on at least one channel into information on a plurality of different channels; or,

[0464] Perform convolution operation.

[0465] In a possible implementation, the operation of splitting the information on at least one channel into information on multiple different channels is achieved through an inverse pixel reconstruction layer.

[0466] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or,

[0467] The first image format is YUV422 format, and the second image format is YUV420 format.

[0468] In one possible implementation, the number of channel dimensions of the second information and the fourth information is 8, the height dimension of the spatial dimension of the second information is H / 2, and the width dimension of the spatial dimension of the second information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0469] In a possible implementation, the obtaining module 1201 is further configured to:

[0470] Acquire fifth information, where the fifth information is information of a brightness component Y;

[0471] The processing module is further configured to perform size transformation on the fifth information to obtain sixth information; the size of the sixth information in a spatial dimension is consistent with that of the second information;

[0472] The processing module is specifically configured to encode a fusion result of the second information and the fourth information through a coding network.

[0473] In a possible implementation, the first information or the third information further includes information of a brightness component.

[0474] Referring to FIG. 13 , an embodiment of a decoding device 1300 in an embodiment of the present application includes:

[0475] An acquisition module 1301 is configured to acquire first information based on a first bitstream; and acquire third information based on a second bitstream, wherein the third information has the same dimensions as the first information in terms of spatial and channel dimensions.

[0476] The introduction of the acquisition module 1301 can refer to the introduction of steps 701 and 704 in the above embodiment, and will not be repeated here.

[0477] Processing module 1302 is used to resize the first information to obtain second information, where the second information includes information about the chrominance component of the first image format; reconstruct the first image based on the second information; obtain third information based on the second code stream, where the sizes of the third information in the spatial dimension and the channel dimension are consistent with those of the first information; resize the third information to obtain fourth information; the fourth information includes information about the chrominance component of the second image format, where the second image format is different from the first image format; and reconstruct the second image based on the fourth information.

[0478] For the introduction of the processing module 1302 , reference may be made to the introduction of steps 702 , 703 , 705 and 706 in the above embodiment, which will not be repeated here.

[0479] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the fourth information in the spatial dimension is larger than or equal to the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

[0480] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is less than or equal to the amount of non-redundant information contained in the third information.

[0481] In a possible implementation, the size conversion includes at least one of the following methods:

[0482] Delete information of some channels;

[0483] Combine different data from multiple channels onto the same channel; or,

[0484] Perform convolution operation.

[0485] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0486] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or,

[0487] The first image format is YUV422 format, and the second image format is YUV420 format.

[0488] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0489] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0490] In addition, an embodiment of the present application further provides an image decoding device, the device comprising:

[0491] An acquisition module, configured to acquire first information according to the first code stream;

[0492] A processing module is used to resize the first information to obtain one or more of second information, third information, or fourth information; wherein the second information is information of the chrominance component of the first image format, the third information is information of the chrominance component of the second image format, and the fourth information is information of the chrominance component of the third image format, and the first image format, the second image format, and the third image format are different.

[0493] In one possible implementation, the size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information, the size of the third information in the spatial dimension is larger than or equal to the first information, and the size of the third information in the channel dimension is smaller than that of the first information.

[0494] In a possible implementation, the amount of non-redundant information contained in the second information is less than or equal to the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the third information is less than or equal to the amount of non-redundant information contained in the first information.

[0495] In a possible implementation, the processing module is further configured to:

[0496] The first image is reconstructed based on at least one of the second information, the third information, or the fourth information.

[0497] In a possible implementation, the size transformation includes at least one of the following methods: deleting information of some channels; combining different data of multiple channels into the same channel; and performing a convolution operation.

[0498] In a possible implementation, the operation of combining different data of multiple channels into the same channel is implemented through a pixel reassembly layer.

[0499] In a possible implementation, the first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, the first image format is YUV422 format, and the second image format is YUV420 format.

[0500] In a possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0501] In a possible implementation, the intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

[0502] Next, a device provided in an embodiment of the present application is introduced. The device can be an encoding device or a decoding device. Please refer to Figure 14. Figure 14 is a structural diagram of a device provided in an embodiment of the present application. The device 1400 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Among them, the device 1400 can be deployed with the device described in the corresponding embodiment of Figure 12 or Figure 13. Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403 and a memory 1404 (wherein the number of processors 1403 in the device 1400 can be one or more, and Figure 14 takes one processor as an example), wherein the processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403 and the memory 1404 can be connected via a bus or other means.

[0503] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0504] Processor 1403 controls the operation of the device. In specific applications, the various components of the device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0505] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1403. Processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1403. The above processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1404, and processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0506] Receiver 1401 can be used to receive input digital or character information and generate signal input related to device settings and function control. Transmitter 1402 can be used to output digital or character information through the first interface. Transmitter 1402 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1402 can also include a display device such as a display screen.

[0507] The embodiment of the present application also provides a device, which can be an encoding device or a decoding device. Please refer to Figure 15, which is a structural diagram of a device provided by an embodiment of the present application. The device 1500 can be deployed with the device described in the corresponding embodiment of Figure 12 or Figure 13. Specifically, the device 1500 is implemented by one or more servers. The device 1500 may have relatively large differences due to different configurations or performances. It can include one or more central processing units (CPU) 1522 (for example, one or more processors) and memory 1532, one or more storage media 1530 (for example, one or more mass storage devices) for storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be temporary storage or persistent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device. Furthermore, the central processing unit 1522 may be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the device 1500 .

[0508] The device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0509] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation method of the above-mentioned image processing device.

[0510] An embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned image processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data to the at least one processor. In one possible design, the chip system may also include a memory for storing the necessary program instructions and data for the terminal device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0511] An embodiment of the present application further provides a readable storage medium, on which a code stream encoded according to the method in the embodiments shown in FIG. 1 a to FIG. 6 is stored.

[0512] An embodiment of the present application further provides a device for storing a code stream, characterized in that it includes at least one storage medium and a communication interface; the communication interface is used to receive or send the code stream; the at least one storage medium is used to store the code stream; the code stream is encoded by an encoder according to the method in the embodiments shown in Figures 1a to 6 above.

[0513] An embodiment of the present application further provides a method for storing a code stream, the method comprising: receiving a code stream through a communication interface; and storing the code stream in one or more storage media, wherein the code stream is encoded by an encoder according to the method in the embodiments shown in FIG. 1 a to FIG. 6 .

[0514] An embodiment of the present application further provides a system for distributing a bitstream, the system comprising at least one storage medium and a video streaming device; the at least one storage medium is used to store the bitstream, where the bitstream is encoded by the encoder according to the method in the embodiments shown in Figures 1a to 6 above; and the video streaming device is used to respond to a request from a decoder so that the target bitstream in the at least one storage medium is sent to the decoder.

[0515] An embodiment of the present application further provides a method for distributing a bitstream, the method comprising: receiving a first request; selecting a target bitstream from at least one storage medium in response to the first request; and sending the target bitstream to a destination device; the at least one storage medium being used to store the bitstream, where the bitstream is encoded by an encoder according to the method in the embodiments shown in FIG. 1A to FIG. 6 .

[0516] An embodiment of the present application also provides a system for processing a code stream, the system including an image source device, an encoder device, one or more storage media, and a destination device; the image source device is used to provide image data; the encoder device is used to obtain image data from the image source device through an interface and encode the image data to obtain one or more code streams, where the code streams are encoded by the encoder according to the method in the embodiments shown in Figures 1a to 13 above; the encoder device is used to store the one or more code streams in one or more storage media; or the encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; the encoder device is used to transmit the transmission code stream to the destination device via a communication link or communication network; the destination device is used to decapsulate the transmission code stream to obtain one or more code streams; and the destination device is used to decode the one or more code streams to obtain decoded data.

[0517] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0518] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0519] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0520] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0521] Through the description of the above embodiments, it is clear to those skilled in the art that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be implemented by means of dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, device, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0522] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0523] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, device or data center to another website, computer, device or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a device integrated with one or more available media, a data center, etc. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. An image coding method, characterized in that: The method comprises: Acquire first information, where the first information includes information of a chrominance component of a first image format; Performing a size transformation on the first information to obtain second information; encoding the second information by means of a coding network or a portion of a coding network; Acquire third information, where the third information includes information of a chrominance component of a second image format; the second image format is different from the first image format; Performing a size transformation on the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with that of the second information; The fourth information is encoded by the encoding network or a part of the encoding network.

2. The method according to claim 1, characterized in that The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to that of the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

3. The method according to claim 1 or 2, characterized in that: The amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

4. The method according to any one of claims 1 to 3, characterized in that: The size transformation includes at least one of the following: Copy the information and use the copied information as the information on the new channel; Splitting information on at least one channel into information on a plurality of different channels; or, Perform convolution operation.

5. The method according to claim 4, characterized in that The operation of splitting the information on at least one channel into information on a plurality of different channels is achieved through an inverse pixel recombination layer.

6. The method according to any one of claims 1 to 5, characterized in that: The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

7. The method according to any one of claims 1 to 6, characterized in that: The number of channel dimensions of the second information and the fourth information is 8, the size of the spatial dimension of the second information includes height and width, the height of the second information is H / 2, and the width of the second information is W / 2, wherein H is the height of the first image, and W is the width of the first image.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Acquire fifth information, where the fifth information is information of a brightness component Y; Performing a size transformation on the fifth information to obtain sixth information; the size of the sixth information in a spatial dimension is consistent with that of the second information; The encoding of the second information by means of a coding network or a portion of a coding network comprises: The fusion result of the second information and the fourth information is encoded through a coding network.

9. The method according to any one of claims 1 to 8, characterized in that: The first information or the third information further includes information of a brightness component.

10. An image decoding method, characterized in that: The method comprises: Acquire first information according to the first code stream; Performing a size conversion on the first information to obtain second information, where the second information includes information of a chrominance component in a first image format; reconstructing the first image according to the second information; Acquire third information according to the second bitstream, wherein the sizes of the third information in the spatial dimension and the channel dimension are consistent with those of the first information; Performing a size conversion on the third information to obtain fourth information; the fourth information includes information of a chrominance component in a second image format, and the second image format is different from the first image format; The second image is reconstructed according to the fourth information.

11. The method according to claim 10, characterized in that The first information or the third information is an output of a decoding network or an output of an intermediate layer of a decoding network.

12. The method according to claim 10 or 11, characterized in that: The size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information. The size of the fourth information in the spatial dimension is larger than or equal to that of the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

13. The method according to any one of claims 10 to 12, characterized in that: The size transformation includes at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

14. The method according to claim 13, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

15. The method according to any one of claims 10 to 14, characterized in that: The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

16. The method according to any one of claims 10 to 15, characterized in that: The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

17. The method according to any one of claims 10 to 16, characterized in that: The intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

18. An image encoding device, characterized in that: The device comprises: an acquisition module, configured to acquire first information, wherein the first information includes information of a chrominance component in a first image format; and acquire third information, wherein the third information includes information of a chrominance component in a second image format; wherein the second image format is different from the first image format; A processing module is used to resize the first information to obtain second information; encode the second information through a coding network or a part of the coding network; resize the third information to obtain fourth information; the size of the fourth information in the spatial dimension and the channel dimension is consistent with the second information; and encode the fourth information through the coding network.

19. The device according to claim 18, characterized in that The size of the second information in the spatial dimension is smaller than that of the first information, and the size of the second information in the channel dimension is larger than that of the first information; the size of the fourth information in the spatial dimension is smaller than or equal to that of the third information, and the size of the fourth information in the channel dimension is larger than that of the third information.

20. The device according to claim 18 or 19, characterized in that The amount of non-redundant information contained in the second information is consistent with the amount of non-redundant information contained in the first information; the amount of non-redundant information contained in the fourth information is consistent with the amount of non-redundant information contained in the third information.

21. The device according to any one of claims 18 to 20, characterized in that The size transformation includes at least one of the following: Copy the information and use the copied information as the information on the new channel; Splitting information on at least one channel into information on a plurality of different channels; or, Perform convolution operation.

22. The device according to claim 21, characterized in that The operation of splitting the information on at least one channel into information on a plurality of different channels is achieved through an inverse pixel recombination layer.

23. The device according to any one of claims 18 to 22, characterized in that The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

24. The device according to any one of claims 18 to 23, characterized in that The number of channel dimensions of the second information and the fourth information is 8, the size of the spatial dimension of the second information includes height and width, the height of the second information is H / 2, and the width of the second information is W / 2, wherein H is the height of the first image, and W is the width of the first image.

25. The device according to any one of claims 18 to 24, characterized in that The acquisition module is further used for: Acquire fifth information, where the fifth information is information of a brightness component Y; The processing module is further used to perform a size transformation on the fifth information to obtain sixth information; the size of the sixth information in the spatial dimension is consistent with that of the second information; The processing module is specifically used to encode the fusion result of the second information and the fourth information through a coding network.

26. The device according to any one of claims 18 to 25, characterized in that The first information or the third information further includes information of a brightness component.

27. An image decoding device, characterized in that: The device comprises: An acquisition module, configured to acquire first information according to the first bit stream; and acquire third information according to the second bit stream, wherein the size of the third information in the spatial dimension and the channel dimension is consistent with that of the first information; A processing module is used to resize the first information to obtain second information, wherein the second information includes information of chromaticity components in a first image format; reconstruct the first image based on the second information; resize the third information to obtain fourth information; the fourth information includes information of chromaticity components in a second image format, and the second image format is different from the first image format; and reconstruct the second image based on the fourth information.

28. The device according to claim 27, characterized in that The first information or the third information is an output of a decoding network or an output of an intermediate layer of a decoding network.

29. The device according to claim 27 or 28, characterized in that The size of the second information in the spatial dimension is larger than that of the first information, and the size of the second information in the channel dimension is smaller than that of the first information. The size of the fourth information in the spatial dimension is larger than or equal to that of the third information, and the size of the fourth information in the channel dimension is smaller than that of the third information.

30. The device according to any one of claims 27 to 29, characterized in that The size transformation includes at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

31. The device according to claim 30, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

32. The device according to any one of claims 27 to 31, characterized in that The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format; or, The first image format is YUV422 format, and the second image format is YUV420 format.

33. The device according to any one of claims 27 to 32, characterized in that The number of channel dimensions of the first information is 8, the height dimension of the spatial dimension of the first information is H / 2, and the width dimension of the spatial dimension of the first information is W / 2, wherein H is the height dimension of the first image, and W is the width dimension of the first image.

34. The device according to any one of claims 27 to 33, characterized in that The intermediate layer is a network layer before the last convolutional layer of the decoding network or a network layer before the last pixel reconstruction layer.

35. An image decoding method, characterized in that: The method comprises: Acquire first information according to the first code stream; resizing the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of chrominance components in a first image format, the third information is information of chrominance components in a second image format, the fourth information is information of chrominance components in a third image format, and the first image format, the second image format and the third image format are different; and Perform one or more of the following steps: reconstructing the first image according to the second information; reconstructing the second image according to the third information; or, The third image is reconstructed according to the fourth information.

36. The method according to claim 35, characterized in that The first information is the output of the decoding network or the output of an intermediate layer of the decoding network.

37. The method according to claim 35 or 36, characterized in that The sizes of the second information, the third information and the fourth information in the spatial dimension are greater than or equal to the first information, and the sizes of the second information, the third information and the fourth information in the channel dimension are smaller than the first information.

38. The method according to any one of claims 35 to 37, characterized in that: The amount of non-redundant information included in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information included in the first information.

39. The method according to any one of claims 35 to 38, characterized in that The first size transformation and the second size transformation include at least one of the following methods: Delete information of some channels; Combine different data from multiple channels onto the same channel; or, Perform convolution operation.

40. The method according to claim 39, characterized in that The operation of combining different data of multiple channels into the same channel is achieved through a pixel reorganization layer.

41. An image decoding device, characterized in that: The device comprises: An acquisition module, used for acquiring first information according to the first code stream; a processing module, configured to perform a size conversion on the first information to obtain one or more of second information, third information or fourth information; wherein the second information is information of a chroma component in a first image format, the third information is information of a chroma component in a second image format, the fourth information is information of a chroma component in a third image format, and the first image format, the second image format and the third image format are different; and performing one or more of the following steps: reconstructing the first image according to the second information; reconstructing the second image according to the third information; or, The third image is reconstructed according to the fourth information.

42. A decoding device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the decoding device performs the method as described in any one of claims 10 to 17, and 35 to 40.

43. A coding device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the encoding device executes the method as described in any one of claims 1 to 9.

44. A coding and decoding system, characterized in that: The encoding and decoding system includes the decoding device as claimed in claim 42 and / or the encoding device as claimed in claim 43.

45. A computer-readable storage medium, characterized in that The medium stores instructions, and when the instructions are executed by a computer, the method of any one of claims 1 to 17, and 35 to 40 is implemented.

46. ​​A computer program product, characterized in that The invention comprises instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 17, and 35 to 40.

47. A readable storage medium, characterized in that The readable medium stores a code stream encoded according to any one of the encoding methods of claims 1 to 9.

48. A device for storing a code stream, characterized in that: comprising at least one storage medium and a communication interface; The communication interface is used to receive or send code streams; The at least one storage medium is used to store the code stream; The code stream is obtained by encoding by an encoder according to any encoding method of claims 1 to 9.

49. A method for storing a code stream, characterized in that: include: Receiving a code stream via a communication interface; The code stream is stored in one or more storage media, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9.

50. A system for distributing code streams, characterized in that: including at least one storage medium and a video streaming device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9; The video stream device is used to respond to a request of a decoder so that the code stream in the at least one storage medium can be sent to the decoder.

51. A method for distributing code streams, characterized in that: include: receiving a first request; In response to the first request, selecting a code stream from at least one storage medium; Sending the code stream to a destination device; The at least one storage medium is used to store a code stream, and the code stream is an encoder Obtained by encoding according to any one of the encoding methods of claims 1 to 9.

52. A system for processing a code stream, characterized in that: It includes an image source device, an encoder device, one or more storage media and a destination device; The image source device is used to provide image data; The encoder device is used to obtain the image data of the image source device through an interface, and encode the image data to obtain one or more code streams, wherein the code streams are encoded by the encoder according to any one of the encoding methods of claims 1 to 9; The encoder device is used to store the one or more code streams in one or more storage media; or The encoder device is used to encapsulate the one or more code streams to obtain a transmission code stream; The encoder device is used to transmit the transmission code stream to the destination device through a communication link or a communication network; The destination device is used to decapsulate the transport code stream to obtain the one or more code streams; The destination device is used to decode the one or more code streams to obtain decoded data.

Images