Image encoding method, image encoding method, and related devices

Abstract

This application discloses an image encoding method, an image decoding method, and related devices relating to the encoding and decoding field. The image decoding method includes the steps of: obtaining first information based on a first bitstream; performing a size transformation on the first information to obtain second information, the second information including information about the color difference components of a first image format; reconstructing a first image based on the second information; obtaining third information based on a second bitstream, the size of the third information in spatial and channel dimensions matching the size of the first information; performing a size transformation on the third information to obtain fourth information, the fourth information including information about the color difference components of a second image format; and reconstructing a second image based on the fourth information. In this application, size transformations may be performed on information of the same size in the decoding process to obtain information about the color difference components of different image formats, and as a result, the computational power and time overhead are consistent in the process of the decoding network decoding different image formats, thus improving the stability of information processing.

Description

TECHNICAL FIELD

[0001] This application claims the priority of Chinese Patent Application No. 202311361171.2 entitled "IMAGE ENCODING METHOD, IMAGE DECODING METHOD, AND RELATED DEVICE" filed with the China National Intellectual Property Administration on October 18, 2023, the entire content of which is incorporated herein by reference.

[0002] The present application relates to the field of artificial intelligence, and in particular, to an image encoding method, an image decoding method, and a related device. BACKGROUND ART

[0003] Image / video compression is a technology that uses data features such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / videos with as few bits as possible in a lossy or lossless manner. This technology can realize efficient transmission and storage of data information, and plays an important role in the current media era where the types of transmitted data are increasingly diversified and the data volume is increasingly large. Data compression is classified into lossy compression and lossless compression. Lossy compression achieves a high compression ratio at the expense of a certain degree of degradation in image / video quality, while lossless compression does not cause loss of image / video quality.

[0004] Currently, JPEG AI can encode images in different formats such as YUV420, YUV444, and YUV422 by using different downsampling multiples. However, different formats greatly affect the computing power of the current JPEG AI network. SUMMARY OF THE INVENTION

[0005] The present application provides an image encoding method and an image decoding method, such that when an encoding network or a decoding network processes information of different (or at least two) image formats, the computational capability and time overhead are consistent (or substantially consistent, and the overhead difference is at least smaller than the overhead difference for processing original information of different formats), thereby improving the stability of information processing. [Means for Solving the Problems]

[0006] According to a first aspect, the present application provides an image encoding method. The method comprises the steps of: acquiring first information, wherein the first information comprises information related to a color difference component of a first image format; performing size conversion on the first information to acquire second information; encoding the second information using an encoding network or a part of the encoding network; acquiring third information, wherein the third information comprises information related to a color difference component of a second image format, and the second image format is different from the first image format; performing size conversion on the third information to acquire fourth information, wherein the size of the fourth information in the spatial dimension and the channel dimension matches the size of the second information; and encoding the fourth information using the encoding network or a part of the encoding network.

[0007] In existing implementations, information regarding color difference components is directly input to the coding network, which then encodes the information. Because the size of information differs across different image formats, i.e., the size of information input to the coding network differs, the computational power and time overhead differ when the coding network processes information from different image formats. As a result, the stability of information processing is insufficient. In this embodiment of the present application, information regarding color difference components from different image formats is converted to information of the same size by size conversion, and this information is input to the coding network. As a result, when the coding network processes information from different (or at least two) image formats, the computational power and time overhead are consistent (or nearly consistent, with the difference in overhead being at least less than the difference in overhead for processing the original information from different formats), thereby improving the stability of information processing. This is superior in terms of both the machine experience and the user experience.

[0008] In addition, in the prior art, if a model is trained for a low-resolution image format (e.g., YUV420), the model has insufficient encoding performance for a high-resolution format (e.g., YUV444). (If a standard 444 encoding model is required, a separate model must be trained.) In this embodiment of the present application, a single model trained at once may be used to encode multiple formats, thereby providing good and stable encoding performance.

[0009] Encoding networks are sometimes also called analytic transformation networks.

[0010] In one possible implementation, the size of the second piece of information in the spatial dimension is smaller than the size of the first piece of information, the size of the second piece of information in the channel dimension is larger than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is less than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is larger than the size of the third piece of information.

[0011] In one possible implementation, the amount of non-redundant information in the second piece of information is equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is equal to the amount of non-redundant information in the third piece of information.

[0012] In one possible implementation, information about color difference components in different formats carries different amounts of information due to their different sizes (in the case of low-resolution image formats, downsampling is performed, so the amount of information carried is less than that of high-resolution image formats). Here, "amount of information" refers to the amount of effective information, i.e., the amount of non-redundant information. For example, if the same information is duplicated multiple times, the amount of information increases, but the amount of non-redundant information remains unchanged. In this embodiment of the present application, size conversion matches the size of information about color difference components in different (or at least two) image formats, maintaining stability in computational overhead (including computational power overhead or time overhead). However, even if size conversion is performed on information about color difference components, it is preferable that the bitstream obtained throughout the encoding process matches the bitstream obtained by encoding before size conversion. If the bitstreams do not match, using different image formats is not meaningful. To ensure that the result of the encoding process (i.e., the bitstream) matches (or nearly matches) the result of existing solutions, it is necessary to ensure that the amount of effective information remains unchanged before and after size conversion.

[0013] In one possible implementation, the size transformation includes at least one of the following: performing information duplication and using the duplicated information as information on the new channel; splitting the information on at least one channel into information on multiple different channels; or performing a convolution operation.

[0014] In one possible implementation, the operation of splitting information on at least one channel into information on multiple different channels is performed by using a pixel unshuffle layer.

[0015] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0016] In one possible implementation, the number of channel dimensions for the second and fourth pieces of information is 8, the height dimension of the second piece of information in spatial dimensions is H / 2, the width dimension of the second piece of information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0017] In one possible implementation, the method further includes the steps of: obtaining a fifth piece of information, the fifth piece of information relating to a luminance component Y; and performing a size transformation on the fifth piece of information to obtain a sixth piece of information, the size of the sixth piece of information in spatial dimensions being equal to the size of the second piece of information, wherein the step of encoding the second piece of information using an encoding network or a portion of an encoding network includes the step of encoding the result of merging the second piece of information with the fourth piece of information using an encoding network.

[0018] In one possible implementation, the first or third piece of information further includes information relating to the luminance component.

[0019] According to a second aspect, the present application provides an image decoding method. The method includes the steps of: obtaining first information based on a first bitstream; performing a size transformation on the first information to obtain second information, wherein the second information includes information about the color difference components of a first image format; reconstructing a first image based on the second information; obtaining third information based on a second bitstream, wherein the size of the third information in spatial and channel dimensions matches the size of the first information; performing a size transformation on the third information to obtain fourth information, wherein the fourth information includes information about the color difference components of a second image format, wherein the second image format is different from the first image format; and reconstructing a second image based on the fourth information.

[0020] In existing implementations, when a decoding network performs decoding of different (or at least two) image formats, the decoding network or its intermediate layers output information of different sizes, resulting in varying computational power and time overhead. Consequently, the stability of information processing is insufficient. In this embodiment of the present application, the decoding network or its intermediate layers may output information of the same size (same channel dimension and same spatial dimension) to obtain information about the color difference components of different image formats, resulting in consistent (or nearly consistent) computational power and time overhead when the decoding network performs decoding of different (or at least two) image formats, thus improving the stability of information processing. This is superior in terms of both machine experience and user experience.

[0021] Decoding networks are sometimes called synthetic transform networks.

[0022] In addition, there is no coupling relationship between the input and output. The input of YUV444 can produce both the output of 444 and the output of 420, or the output of 422. Furthermore, in this embodiment of the present application, the complexity of decoding 444 and 422 is approximately the same as the complexity of decoding 420, and is lower than the complexity of decoding 444 and 422 by using the original JPEG AI.

[0023] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is greater than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is less than the size of the third piece of information.

[0024] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is less than or equal to the amount of non-redundant information in the third piece of information.

[0025] In one possible implementation, the size transformation includes at least one of the following: deleting some of the information from a channel, combining different data from multiple channels into the same channel, or performing a convolution operation.

[0026] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0027] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0028] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, and the width dimension of the first information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0029] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0030] According to a third aspect, the present application provides an image coding device. This device is An acquisition module is configured to acquire first information, which includes information about the color difference components of a first image format, and third information, which includes information about the color difference components of a second image format, wherein the second image format is different from the first image format. A processing module configured to perform a size transformation on the first information to obtain the second information, encode the second information using an encoding network or a part of an encoding network, perform a size transformation on the third information to obtain the fourth information, the size of the fourth information in the spatial and channel dimensions matches the size of the second information, and encode the fourth information using an encoding network. Includes.

[0031] In one possible implementation, the size of the second piece of information in the spatial dimension is smaller than the size of the first piece of information, the size of the second piece of information in the channel dimension is larger than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is less than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is larger than the size of the third piece of information.

[0032] In one possible implementation, the amount of non-redundant information in the second piece of information is equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is equal to the amount of non-redundant information in the third piece of information.

[0033] In one possible implementation, size conversion is performed. Performing information replication and using the replicated information as information on the new channel, Splitting information on at least one channel into information on multiple different channels, or Performing a convolution operation It includes at least one of the following.

[0034] In one possible implementation, the operation of splitting information on at least one channel into information on multiple different channels is performed by using a pixel unshuffle layer.

[0035] In one possible implementation, the first image format is the YUV444 format, and the second image format is either the YUV422 or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format.

[0036] In one possible implementation, the number of channel dimensions for the second and fourth pieces of information is 8, the height dimension of the second piece of information in spatial dimensions is H / 2, the width dimension of the second piece of information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0037] In one possible implementation, the acquisition module is: The fifth piece of information is obtained, and this fifth piece of information is information about the luminance component Y. It is further configured in this way, The processing module is further configured to perform a size transformation on the fifth information in order to obtain the sixth information, such that the size of the sixth information in spatial dimensions matches the size of the second information. The processing module is specifically configured to encode the fusion result of the second and fourth pieces of information using an encoding network.

[0038] In one possible implementation, the first or third piece of information further includes information relating to the luminance component.

[0039] According to a fourth aspect, the present application provides an image decoding device. This device is An acquisition module configured to acquire first information based on a first bitstream, acquire third information based on a second bitstream, and such that the size of the third information in the spatial and channel dimensions matches the size of the first information, A processing module configured to perform a size transformation on the first information in order to obtain second information, the second information including information about the color difference components of the first image format, and to reconstruct the first image based on the second information, and to perform a size transformation on the third information in order to obtain fourth information, the fourth information including information about the color difference components of the second image format, the second image format being different from the first image format, and to reconstruct the second image based on the fourth information, and Includes.

[0040] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is greater than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is less than the size of the third piece of information.

[0041] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is less than or equal to the amount of non-redundant information in the third piece of information.

[0042] In one possible implementation, size conversion is as follows: A method of deleting some of the information in a channel. A method of combining different data from multiple channels into the same channel, or Method for performing convolution operations It includes at least one of the following.

[0043] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0044] In one possible implementation, the first image format is the YUV444 format, and the second image format is either the YUV422 or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format.

[0045] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, and the width dimension of the first information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0046] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0047] According to a fifth aspect, the present application provides an image decoding method. The method includes the steps of: obtaining first information based on a first bitstream; performing a size transformation on the first information to obtain one or more of second information, third information, or fourth information, wherein the second information is information relating to the color difference components of a first image format, the third information is information relating to the color difference components of a second image format, and the fourth information is information relating to the color difference components of a third image format, and the first, second, and third image formats are different from each other; and performing one or more of the steps of: reconstructing a first image based on the second information, reconstructing a second image based on the third information, or reconstructing a third image based on the fourth information.

[0048] In existing implementations, when a decoding network performs decoding of different (or at least two) image formats, the decoding network or its intermediate layers output information of different sizes, resulting in varying computational power and time overhead. Consequently, the stability of information processing is insufficient. In this embodiment of the present application, the decoding network or its intermediate layers may output information of the same size (same channel dimension and same spatial dimension) to obtain information about the color difference components of different image formats, resulting in consistent (or nearly consistent) computational power and time overhead when the decoding network performs decoding of different (or at least two) image formats, thus improving the stability of information processing. This is superior in terms of both machine experience and user experience.

[0049] In addition, a size transformation may be performed on the first information (e.g., the first information) obtained from the same bitstream in order to generate color difference information for multiple image formats. In this way, there is no coupling relationship between the input and output. For example, a YUV444 input can produce both a 444 output and a 420 output, or a 422 output. Furthermore, in this embodiment of the present application, the complexity of decoding 444 and 422 is approximately the same as the complexity of decoding 420, and is lower than the complexity of decoding 444 and 422 by using the original JPEG AI.

[0050] In one possible implementation, the sizes of the second, third, and fourth pieces of information in the spatial dimension are greater than or equal to the size of the first piece of information, while the sizes of the second, third, and fourth pieces of information in the channel dimension are smaller than the size of the first piece of information.

[0051] In one possible implementation, the amount of non-redundant information contained in the second, third, and fourth pieces of information is less than or equal to the amount of non-redundant information contained in the first piece of information.

[0052] In one possible implementation, the size transformation includes at least one of the following: deleting some of the information from a channel, combining different data from multiple channels into the same channel, or performing a convolution operation.

[0053] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0054] In one possible implementation, the first image format is the YUV444 format, the second image format is the YUV422 format, and the third image format is the YUV420 format.

[0055] In one possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0056] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0057] According to a sixth aspect, the present application provides an image decoding device. This device is An acquisition module configured to acquire first information based on a first bitstream, A size transformation is performed on the first information in order to obtain one or more of the second, third, or fourth pieces of information, wherein the second piece of information is information about the color difference components of the first image format, the third piece of information is information about the color difference components of the second image format, and the fourth piece of information is information about the color difference components of the third image format, and the first, second, and third image formats are different from each other. Steps to reconstruct the first image based on the second piece of information, A step of reconstructing a second image based on third information, or Steps to reconstruct the third image based on the fourth piece of information, A processing module configured to perform one or more of the following: Includes.

[0058] It should be understood that the processing module described in the sixth aspect may have the ability to convert information obtained from the same bitstream into color difference information in multiple image formats. When the processing model is executed, one or more of the multiple image formats may be selected based on the requirements for generation (e.g., user-specified or device selection).

[0059] In one possible implementation, the first piece of information is the output of the decoding network or the output of the intermediate layer of the decoding network.

[0060] In one possible implementation, the sizes of the second, third, and fourth pieces of information in the spatial dimension are greater than or equal to the size of the first piece of information, while the sizes of the second, third, and fourth pieces of information in the channel dimension are smaller than the size of the first piece of information.

[0061] In one possible implementation, the amount of non-redundant information contained in the second, third, and fourth pieces of information is less than or equal to the amount of non-redundant information contained in the first piece of information.

[0062] In one possible implementation, the size transformation includes at least one of the following: deleting some of the information from a channel, combining different data from multiple channels into the same channel, or performing a convolution operation.

[0063] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0064] In one possible implementation, the first image format is the YUV444 format, the second image format is the YUV422 format, and the third image format is the YUV420 format.

[0065] In one possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0066] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0067] According to a seventh aspect, the present application provides an image decoding method. The method includes the steps of: obtaining first information based on a first bitstream; performing a first resizing transformation on the first information to obtain second information, wherein the second information includes information about the color difference components of a first image format; reconstructing a first image based on the second information; performing a second resizing transformation on the first information to obtain third information, wherein the third information includes information about the color difference components of a second image format, and the second image format is different from the first image format; and reconstructing a second image based on the third information.

[0068] In existing implementations, when a decoding network performs decoding of different (or at least two) image formats, the decoding network or its intermediate layers output information of different sizes, resulting in varying computational power and time overhead. Consequently, the stability of information processing is insufficient. In this embodiment of the present application, the decoding network or its intermediate layers may output information of the same size (same channel dimension and same spatial dimension) to obtain information about the color difference components of different image formats, resulting in consistent (or nearly consistent) computational power and time overhead when the decoding network performs decoding of different (or at least two) image formats, thus improving the stability of information processing. This is superior in terms of both machine experience and user experience.

[0069] In addition, a size transformation may be performed on the first information (e.g., the first information) obtained from the same bitstream in order to generate color difference information for multiple image formats. In this way, there is no coupling relationship between the input and output. For example, a YUV444 input can produce both a 444 output and a 420 output, or a 422 output. Furthermore, in this embodiment of the present application, the complexity of decoding 444 and 422 is approximately the same as the complexity of decoding 420, and is lower than the complexity of decoding 444 and 422 by using the original JPEG AI.

[0070] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the third piece of information in the spatial dimension is greater than or equal to the size of the first piece of information, and the size of the third piece of information in the channel dimension is less than the size of the first piece of information.

[0071] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the third piece of information is less than or equal to the amount of non-redundant information in the first piece of information.

[0072] In one possible implementation, the size transformation includes at least one of the following: deleting some of the information from a channel, combining different data from multiple channels into the same channel, or performing a convolution operation.

[0073] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0074] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0075] In one possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0076] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0077] According to the eighth aspect, the present application provides an image decoding device. This device is An acquisition module configured to acquire first information based on a first bitstream, A processing module configured to perform a first size transformation on the first information in order to obtain second information, the second information including information about the color difference components of the first image format, and to reconstruct the first image based on the second information, or to perform a second size transformation on the first information in order to obtain third information, the third information including information about the color difference components of the second image format, the second image format being different from the first image format, and to reconstruct the second image based on the third information, and Includes.

[0078] It should be understood that the processing module described in the eighth aspect may have the ability to convert information obtained from the same bitstream into color difference information in multiple image formats. When the processing model is executed, one or more of the multiple image formats may be selected based on the requirements for generation (e.g., user-specified or device selection).

[0079] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the third piece of information in the spatial dimension is greater than or equal to the size of the first piece of information, and the size of the third piece of information in the channel dimension is less than the size of the first piece of information.

[0080] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the third piece of information is less than or equal to the amount of non-redundant information in the first piece of information.

[0081] In one possible implementation, the size transformation includes at least one of the following: a method of deleting some information from a channel; a method of combining different data from multiple channels into the same channel; and a method of performing a convolution operation.

[0082] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0083] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0084] In one possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0085] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0086] A ninth aspect of the embodiments of this application provides an encoding device including a processor. The processor is coupled to a memory, which is configured to store a program or instruction. When the program or instruction is executed by the processor, the encoding device is enabled to carry out a method according to the first embodiment or any one of the possible implementations of the first embodiment.

[0087] A tenth aspect of the embodiments of this application provides a decoding device including a processor. The processor is coupled to a memory, which is configured to store a program or instruction. When the program or instruction is executed by the processor, the decoding device is enabled to perform a method according to the second aspect or any possible implementation of the second aspect, a method according to the fifth aspect or any possible implementation of the fifth aspect, or a method according to the seventh aspect or any possible implementation of the seventh aspect.

[0088] An eleventh embodiment of the present application provides an encoding and decoding system including an encoding device according to the ninth embodiment and / or a decoding device according to the tenth embodiment.

[0089] A twelfth aspect of the embodiments of this application provides a computer-readable medium that stores a computer program or instruction. When the computer program or instruction is executed on a computer, the computer is enabled to perform a method according to the first aspect or any possible implementation of the first aspect, or the computer is enabled to perform a method according to the second aspect or any possible implementation of the second aspect, the fifth aspect or any possible implementation of the fifth aspect, or the seventh aspect or any possible implementation of the seventh aspect.

[0090] A thirteenth embodiment of the embodiments of this application provides a computer program product. When the computer program product is executed on a computer, the computer is enabled to perform a method according to the first embodiment or any possible implementation of the first embodiment, or the computer is enabled to perform a method according to the second embodiment or any possible implementation of the second embodiment, the fifth embodiment or any possible implementation of the fifth embodiment, or the seventh embodiment or any possible implementation of the seventh embodiment.

[0091] A fourteenth aspect of the embodiments of this application provides a readable storage medium. The readable medium stores a bitstream obtained by encoding using a method according to the first embodiment or one of the possible implementations of the first embodiment.

[0092] A fifteenth aspect of the embodiments of this application provides a bitstream storage device comprising at least one storage medium and a communication interface, the communication interface being configured to receive or transmit a bitstream, the at least one storage medium being configured to store a bitstream, and the bitstream being obtained by encoding by an encoder using a method according to the first embodiment or one of the possible implementations of the first embodiment.

[0093] A sixteenth aspect of the embodiments of this application provides a bitstream storage method. The method includes the steps of: receiving a bitstream via a communication interface; and storing the bitstream in one or more storage media, wherein the bitstream is obtained by an encoder by encoding it using a method according to the first embodiment or one of the possible implementations of the first embodiment.

[0094] A 17th aspect of the embodiments of this application provides a bitstream distribution system. The system includes at least one storage medium and a video stream device. The at least one storage medium is configured to store a bitstream, which is obtained by encoding by an encoder using the method according to the first embodiment or one of the possible implementations of the first embodiment. The video stream device is configured to respond to a request from a decoder, thereby transmitting a target bitstream in the at least one storage medium to the decoder.

[0095] An 18th aspect of the embodiments of this application provides a bitstream distribution method. The method includes the steps of receiving a first request, selecting a target bitstream from at least one storage medium in response to the first request, and transmitting the target bitstream to a destination device. The at least one storage medium is configured to store a bitstream, which is obtained by encoding by an encoder using the method according to the first embodiment or one of the possible implementations of the first embodiment.

[0096] A 19th aspect of the embodiments of this application provides a bitstream processing system. The system includes an image source device, an encoder device, one or more storage media, and a destination device. The image source device is configured to provide image data. The encoder device is configured to acquire image data from the image source device via an interface and to encode the image data to acquire one or more bitstreams, which are obtained by encoding the bitstreams by the encoder using a method according to the first embodiment or one of the possible implementations of the first embodiment. The encoder device is configured to store one or more bitstreams in one or more storage media. Alternatively, the encoder device is configured to encapsulate one or more bitstreams to acquire a transmit bitstream. The encoder device is configured to transmit the transmit bitstream to the destination device via a communication link or communication network. The destination device is configured to decapsulate the transmit bitstream to acquire one or more bitstreams. The destination device is configured to decode one or more bitstreams to acquire the decoded data.

[0097] According to a 20th aspect, the present application provides a chip system. The chip system includes a processor configured to support the device in performing some or all of the functions in the preceding aspects, such as transmitting or processing data or information in the manner described above. In one possible design, the chip system further includes memory. The memory is configured to store program instructions and data required by the device or the device. The chip system may include a chip, or it may include a chip and other separate components. [Brief explanation of the drawing]

[0098] [Figure 1a] This is a diagram illustrating an application scenario according to one embodiment of this application. [Figure 1b]This is a diagram illustrating an application scenario according to one embodiment of this application. [Figure 2a] This figure shows an application scenario according to one embodiment of this application. [Figure 2b] This figure shows an application scenario according to one embodiment of this application. [Figure 3] This figure shows an embodiment of an image processing method according to one embodiment of this application. [Figure 4] This figure shows one embodiment of an image processing method according to one embodiment of the present application. [Figure 5] This figure shows one embodiment of an image processing method according to one embodiment of the present application. [Figure 6] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 7] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 8a] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 8b] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 9] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 10] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 11] This figure shows one embodiment of an image processing method according to one embodiment of this application. [Figure 12] This is a diagram showing the structure of an image encoding device according to one embodiment of this application. [Figure 13] This is a diagram showing the structure of an image decoding device according to one embodiment of this application. [Figure 14] This is a diagram showing the structure of a device according to one embodiment of this application. [Figure 15] This is a diagram showing the structure of a device according to one embodiment of this application. [Modes for carrying out the invention]

[0099] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings of embodiments of the present invention. The terms used in the embodiments of the present invention are intended solely to describe specific embodiments of the present invention and are not intended to limit the present invention.

[0100] Embodiments of this application will be described below with reference to the attached drawings. Those skilled in the art will know that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0101] In the specification, claims, and accompanying drawings of this application, terms such as “first” and “second” are intended to distinguish similar subjects but do not necessarily indicate a particular order or arrangement. It should be understood that such terms are interchangeable in appropriate contexts and are merely a means of identification used when subjects having the same attributes are described in the embodiments of this application. In addition, the terms “includes” and “having,” and any variations thereof, mean to cover non-exclusive inclusion, meaning that a process, method, system, product, or device including a set of units is not necessarily limited to those units and may include other units not expressly enumerated, or other units specific to such process, method, product, or device.

[0102] The following describes application scenarios to which the method provided in the embodiments of this application may be applied.

[0103] The methods provided in the embodiments of this application may be applied to service scenarios such as public cloud, private cloud, video surveillance, live streaming technology, storage, and transmission.

[0104] Figure 1a shows an example of an application scenario according to one embodiment of the present application. The application scenario may also be understood as an encoding and decoding application scenario. The application scenario includes an AI encoding unit, a storage / transmission unit, and an AI decoding unit. The method provided in the embodiments of the present application may be applied to an AI encoding unit / AI decoding unit.

[0105] The AI ​​coding unit is configured to encode input data (e.g., images or videos) according to the coding method provided in the embodiments of this application.

[0106] The storage / transmission unit is configured to store (for terminals, video surveillance, public clouds, private clouds, etc.) or transmit (for public clouds, private clouds, live streaming technologies, etc.) encoded input data.

[0107] The AI ​​decoding unit is configured to decode input data encoded according to the decoding method provided in the embodiments of this application to obtain reconstructed data (which may also be understood as data obtained by reconstructing the encoded features of the input data).

[0108] The aforementioned exemplary application scenarios include several cases, which are described in detail below.

[0109] The first case is a storage application scenario.

[0110] Storage application scenarios include terminal albums and video surveillance. Figure 1b is a diagram of a scenario. The scenario includes an AI encoding unit, a file storage unit, a file loading unit, and an AI decoding unit.

[0111] The AI ​​encoding unit is configured to encode input data (e.g., an image or video) according to the encoding method provided in the embodiments of this application in order to obtain a binary file.

[0112] The file storage unit is configured to store the binary files output by the AI ​​coding unit.

[0113] The file load unit is configured to load binary files.

[0114] The AI ​​decoding unit is configured to decode a binary file loaded according to the decoding method provided in the embodiments of this application to obtain reconstructed data (which may also be understood as data obtained by reconstructing the encoded features of the input data).

[0115] The second case is an interaction scenario between a device and the cloud.

[0116] The interaction scenarios between the terminal and the cloud include live streaming scenarios and public clouds. Figure 2a is a diagram of a scenario. The scenario includes an encoding unit, a decoding unit, an AI encoding unit, an AI decoding unit, an encoding unit, and a decoding unit.

[0117] The terminal side includes an encoding unit and a decoding unit, while the cloud side includes a decoding unit, an encoding unit, an AI encoding unit, and an AI decoding unit.

[0118] The terminal acquires input data and uses an encoding unit to encode the input data and obtain encoded features. The terminal sends the encoded features to the cloud. After receiving the encoded features, the cloud uses a decoding unit to decode the encoded features and obtain reconstructed data. The reconstructed data is then encoded using an AI encoding unit according to the encoding method provided in the embodiments of this application to obtain a bitstream / file. The cloud uses an AI decoding unit to decode the bitstream / file according to the decoding method provided in the embodiments of this application and obtain reconstructed data. The cloud uses an encoding unit to encode the reconstructed data and obtain encoded features, and sends the encoded features to the terminal. After receiving the encoded features, the terminal uses a decoding unit to decode the encoded features and obtain reconstructed data.

[0119] It should be understood that the encoding methods used by the encoding unit and AI encoding unit in Figure 2a may not be consistent (or, the operation of the decoding unit and AI encoding unit in the cloud may be called a transcoding operation, and the operation of the AI ​​decoding unit and encoding unit in the cloud may be called a transcoding operation). For example, the cloud typically has more computing resources than a terminal. Therefore, an algorithm with high computing requirements and higher compression efficiency (AI encoding) may be used to compress (transcode) the input data again to achieve a higher compression ratio and save storage space in the cloud. In the transcoding process, decoding is performed first, and then encoding is performed in a different manner.

[0120] It should be understood that some of the scenarios mentioned above are merely examples. Other scenarios may exist in actual applications, and these are not limited to those described here.

[0121] The following describes the hardware structure of a chip according to one embodiment of this application.

[0122] Figure 2b shows the hardware structure of a chip according to one embodiment of the present invention. The chip includes a neural network processing unit 40.

[0123] The neural network processing unit 40 can be any processor suitable for large-scale exclusive OR operations, such as a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). An NPU is used as an example. The neural network processing unit 40 functions as a coprocessor and is implemented on a host central processing unit (CPU). The host CPU assigns tasks. The core of the NPU is the arithmetic circuit 403. The controller 404 controls the arithmetic circuit 403 to extract data from memory (weight memory or input memory) and perform operations.

[0124] In some implementations, the arithmetic circuit 403 contains multiple processing units (process engines, PEs). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may be a one-dimensional systolic array, or it may be other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.

[0125] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 fetches data corresponding to matrix B from the weight memory 402 and buffers that data in each PE within the arithmetic circuit. The arithmetic circuit fetches data for matrix A from the input memory 401, performs matrix operations on the data of matrices A and B, obtains a partial or final result of the matrix, and stores the result in the accumulator 408.

[0126] The vector computation unit 407 can perform further processing on the outputs of arithmetic circuits such as vector multiplication, vector addition, exponential operations, logarithmic operations, and size comparisons. For example, the vector computation unit 407 may be configured to perform network computations such as pooling, batch normalization, or local response normalization in the non-convolutional / non-FC layers of a neural network.

[0127] In some implementations, the vector computation unit 407 stores the processed output vector in the integrated memory 406. For example, the vector computation unit 407 may apply a nonlinear function to the output of the arithmetic circuit 403, e.g., a vector of cumulative values, to generate activation values. In some implementations, the vector computation unit 407 generates normalized values, composite values, or both. In some implementations, the processed output vector can be used, for example, as an activation input for the arithmetic circuit 403 used in subsequent layers in the neural network.

[0128] The integrated memory 406 is configured to store input and output data.

[0129] Regarding weight data, the direct memory access controller (DMAC) 405 directly transfers input data in external memory to input memory 401 and / or integrated memory 406, stores weight data in external memory in weight memory 402, and stores data in integrated memory 406 in external memory.

[0130] The bus interface unit (BIU) 410 is configured to implement interaction between the host CPU, DMAC, and instruction fetch buffer 409 via the bus.

[0131] An instruction fetch buffer 409 connected to the controller 404 is configured to store instructions used by the controller 404.

[0132] The controller 404 is configured to control the operation process of the arithmetic accelerator by calling instructions buffered in the instruction fetch buffer 409.

[0133] Generally, the integrated memory 406, input memory 401, weight memory 402, and instruction fetch buffer 409 are on-chip memory. External memory is memory outside the NPU. External memory may be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other read / write memory.

[0134] The methods provided in embodiments of this application are described below. These methods may be performed by an encoding device or by components of an encoding device (e.g., a processor, chip, or chip system). The encoding device may be a cloud device or a terminal device. Naturally, the methods may also be performed by a system including both cloud and terminal devices. Optionally, the methods may be processed by a processor such as a CPU, NPU, or GPU within the encoding device. This is not limited to these methods.

[0135] Since the embodiments of this application relate to large-scale applications of neural networks, for the sake of ease of understanding, the following will first explain the terms and concepts related to neural networks that may be used in the embodiments of this application.

[0136] (1) Neural Network

[0137] A neural network may include neurons. Neurons are x s The arithmetic unit may also use intercept 1 as input, and the output of the arithmetic unit may be as follows:

number

[0138] (2) Deep neural networks

[0139] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. A DNN can be divided based on the location of different layers, and the neural network within a DNN may be classified into three types: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and all intermediate layers are hidden layers. The layers are fully connected. Specifically, any neuron in the i-th layer is always connected to any neuron in the (i+1)-th layer.

[0140] While DNNs may appear complex, they are not actually complex in terms of the work performed at each layer. Simply put, the work at each layer is governed by the following linear relationship:

number

number

number

number

number

number

number

number

[0141] In conclusion, the coefficient from the k-th neuron in layer (L-1) to the j-th neuron in layer L is

number

[0142] Note that there is no parameter W for the input layer. In deep neural networks, increasing the number of hidden layers allows the network to more accurately describe complex real-world cases. Theoretically, models with more parameters have higher complexity and greater "capacity." This indicates that the model can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and the ultimate goal of training is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by vectors W in many layers).

[0143] (3) Convolutional Neural Networks

[0144] A convolutional neural network (CNN) is a deep neural network that has a convolutional structure. A convolutional neural network includes a feature extractor, which consists of convolutional layers and subsampling layers, and the feature extractor can be considered a filter. A convolutional layer is a layer of neurons in a convolutional neural network where a convolutional operation is performed on an input signal. In the convolutional layers of a convolutional neural network, a single neuron may be connected to only some neighboring layer neurons. A single convolutional layer typically contains several feature planes, each feature plane may contain some neurons in a rectangular arrangement. Neurons within the same feature plane share weights, and these shared weights are referred to herein as the convolutional kernel. Weight sharing can be understood as the image information extraction scheme being location-independent. The convolutional kernel can be initialized in the form of a random-size matrix. In the process of training a convolutional neural network, the convolutional kernel can acquire appropriate weights through learning. In addition, a direct benefit of weight sharing is that it reduces the number of connections between layers in a convolutional neural network, thereby lowering the risk of overfitting.

[0145] (4) Concatenation Concat

[0146] A concatenation operation is the process of connecting multiple tensors of a specific dimension. All tensors must have the same shape (except for the connecting dimension). In this patent, a concatenation operation means a connection on a channel.

[0147] (5) Bitrate: Indicates the average coding length required to encode a unit pixel in an image compression task.

[0148] (6) Rate Distortion Performance: This is an index used to measure the performance of the compression algorithm. It takes into account both the bitrate and distortion of the decoded image.

[0149] (7) YUV: which is a color format. The YUV and RGB formats are mutually convertible. YUV is usually applied to various media processing algorithms to reduce color difference bandwidth. The following three YUV formats are commonly used: YUV4:4:4 (YUV444): full sampling is performed on the color difference components (UV) to obtain Y[1,H,W] and UV[2,H,W]; YUV4:2:2 (YUV422): 2:1 horizontal sampling and full vertical sampling are performed on the color difference components (UV) to obtain Y[1,H,W] and UV[2,H,W / 2]; and YUV4:2:0 (YUV420): 2:1 horizontal sampling and 2:1 vertical sampling are performed on the color difference components (UV) to obtain Y[1,H,W] and UV[2,H / 2,W / 2].

[0150] In addition to the default format YUV422, the present invention also supports another possible YUV422 format, in which full horizontal sampling and 2:1 vertical sampling are performed on the color difference components (UV) to obtain Y[1,H,W] and UV[2,H / 2,W].

[0151] (8) Pixel Shuffle

[0152] Pixel Shuffle is mainly used to obtain a high-resolution feature map from a low-resolution feature map through shuffling of multiple channels. In this specification, Pixel Shuffle is described as PS(s1,s2), where S1 and S2 refer to upsampling coefficients, which are usually integers greater than 1. [C in ,h in ,w in is input, and [C out ,h out ,w out is output, h out =s1·h in , w out =s2·w in , and C out =C in / (s1·s2),

number

number

[0153] (9) Pixel Unshuffle

[0154] Pixel unshuffle is the inverse operation of PixelShuffle and is primarily used to obtain a low-resolution feature map from a high-resolution feature map via a multi-channel shuffle. In this embodiment of the present application, pixel unshuffle is written as UPS(s1,s2), where s1 and s2 refer to downsampling coefficients, which are typically integers greater than 1. [C in ,h in ,w in ] is entered, [C out ,h out ,w out ] is output,

number

[0155] (10)s UV and s Y

[0156] Upsampling and downsampling factors: Downsampling factors are all used in the encoder, and upsampling factors are all used in the decoder. UV =[s1,s2] represents that s1 is the vertical sampling multiplier for UVs, s2 is the horizontal sampling multiplier for UVs, and both s1 and s2 are integers greater than or equal to 1. In JPEG AI, bilinear (bicubic) interpolation is used for upsampling, and sampling at (sample) intervals is used for downsampling. This application also supports upsampling and downsampling in other ways.

[0157] (11) Color conversion (ColorTr)

[0158] Color conversion is used to convert an input image into primary and secondary components. The color conversion used to convert RGB pictures to YUV format, and used in JPEG AI, is defined by the international standard ITU-R BT.709. The input consists of three components: R[H,W], G[H,W], and B[H,W], and the spatial shape of the input and output is H×W. Further details are explained below. Y[i,j]=0.2126·R[i,j]+0.7152·G[i,j]+0.0722·B[i,j] U[i,j]=(-0.2126·R[i,j]-0.7152·G[i,j]+0.9278·B[i,j]) / 1.8556 V[i,j]=(0.7874·R[i,j]-0.7152·G[i,j]-0.0722·B[i,j]) / 1.5748

[0159] For subscripts [i,j], 0≦i <H、および0≦j<Wである。

[0160] (12) Inverse Color Transform (invColorTr): The inverse transformation of ColorTr.

[0161] Image / video compression is a technique that uses data features such as spatial redundancy, visual redundancy, and statistical redundancy to represent images / videos with as few bits as possible, either irreversibly or reversibly. This technique enables the effective transmission and storage of data information and plays a crucial role in today's media age, where the types of data transmitted are becoming increasingly diverse and the amount of data is constantly growing. Data compression is classified into irreversible and reversible compression. Irresistible compression achieves a high compression ratio at the expense of a degree of image / video quality degradation, while reversible compression does not cause any loss of image / video quality.

[0162] Currently, JPEG AI can encode images in different formats such as YUV420, YUV444, and YUV422 by using different downsampling multiples. However, different formats significantly impact the computational power of the current JPEG AI network.

[0163] Embodiments of this application first provide an explanation by using an example in which the application scenario is a terminal device.

[0164] For example, the terminal device may be a mobile phone, tablet computer, notebook computer, or intelligent wearable device, and the terminal device may perform compression on the acquired picture. In another example, the terminal device may be a virtual reality (VR) device. In yet another example, embodiments of this application may also be applied to intelligent monitoring, in which a camera may be configured. In this case, the intelligent monitoring may use the camera to acquire the picture to be compressed. It should be understood that embodiments of this application may be further applied to other scenarios where image compression is required. Other application scenarios are not listed one by one here.

[0165] Figure 3 shows one embodiment of an image coding method according to one embodiment of the present application. As shown in Figure 3, the image coding method provided in this embodiment of the present application includes the following steps.

[0166] 301: Obtain the first piece of information, which includes information about the color difference components of the first image format.

[0167] In this embodiment of the present application, the first information may include information relating to the color difference components of a first image in a first image format.

[0168] Regarding the first image: In one possible implementation, the first image may be the image to be encoded. The first image may be an image captured by a terminal device using a camera, or the first image may be an image acquired from a terminal device (for example, an image stored in the terminal device's album, or a picture acquired from the cloud by the terminal device). It should be understood that the first image may be an image having image encoding requirements. The source of the first image is not limited in this application.

[0169] Information regarding color difference components: In one possible implementation, the input image (e.g., the first image in this embodiment of the present application) may be preprocessed to convert the image into primary and secondary components. The primary component may be the luminance component, and the secondary component may be the chrominance component. The primary component (i.e., the luminance component) may be, but not limited to, component Y, and the secondary component (i.e., the chrominance component) may be, but not limited to, component UV. For example, components Y and UV may be obtained by splitting using ColorTr. YUV is a type of compiled true-color color space, and Y'UV, YUV, YCbCr, YPbPr, etc., are all called YUV and may overlap with each other. "Y" represents luminance (Luminance or Luma), i.e., grayness value. "U" and "V" represent chrominance (Chrominance or Chroma) and are used to describe the color and saturation of the image and to specify the color of a pixel.

[0170] In this embodiment of the present application, a first image is acquired, and chromatic difference and luminance components are acquired by preprocessing the first image, or the chromatic difference and luminance components corresponding to the first image are acquired directly without acquiring the first image.

[0171] In one possible implementation, information regarding the color difference components and luminance components of the image format may be obtained. The image format may be YUV444, YUV422, or YUV420.

[0172] In the case of the YUV444 format, the secondary component of the first image may be used directly as information about the color difference components of the YUV444 format (for example, the size is [2, H, W], where H is the height of the first image and W is the width of the first image).

[0173] In the case of the YUV422 format, resampling (e.g., downsampling) may be performed on the quadratic components of the first image to obtain information about the color difference components of the YUV422 format (for example, the size is [2, H, W / 2], where H is the height of the first image and W is the width of the first image).

[0174] In the case of the YUV420 format, resampling (e.g., downsampling) may be performed on the quadratic components of the first image to obtain information about the color difference components of the YUV420 format (for example, the size is [2, H / 2, W / 2], where H is the height of the first image and W is the width of the first image).

[0175] In this embodiment of the present application, the first information is information relating to the color difference components of a first image format, the third information is information relating to the color difference components of a second image format, and the resolution of the first image format is higher than the resolution of the second image format. For example, the first image format is the YUV444 format and the second image format is the YUV422 format or the YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0176] 302: Perform a size conversion on the first piece of information in order to obtain the second piece of information.

[0177] In one possible implementation, in this embodiment of the present application, size transformations can be performed on information regarding chromatic difference components of different (or at least two) image formats such that the size of the spatial dimension information is consistent and the size of the channel dimension information is consistent. For example, size transformations can be performed separately for first information in a first image format and third information in a second image format such that the transformed first information (i.e., second information) and the transformed third information (i.e., fourth information) are the same size (i.e., the channel size and spatial size are the same).

[0178] In existing implementations, information regarding color difference components is directly input to the coding network, which then encodes the information. Because the size of information differs across different image formats, i.e., the size of information input to the coding network differs, the computational power and time overhead differ when the coding network processes information from different image formats. As a result, the stability of information processing is insufficient. In this embodiment of the present application, information regarding color difference components from different image formats is converted to information of the same size by size conversion, and this information is input to the coding network. As a result, when the coding network processes information from different (or at least two) image formats, the computational power and time overhead are consistent (or nearly consistent, with the difference in overhead being at least less than the difference in overhead for processing the original information from different formats), thereby improving the stability of information processing. This is superior in terms of both the machine experience and the user experience.

[0179] Encoding networks are sometimes called analytic transformation networks or signal encoders. Encoders are implemented based on neural networks and are configured to encode (or sometimes called perform feature extraction) input information to obtain a feature map. The feature map can be used to obtain a bitstream using quantization and entropy coding modules. Because encoders are implemented based on neural networks, they are sometimes called AI encoders.

[0180] In this embodiment of the present application, a size transformation performed on the first information can reduce the size of the information in the spatial dimension and increase the size of the information in the channel dimension. The size of the second information in the spatial dimension is smaller than the size of the first information, and the size of the second information in the channel dimension is larger than the size of the first information.

[0181] The spatial dimension may further include height and width dimensions, and the size of the second piece of information in the spatial dimension being smaller than the size of the first piece of information can be understood as the size of the second piece of information in at least one of the height and width dimensions being smaller than the size of the first piece of information (naturally, the size of the second piece of information in one of the spatial dimensions cannot be larger than the size of the first piece of information).

[0182] For example, the height dimension of the second piece of information in spatial dimensions is smaller than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is smaller than the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is larger than the size of the first piece of information.

[0183] For example, the height dimension of the second piece of information in spatial dimensions is smaller than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is equal to the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is larger than the size of the first piece of information.

[0184] The two examples mentioned above can be understood as cases where the size of the second piece of information in spatial dimension is smaller than the size of the first piece of information.

[0185] The size transformation performed on the first information in this embodiment of the present application will be described in detail below.

[0186] In one possible implementation, information about color difference components in different formats carries different amounts of information due to their different sizes (in the case of low-resolution image formats, downsampling is performed, so the amount of information carried is less than that of high-resolution image formats). Here, the amount of information refers to the amount of effective information, i.e., the amount of non-redundant information. For example, if the same information is duplicated multiple times, the amount of information increases, but the amount of non-redundant information remains unchanged. The amount of non-redundant information is the amount of information that substantially affects the encoding or decoding result.

[0187] In this embodiment of the present application, the size conversion matches the size of information regarding chromatic difference components of different (or at least two) image formats, while maintaining stability in computational overhead (including computational power overhead or time overhead). However, even if size conversion is performed on the information regarding chromatic difference components, it is preferable that the bitstream obtained throughout the encoding process matches the bitstream obtained by encoding before size conversion. If the bitstreams do not match, it is not worthwhile to use different image formats. To ensure that the result of the encoding process (i.e., the bitstream) matches (or nearly matches) the result of existing solutions, it is necessary to ensure that the amount of effective information remains unchanged before and after size conversion.

[0188] Specifically, the amount of non-redundant information in the second piece of information obtained by performing a size transformation on the first piece of information matches the amount of non-redundant information in the first piece of information. Here, "amount of information" refers to the amount of effective information, i.e., the amount of non-redundant information. For example, if the same information is duplicated multiple times, the amount of information increases, but the amount of non-redundant information remains unchanged.

[0189] Size transformations may involve operations such as information partitioning, information duplication, and convolution. These are described separately below.

[0190] 1. Information division In one possible implementation, performing a size transformation on the first information may involve splitting the information on at least one channel of the first information into information on multiple different channels. In other words, some of the information within the first information may be placed on extended channels.

[0191] In other words, to increase the number of channels in the first information while ensuring that the total amount of information remains unchanged, the channels of the first information may be expanded, and the information in the expanded channels must come from existing data within the first information. In this case, a portion of the information within the first information may be extracted and used as the information for the expanded channels, and this portion of the extracted information is removed from the original channels. This is equivalent to the original information of the first information being split to obtain more channels, thereby reducing the size of the first information in spatial dimensions.

[0192] A characteristic of the information regarding the color difference component is that the size of the information on all channels is the same; therefore, the same size of information can be extracted from the information on each channel (since the size of the extracted information is the same, the size of the information on all channels of the second information obtained by deleting a portion of this information is the same).

[0193] In one possible implementation, different data on the same channel within the first piece of information may be placed on a different, newly extended channel.

[0194] For example, channel 1 of the first piece of information (including channels 1 and 2) may contain elements A1, A2, and A3.

[0195] Elements A1, A2, and A3 may be removed from channel 1, and elements A1, A2, and A3 will be used as data for channel 3 (a channel not originally included in the first information).

[0196] Alternatively, elements A1, A2, and A3 may be removed from channel 1, with element A1 being used as data on channel 3 (a channel not originally included in the first information), element A2 being used as data on channel 4 (a channel not originally included in the first information), and element A3 being used as data on channel 5 (a channel not originally included in the first information).

[0197] In one possible implementation, data from different channels within the first information can be extracted, combined, and placed on a different, newly extended channel. In other words, data on the same newly extended channel may come from different channels.

[0198] For example, channel 1 of the first information (including channel 1 and channel 2) may include elements A1, A2, and A3, and channel 2 may include elements B1, B2, and B3.

[0199] Elements A1, A2, and A3 may be removed from channel 1, elements B1, B2, and B3 may be removed from channel 2, elements A1, A2, and A3 are used as data on channel 3 (a channel not originally included in the first information), and elements B1, B2, and B3 are used as data on channel 4 (a channel not originally included in the first information).

[0200] Alternatively, elements A1, A2, and A3 may be removed from channel 1, elements B1, B2, and B3 may be removed from channel 2, elements A1 and B1 may be used as data on channel 3 (a channel not originally included in the first information), elements A2 and B2 may be used as data on channel 4 (a channel not originally included in the first information), and elements A3 and B3 may be used as data on channel 5 (a channel not originally included in the first information).

[0201] In one possible implementation, the size conversion method may include a pixel unshuffle (UPS), i.e., the UPS may be performed on the first information.

[0202] Figure 4 shows the processing result of UPS. In Figure 4, an example is used to illustrate that each C channel of the first information is expanded to 4 channels of data, and the data of an information block with a size of 2*2 (height 2 and width 2) is expanded to 4 channels of data. In this case, the number of channels is also changed from C to 4C. If the spatial size of the first information is H*W, the number of channels in the second information obtained by the size conversion is H / 2*W / 2.

[0203] In one possible implementation, the first information is information about the color difference components of YUV444, the size of the first information in channel dimensions is 2, the height dimension of the first information in spatial dimensions is H, and the width dimension of the first information in spatial dimensions is W; the number of channels of the second information is 8, the height dimension of the second information in spatial dimensions is H / 2, and the width dimension of the second information in spatial dimensions is W / 2. The size conversion method in this example may be the same as that shown in Figure 4. Details are not repeated here.

[0204] 2. Information duplication In one possible implementation, in addition to increasing the number of channels through information partitioning, further information duplication may be performed during size conversion. This approach is more necessary when the first information is in a low-resolution image format (at least not the highest resolution, e.g., YUV422, YUV420). This is because the amount of information (i.e., information size) regarding the chrominance components of a low-resolution image format is less than the amount of information regarding the chrominance components of a high-resolution image format. When it is guaranteed that the amount of non-redundant information remains unchanged before and after size conversion, if only operations such as information partitioning that do not increase the amount of data (note that the amount of data and the amount of non-redundant information mentioned above are not the same concept; the amount of data is the total amount of information contained in the information, counting both redundant and non-redundant information) are performed, the size of the information regarding the chrominance components of a low-resolution (at least not the highest resolution) image format will certainly be smaller than the size of the information regarding the chrominance components of a high-resolution image format. In this case, duplication operations may be performed on the information regarding the chrominance components of the low-resolution (at least not the highest resolution) image format in order to increase the amount of data. Compared to the data before replication, the amount of non-redundant information in the information obtained through replication remains unchanged; only the amount of data included increases (i.e., the size increases).

[0205] In one possible implementation, information (all or part of the information) from the first information may be duplicated, and the duplicated information is used as information on the new channel. The use of duplicated information as information on the new channel can be understood as an increase in the number of channels compared to before the duplication, and the data included in the newly added channels includes the duplicated data.

[0206] In one possible implementation, information duplication may be performed on the first information, the duplicated information may be used as information on a new channel, and the information on at least one channel may be divided into information on multiple different channels. There is no restriction on the time order between the two operations, nor is there a limit to the number of each type of operation. For example, information duplication may be performed first, followed by information division, and then duplication, provided that the amount of non-redundant information remains unchanged and it is guaranteed that the information can be transformed to the size that needs to be transformed. The types of operations, the amount of operations, the combination of operations, and the order of operations are not limited to the embodiments of this application.

[0207] The order of arrangement on the channel after replication is not limited in this application.

[0208] For example, the information before duplication is channel 1, channel 2, channel 3, and channel 4 based on the channel order, and each of channels 1, 2, 3, and 4 may be duplicated once to obtain data for the new channels. If each duplicated channel is used as a channel adjacent to the channel before duplication, the information for channels 1, 1, 2, 2, 3, 3, 3, 4, and 4 may be obtained, or if each duplicated channel is placed after the channels of the first information according to the ordering relationship between the channels before duplication, the information for channels 1, 2, 3, 4, 1, 2, 3, and 4 may be obtained. The channel order between the information for the new channels and the information for the original channels is not limited in this application.

[0209] For example, in one possible implementation, the first information is information about the color difference components in YUV422, the size of the first information in the channel dimension is 2, the height dimension of the first information in the spatial dimension is H, the width dimension of the first information in the spatial dimension is W / 2, and the size may also be described as (2, H, W / 2); the number of channels of the second information is 8, the height dimension of the second information in the spatial dimension is H / 2, and the width dimension of the second information in the spatial dimension is W / 2.

[0210] In this example, to obtain information of size (4, H / 2, W / 2), information partitioning may be performed first on the first piece of information (whose size is (2, H, W / 2)), and then information duplication may be performed to obtain the second piece of information (8, H / 2, W / 2); or, in this example, to obtain information of size (4, H, W / 2), information duplication may be performed first on the first piece of information (whose size is (2, H, W / 2)), and then information partitioning may be performed to obtain the second piece of information (8, H / 2, W / 2).

[0211] In one possible implementation, the aforementioned operations of information duplication or information partitioning may be replaced with convolution.

[0212] In one possible implementation, size conversion may be performed by an alternative method, provided that the amount of non-redundant information remains unchanged and it is guaranteed that the information can be converted to the size to which it needs to be converted. This is not limited to the present application.

[0213] If the first information is information about the color difference components of YUV420, it should be understood that operations such as duplication, splitting, and convolution may also be performed on the first information. This will be explained in subsequent embodiments.

[0214] 303: Encode second information using an encoding network or a portion of an encoding network.

[0215] It should be understood that step 302 may operate independently of the coding network, or it may be performed within the coding network, for example, using an internal network layer close to the input of the coding network. Separate explanations are provided below.

[0216] 1. Step 302 is an operation independent of the coding network. In one possible implementation, if step 302 operates independently of the coding network, the second information may be coded using the coding network. Compared to conventional techniques, step 302 is an additional step that precedes the coding network.

[0217] 2. Step 302 is performed using a network layer that is close to the input of the coding network and is located within the coding network. In one possible implementation, when step 302 is performed using a network layer that is close to the input of the coding network and is within the coding network, the network layer that is close to the input of the coding network and is within the existing coding network may be improved. For example, after the calculated values ​​of the UPS layer or convolutional layer are changed, the second information is the output of the network layer in the analytic transformation network.

[0218] In one possible implementation, if step 302 is performed using a network layer that is close to and within the encoding network, the second information may be encoded using a portion of the encoding network (i.e., a network connected after the improved network layer).

[0219] It should be understood that there is also information exchange between information about the primary component (e.g., information about component Y) and information about the secondary component (e.g., information about component UV), and as a result, the compression effect may be further improved.

[0220] In one possible implementation, the information exchange may be performed before size conversion, or may be performed after size conversion and before input to the encoding network. In the following, descriptions are provided separately.

[0221] 1. Before size conversion In one possible implementation, the first information may further include information related to the luma component. That is, the first information needs to include at least information of the chrominance component, but information of the luma component may also be fused. For example, the first information may include a fusion result (e.g., a concatenated result) of the information related to the luma component and the information related to the chrominance component.

[0222] For example, the first information may include a fusion result (e.g., a concatenated result) of the information related to the luma component and the information related to the chrominance component in the YUV444 format.

[0223] For example, the first information may include a fusion result (e.g., a concatenated result) of the information related to the luma component and the information related to the chrominance component in the YUV422 format.

[0224] 2. After size conversion and before input to the encoding network In one possible implementation, the second information and the information related to the luma component may be fused (e.g., concatenated), and the fusion result is input to the encoding network.

[0225] In one possible implementation, the fusion is obtained by concatenation in the channel dimension. Since the size of the second information is converted compared with the size of the first information, in order to fuse the second information and the information related to the luma component, the size of the information related to the luma component (i.e., the fifth information in this embodiment of the present application) is also converted to obtain information whose size is at least the same as the size of the second information in the spatial dimension (i.e., the sixth information in this embodiment of the present application), and as a result, the second information and the sixth information can be concatenated in the channel dimension.

[0226] Specifically, in one possible implementation, a fifth piece of information may be obtained, which is information relating to the luminance component, and a size transformation may be performed on the fifth piece of information in order to obtain the sixth piece of information. The size of the sixth piece of information in spatial dimensions is the same as the size of the second piece of information. Furthermore, the fusion result of the second and fourth pieces of information (e.g., concatenated result) may be encoded using an encoding network.

[0227] 304: A third piece of information is obtained, which includes information about the color difference components of the second image format, and the second image format is different from the first image format.

[0228] In this embodiment of the present application, the third information may include information relating to the color difference components of a second image in a second image format, and the second image may be different from or the same as the first image.

[0229] The resolution of the second image format may be lower than the resolution of the first image format. For a description of step 304, please refer to the description of step 301 in the above embodiment. Similarities are not described again in this specification.

[0230] 305: To obtain the fourth piece of information, a size transformation is performed on the third piece of information, so that the size of the fourth piece of information in the spatial and channel dimensions matches the size of the second piece of information.

[0231] In one possible implementation, the first image format is YUV444, the second image format is YUV422, the size of the fourth piece of information in the spatial dimension is smaller than the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is larger than the size of the third piece of information.

[0232] For example, in one possible implementation, the third information is information about the color difference components in YUV422, the size of the third information in the channel dimension is 2, the height dimension of the third information in the spatial dimension is H, the width dimension of the third information in the spatial dimension is W / 2, and the size may also be described as (2, H, W / 2); the number of channels of the fourth information is 8, the height dimension of the fourth information in the spatial dimension is H / 2, and the width dimension of the fourth information in the spatial dimension is W / 2.

[0233] In this example, information partitioning may be performed first on a third piece of information (whose size is (2,H,W / 2)) in order to obtain information with size (4,H / 2,W / 2), and then information duplication may be performed to obtain a fourth piece of information (8,H / 2,W / 2); or, in this example, information duplication may be performed first on a third piece of information (whose size is (2,H,W / 2)) in order to obtain information with size (4,H,W / 2), and then information partitioning may be performed to obtain a fourth piece of information (8,H / 2,W / 2).

[0234] In one possible implementation, the first image format is YUV422, the second image format is YUV420, the size of the fourth piece of information in the spatial dimension is equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is greater than the size of the third piece of information.

[0235] For example, in one possible implementation, the third information is information about the color difference component in YUV420, the size of the third information in channel dimensions is 2, the height dimension of the third information in spatial dimensions is H / 2, the width dimension of the third information in spatial dimensions is W / 2, and the size may also be described as (2, H / 2, W / 2); the number of channels of the fourth information is 8, the height dimension of the fourth information in spatial dimensions is H / 2, and the width dimension of the fourth information in spatial dimensions is W / 2.

[0236] In this example, to obtain information with size (8, H / 2, W / 2), information duplication may be performed on a third piece of information (whose size is (2, H / 2, W / 2)).

[0237] In one possible implementation, the amount of non-redundant information contained in the fourth piece of information is equal to the amount of non-redundant information contained in the third piece of information.

[0238] 306: Encode a fourth piece of information using an encoding network or a portion of an encoding network.

[0239] For a description of step 306, please refer to the description of step 303 in the previously described embodiment. Similarities are not described again in this specification.

[0240] It should be understood that steps 301 to 303 are the encoding process for the first information, and steps 304 to 306 are the encoding process for the third information. The temporal order relationship between the first information encoding process and the second information encoding process is not limited to the embodiments of this application. For example, steps 301 to 303 may be performed before steps 304 to 306, or steps 301 to 303 may be performed after steps 304 to 306, or steps 301 to 303 may be performed simultaneously with steps 304 to 306.

[0241] The image encoding method in this embodiment will be described below with reference to Figure 5. This method will be described using, as an example, the encoding processes for three image formats: YUV444, YUV422, and YUV420. In Figure 5, information regarding chromatic difference components and luminance information are exchanged after size conversion.

[0242] Please refer to Figure 5. This method includes the following:

[0243] An image in RGB format is input. The size of the image is [3,H,W], where 3 is the size of the channel dimension, H is the height of the spatial dimension, and W is the width of the spatial dimension.

[0244] The image in RGB format is converted by color conversion ColorTr into information in YUV444 format, that is, YUV[3,H,W]. YUV[3,H,W] may be divided into information related to the color difference component UV[2,H,W] and information related to the luminance component Y[1,H,W].

[0245] Information processing for component Y: x Y [1,H,W] is generated from Y[1,H,W] without downsampling, and transmitted to the encoding network (Signal Encoder_Y) for encoding by the encoding network.

[0246] Processing for component UV: One of the three formats may be selected during one processing procedure.

[0247] YUV444 encoding: Pixel unshuffle UPS(2,2) is performed on the input component UV (UV[2,H,W]) (the number of channels is 2, the height is H, and the width is W), to generate x UV [8,H / 2,W / 2]. Auxiliary information [Formula] (A consistent size in the spatial dimension with x UV is maintained) is generated by performing UPS(2,2) on Y[1,H,W], [Formula] and x UV are concatenated (concat) in the channel dimension, and then transmitted together to Signal Encoder_UV.

[0248] The input component UV(UV[2,H,W]) may be the first information described in the above embodiment, x UV [8,H / 2,W / 2] may be the second piece of information described in the above embodiment, and Y[1,H,W] may be the fifth piece of information described in the above embodiment, and auxiliary information

number

number

number

[0249] YUV420 encoding: Downsampling of input component UV (UV[2,H,W]) UV =[2,2] is executed to generate UV[2,H / 2,W / 2] in 420 format. x UV [8,H / 2,W / 2] is a 4x repeat (4)(cin) channel dimension. iis the channel before replication, i is the channel number, and cout is the channel after replication. 4i+j =cin i (where i=0~1, j=0~3) is generated by (because x UV (This is because the number of pixels required for the unified shape [8,H / 2,W / 2] is four times that of the current shape [2,H / 2,W / 2].) (Supplementary information)

number

number

[0250] UV[2,H / 2,W / 2] may be the third piece of information described in the above embodiment, x UV [8,H / 2,W / 2] may be the fourth piece of information described in the above embodiment, and Y[1,H,W] may be the fifth piece of information described in the above embodiment, and auxiliary information

number

number

number

[0251] In the YUV422 encoding process, downsampling is performed on UV[2,H,W]. UV =[1,2] is executed to generate UV[2,H,W / 2] in 422 format. UV'[4,H / 2,W / 2] is generated by UPS(2,1), then x UV [8,H / 2,W / 2] is a double-dimension repeat(2)(cin) of the channel dimension. i is the channel before replication, i is the channel number, and cout is the channel after replication. 2i+j =cin i (where i=0~3, j=0~1) is generated by (because x UV (This is because the number of pixels required for the unified shape [8,H / 2,W / 2] is twice that of the current shape [4,H / 2,W / 2].) (Supplementary information)

number

number

[0252] The input component UV(UV[2,H,W / 2]) may be the first information described in the above embodiment, x UV [8,H / 2,W / 2] may be the second piece of information described in the above embodiment, and Y[1,H,W] may be the fifth piece of information described in the above embodiment, and auxiliary information

number

number

number

[0253] UV[2,H,W / 2] may be the third piece of information described in the above embodiment, x UV [8,H / 2,W / 2] may be the fourth piece of information described in the above embodiment, and Y[1,H,W] may be the fifth piece of information described in the above embodiment, and auxiliary information

number

number

number

[0254] The image encoding method in this embodiment will be described below with reference to Figure 6. This method will be described using the encoding process of the YUV444 image format as an example. In Figure 6, unlike Figure 5, information regarding the chromatic difference component and luminance information are exchanged before the size conversion.

[0255] Please refer to Figure 6. This method includes the following:

[0256] Information regarding component Y and component UV may first be combined by channel, and then UPS is performed. The input picture is RGB[3,H,W], and RGB is converted to YUV[3,H,W] in 444 format by the color conversion ColorTr. If the input is in YUV444 format, this step is skipped.

[0257] Processing of information regarding component Y: x Y [1,H,W] is generated from Y[1,H,W] without downsampling and sent to Signal Encoder_Y.

[0258] Processing of information regarding UV components: Y[1,H,W] and UV[2,H,W] are joined in the channel dimension to obtain YUV[3,H,W], x UV [12,H / 2,W / 2] is generated by UPS(2,2) and sent to Signal Encoder_UV.

[0259] In Figure 6, information regarding component Y and component UV may first be combined by a channel, and for the subsequent step of UPS being performed, please refer to the description of information exchange between the primary component information and the secondary component information before size conversion in the embodiments described above.

[0260] Figure 7 shows one embodiment of an image decoding method according to one embodiment of the present application. As shown in Figure 7, the image decoding method provided in this embodiment of the present application includes the following steps.

[0261] 701: Retrieve the first piece of information based on the first bitstream.

[0262] The first bitstream may be the encoding result obtained by encoding the first image.

[0263] The first piece of information may be the output of the decoding network or the output of the intermediate layer of the decoding network.

[0264] On the decoder side, in the process of the decoding network decoding different image formats, the decoding network or the intermediate layer of the decoding network may output information of the same size (same size in channel dimensions and same size in spatial dimensions). In this embodiment of the present application, the output of the decoding network includes information about the chromatic difference component, the output of the intermediate layer of the decoding network is an intermediate quantity, and the information about the chromatic difference component or the intermediate quantity is first information.

[0265] In one possible implementation, during the process of a decoding network decoding different (or at least two) image formats, the decoding network or an intermediate layer of the decoding network may output information of the same size (same size in channel dimensions and same size in spatial dimensions). In addition, in order to obtain information about the chromatic difference components of different image formats, size transformations may be performed on the information to obtain information about the chromatic difference components that satisfy the size requirements specified in the corresponding image formats.

[0266] In existing implementations, when a decoding network performs decoding of different (or at least two) image formats, the decoding network or its intermediate layers output information of different sizes, resulting in varying computational power and time overhead. Consequently, the stability of information processing is insufficient. In this embodiment of the present application, the decoding network or its intermediate layers may output information of the same size (same channel dimension and same spatial dimension) to obtain information about the color difference components of different image formats, resulting in consistent (or nearly consistent) computational power and time overhead when the decoding network performs decoding of different (or at least two) image formats, thus improving the stability of information processing. This is superior in terms of both machine experience and user experience.

[0267] In addition, there is no coupling relationship between the input and output. The input of YUV444 can produce both the output of 444 and the output of 420, or the output of 422. Furthermore, in this embodiment of the present application, the complexity of decoding 444 and 422 is approximately the same as the complexity of decoding 420, and is lower than the complexity of decoding 444 and 422 by using the original JPEG AI.

[0268] Decoding networks are sometimes called signal decoders. Decoders are implemented based on neural networks and are configured to decode input information (e.g., entropy decoding results) to obtain YUV information (or sometimes called performing feature reconstruction). Because decoders are implemented based on neural networks, they are sometimes called AI decoders.

[0269] 702: A size transformation is performed on the first information to obtain the second information, and the second information includes information about the color difference components of the first image format.

[0270] In this embodiment of the present application, in contrast to the embodiment corresponding to Figure 3, the size transformation performed on the first information can increase the size of the information in the spatial dimension (optionally, for information in YUV420 format, the size of the spatial dimension may remain unchanged) and decrease the size of the information in the channel dimension. The size of the second information in the spatial dimension is greater than or equal to the size of the first information, and the size of the second information in the channel dimension is less than the size of the first information.

[0271] The spatial dimension may further include height and width dimensions, and the size of the second piece of information in the spatial dimension being larger than the size of the first piece of information can be understood as the size of the second piece of information in at least one of the height and width dimensions being larger than the size of the first piece of information (naturally, the size of the second piece of information in one of the spatial dimensions cannot be smaller than the size of the first piece of information).

[0272] For example, the height dimension of the second piece of information in spatial dimensions is greater than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is greater than the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is smaller than the size of the first piece of information.

[0273] For example, the height dimension of the second piece of information in spatial dimensions is greater than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is equal to the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is smaller than the size of the first piece of information.

[0274] The two examples mentioned above can be understood as cases where the size of the second piece of information in spatial dimension is larger than the size of the first piece of information.

[0275] The size transformation performed on the first information in this embodiment of the present application will be described in detail below.

[0276] The size conversion in this embodiment of the present application allows information of the same size to be converted into information of a size specified in the corresponding image format.

[0277] Size transformations can involve operations such as information merging, information selection, and convolution. These are described separately below.

[0278] 1. Information Merging In one possible implementation, performing a size transformation on the first piece of information may involve combining different data from multiple channels into the same channel.

[0279] In other words, to reduce the number of channels in the first information while ensuring that the overall amount of information remains unchanged, the channels in the first information may be reduced, and the information from the reduced channels must be merged onto the existing channels in the first information. In this case, the information from some channels in the first information may be extracted and merged with the information from other channels, thereby reducing the size of the first information in the channel dimension.

[0280] In one possible implementation, the size conversion method may include a pixel shuffle (PS), i.e., UPS may be performed on the first information.

[0281] Figure 8a shows the processing result of PS. In Figure 8a, an example is used to illustrate that information from every fourth channel within the 4C channels in the first information is merged into data in one channel, where four different data points at the same position on the four channels are merged into one channel. In this case, the number of channels is also changed from 4C to C. If the spatial size of the first information is H / 2*W / 2, the number of channels in the second information obtained by the size transformation is H*W.

[0282] In one possible implementation, the second information is information about the color difference components of YUV444, the size of the second information in channel dimensions is 2, the height dimension of the second information in spatial dimensions is H, and the width dimension of the second information in spatial dimensions is W; the number of channels in the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, and the width dimension of the first information in spatial dimensions is W / 2. The size conversion scheme in this example may be the same as that shown in Figure 8a. Details are not repeated here.

[0283] 2. Information Selection Selection involves selecting information from certain channels within the first set of information and then merging that information into new data where the size of the spatial dimension remains unchanged but the size of the channel dimension decreases. This is equivalent to deleting the information from the unselected channels and retaining only the information from the selected channels.

[0284] In one possible implementation, in addition to increasing the number of channels by information merging, further information selection may be performed during size conversion. This approach is more necessary when the second information is a low-resolution image format (at least not the highest resolution, e.g., YUV422, YUV420). This is because the amount of information (i.e., information size) regarding the chrominance components of a low-resolution image format is less than the amount of information regarding the chrominance components of a high-resolution image format. If only operations such as information merging are performed that do not reduce the amount of data (note that the amount of data and the amount of non-redundant information mentioned above are not the same concept; the amount of data is the total amount of information contained in the information, counting both redundant and non-redundant information), the size of the information regarding the chrominance components of a low-resolution (at least not the highest resolution) image format will certainly be smaller than the size of the information regarding the chrominance components of a high-resolution image format. As a result, the size requirements specified by the image format cannot be met. In this case, when the information regarding the chrominance components of a low-resolution (at least not the highest resolution) image format is reconstructed, a selection operation may be performed on the first information to reduce the amount of data. The information obtained by the selection operation may also have a reduced amount of non-redundant information compared to the data before selection.

[0285] In one possible implementation, information selection may be performed on the first information, and different data from multiple channels may be combined into the same channel. There is no restriction on the time order between the two operations, nor is there a limit to the number of operations of each type. For example, if the information can be transformed to the size that needs to be transformed, information selection may be performed first, followed by information merging, and then selection. The types of operations, the amount of operations, the combination of operations, and the order of operations are not limited to the embodiments of this application.

[0286] For example, in one possible implementation, the second information is information about the color difference components in YUV422, the size of the second information in the channel dimension is 2, the height dimension of the second information in the spatial dimension is H, the width dimension of the second information in the spatial dimension is W / 2, and the size may also be described as (2, H, W / 2); the number of channels of the first information is 8, the height dimension of the first information in the spatial dimension is H / 2, and the width dimension of the first information in the spatial dimension is W / 2.

[0287] In this example, to obtain information with a size of (4, H / 2, W / 2), information selection may be performed first on the first piece of information (whose size is (8, H / 2, W / 2)), and then information merging may be performed to obtain the second piece of information (2, H, W / 2); or, in this example, to obtain information with a size of (4, H, W / 2), information merging may be performed first on the first piece of information (whose size is (8, H / 2, W / 2)), and then information selection may be performed to obtain the second piece of information (2, H, W / 2).

[0288] In one possible implementation, the aforementioned operations of information selection or information merging may be replaced by convolution.

[0289] In one possible implementation, size conversion may be performed by an alternative method, provided that the amount of non-redundant information remains unchanged and it is guaranteed that the information can be converted to the size to which it needs to be converted. This is not limited to the present application.

[0290] Step 702 may be performed independently of the decoding network, or it may be performed within the decoding network, for example, using an internal network layer close to the output of the decoding network (e.g., a network layer before the last convolutional layer of the decoding network, or a network layer before the last pixel shuffle layer of the decoding network). These will be explained separately below.

[0291] 1. Step 702 is an operation independent of the decoding network. In one possible implementation, if step 702 operates independently of the decoding network, the decoding network may output the first information. Compared to conventional techniques, step 702 is an additional step that follows the decoding network.

[0292] 2. Step 702 is performed using a network layer that is close to the output of the decoding network and is located within the decoding network. In one possible implementation, when step 702 is performed using a network layer that is close to the output of the decoding network and is located within the decoding network, the existing network layer that is close to the output of the decoding network and is located within the decoding network may be improved. For example, after the calculation values ​​of the UPS layer or convolutional layer are changed, the first information is the output of the network layer in the composite transformation network.

[0293] 703: Reconstruct the first image based on the second piece of information.

[0294] When converting to RGB, further steps such as upsampling, combining information on color difference components and luminance components, ICCI filtering, and generating an RGB image using the color conversion module invColorTr may be performed, but are not limited to these.

[0295] 704: A third piece of information is obtained based on the second bitstream, and the size of the third piece of information in the spatial and channel dimensions matches the size of the first piece of information.

[0296] The second bitstream may be the encoded result obtained by encoding the second image.

[0297] The third piece of information may be the output of the decoding network or the output of the intermediate layer of the decoding network.

[0298] 705: A size transformation is performed on the third information to obtain the fourth piece of information, and the fourth piece of information includes information about the color difference components of the second image format, which is different from the first image format.

[0299] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0300] For the relationship and explanation between the first and second image formats, please refer to the description of the embodiments above. Details will not be repeated here.

[0301] In one possible implementation, the size of the fourth piece of information in the spatial dimension is greater than or equal to the size of the third piece of information, while the size of the fourth piece of information in the channel dimension is less than the size of the third piece of information.

[0302] In one possible implementation, the amount of non-redundant information contained in the fourth piece of information is equal to the amount of non-redundant information contained in the third piece of information.

[0303] 706: Reconstruct the second image based on the fourth piece of information.

[0304] It should be understood that steps 701 to 703 are the decoding process for the first bitstream, and steps 704 to 706 are the decoding process for the second bitstream. The temporal order relationship between the decoding process for the first bitstream and the decoding process for the second bitstream is not limited to the embodiments of this application. For example, steps 701 to 703 may be performed before steps 704 to 706, or steps 701 to 703 may be performed after steps 704 to 706, or steps 701 to 703 may be performed simultaneously with steps 704 to 706.

[0305] Figure 8b shows one embodiment of an image decoding method according to one embodiment of the present application. As shown in Figure 8b, the image decoding method provided in this embodiment of the present application includes the following steps:

[0306] 801: Retrieve the first piece of information based on the first bitstream.

[0307] The first piece of information may be the output of the decoding network or the output of the intermediate layer of the decoding network.

[0308] On the decoder side, in the process of the decoding network decoding different image formats, the decoding network or the intermediate layer of the decoding network may output information of the same size (same size in channel dimensions and same size in spatial dimensions). In this embodiment of the present application, the output of the decoding network includes information about the chromatic difference component, the output of the intermediate layer of the decoding network is an intermediate quantity, and the information about the chromatic difference component or the intermediate quantity is first information.

[0309] In one possible implementation, during the process of a decoding network decoding different (or at least two) image formats, the decoding network or an intermediate layer of the decoding network may output information of the same size (same size in channel dimensions and same size in spatial dimensions). In addition, in order to obtain information about the chromatic difference components of different image formats, size transformations may be performed on the information to obtain information about the chromatic difference components that satisfy the size requirements specified in the corresponding image formats.

[0310] In existing implementations, when a decoding network performs decoding of different (or at least two) image formats, the decoding network or its intermediate layers output information of different sizes, resulting in varying computational power and time overhead. Consequently, the stability of information processing is insufficient. In this embodiment of the present application, the decoding network or its intermediate layers may output information of the same size (same channel dimension and same spatial dimension) to obtain information about the color difference components of different image formats, resulting in consistent (or nearly consistent) computational power and time overhead when the decoding network performs decoding of different (or at least two) image formats, thus improving the stability of information processing. This is superior in terms of both machine experience and user experience.

[0311] In addition, a size transformation may be performed on the first information (e.g., the first information) obtained from the same bitstream in order to generate color difference information for multiple image formats. In this way, there is no coupling relationship between the input and output. For example, a YUV444 input can produce both a 444 output and a 420 output, or a 422 output. Furthermore, in this embodiment of the present application, the complexity of decoding 444 and 422 is approximately the same as the complexity of decoding 420, and is lower than the complexity of decoding 444 and 422 by using the original JPEG AI.

[0312] A coding-synthesizing-transformation network, or signal decoder, is implemented based on a neural network and configured to decode input information (e.g., the entropy decoding result) to obtain YUV information (sometimes called performing feature reconstruction). Because decoders are implemented based on neural networks, they are sometimes called AI decoders.

[0313] 802: Perform a size transformation on the first information in order to obtain one or more of the second, third, or fourth pieces of information, the second piece of information being information about the color difference components of the first image format, the third piece of information being information about the color difference components of the second image format, the fourth piece of information being information about the color difference components of the third image format, and the first, second, and third image formats being different from each other.

[0314] For example, when a scale transformation is performed, at least one of the following operations may be performed to obtain information about the color difference of one or more image formats. In other words, in this embodiment of the present application, for the same information piece, information about the color difference of multiple image formats may be obtained using different scale transformation schemes in order to reconstruct images of the corresponding image formats.

[0315] To obtain the second piece of information, a first size transformation is performed on the first piece of information, and the second piece of information includes information about the color difference components of the first image format.

[0316] To obtain the second piece of information, a second size transformation is performed on the first piece of information, and the second piece of information includes information about the color difference components of the first image format.

[0317] A third size transformation is performed on the first information to obtain the second information, and the second information includes information about the color difference components of the first image format.

[0318] The second piece of information is used as an example. In this embodiment of the present application, in contrast to the embodiment corresponding to Figure 3, the size transformation performed on the first piece of information can increase the size of the information in the spatial dimension (optionally, for information in YUV420 format, the size of the spatial dimension may remain unchanged) and decrease the size of the information in the channel dimension. The size of the second piece of information in the spatial dimension is greater than or equal to the size of the first piece of information, and the size of the second piece of information in the channel dimension is less than the size of the first piece of information.

[0319] The spatial dimension may further include height and width dimensions, and the size of the second piece of information in the spatial dimension being larger than the size of the first piece of information can be understood as the size of the second piece of information in at least one of the height and width dimensions being larger than the size of the first piece of information (naturally, the size of the second piece of information in one of the spatial dimensions cannot be smaller than the size of the first piece of information).

[0320] For example, the height dimension of the second piece of information in spatial dimensions is greater than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is greater than the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is smaller than the size of the first piece of information.

[0321] For example, the height dimension of the second piece of information in spatial dimensions is greater than the height dimension of the first piece of information, the width dimension of the second piece of information in spatial dimensions is equal to the width dimension of the first piece of information, and the size of the second piece of information in channel dimensions is smaller than the size of the first piece of information.

[0322] The two examples mentioned above can be understood as cases where the size of the second piece of information in spatial dimension is larger than the size of the first piece of information.

[0323] The size transformation performed on the first information in this embodiment of the present application will be described in detail below.

[0324] The size conversion in this embodiment of the present application allows information of the same size to be converted into information of a size specified in the corresponding image format.

[0325] Size transformations can involve operations such as information merging, information selection, and convolution. These are described separately below.

[0326] 1. Information Merging In one possible implementation, performing a size transformation on the first piece of information may involve combining different data from multiple channels into the same channel.

[0327] In other words, to reduce the number of channels in the first information while ensuring that the overall amount of information remains unchanged, the channels in the first information may be reduced, and the information from the reduced channels must be merged onto the existing channels in the first information. In this case, the information from some channels in the first information may be extracted and merged with the information from other channels, thereby reducing the size of the first information in the channel dimension.

[0328] In one possible implementation, the size conversion method may include a pixel shuffle (PS), i.e., UPS may be performed on the first information.

[0329] Figure 8a shows the processing result of PS. In Figure 8a, an example is used to illustrate that the information from four channels in the first information is merged into data from one channel, where four data fragments at the same position on the four channels are merged into a data block of one channel. In this case, the number of channels is also changed from 4 to 1. If the number of channels in the first information is 4C, the number of channels in the second information obtained by the size transformation is C. If the spatial size of the first information is H / 2*W / 2, the number of channels in the second information obtained by the size transformation is H*W.

[0330] In one possible implementation, the second information is information about the color difference components of YUV444, the size of the second information in channel dimensions is 2, the height dimension of the second information in spatial dimensions is H, and the width dimension of the second information in spatial dimensions is W; the number of channels in the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, and the width dimension of the first information in spatial dimensions is W / 2. The size conversion scheme in this example may be the same as that shown in Figure 8a. Details are not repeated here.

[0331] 2. Information Selection Selection involves selecting information from certain channels within the first set of information and then merging that information into new data where the size of the spatial dimension remains unchanged but the size of the channel dimension decreases. This is equivalent to deleting the information from the unselected channels and retaining only the information from the selected channels.

[0332] In one possible implementation, in addition to increasing the number of channels by information merging, further information selection may be performed during size conversion. This approach is more necessary when the second information is a low-resolution image format (at least not the highest resolution, e.g., YUV422, YUV420). This is because the amount of information (i.e., information size) regarding the chrominance components of a low-resolution image format is less than the amount of information regarding the chrominance components of a high-resolution image format. If only operations such as information merging are performed that do not reduce the amount of data (note that the amount of data and the amount of non-redundant data mentioned above are not the same concept; the amount of data is the total amount of information contained in the information, counting both redundant and non-redundant data), the size of the information regarding the chrominance components of a low-resolution (at least not the highest resolution) image format will certainly be smaller than the size of the information regarding the chrominance components of a high-resolution image format. As a result, the size requirements specified by the image format cannot be met. In this case, when the information regarding the chrominance components of a low-resolution (at least not the highest resolution) image format is reconstructed, a selection operation may be performed on the first information to reduce the amount of data. The information obtained by the selection operation may also have a reduced amount of non-redundant data compared to the data before selection.

[0333] In one possible implementation, information selection may be performed on the first information, and different data from multiple channels may be combined into the same channel. There is no restriction on the time order between the two operations, nor is there a limit to the number of operations of each type. For example, information selection may be performed first, followed by information merging, and then selection, as long as it is guaranteed that the information can be transformed to the size to which it needs to be transformed. The types of operations, the number of operations, the combination of operations, and the order of operations are not limited to the embodiments of this application.

[0334] For example, in one possible implementation, the second information is information about the color difference components in YUV422, the size of the second information in the channel dimension is 2, the height dimension of the second information in the spatial dimension is H, the width dimension of the second information in the spatial dimension is W / 2, and the size may also be described as (2, H, W / 2); the number of channels of the first information is 8, the height dimension of the first information in the spatial dimension is H / 2, and the width dimension of the first information in the spatial dimension is W / 2.

[0335] In this example, to obtain information with a size of (4, H / 2, W / 2), information selection may be performed first on the first piece of information (whose size is (8, H / 2, W / 2)), and then information merging may be performed to obtain the second piece of information (2, H, W / 2); or, in this example, to obtain information with a size of (4, H, W / 2), information merging may be performed first on the first piece of information (whose size is (8, H / 2, W / 2)), and then information selection may be performed to obtain the second piece of information (2, H, W / 2).

[0336] In one possible implementation, the aforementioned operations of information selection or information merging may be replaced by convolution.

[0337] In one possible implementation, size conversion may be performed by an alternative method, provided that the amount of non-redundant data of the information remains unchanged and it is guaranteed that the information can be converted to the size to which it needs to be converted. This is not limited to the present application.

[0338] Step 802 may be performed independently of the decoding network, or it may be performed within the decoding network, for example, using an internal network layer close to the output of the decoding network (e.g., a network layer before the last convolutional layer of the decoding network, or a network layer before the last pixel shuffle layer of the decoding network). These will be explained separately below.

[0339] 1. Step 802 is an operation independent of the decoding network. In one possible implementation, if step 802 operates independently of the decoding network, the decoding network may output the first information. Compared to conventional techniques, step 802 is an additional step that follows the decoding network.

[0340] 2. Step 802 is performed using a network layer that is close to the output of the decoding network and is located within the decoding network. In one possible implementation, when step 802 is performed using a network layer that is close to the output of the decoding network and is located within the decoding network, the existing network layer that is close to the output of the decoding network and is located within the decoding network may be improved. For example, after the calculation values ​​of the UPS layer or convolutional layer are changed, the first information is the output of the network layer in the composite transformation network.

[0341] 803: Perform one or more of the following steps: reconstruct the first image based on the second piece of information, reconstruct the second image based on the third piece of information, or reconstruct the third image based on the fourth piece of information.

[0342] In one possible implementation, the image can be reconstructed based on at least one of the second, third, or fourth pieces of information.

[0343] When converting to RGB, further steps such as upsampling, combining information on color difference components and luminance components, ICCI filtering, and generating an RGB image using the color conversion module invColorTr may be performed, but are not limited to these.

[0344] The image decoding method in this embodiment will be described below with reference to Figure 9. As an example, the method will be described using the decoding processes for three image formats: YUV444, YUV422, and YUV420. In Figure 9, size conversion is performed by a separate module independent of the decoding network.

[0345] The decoding network (Signal Decoder) reconstructs the Y and UV (i.e., the first information in the embodiment corresponding to Figure 7). The reconstructed UV features have a unified shape [8,H / 2,W / 2], and the UV features with the unified shape are then converted to different shape formats (YUV444, YUV420, and YUV422) using pixel shuffle (PS).

[0346] Please refer to Figure 9. This method includes the following:

[0347]

number

number

[0348] Information processing for component Y: The Signal Decoder is,

number

number

[0349] Treatment for UV components: During a single process, one of the three formats may be selected.

[0350] YUV420 Decoding: Signal Decoder is

number

number

number

number

number

number

[0351]

number

number

number

number

number

[0352] YUV444 Decoding: Signal Decoder is

number

number

number

number

[0353]

number

number

number

number

number

[0354] YUV422 Decoding: Signal Decoder is

number

number

number

number

number

[0355]

number

number

number

number

number

[0356] Optional,

number

number

number

number

number

number

number

number

[0357] The image decoding method in this embodiment will be described below with reference to Figure 10. Unlike Figure 9, in Figure 10, size conversion is performed using the pixel shuffle PS layer of the decoding network.

[0358] In existing implementations, the last layer of the decoding network signal decoder is a pixel shuffle PS(4,4) layer, which may be combined with a size transformation PS in this embodiment of the present application. In this embodiment, the size transformation operation and the last layer PS in the signal decoder may be merged into a single layer PS.

[0359] Please refer to Figure 10. This method includes the following:

[0360] Information processing for component Y:

[0361] The Signal Decoder is,

number

number

[0362] Treatment for UV components: During a single process, one of the three formats may be selected.

[0363] YUV420 Decoding: Part 1 of the Signal Decoder (all layers except the pixel shuffle layer in the last layer) outputs the feature L[128,H / 8,W / 8]. To generate L'[32,H / 8,W / 8], 32 channels are selected from L (channels corresponding to the value obtained by dividing 0 by 4 are selected from channels 0 to 127) (the number of UV pixels required for the 420 format is 1 / 4 of the number of UV pixels required for the 444 format), and PS(4,4)

number

number

number

number

[0364] Feature L may be the first piece of information described in the embodiment of Figure 7.

number

number

[0365] YUV444 Decoding: Part 1 of the Signal Decoder (all layers except the last layer, pixel shuffle) outputs L[128,H / 8,W / 8]. PS(8,8) is,

number

number

[0366] Feature L may be the first piece of information described in the embodiment of Figure 7.

number

number

number

[0367] YUV422 Decoding: Part 1 of the Signal Decoder (all layers except the last layer, pixel shuffle) outputs L[128, H / 8, W / 8].

number

number

number

number

number

[0368] Feature L may be the first piece of information described in the embodiment of Figure 7.

number

number

[0369] The image decoding method in this embodiment will be described below with reference to Figure 11. Unlike Figure 10, in Figure 11, size transformation is performed using the convolutional layer of the decoding network.

[0370] This embodiment describes possible modifications to the decoder structure. In this embodiment, the feature L reconstructed by part 1 of the decoder no longer has a unified number of channels, but the decoder Signal Decoder reconstructs L using different channels according to different required output formats. Part 1 of the decoder includes all layers except the last pixel shuffle (PS) layer and the last convolutional layer.

[0371] Please refer to Figure 9. This method includes the following:

[0372] Information processing for component Y: The Signal Decoder is,

number

number

[0373] Treatment for UV components: During a single process, one of the three formats may be selected.

[0374] YUV420 Decoding: Part 1 in the Signal Decoder outputs the feature L'[Ch,H / 8,W / 8], and L'[32,H / 8,W / 8] is generated by using a convolutional layer CONV(3×3,32) with 32 output channels. PS(4,4) is,

number

number

number

number

[0375] L' may be the first piece of information described in the embodiment of Figure 7,

number

number

[0376] YUV444 Decoding: Part 1 in the Signal Decoder outputs the feature L'[Ch,H / 8,W / 8], and L'[128,H / 8,W / 8] is generated by using a convolutional layer CONV(3×3,128) with 128 output channels. PS(8,8) is,

number

number

[0377] L' may be the first piece of information described in the embodiment of Figure 7,

number

number

[0378] YUV422 Decoding: Part 1 in the Signal Decoder outputs the feature L'[Ch,H / 8,W / 8], and L'[64,H / 8,W / 8] is generated by using a convolutional layer CONV(3×3,64) with 64 output channels. PS(8,4) is,

number

number

number

number

[0379] L' may be the first piece of information described in the embodiment of Figure 7,

number

number

[0380] The encoding method and decoding method in the embodiments of this application have been described above. The apparatus 1200 in one embodiment of this application will now be described. Please refer to Figure 12. One embodiment of the encoding apparatus in one embodiment of this application includes the following modules.

[0381] The acquisition module 1201 is configured to acquire first information, which includes information about the color difference components of a first image format, and to acquire third information, which includes information about the color difference components of a second image format, wherein the second image format is different from the first image format.

[0382] For a description of the acquisition module 1201, please refer to the descriptions of steps 301 and 304 of the above embodiment. Details will not be repeated here.

[0383] The processing module 1202 is configured to perform a size transformation on the first information to obtain the second information, encode the second information using the encoding network or a part of the encoding network, perform a size transformation on the third information to obtain the fourth information, the size of the fourth information in the spatial and channel dimensions matches the size of the second information, and encode the fourth information using the encoding network.

[0384] For a description of the processing module 1202, please refer to the descriptions of steps 302, 303, 305, and 306 in the above-described embodiment. Details will not be repeated here.

[0385] In one possible implementation, the size of the second piece of information in the spatial dimension is smaller than the size of the first piece of information, the size of the second piece of information in the channel dimension is larger than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is less than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is larger than the size of the third piece of information.

[0386] In one possible implementation, the amount of non-redundant information in the second piece of information is equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is equal to the amount of non-redundant information in the third piece of information.

[0387] In one possible implementation, size conversion is performed. Performing information replication and using the replicated information as information on the new channel, Splitting information on at least one channel into information on multiple different channels, or Performing a convolution operation It includes at least one of the following.

[0388] In one possible implementation, the operation of splitting information on at least one channel into information on multiple different channels is performed by using a pixel unshuffle layer.

[0389] In one possible implementation, the first image format is the YUV444 format, and the second image format is either the YUV422 or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format.

[0390] In one possible implementation, the number of channel dimensions for the second and fourth pieces of information is 8, the height dimension of the second piece of information in spatial dimensions is H / 2, the width dimension of the second piece of information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0391] In one possible implementation, the acquisition module 1201 is: The fifth piece of information is obtained, and this fifth piece of information is information about the luminance component Y. It is further configured in this way, The processing module is further configured to perform a size transformation on the fifth information in order to obtain the sixth information, such that the size of the sixth information in spatial dimensions matches the size of the second information. The processing module is specifically configured to encode the fusion result of the second and fourth pieces of information using an encoding network.

[0392] In one possible implementation, the first or third piece of information further includes information relating to the luminance component.

[0393] Please refer to Figure 13. One embodiment of the decoding device 1300 in one embodiment of this application includes the following module.

[0394] The acquisition module 1301 is configured to acquire first information based on a first bitstream, and third information based on a second bitstream, such that the size of the third information in the spatial and channel dimensions matches the size of the first information.

[0395] For a description of the acquisition module 1301, please refer to the descriptions of steps 701 and 704 of the above embodiment. Details will not be repeated here.

[0396] The processing module 1302 is configured to perform a size transformation on the first information in order to obtain a second piece of information, the second piece of information containing information about the color difference components in the first image format, to reconstruct the first image based on the second piece of information, to obtain a third piece of information based on the second bitstream, the size of the third piece of information in spatial and channel dimensions matching the size of the first piece of information, to perform a size transformation on the third information in order to obtain a fourth piece of information, the fourth piece of information containing information about the color difference components in the second image format, the second image format being different from the first image format, and to reconstruct the second image based on the fourth piece of information.

[0397] For a description of the processing module 1302, please refer to the descriptions of steps 702, 703, 705, and 706 in the above-described embodiment. Details will not be repeated here.

[0398] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the fourth piece of information in the spatial dimension is greater than or equal to the size of the third piece of information, and the size of the fourth piece of information in the channel dimension is less than the size of the third piece of information.

[0399] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the fourth piece of information is less than or equal to the amount of non-redundant information in the third piece of information.

[0400] In one possible implementation, size conversion is as follows: A method of deleting some of the information in a channel. A method of combining different data from multiple channels into the same channel, or Method for performing convolution operations It includes at least one of the following.

[0401] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0402] In one possible implementation, the first image format is the YUV444 format, and the second image format is either the YUV422 or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format.

[0403] In one possible implementation, the number of channel dimensions of the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, and the width dimension of the first information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0404] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0405] In addition, one embodiment of this application further provides an image decoding device. This device is An acquisition module configured to acquire first information based on a first bitstream, A processing module configured to perform a size transformation on first information in order to obtain one or more of second, third, or fourth pieces of information, wherein the second piece of information is information about the color difference components of a first image format, the third piece of information is information about the color difference components of a second image format, and the fourth piece of information is information about the color difference components of a third image format, and the first, second, and third image formats are different from each other. Includes.

[0406] In one possible implementation, the size of the second piece of information in the spatial dimension is greater than the size of the first piece of information, the size of the second piece of information in the channel dimension is less than the size of the first piece of information, the size of the third piece of information in the spatial dimension is greater than or equal to the size of the first piece of information, and the size of the third piece of information in the channel dimension is less than the size of the first piece of information.

[0407] In one possible implementation, the amount of non-redundant information in the second piece of information is less than or equal to the amount of non-redundant information in the first piece of information, and the amount of non-redundant information in the third piece of information is less than or equal to the amount of non-redundant information in the first piece of information.

[0408] In one possible implementation, the processing module is: Reconstruct the first image based on at least one of the second, third, or fourth pieces of information. It is further configured in this way.

[0409] In one possible implementation, the size transformation includes at least one of the following: a method of deleting some information from a channel; a method of combining different data from multiple channels into the same channel; and a method of performing a convolution operation.

[0410] In one possible implementation, the operation of combining different data from multiple channels into the same channel is performed by using a pixel shuffling layer.

[0411] In one possible implementation, the first image format is the YUV444 format and the second image format is the YUV422 or YUV420 format, or the first image format is the YUV422 format and the second image format is the YUV420 format.

[0412] In one possible implementation, the number of channel dimensions of the first information is 8, the spatial dimensions of the first information include height and width, the height of the first information is H / 2, and the width of the first information is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

[0413] In one possible implementation, the intermediate layer is the network layer preceding the last convolutional layer of the decoding network or the network layer preceding the last pixel shuffle layer of the decoding network.

[0414] The following describes a device provided in embodiments of this application. The device may be an encoding device or a decoding device. Figure 14 is a diagram of the structure of a device according to one embodiment of this application. The device 1400 may be specifically represented as a virtual reality (VR) device, a mobile phone, a tablet computer, a notebook computer, an intelligent wearable device, etc., but is not limited thereto. The device described in the embodiment corresponding to Figure 12 or Figure 13 may be deployed on the device 1400. Specifically, the device 1400 includes a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (the device 1400 may have one or more processors 1403, and Figure 14 uses an example with one processor). The processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of this application, the receiver 1401, transmitter 1402, processor 1403, and memory 1404 may be connected to each other via a bus or in another manner.

[0415] Memory 1404 includes read-only memory and random access memory, and may provide instructions and data to processor 1403. A portion of memory 1404 may further include non-volatile random access memory (NVRAM). Memory 1404 stores processor instructions and operation instructions, executable modules or data structures, or subsets thereof, or extensions thereof. Operation instructions may include various operation instructions for performing various operations.

[0416] Processor 1403 controls the operation of the device. In certain applications, the components of the device are coupled to each other through a bus system. In addition to the data bus, the bus system may further include a power bus, a control bus, a state signal bus, and so on. However, for the sake of clarity, various buses are referred to as the bus system in the diagram.

[0417] The methods disclosed in the aforementioned embodiments of this application may be applied to or implemented by a processor 1403. The processor 1403 may be an integrated circuit chip having signal processing capabilities. In the implementation process, the steps in the aforementioned methods may be implemented using hardware integrated logic circuits within the processor 1403 or using instructions in the form of software. The processor 1403 may be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, or may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 1403 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps in the methods disclosed with reference to embodiments of this application may be performed and completed directly by a hardware decoding processor, or by using a combination of hardware and software modules within the decoding processor. The software modules may reside in mature storage media of the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory 1404, and the processor 1403 reads information from memory 1404 and completes the steps in the aforementioned methods together with the hardware of the processor 1403.

[0418] The receiver 1401 may be configured to receive input numeric or character information and generate signal inputs for associated settings and function controls of the device. The transmitter 1402 may be configured to output numeric or character information via the first interface. The transmitter 1402 may be further configured to send commands to a disk group via the first interface to modify data in the disk group. The transmitter 1402 may further include a display device such as a display.

[0419] One embodiment of this application further provides a device, which may be an encoding device or a decoding device. Figure 15 is a diagram of the structure of a device according to one embodiment of this application. The device described in the embodiment corresponding to Figure 12 or Figure 13 may be deployed on device 1500. Specifically, device 1500 is implemented by one or more servers. Device 1500 may vary considerably in configuration or performance and may include one or more central processing units (CPUs) 1522 (e.g., one or more processors), memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 may perform temporary or persistent storage. The program stored in storage media 1530 may include one or more modules (not shown), each module may include a set of instruction operations for the device. Furthermore, the central processing unit 1522 may be configured to communicate with the storage medium 1530 and execute a series of instruction operations within the storage medium 1530 on the device 1500.

[0420] Device 1500 may further include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541 such as Windows Server®, Mac OS X®, Unix®, Linux®, or FreeBSD®.

[0421] One embodiment of this application further provides a computer program product (or referred to as a computer program) that stores one or more computers. When the computer program product is executed by a processor, the processor performs the method in a possible implementation of the aforementioned image processing device.

[0422] One embodiment of this application further provides a chip system. The chip system includes at least one processor configured to support a terminal device in performing the functions of the aforementioned image processing device in a possible implementation. Optionally, the chip system further includes an interface circuit, which provides program instructions and / or data to the at least one processor. In a possible design, the chip system may further include memory, which is configured to store program instructions and data required by the terminal device. The chip system may include a chip, or a chip and other separate components.

[0423] One embodiment of this application further provides a readable storage medium. The readable medium stores a bitstream obtained by encoding according to the method of the embodiment shown in Figures 1a to 6.

[0424] One embodiment of the present application further provides a bitstream storage device comprising at least one storage medium and a communication interface. The communication interface is configured to receive or transmit a bitstream. At least one storage medium is configured to store a bitstream. The bitstream is obtained by encoding it with an encoder according to the method of the embodiments shown in Figures 1a to 6.

[0425] One embodiment of this application further provides a bitstream storage method. The method includes the steps of: receiving a bitstream via a communication interface; and storing the bitstream in one or more storage media, wherein the bitstream is obtained by encoding it with an encoder according to the methods of the embodiments shown in Figures 1a to 6.

[0426] One embodiment of this application further provides a bitstream distribution system. The system includes at least one storage medium and a video stream device. The at least one storage medium is configured to store a bitstream, which is obtained by encoding with an encoder according to the methods of the embodiments shown in Figures 1a to 6. The video stream device is configured to respond to a request from a decoder, thereby transmitting a target bitstream in the at least one storage medium to the decoder.

[0427] One embodiment of this application further provides a bitstream distribution method. The method includes the steps of receiving a first request, selecting a target bitstream from at least one storage medium in response to the first request, and transmitting the target bitstream to a destination device. The at least one storage medium is configured to store a bitstream, which is obtained by encoding with an encoder according to the methods of the embodiments shown in Figures 1a to 6.

[0428] One embodiment of the present application further provides a bitstream processing system. The system includes an image source device, an encoder device, one or more storage media, and a destination device. The image source device is configured to provide image data. The encoder device is configured to acquire image data from the image source device via an interface and to encode the image data to acquire one or more bitstreams, which are obtained by encoding the bitstreams with an encoder according to the methods of the embodiments shown in Figures 1a to 13. The encoder device is configured to store one or more bitstreams in one or more storage media. Alternatively, the encoder device is configured to encapsulate one or more bitstreams to acquire a transmit bitstream. The encoder device is configured to transmit the transmit bitstream to the destination device via a communication link or communication network. The destination device is configured to decapsulate the transmit bitstream to acquire one or more bitstreams. The destination device is configured to decode one or more bitstreams to acquire the decoded data.

[0429] For the purpose of a simplified explanation, it will be readily apparent to those skilled in the art that detailed working processes of the aforementioned systems, apparatus, and units should be referred to the corresponding processes in the embodiments of the methods described above. Details will not be repeated here.

[0430] It should be understood that in some embodiments provided in this application, the disclosed systems, apparatus, and methods may be implemented in other ways. For example, the embodiments of the apparatus described above are merely examples. For example, the division into units is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the mutual coupling, direct coupling, or communication connection shown or discussed may be implemented using some interfaces. Indirect coupling or communication connection between apparatus or units may be implemented in an electrical, mechanical, or other form.

[0431] Units described as separate parts may or may not be physically separate, and parts shown as units may or may not be physical units, may be located in one location, or may be distributed across multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0432] Furthermore, it should be noted that the embodiments of the apparatus described are merely examples. Units described as separate parts may or may not be physically separate, and parts shown as units may or may not be physical units, may be located in one location, or may be distributed across multiple network units. Some or all modules may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments. In addition, in the accompanying drawings of embodiments of the apparatus provided by this application, the connection relationships between modules indicate that the modules have communication connections with one another, which may be specifically implemented as one or more communication buses or signal cables.

[0433] Based on the above-described implementation forms, those skilled in the art will clearly understand that the present application can be implemented by software in addition to the necessary general-purpose hardware, or by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, and dedicated components. In general, any function performed by a computer program can be easily implemented using the corresponding hardware. Furthermore, the specific hardware structure used to achieve the same function may take various forms, for example, analog circuits, digital circuits, or dedicated circuits. However, with respect to the present application, the software program implementation form is the better implementation form in most cases. Based on this understanding, the technical solutions of the present application, either essentially or contributing to the prior art, may be implemented in the form of a software product. The computer software product is stored on a readable storage medium, such as a computer floppy disk, USB flash drive, removable hard disk, ROM, RAM, magnetic disk, or optical disk, and includes several instructions for instructing a computer device (which may be a personal computer, device, or network device) to perform the methods described in the embodiments of the present application.

[0434] All or part of the embodiments described above may be implemented using software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product.

[0435] A computer program product includes one or more computer instructions. When a computer program instruction is loaded and executed on a computer, all or part of the procedure or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted by wired means (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless means (e.g., infrared, radio, or microwave) from one website, computer, device, or data center to another website, computer, device, or data center. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a device or data center integrating one or more available media. The available media may be magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk (SSD)). [Explanation of symbols]

[0436] 40 Neural Network Processing Units 401 Input Memory 402 Weight memory 403 Arithmetic circuit 404 Controller 405 Direct Memory Access Controller 406 Integrated Memory 407 Vector Computation Unit 408 Accumulator 409 Instruction Fetch Buffer 410 Bus Interface Unit 1200 encoding device 1201 Acquisition Module 1202 Processing Module 1300 Decoder 1301 Acquisition Module 1302 Processing Module 1400 devices 1401 Receiver 1402 Transmitter 1403 Processor 14031 Application Processor 14032 Communication Processor 1404 memory 1500 devices 1522 Central Processing Unit 1526 Power supply 1530 Storage medium 1532 memory 1541 Operating Systems 1542 Application Programs 1544 data 1550 Wired or wireless network interface 1558 Input / Output Interface

Claims

1. An image encoding method, wherein the method is A step of obtaining first information, wherein the first information includes information about the color difference components of a first image format. The steps include performing a size transformation on the first information in order to obtain a second piece of information, The steps include encoding the second information using an encoding network or a portion of an encoding network, A step of obtaining a third piece of information, wherein the third piece of information includes information about the color difference components of a second image format, and the second image format is different from the first image format. A step of performing a size transformation on the third information in order to obtain a fourth piece of information, wherein the size of the fourth piece of information in the spatial dimension and channel dimension matches the size of the second piece of information. The steps of encoding the fourth information using the encoding network or a part of the encoding network Image encoding methods, including those mentioned above.

2. The method according to claim 1, wherein the size of the second information in the spatial dimension is smaller than the size of the first information, the size of the second information in the channel dimension is larger than the size of the first information, the size of the fourth information in the spatial dimension is less than or equal to the size of the third information, and the size of the fourth information in the channel dimension is larger than the size of the third information.

3. The method according to claim 1 or 2, wherein the amount of non-redundant information contained in the second information is the same as the amount of non-redundant information contained in the first information, and the amount of non-redundant information contained in the fourth information is the same as the amount of non-redundant information contained in the third information.

4. The aforementioned size conversion is as follows: Performing information replication and using the replicated information as information on the new channel, Splitting information on at least one channel into information on multiple different channels, or Performing a convolution operation The method according to any one of claims 1 to 3, comprising at least one of the following.

5. The method according to claim 4, wherein the operation of dividing the information on at least one channel into the information on a plurality of different channels is performed by using a pixel unshuffle layer.

6. The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format. The method according to any one of claims 1 to 5.

7. The method according to any one of claims 1 to 6, wherein the number of channel dimensions of the second information and the fourth information is 8, the size of the second information in the spatial dimension includes height and width, the height of the second information is H / 2, the width of the second information is W / 2, where H is the height of the first image and W is the width of the first image.

8. The aforementioned method, A step of obtaining a fifth piece of information, wherein the fifth piece of information is information relating to the luminance component Y, A step of performing a size transformation on the fifth information in order to obtain a sixth piece of information, wherein the size of the sixth piece of information in spatial dimensions is equal to the size of the second piece of information. It further includes, The step of encoding the second information using the encoding network or the part of the encoding network is: The step of encoding the fusion result of the second information and the fourth information using the aforementioned encoding network. including, The method according to any one of claims 1 to 7.

9. The method according to any one of claims 1 to 8, wherein the first information or the third information further includes information relating to a luminance component.

10. An image decoding method, wherein the method is A step of obtaining first information based on a first bitstream, A step of performing a size transformation on the first information in order to obtain second information, wherein the second information includes information about the color difference components of the first image format, The steps include: reconstructing the first image based on the second information described above; A step of obtaining third information based on a second bitstream, wherein the size of the third information in the spatial dimension and the channel dimension matches the size of the first information. A step of performing a size conversion on the third information in order to obtain a fourth piece of information, wherein the fourth piece of information includes information about the color difference components of a second image format, and the second image format is different from the first image format, and The steps include: reconstructing a second image based on the fourth piece of information described above; Image decoding methods, including those mentioned above.

11. The method according to claim 10, wherein the first information or the third information is the output of the decoding network or the output of an intermediate layer of the decoding network.

12. The method according to claim 10 or 11, wherein the size of the second information in the spatial dimension is greater than the size of the first information, the size of the second information in the channel dimension is less than the size of the first information, the size of the fourth information in the spatial dimension is greater than or equal to the size of the third information, and the size of the fourth information in the channel dimension is less than the size of the third information.

13. The aforementioned size conversion is as follows: A method of deleting some of the information in a channel. A method of combining different data from multiple channels into the same channel, or Method for performing convolution operations The method according to any one of claims 10 to 12, comprising at least one of the following:

14. The method according to claim 13, wherein the operation of combining the different data of the plurality of channels into the same channel is performed by using a pixel shuffle layer.

15. The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format. The method according to any one of claims 10 to 14.

16. The method according to any one of claims 10 to 15, wherein the number of channel dimensions of the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, the width dimension of the first information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

17. The method according to any one of claims 10 to 16, wherein the intermediate layer is a network layer preceding the last convolutional layer of the decoding network or a network layer preceding the last pixel shuffle layer of the decoding network.

18. An image encoding device, wherein the device is An acquisition module is configured to acquire first information, the first information including information about the color difference components of a first image format, and third information including information about the color difference components of a second image format, wherein the second image format is different from the first image format. A processing module configured to perform a size transformation on the first information in order to obtain second information, encode the second information using an encoding network or a part of an encoding network, perform a size transformation on the third information in order to obtain fourth information, the size of the fourth information in the spatial and channel dimensions being the same as the size of the second information, and encode the fourth information using the encoding network. An image encoding device equipped with the following features.

19. The apparatus according to claim 18, wherein the size of the second information in the spatial dimension is smaller than the size of the first information, the size of the second information in the channel dimension is larger than the size of the first information, the size of the fourth information in the spatial dimension is less than or equal to the size of the third information, and the size of the fourth information in the channel dimension is larger than the size of the third information.

20. The apparatus according to claim 18 or 19, wherein the amount of non-redundant information contained in the second information is the same as the amount of non-redundant information contained in the first information, and the amount of non-redundant information contained in the fourth information is the same as the amount of non-redundant information contained in the third information.

21. The aforementioned size conversion is as follows: Performing information replication and using the replicated information as information on the new channel, Splitting information on at least one channel into information on multiple different channels, or Performing a convolution operation The apparatus according to any one of claims 18 to 20, comprising at least one of the following.

22. The apparatus according to claim 21, wherein the operation of dividing the information on at least one channel into the information on a plurality of different channels is performed by using a pixel unshuffle layer.

23. The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format. The apparatus according to any one of claims 18 to 22.

24. The apparatus according to any one of claims 18 to 23, wherein the number of channel dimensions of the second information and the fourth information is 8, the size of the second information in the spatial dimension includes height and width, the height of the second information is H / 2, the width of the second information is W / 2, where H is the height of the first image and W is the width of the first image.

25. The acquisition module, A fifth piece of information is obtained, and the said fifth piece of information is information relating to the luminance component Y. It is further configured in this way, The processing module is further configured to perform a size transformation on the fifth information in order to obtain the sixth information, such that the size of the sixth information in spatial dimensions matches the size of the second information. The processing module is particularly configured to encode the fusion result of the second information and the fourth information using the encoding network. The apparatus according to any one of claims 18 to 24.

26. The apparatus according to any one of claims 18 to 25, wherein the first information or the third information further includes information relating to a luminance component.

27. An image decoding device, wherein the device is An acquisition module configured to acquire first information based on a first bitstream, acquire third information based on a second bitstream, and such that the size of the third information in the spatial and channel dimensions matches the size of the first information, A processing module configured to perform a size transformation on the first information in order to obtain a second piece of information, the second piece of information including information about the color difference components of the first image format, and to reconstruct the first image based on the second piece of information, and to perform a size transformation on the third information in order to obtain a fourth piece of information, the fourth piece of information including information about the color difference components of the second image format, the second image format being different from the first image format, and to reconstruct the second image based on the fourth piece of information, and An image decoding device equipped with the following features.

28. The apparatus according to claim 27, wherein the first information or the third information is the output of the decoding network or the output of the intermediate layer of the decoding network.

29. The apparatus according to claim 27 or 28, wherein the size of the second information in the spatial dimension is greater than the size of the first information, the size of the second information in the channel dimension is less than the size of the first information, the size of the fourth information in the spatial dimension is greater than or equal to the size of the third information, and the size of the fourth information in the channel dimension is less than the size of the third information.

30. The aforementioned size conversion is as follows: A method of deleting some of the information in a channel. A method of combining different data from multiple channels into the same channel, or Method for performing convolution operations The apparatus according to any one of claims 27 to 29, comprising at least one of the following.

31. The apparatus according to claim 30, wherein the operation of combining the different data of the plurality of channels into the same channel is performed by using a pixel shuffle layer.

32. The first image format is YUV444 format, and the second image format is YUV422 format or YUV420 format, or The first image format is the YUV422 format, and the second image format is the YUV420 format. The apparatus according to any one of claims 27 to 31.

33. The apparatus according to any one of claims 27 to 32, wherein the number of channel dimensions of the first information is 8, the height dimension of the first information in spatial dimensions is H / 2, the width dimension of the first information in spatial dimensions is W / 2, where H is the height dimension of the first image and W is the width dimension of the first image.

34. The apparatus according to any one of claims 27 to 33, wherein the intermediate layer is a network layer preceding the last convolutional layer of the decoding network or a network layer preceding the last pixel shuffle layer of the decoding network.

35. An image decoding method, wherein the method is A step of obtaining first information based on a first bitstream, A step of performing a size transformation on the first information in order to obtain one or more of the second, third, or fourth pieces of information, wherein the second information is information relating to the color difference components of the first image format, the third information is information relating to the color difference components of the second image format, the fourth information is information relating to the color difference components of the third image format, and the first, second, and third image formats are different from each other. below: The step of reconstructing the first image based on the second information described above, The step of reconstructing a second image based on the third piece of information, or The step of reconstructing a third image based on the fourth piece of information described above, Steps to perform one or more of the following: Image decoding methods, including those mentioned above.

36. The method according to claim 35, wherein the first information is the output of the decoding network or the output of an intermediate layer of the decoding network.

37. The method according to claim 35 or 36, wherein the sizes of the second, third, and fourth pieces of information in the spatial dimension are greater than or equal to the size of the first piece of information, and the sizes of the second, third, and fourth pieces of information in the channel dimension are smaller than the size of the first piece of information.

38. The method according to any one of claims 35 to 37, wherein the amount of non-redundant information contained in the second information, the third information, and the fourth information is less than or equal to the amount of non-redundant information contained in the first information.

39. The first size transformation and the second size transformation are as follows: A method of deleting some of the information in a channel. A method of combining different data from multiple channels into the same channel, or Method for performing convolution operations The method according to any one of claims 35 to 38, comprising at least one of the following:

40. The method according to claim 39, wherein the operation of combining the different data of the plurality of channels into the same channel is performed by using a pixel shuffle layer.

41. An image decoding device, wherein the device is An acquisition module configured to acquire first information based on a first bitstream, A size transformation is performed on the first information in order to obtain one or more of the second, third, or fourth pieces of information, wherein the second piece of information is information about the color difference components of the first image format, the third piece of information is information about the color difference components of the second image format, and the fourth piece of information is information about the color difference components of the third image format, and the first, second, and third image formats are different from each other, as follows: The step of reconstructing the first image based on the second information described above, The step of reconstructing a second image based on the third piece of information, or The step of reconstructing a third image based on the fourth piece of information described above, A processing module configured to perform one or more of the following: An image decoding device equipped with the following features.

42. A decoding device comprising a processor, wherein the processor is coupled to a memory, the memory is configured to store a program or instruction, and when the program or instruction is executed by the processor, the decoding device is enabled to perform the method according to any one of claims 10 to 17 and 35 to 40.

43. An encoding device comprising a processor, wherein the processor is coupled to a memory, the memory is configured to store a program or instruction, and when the program or instruction is executed by the processor, the encoding device is enabled to perform the method according to any one of claims 1 to 9.

44. An encoding and decoding system comprising the decoding device according to claim 42 and / or the encoding device according to claim 43.

45. A computer-readable storage medium, wherein the medium stores instructions, and when an instruction is executed by a computer, the method according to any one of claims 1 to 17 and 35 to 40 is performed.

46. A computer program product comprising instructions, wherein when the instructions are executed on a computer, the computer is enabled to perform the method according to any one of claims 1 to 17 and 35 to 40.

47. A readable storage medium, wherein the readable storage medium stores a bitstream obtained by encoding using the encoding method described in any one of claims 1 to 9.

48. A bitstream storage device comprising at least one storage medium and a communication interface, The aforementioned communication interface is configured to receive or transmit a bitstream. The at least one storage medium is configured to store the bitstream, The bitstream is obtained by encoding it with an encoder using the encoding method described in any one of claims 1 to 9. Bitstream storage device.

49. A bitstream storage method, The steps include receiving a bitstream via a communication interface, A step of storing the bitstream in one or more storage media, wherein the bitstream is obtained by encoding it with an encoder using the encoding method described in any one of claims 1 to 9. A bitstream storage method that includes this.

50. A bitstream distribution system comprising at least one storage medium and a video stream device, The at least one storage medium is configured to store a bitstream, and the bitstream is obtained by encoding it with an encoder using the encoding method described in any one of claims 1 to 9. The video stream device is configured to respond to requests from the decoder, and as a result, the bitstream in the at least one storage medium is transmitted to the decoder. Bitstream distribution system.

51. A bitstream distribution method, The steps include receiving a first request, The steps of selecting a bitstream from at least one storage medium in response to the first request described above, The steps include sending the bitstream to a destination device, Includes, The at least one storage medium is configured to store a bitstream, and the bitstream is obtained by encoding it with an encoder using the encoding method described in any one of claims 1 to 9. Bitstream distribution method.

52. A bitstream processing system comprising an image source device, an encoder device, one or more storage media, and a destination device, The image source device is configured to provide image data, The encoder device is configured to acquire the image data from the image source device via an interface and to encode the image data to acquire one or more bitstreams, the bitstreams being obtained by encoding them with the encoder using the encoding method described in any one of claims 1 to 9. The encoder device is configured to store the one or more bitstreams in the one or more storage medium, or The encoder device is configured to encapsulate one or more bitstreams in order to obtain a transmission bitstream. The encoder device is configured to transmit the transmission bitstream to the destination device via a communication link or communication network. The destination device is configured to decapsulate the transmission bitstream in order to obtain one or more bitstreams. The destination device is configured to decode one or more bitstreams in order to obtain the decoded data. Bitstream processing system.

Images