Image decompression method and apparatus

Abstract

This application provides an image decompression method and apparatus. The image decompression method in this application includes the steps of: obtaining a first feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image; obtaining a second feature tensor, wherein the second feature tensor corresponds to a second component of the reconstructed image; performing concatenation and convolution on the first and second feature tensors to obtain a third feature tensor; obtaining a fourth feature tensor based on the third and second feature tensors; and obtaining a reconstructed image based on the first and fourth feature tensors. In this embodiment of the application, the structure for fusing the Y and UV components is optimized. This reduces the number of channels, saving computational power and achieving a lighter weight. Even if the number of channels is not reduced, performance can still be significantly improved.

Description

Technical Field

[0001] This application relates to image processing technology, and particularly to an image decompression method and apparatus.

Background Art

[0002] Since a convolutional neural network (CNN) has better performance than conventional algorithms in computer vision tasks such as image recognition and object detection, more researchers have begun to explore deep learning-based image / video compression methods. Some researchers are designing end-to-end deep learning image / video compression algorithms. For example, modules such as an encoding network, an entropy estimation network, an entropy encoding engine, an entropy decoding engine, and a decoding network are integrated as a whole, or modules such as a prediction module and a residual compression module are integrated as a whole. The encoding network and the decoding network are also called a conversion module and an inverse conversion module, and typically include convolutional layers and non-linear conversion units.

[0003] The Joint Photographic Experts Group (JPEG) Artificial Intelligence (AI) is the first international standard for AI image compression. In its typical structure, the encoder side performs RGB-YUV color space conversion on the image to decompose the image into a Y component and a UV component, then performs separate encoding and transmission, and the decoder side restores the Y component and the UV component separately, and then performs YUV-RGB color space conversion to obtain an RGB image. In the above process, for a better compression effect, there is some specific information exchange between the Y component and the UV component. The information exchange between different components is particularly important for performance optimization.

Summary of the Invention

[0004] This application provides an image decompression method and apparatus that can improve performance.

[0005] According to a first aspect, the application provides an image decompression method comprising the steps of: obtaining a first feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image; obtaining a second feature tensor, wherein the second feature tensor corresponds to a second component of the reconstructed image; performing concatenation and convolution on the first and second feature tensors to obtain a third feature tensor; obtaining a fourth feature tensor based on the third and second feature tensors; and obtaining a reconstructed image based on the first and fourth feature tensors.

[0006] In the embodiments of this application, the structure for fusing the Y and UV components is optimized. A fourth feature tensor (corresponding to the UV component of the reconstructed image) is first obtained based on a third feature tensor (i.e., the tensor output after concatenating and convolving the first feature tensor (corresponding to the Y component of the reconstructed image) and the second feature tensor (corresponding to the UV component of the reconstructed image)) and the second feature tensor, and then a reconstructed image is obtained based on the first and fourth feature tensors. There are three cases for the number of channels in the fourth feature tensor: it is less than, greater than, or equal to the sum of the number of channels in the first and second feature tensors. If it is less than, the number of channels in the fourth feature tensor is reduced, saving computational power and achieving a lighter implementation. If it is greater than or equal to, even if the number of channels in the fourth feature tensor is not reduced, the fourth feature vector can be obtained by performing operations only on the third and second feature tensors, resulting in a significant improvement in performance.

[0007] Optionally, the first component may be the Y component of the image represented using the YUV color space, and the second component may be the UV component of the image represented using the YUV color space.

[0008] It should be noted that the first and second components may be replaced with other components based on the image representation scheme. This is not particularly limited to the embodiments of this application.

[0009] As described above, in the JPEG AI structure, the encoder decomposes the image into Y and UV components, then performs separate encoding and transmission, while the decoder separately decodes and reconstructs the Y and UV components. It can be seen that the Y component (i.e., the first component) and the UV component (i.e., the second component) are processed independently across two separate paths. In the path for processing the UV component on the decoder side, the Y and UV components are fused.

[0010] Concatenating a first feature tensor with a second feature tensor can also be done by performing channel concatenation on them. For example, if the scale of the first feature tensor is [128,H,W] and the scale of the second feature tensor is [64,H,W], the scale of the concatenated feature tensor is [192,H,W]. That is, the width and height remain unchanged, and the number of channels is the sum of the number of channels in those features. Next, convolution is performed on the concatenated feature tensor. Convolution may increase, decrease, or leave the number of channels unchanged. Correspondingly, the number of channels in the output third feature tensor increases, decreases, or remains unchanged. For example, the scale of the third feature tensor may be [64,H,W], [128,H,W], or [192,H,W]. The first element of the above scale represents the number of channels in the feature tensor.

[0011] Optionally, the third and second feature tensors may be concatenated to obtain a fourth feature tensor.

[0012] The concatenation is channel concatenation. That is, channel concatenation is performed on the third feature tensor and the second feature tensor.

[0013] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. If the number of channels in the fourth feature tensor is less than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is greater than the sum of the number of channels in the first and second feature tensors, after convolution, the element-wise addition is changed to concatenation, and the element-wise addition of the convolution output with the fused feature tensor of the Y and UV components is changed to concatenation of the output with only the UV component, resulting in a significant improvement in performance.

[0014] Optionally, channel extraction may be performed on the second feature tensor to obtain the fifth feature tensor, and the third and fifth feature tensors may be concatenated to obtain the fourth feature tensor.

[0015] To obtain the fifth feature tensor, channel reduction may be performed on the second feature tensor, and then the third and fifth feature tensors may be concatenated.

[0016] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. If the number of channels in the fourth feature tensor is less than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is greater than the sum of the number of channels in the first and second feature tensors, after convolution, the element-wise addition is changed to concatenation, and the element-wise addition of the convolution output with the fused feature tensor of the Y and UV components is changed to concatenation of the output with only the UV component, resulting in a significant improvement in performance.

[0017] Optionally, the third and second feature tensors may be added together to obtain the fourth feature tensor.

[0018] If the number of channels in the third feature tensor is equal to the number of channels in the second feature tensor, they may be added element by element. If the number of channels in the third feature tensor is not equal to the number of channels in the second feature tensor, they may be added by adding the channels of the feature tensor with a smaller number of channels to some of the channels of the feature tensor with a larger number of channels, element by element. For example, the third feature tensor has 64 channels and the second feature tensor has 32 channels. The 32 channels of the second feature tensor and the first 32 channels of the third feature tensor may be added element by element, and the 32 channels of the second feature tensor and the last 32 channels of the third feature tensor may be added element by element, and so on. The method of addition in this case is not particularly limited in the embodiments of this application.

[0019] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. If the number of channels in the fourth feature tensor is less than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is greater than the sum of the number of channels in the first and second feature tensors, after convolution, the element-wise addition of the convolution output and the fused feature tensor of the Y and UV components can be changed to an element-wise addition of the output and the UV component only, resulting in a significant improvement in performance.

[0020] Optionally, channel extraction may be performed on the second feature tensor to obtain the fifth feature tensor, and the third and fifth feature tensors may be added together to obtain the fourth feature tensor.

[0021] If the number of channels in the third feature tensor is less than the number of channels in the second feature tensor, channel reduction may be performed on the second feature tensor to obtain the fifth feature tensor such that the number of channels in the fifth feature tensor is equal to the number of channels in the second feature tensor, and then the third and fifth feature tensors may be added element by element.

[0022] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. In the above case, if the number of channels in the fourth feature tensor is less than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is greater than the sum of the number of channels in the first and second feature tensors, after convolution, the element-wise addition of the convolution output and the fused feature tensor of the Y and UV components can be changed to an element-wise addition of the output and the UV component only, resulting in a significant improvement in performance.

[0023] The UV component of the reconstructed image may be obtained from the fourth feature tensor obtained through fusion through some processing (see the embodiments below), and then combined with the Y component of the reconstructed image to form a complete reconstructed image.

[0024] According to a second aspect, the application provides an image decompression device comprising: an acquisition module configured to acquire a first feature tensor and a second feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image and the second feature tensor corresponds to a second component of the reconstructed image; a processing module configured to perform concatenation and convolution on the first and second feature tensors to acquire a third feature tensor and to acquire a fourth feature tensor based on the third and second feature tensors; and a reconstruction module configured to acquire a reconstructed image based on the first and fourth feature tensors.

[0025] In possible implementations, the processing module is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain a fourth feature tensor.

[0026] In possible implementations, the processing module is specifically configured to add the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

[0027] In a possible implementation, the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain the fifth feature tensor, and then concatenate the third and fifth feature tensors to obtain the fourth feature tensor.

[0028] In a possible implementation, the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain the fifth feature tensor, and then add the third and fifth feature tensors to obtain the fourth feature tensor.

[0029] In a possible implementation, the first component is the Y component, and the second component is the UV component.

[0030] According to a third aspect, the application provides an electronic device including one or more processors and a memory configured to store one or more programs. When one or more programs are executed by one or more processors, one or more processors can implement a method according to any one of the implementations of the first aspect.

[0031] According to a fourth aspect, the application provides a computer-readable storage medium containing a computer program. When the computer program is executed on a computer, the computer is able to perform a method according to any one of the implementations of the first aspect.

[0032] According to a fifth aspect, the application provides a computer program. When the computer program is executed by a computer, the computer is configured to perform a method according to any one of the implementations of the first aspect.

[0033] According to a sixth aspect, the application further provides a computer program product, the computer program product including computer program code, when the computer program code is executed on a computer, the computer is able to perform operations and / or processes that are performed by an electronic device in any one of the embodiments of the method described above. [Brief explanation of the drawing]

[0034] [Figure 1] This is an illustrative diagram of an end-to-end deep learning image coding framework. [Figure 2] This is an illustrative diagram of an end-to-end deep learning video coding framework. [Figure 3] This is an illustrative diagram of an application scenario according to the embodiments of this application. [Figure 4] This is an illustrative diagram of an application scenario according to the embodiments of this application. [Figure 5a] This is a diagram illustrating a typical structure in the JPEG AI standard. [Figure 5b] This is a diagram of the structure within the thick-lined box in Figure 5a. [Figure 6] This is a flowchart of process 600 of the image decompression method according to an embodiment of this application. [Figure 7a] This is a diagram of the fusion structure. [Figure 7b] This is a diagram of the fusion structure. [Figure 7c] This is a diagram of the fusion structure. [Figure 7d] This is a diagram of the fusion structure. [Figure 8] This is a diagram of the structure of a JPEG AI using a fusion structure. [Figure 9]This is a diagram of the structure of a JPEG AI using a fusion structure. [Figure 10] This is a diagram of the structure of a JPEG AI using a fusion structure. [Figure 11] This is a diagram of the structure of a JPEG AI using a fusion structure. [Figure 12] This is a diagram showing the structure of an image defrosting device 1200 according to an embodiment of this application. [Modes for carrying out the invention]

[0035] To clarify the purpose, technical solution, and merits of this application, the technical solution described herein will be clearly and completely described below with reference to the accompanying drawings. It will be apparent that the embodiments described are not all but a part of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort will fall within the scope of protection of this application.

[0036] In the specification, embodiments, claims, and accompanying drawings of this application, terms such as “First” and “Second” are used solely for distinction and descriptive purposes and should not be understood as indicating or implying relative importance or order. Furthermore, “Includes,” “Has,” and any variation thereof are intended to cover non-exclusive inclusion, such as a series of steps or units. A method, system, product, or device is not necessarily limited to the explicitly enumerated steps or units and may include other steps or units that are not explicitly enumerated or that are specific to such process, method, product, or device.

[0037] In this application, it should be understood that “at least one (item)” means one or more, and “multiple” means two or more. The term “and / or” is used to describe an association between related objects and indicates that there may be three such relationships. For example, “A and / or B” may mean that only A exists, only B exists, and both A and B exist, and A and B may be singular or plural. The letter “ / ” generally indicates an “or” relationship between related objects. The expression “at least one of the following items” or similar expressions means any combination of these items, including either a single item or a combination of multiple items. For example, at least one of a, b or c may mean a, b, c, a and b, a and c, b and c, or a, b and c, and a, b and c may be singular or plural.

[0038] Figure 1 is an illustrative diagram of an end-to-end deep learning image coding framework. As shown in Figure 1, the image coding framework includes an encoder side and a decoder side. The encoder side includes an encoding network (Encoder), a quantization module, a hyperencoder, a hyperencoder decoding network, an entropy estimation and entropy coding engine (common entropy coding engines include arithmetic encoders (AE), asymmetric numeric systems (ANS), etc.). The decoder side includes a hyperdecoder, a decoding network (Decoder), an entropy estimation and entropy decoding engine (common entropy decoding engines include arithmetic decoders (AD), ANS, etc.).

[0039] On the encoder side, the original image is processed by the encoding network and then converted from an image region to a feature region. The converted image features are processed by the quantization module and the entropy encoding engine and then encoded into a bitstream to be transmitted or stored. On the decoder side, the bitstream is processed by the entropy decoding engine and then decoded into image features. The image features are processed by the decoding network and then converted from the feature region to an image region to obtain a reconstructed image. The entropy estimation network estimates and obtains estimated probability values ​​for each feature element based on the image features. These probability values ​​are used in the processing of the entropy encoding engine and the entropy decoding engine.

[0040] In this embodiment, both the encoding network (Encoder) and the decoding network (Decoder) have nonlinear transformation units.

[0041] Figure 2 is an illustrative diagram of an end-to-end deep learning video coding framework. As shown in Figure 2, the video coding framework includes a predict model module and a residual compress module.

[0042] The prediction module predicts the current frame by using the reconstructed image of the previous frame to obtain the predicted image. The residual compression module compresses the residual between the original image and the predicted image of the current frame, then decompresses the compressed residual to obtain the reconstructed residual, and sums the reconstructed residual with the predicted image to obtain the reconstructed image of the current frame. Both the coding subnetwork and the decoding subnetwork within the prediction module and residual compression module have nonlinear transformation units.

[0043] In this embodiment, both the predict model module and the residual compress module have a nonlinear transformation unit.

[0044] Figure 3 is an exemplary diagram of an application scenario according to an embodiment of this application. As shown in Figure 3, the application scenario may be a service related to the capture, storage, or transmission of images / videos in a terminal, cloud server, or video surveillance, for example, shooting / video recording by a terminal, album, cloud album, or video surveillance.

[0045] Encoder side: The camera captures images / videos. An artificial intelligence (AI) image / video encoding network performs feature extraction on the images / videos to obtain low-redundancy image features, and then performs entropy encoding based on the image features to obtain a bitstream / image file.

[0046] Decoder side: When an image / video needs to be output, the AI ​​image / video decoding network performs entropy decoding on the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain a reconstructed image / video.

[0047] The storage / transmission module stores bitstream / image files acquired through compression for different services (e.g., captured by terminals, video surveillance, or cloud servers) or transmits them (e.g., cloud services or live broadcast technology).

[0048] Figure 4 is an exemplary diagram of an application scenario according to an embodiment of this application. As shown in Figure 4, the application scenario may be a service related to the capture, storage, or transmission of images / videos in the cloud or video surveillance, such as a cloud album, video surveillance, or live broadcast.

[0049] Encoder side: The local side acquires the image / video, encodes it to obtain a compressed image / video, and then sends the compressed image / video to the cloud. The cloud side decodes the compressed image / video to obtain the image / video, then performs AI encoding on the image / video to obtain a bitstream / image file, and stores the bitstream / image file.

[0050] Decoder side: When the local side needs to retrieve images / videos from the cloud side, the cloud side performs AI decoding on the bitstream / image file to retrieve the images / videos, encodes the images / videos to obtain compressed images / videos, and sends the compressed images / videos to the local side. The local side decodes the compressed images / videos to retrieve the images / videos. For the structure of the cloud side and the use of each module, refer to the structure and use of each module in Figure 4. Details are not described here in the embodiments of this application.

[0051] Figure 5a shows a typical structure in the JPEG AI standard. As shown in Figure 5a, the encoder performs RGB-YUV color space conversion to decompose the image into Y and UV components, and then performs separate encoding and transmission. The decoder separately reconstructs the Y and UV components, and then performs YUV-RGB color space conversion to obtain an RGB image. Some specific information exchange exists between the Y and UV components on both the encoder and decoder sides for better compression.

[0052] Figure 5b is a diagram of the structure within the thick-lined box in Figure 5a. As shown in Figure 5b,

number

number

[0053] (1)

number

number

[0054] (2)

number

[0055] (3) Input x1 into the convolution (Conv) module and output x2. The scale of x2 is [192, H, W]. In the convolution module, 192 represents the number of output channels of the convolution module, 3x3 represents the size of the convolution kernel of the convolution module, and S1 represents the stride of the convolution module.

[0056] (4) Add x1 and x2 element by element to obtain x3. x3 The scale is [192, H, W].

[0057] The structure within the thick-lined box is a ResBlock structure and has the following characteristics: The input and output feature tensors of the convolutional module have the same number of channels, and as a result, the feature tensors before and after the convolution can be directly added through skip connections. However, in the JPEG AI structure, the input feature tensor is formed by concatenating the Y and UV components, often resulting in a large number of channels. If the number of channels is reduced to achieve weight reduction, the computational complexity is reduced, but the performance deteriorates sharply.

[0058] To solve the technical problems described above, embodiments of this application provide an image decompression method for improving the performance of image / video compression algorithms in the above-described encoding / decoding networks and application scenarios. It should be noted that the image in embodiments of this application may be a separate image, such as a photograph or picture, or an image frame within a video. In other words, the method in embodiments of this application may be used to process a separate image or a sequence of image frames within a video. Unless otherwise specified herein, video / images are collectively referred to as images.

[0059] Figure 6 is a flowchart of process 600 of the image decompression method according to an embodiment of this application. Process 600 may be performed by the decoder side in the above embodiment. Process 600 is described as a series of steps or operations. It should be understood that the steps or operations of process 600 may be performed in various orders and / or simultaneously, and are not limited to the execution order shown in Figure 6. Process 600 includes the following steps:

[0060] Step 601: Obtain the first feature tensor. The first feature tensor corresponds to the first component of the reconstructed image.

[0061] Step 602: Obtain the second feature tensor. The second feature tensor corresponds to the second component of the reconstructed image.

[0062] Optionally, the first component may be the Y component of the image represented using the YUV color space, and the second component may be the UV component of the image represented using the YUV color space.

[0063] It should be noted that the first and second components may be replaced with other components based on the image representation scheme. This is not particularly limited to the embodiments of this application.

[0064] As described above, in the JPEG AI structure, the encoder decomposes the image into Y and UV components, which are then encoded and transmitted separately, and the decoder decodes and reconstructs the Y and UV components separately. It can be seen that the Y component (i.e., the first component) and the UV component (i.e., the second component) are processed independently on two separate paths. Referring to Figure 5a, the Y and UV components are fused on the path for processing the UV component on the decoder side, i.e., the operation in the thick-lined box shown in Figure 5b is performed. Assume that the structure within the thick-lined box (configured to fuse the Y and UV components) is considered a black box. In this case, the black box has two inputs, namely a first feature tensor (corresponding to the Y component and coming from the path for processing the Y component; see the embodiments below for its specific acquisition process) and a second feature tensor (corresponding to the UV component and coming from the path for processing the UV component; see the embodiments below for its specific acquisition process). The black box has one output (the fourth feature tensor below).

[0065] In embodiments of this application, the structure for achieving the fusion function is improved. Refer to the following steps.

[0066] Step 603: Perform concatenation and convolution on the first and second feature tensors to obtain the third feature tensor.

[0067] Concatenating a first feature tensor with a second feature tensor can also be done by performing channel concatenation on them. For example, if the scale of the first feature tensor is [128,H,W] and the scale of the second feature tensor is [64,H,W], the scale of the concatenated feature tensor is [192,H,W]. That is, the width and height remain unchanged, and the number of channels is the sum of the number of channels. Next, convolution is performed on the concatenated feature tensor. Convolution may increase, decrease, or leave the number of channels unchanged. Correspondingly, the number of channels in the output third feature tensor increases, decreases, or remains unchanged. For example, the scale of the third feature tensor may be [64,H,W], [128,H,W], or [192,H,W].

[0068] Step 604: Obtain the fourth feature tensor based on the third and second feature tensors.

[0069] Optionally, the third and second feature tensors may be concatenated to obtain a fourth feature tensor.

[0070] The concatenation is channel concatenation. That is, channel concatenation is performed on the third feature tensor and the second feature tensor.

[0071] Figure 7a is a diagram of the fused structure. As shown in Figure 7a, the fused structure concatenates the first feature tensor and the second feature tensor, performs convolution to obtain the third feature tensor, and concatenates the third feature tensor and the second feature tensor to obtain the fourth feature tensor. First feature tensor

number

number

[0072] (1)

Number

[0073] (2) x1 is input into the Conv module to output the third feature tensor x2. The scale of x2 is [c UV ', H, W].

[0074] (3)

Number

[0075] It should be noted that the number of channels of the fourth feature tensor is not particularly limited in the embodiments of this application. For example, c UV ' + c UV may be smaller than c Y + c UV , and c UV ' + c UV may be larger than c Y + c UV , and c UV ' + c UV may be equal to c YIt may be smaller, etc. In the above case, if the number of channels in the fourth feature tensor is smaller than the sum of the number of channels in the first and second feature tensors, the number of channels is reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is larger than the sum of the number of channels in the first and second feature tensors, compared to the embodiment shown in Figure 5b, in this embodiment, after convolution, element-wise addition is changed to concatenation, and element-wise addition of the convolution output and the fused feature tensor of the Y and UV components is changed to concatenation of the output and the UV component only, resulting in a significant improvement in performance.

[0076] Optionally, channel extraction may be performed on the second feature tensor to obtain the fifth feature tensor, and the third and fifth feature tensors may be concatenated to obtain the fourth feature tensor.

[0077] To obtain the fifth feature tensor, channel reduction may be performed on the second feature tensor, and then the third and fifth feature tensors may be concatenated.

[0078] Figure 7b is a diagram of the fused structure. As shown in Figure 7b, the fused structure concatenates the first and second feature tensors, performs convolution to obtain the third feature tensor, performs channel extraction on the second feature tensor to obtain the fifth feature tensor, and concatenates the third and fifth feature tensors to obtain the fourth feature tensor. First feature tensor

number

number

[0079] (1)

number

[0080] (2) Input x1 into the Conv module and output the third feature tensor x2. The scale of x2 is [c UV It is ',H,W].

[0081] (3) According to pre-set rules

number

[0082] (4)y UV ' and x2 are input to the Concat module, and channel concatenation is performed on them to obtain the fourth feature tensor x3. The scale of x3 is [c UV '+c UV '',H,W].

[0083] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. For example, c UV '+c UV '' is c Y +c UV It can be even smaller, c UV '+c UV '' is c Y +c UV It can be even larger, cUV '+c UV '' is c Y or c UV It may be smaller, etc. In the above case, if the number of channels in the fourth feature tensor is smaller than the sum of the number of channels in the first and second feature tensors, the number of channels is reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is larger than the sum of the number of channels in the first and second feature tensors, compared to the embodiment shown in Figure 5b, in this embodiment, after convolution, element-wise addition is changed to concatenation, and element-wise addition of the convolution output and the fused feature tensor of the Y and UV components is changed to concatenation of the output and the UV component only, resulting in a significant improvement in performance.

[0084] Optionally, the third and second feature tensors may be added together to obtain the fourth feature tensor.

[0085] If the number of channels in the third feature tensor is equal to the number of channels in the second feature tensor, they may be added element by element. If the number of channels in the third feature tensor is not equal to the number of channels in the second feature tensor, they may be added by adding the channels of the feature tensor with a smaller number of channels to some of the channels of the feature tensor with a larger number of channels, element by element. For example, the third feature tensor has 64 channels and the second feature tensor has 32 channels. The 32 channels of the second feature tensor and the first 32 channels of the third feature tensor may be added element by element, and the 32 channels of the second feature tensor and the last 32 channels of the third feature tensor may be added element by element, and so on. The method of addition in this case is not particularly limited in the embodiments of this application.

[0086] Figure 7c is a diagram of the fused structure. As shown in Figure 7c, the fused structure concatenates the first feature tensor and the second feature tensor, performs convolution to obtain the third feature tensor, and adds the third feature tensor and the second feature tensor to obtain the fourth feature tensor. First feature tensor

number

number

[0087] (1)

number

[0088] (2) Input x1 into the Conv module and output the third feature tensor x2. The scale of x2 is [c UV It is ',H,W].

[0089] (3)

number

[0090] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. For example, c UV '' is c Y +c UV It can be even smaller, c UV '' is c Y +c UV It can be even larger, c UV '' is c Y It may be even smaller, etc. In the above case, if the number of channels in the fourth feature tensor is smaller than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is larger than the sum of the number of channels in the first and second feature tensors, in this embodiment, compared to the embodiment shown in Figure 5b, the element-wise addition of the convolution output and the fused feature tensor of the Y and UV components after convolution is changed to an element-wise addition of only the output and the UV component, and as a result, performance can be greatly improved.

[0091] Optionally, channel extraction may be performed on the second feature tensor to obtain the fifth feature tensor, and the third and fifth feature tensors may be added together to obtain the fourth feature tensor.

[0092] If the number of channels in the third feature tensor is less than the number of channels in the second feature tensor, channel reduction may be performed on the second feature tensor to obtain the fifth feature tensor such that the number of channels in the fifth feature tensor is equal to the number of channels in the second feature tensor, and then the third and fifth feature tensors may be added element by element.

[0093] Figure 7d is a diagram of the fused structure. As shown in Figure 7d, the fused structure concatenates the first and second feature tensors, performs convolution to obtain the third feature tensor, performs channel extraction on the second feature tensor to obtain the fifth feature tensor, and adds the third and fifth feature tensors to obtain the fourth feature tensor. First feature tensor

number

number

[0094] (1)

number

[0095] (2) Input x1 into the Conv module and output the third feature tensor x2. The scale of x2 is [c UV It is ',H,W].

[0096] (3) According to pre-set rules

number

[0097] (4)y UVAdd x2 to obtain the fourth feature tensor x3. The scale of x3 is [c UV It is ',H,W].

[0098] It should be noted that the number of channels in the fourth feature tensor is not particularly limited in the embodiments of this application. For example, c UV ' is c Y +c UV It can be even smaller, c UV ' is c Y +c UV It can be even larger, c UV ' is c Y or c UV It may be even smaller, etc. In the above case, if the number of channels in the fourth feature tensor is smaller than the sum of the number of channels in the first and second feature tensors, the number of channels can be reduced, saving computational power and achieving a lighter implementation. Even if the number of channels in the fourth feature tensor is larger than the sum of the number of channels in the first and second feature tensors, in this embodiment, compared to the embodiment shown in Figure 5b, the element-wise addition of the convolution output and the fused feature tensor of the Y and UV components after convolution is changed to an element-wise addition of only the output and the UV component, and as a result, performance can be greatly improved.

[0099] Step 605: Obtain a reconstructed image based on the first and fourth feature tensors.

[0100] The UV component of the reconstructed image may be obtained from the fourth feature tensor obtained through fusion through some processing (see the embodiments below), and then combined with the Y component of the reconstructed image obtained by processing the first feature tensor to form a complete reconstructed image.

[0101] In embodiments of this application, the structure for fusing the Y and UV components is optimized. This reduces the number of channels, saving computational power and achieving a lighter weight. Even if the number of channels is not reduced, performance can still be significantly improved.

[0102] The technical solutions of this application will be described below through several specific embodiments.

[0103] Figure 8 shows the structure of JEPG AI using a fused structure. As shown in Figure 8, JEPG AI uses the fused structure shown in Figure 7a or Figure 7b, and its processing process is as follows.

[0104] Encoder side:

[0105] 1. Convert the original RGB image to a YUV444 image through color space conversion.

[0106] 2. The YUV444 image obtained in step 1 is decomposed channel by channel to obtain the Y component and UV component.

[0107] 3. The Y component obtained in step 2 is input into the Y component coding (Y Encoder) network, and downsampling is performed four times to obtain y_Y.

[0108] 4. The y_Y obtained in step 3 is processed through a Y-component hyper-encoder network (Y Hyper Encoder Net), and then round-to-nearest quantization is performed to obtain z_Y_hat.

[0109] 5. The z_Y_hat obtained in step 4 is processed through an entropy encoder (lossless encoder) to obtain a first bitstream denoted as bit_z_Y.

[0110] 6. The z_Y_hat obtained in step 4 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0111] 7. The z_Y_hat obtained in step 4 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0112] 8. Subtract mean_Y obtained in step 6 from y_Y obtained in step 3 to obtain residual_Y, and perform nearest neighbor rounding quantization to obtain residual_Y_hat.

[0113] 9. Based on the variance_Y generated in step 7, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the residual_Y_hat obtained in step 8 is processed through an entropy encoder (Lossless Encoder) to obtain a second bitstream denoted as bit_y_Y.

[0114] 10. The Y and UV components obtained in step 2 are each downsampled once. Then, the downsampled components are input into the UV component coding (UV Encoder) network, and the UV Encoder network performs downsampling three times to obtain y_UV. (The UV Encoder network includes three downsampling steps.)

[0115] 11. The y_UV obtained in step 10 is processed through a UV component hyperplier coding network (UV Hyper Encoder Net), and then nearest neighbor rounding quantization is performed to obtain z_UV_hat.

[0116] 12. The z_UV_hat obtained in step 11 is processed through an entropy encoder (lossless encoder) to obtain a third bitstream denoted as bit_z_UV.

[0117] 13. The z_UV_hat obtained in step 11 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0118] 14. The z_UV_hat obtained in step 11 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0119] 15. Subtract the mean_UV output in step 13 from the y_UV output in step 10 to obtain residual_UV, and perform nearest neighbor rounding quantization to obtain residual_UV_hat.

[0120] 16. Based on the variance_UV generated in step 14, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the residual_UV_hat obtained in step 15 is processed through an entropy encoder (Lossless Encoder) to obtain a fourth bitrate denoted as bit_y_UV.

[0121] 17. Obtain the sum of the four bitstreams above as the final encoded bitstream.

[0122] Decoder side:

[0123] 1. The first bitstream bit_z_Y is processed through an entropy decoder (Lossless Decoder) to obtain z_Y_hat.

[0124] 2. The z_Y_hat obtained in step 1 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0125] 3. The z_Y_hat obtained in step 1 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0126] 4. Based on the variance_Y generated in step 2, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the second bitstream bit_y_Y is processed through an entropy decoder (Lossless Decoder) to obtain residual_Y_hat.

[0127] 5. Add residual_Y_hat obtained in step 4 and mean_Y obtained in step 2 to obtain y1_Y.

[0128] 6. The y1_Y obtained in step 5 is processed through a Y-component decoding (Y Decoder) network to obtain the Y component of the reconstructed image.

[0129] 7. The third bitstream, bit_z_UV, is processed through an entropy decoder (Lossless Decoder) to obtain z_UV_hat.

[0130] 8. The z_UV_hat obtained in step 7 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0131] 9. The z_UV_hat obtained in step 7 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0132] 10. Based on the variance_UV generated in step 9, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the fourth bitstream bit_y_UV is processed through an entropy decoder (Lossless Decoder) to obtain residual_UV_hat.

[0133] 11. Add the residual_UV_hat obtained in step 10 and the mean_UV obtained in step 8 to obtain y1_UV.

[0134] 12. Channel concatenation is performed on y1_Y obtained in step 5 and y1_UV obtained in step 11 to obtain y1.

[0135] 13. The y1 obtained in step 12 is processed through a convolutional layer to obtain y2.

[0136] 14. Perform channel concatenation on y2 obtained in step 13 and y1_UV obtained in step 11 to obtain y3.

[0137] 15. The y3 obtained in step 14 is input to the UV Decoder network, and upsampling is performed three times in the UV Decoder network (optionally, the three upsamplings may be performed by performing a 1-scale upsampling through one deconvolution module, and then a 2-scale upsampling through pixel shuffle), and then upsampling is performed once on the output of the UV Decoder (optionally, the upsampling may be performed through nearest neighbor interpolation) to obtain the UV components of the reconstructed image. (The UV Decoder network includes three upsamplings.)

[0138] 16. The Y component obtained in step 6 and the UV component obtained in step 15 are concatenated, and then a color conversion is performed to obtain a reconstructed RGB image.

[0139] Figure 9 shows the structure of JEPG AI using a fused structure. As shown in Figure 9, JEPG AI uses the fused structure shown in Figure 7c or Figure 7d, and its processing process is as follows.

[0140] Encoder side:

[0141] 1. Convert the original RGB image to a YUV444 image through color space conversion.

[0142] 2. The YUV444 image obtained in step 1 is decomposed channel by channel to obtain the Y component and UV component.

[0143] 3. The Y component obtained in step 2 is input into the Y component coding (Y Encoder) network, and downsampling is performed four times to obtain y_Y.

[0144] 4. The y_Y obtained in step 3 is processed through a Y-component hyper-encoder network (Y Hyper Encoder Net), and then round-to-nearest quantization is performed to obtain z_Y_hat.

[0145] 5. The z_Y_hat obtained in step 4 is processed through an entropy encoder (lossless encoder) to obtain a first bitstream denoted as bit_z_Y.

[0146] 6. The z_Y_hat obtained in step 4 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0147] 7. The z_Y_hat obtained in step 4 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0148] 8. Subtract mean_Y obtained in step 6 from y_Y obtained in step 3 to obtain residual_Y, and perform nearest neighbor rounding quantization to obtain residual_Y_hat.

[0149] 9. Based on the variance_Y generated in step 7, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the residual_Y_hat obtained in step 8 is processed through an entropy encoder (Lossless Encoder) to obtain a second bitstream denoted as bit_y_Y.

[0150] 10. The Y and UV components obtained in step 2 are each downsampled once. Then, the downsampled components are input into the UV component coding (UV Encoder) network, and the UV Encoder network performs downsampling three times to obtain y_UV. (The UV Encoder network includes three downsampling steps.)

[0151] 11. The y_UV obtained in step 10 is processed through a UV component hyperplier coding network (UV Hyper Encoder Net), and then nearest neighbor rounding quantization is performed to obtain z_UV_hat.

[0152] 12. The z_UV_hat obtained in step 11 is processed through an entropy encoder (lossless encoder) to obtain a third bitstream denoted as bit_z_UV.

[0153] 13. The z_UV_hat obtained in step 11 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0154] 14. The z_UV_hat obtained in step 11 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0155] 15. Subtract the mean_UV output in step 13 from the y_UV output in step 10 to obtain residual_UV, and perform nearest neighbor rounding quantization to obtain residual_UV_hat.

[0156] 16. Based on the variance_UV generated in step 14, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the residual_UV_hat obtained in step 15 is processed through an entropy encoder (Lossless Encoder) to obtain a fourth bitrate denoted as bit_y_UV.

[0157] 17. Obtain the sum of the four bitstreams above as the final encoded bitstream.

[0158] Decoder side:

[0159] 1. The first bitstream bit_z_Y is processed through an entropy decoder (Lossless Decoder) to obtain z_Y_hat.

[0160] 2. The z_Y_hat obtained in step 1 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0161] 3. The z_Y_hat obtained in step 1 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0162] 4. Based on the variance_Y generated in step 2, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the second bitstream bit_y_Y is processed through an entropy decoder (Lossless Decoder) to obtain residual_Y_hat.

[0163] 5. Add residual_Y_hat obtained in step 4 and mean_Y obtained in step 2 to obtain y1_Y.

[0164] 6. The y1_Y obtained in step 5 is processed through a Y-component decoding (Y Decoder) network to obtain the Y component of the reconstructed image.

[0165] 7. The third bitstream, bit_z_UV, is processed through an entropy decoder (Lossless Decoder) to obtain z_UV_hat.

[0166] 8. The z_UV_hat obtained in step 7 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0167] 9. The z_UV_hat obtained in step 7 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0168] 10. Based on the variance_UV generated in step 9, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the fourth bitstream bit_y_UV is processed through an entropy decoder (Lossless Decoder) to obtain residual_UV_hat.

[0169] 11. Add the residual_UV_hat obtained in step 10 and the mean_UV obtained in step 8 to obtain y1_UV.

[0170] 12. Channel concatenation is performed on y1_Y obtained in step 5 and y1_UV obtained in step 11 to obtain y1.

[0171] 13. The y1 obtained in step 12 is processed through a convolutional layer to obtain y2.

[0172] 14. Add y2 obtained in step 13 and y1_UV obtained in step 11 to obtain y3.

[0173] 15. The y3 obtained in step 14 is input to the UV Decoder network. The UV Decoder network performs upsampling three times, and then performs upsampling once on the output of the UV Decoder to obtain the UV component of the reconstructed image. (The UV Decoder network includes three upsampling steps.)

[0174] 16. The Y component obtained in step 6 and the UV component obtained in step 15 are concatenated, and then a color conversion is performed to obtain a reconstructed RGB image.

[0175] The difference between this embodiment and the embodiment shown in Figure 8 lies in the underlined steps 12-14 on the decoder side described above. In this embodiment, a method of adding two feature tensors is used.

[0176] Figure 10 shows the structure of JEPG AI using a fused structure. As shown in Figure 10, JEPG AI uses the fused structure shown in Figure 7a or Figure 7b, and its processing process is as follows.

[0177] Encoder side:

[0178] 1. Convert the original RGB image to a YUV444 image through color space conversion.

[0179] 2. The YUV444 image obtained in step 1 is decomposed channel by channel to obtain the Y component and UV component.

[0180] 3. The Y component obtained in step 2 is input into the Y component coding (Y Encoder) network, and downsampling is performed four times to obtain y_Y.

[0181] 4. The y_Y obtained in step 3 is processed through a Y-component hyper-encoder network (Y Hyper Encoder Net), and then round-to-nearest quantization is performed to obtain z_Y_hat.

[0182] 5. The z_Y_hat obtained in step 4 is processed through an entropy encoder (lossless encoder) to obtain a first bitstream denoted as bit_z_Y.

[0183] 6. The z_Y_hat obtained in step 4 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0184] 7. The z_Y_hat obtained in step 4 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0185] 8. Subtract mean_Y obtained in step 6 from y_Y obtained in step 3 to obtain residual_Y, and perform nearest neighbor rounding quantization to obtain residual_Y_hat.

[0186] 9. Based on the variance_Y generated in step 7, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the residual_Y_hat obtained in step 8 is processed through an entropy encoder (Lossless Encoder) to obtain a second bitstream denoted as bit_y_Y.

[0187] 10. The Y and UV components obtained in Step 2 are input to the UV Component Coding (UV Encoder) network, and downsampling is performed four times in the UV Encoder network to obtain y_UV. (The UV Encoder network includes four downsampling steps.)

[0188] 11. The y_UV obtained in step 10 is processed through a UV component hyperplier coding network (UV Hyper Encoder Net), and then nearest neighbor rounding quantization is performed to obtain z_UV_hat.

[0189] 12. The z_UV_hat obtained in step 11 is processed through an entropy encoder (lossless encoder) to obtain a third bitstream denoted as bit_z_UV.

[0190] 13. The z_UV_hat obtained in step 11 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0191] 14. The z_UV_hat obtained in step 11 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0192] 15. Subtract the mean_UV output in step 13 from the y_UV output in step 10 to obtain residual_UV, and perform nearest neighbor rounding quantization to obtain residual_UV_hat.

[0193] 16. Based on the variance_UV generated in step 14, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the residual_UV_hat obtained in step 15 is processed through an entropy encoder (Lossless Encoder) to obtain a fourth bitrate denoted as bit_y_UV.

[0194] 17. Obtain the sum of the four bitstreams above as the final encoded bitstream.

[0195] Decoder side:

[0196] 1. The first bitstream bit_z_Y is processed through an entropy decoder (Lossless Decoder) to obtain z_Y_hat.

[0197] 2. The z_Y_hat obtained in step 1 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0198] 3. The z_Y_hat obtained in step 1 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0199] 4. Based on the variance_Y generated in step 2, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the second bitstream bit_y_Y is processed through an entropy decoder (Lossless Decoder) to obtain residual_Y_hat.

[0200] 5. Add residual_Y_hat obtained in step 4 and mean_Y obtained in step 2 to obtain y1_Y.

[0201] 6. The y1_Y obtained in step 5 is processed through a Y-component decoding (Y Decoder) network to obtain the Y component of the reconstructed image.

[0202] 7. The third bitstream, bit_z_UV, is processed through an entropy decoder (Lossless Decoder) to obtain z_UV_hat.

[0203] 8. The z_UV_hat obtained in step 7 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0204] 9. The z_UV_hat obtained in step 7 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0205] 10. Based on the variance_UV generated in step 9, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the fourth bitstream bit_y_UV is processed through an entropy decoder (Lossless Decoder) to obtain residual_UV_hat.

[0206] 11. Add the residual_UV_hat obtained in step 10 and the mean_UV obtained in step 8 to obtain y1_UV.

[0207] 12. Channel concatenation is performed on y1_Y obtained in step 5 and y1_UV obtained in step 11 to obtain y1.

[0208] 13. The y1 obtained in step 12 is processed through a convolutional layer to obtain y2.

[0209] 14. Perform channel concatenation on y2 obtained in step 13 and y1_UV obtained in step 11 to obtain y3.

[0210] 15. The y3 obtained in step 14 is input to the UV Decoder network, and upsampling is performed four times in the UV Decoder network (optionally, the four upsamplings may be performed by performing 2-scale upsampling through two deconvolution modules, and then 2-scale upsampling through pixel shuffle) to obtain the UV component of the reconstructed image. (The UV Decoder network includes four upsamplings.)

[0211] 16. The Y component obtained in step 6 and the UV component obtained in step 15 are concatenated, and then a color conversion is performed to obtain a reconstructed RGB image.

[0212] The difference between this embodiment and the embodiment shown in Figure 8 lies in the underlined step 10 on the encoder side and the underlined step 15 on the decoder side. Specifically, on the encoder side, downsampling is not performed on the Y and UV components, and the number of downsampling cycles in the UV Encoder changes from 3 to 4. On the decoder side, the number of upsampling cycles in the UV Decoder also changes from 3 to 4, and upsampling is not performed on the output of the UV Decoder.

[0213] Figure 11 shows the structure of a JPEG AI using a fused structure. As shown in Figure 11, the JPEG AI uses the fused structure shown in Figure 7a or Figure 7b, and its processing process is as follows.

[0214] Encoder side:

[0215] 1. Convert the original RGB image to a YUV444 image through color space conversion.

[0216] 2. The YUV444 image obtained in step 1 is decomposed channel by channel to obtain the Y component and UV component.

[0217] 3. The Y component obtained in step 2 is input into the Y component coding (Y Encoder) network, and downsampling is performed four times to obtain y_Y.

[0218] 4. Process the y_Y obtained in step 3 through a Y component hyper prior encoding network (Y Hyper Encoder Net), and then perform round-to-nearest quantization to obtain z_Y_hat.

[0219] 5. Process the z_Y_hat obtained in step 4 through a lossless encoder to obtain a first bit stream denoted as bit_z_Y.

[0220] 6. Process the z_Y_hat obtained in step 4 through a Y component hyper prior decoding network (Y Hyper Decoder) to obtain mean_Y.

[0221] 7. Process the z_Y_hat obtained in step 4 through a Y component hyper prior scale decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0222] 8. Subtract the mean_Y obtained in step 6 from the y_Y obtained in step 3 to obtain residual_Y, and perform round-to-nearest quantization to obtain residual_Y_hat.

[0223] 9. Based on the variance_Y generated in step 7, obtain a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y, and process the residual_Y_hat obtained in step 8 through a lossless encoder to obtain a second bit stream denoted as bit_y_Y.

[0224] 10. The Y and UV components obtained in step 2 are each downsampled once. Then, the downsampled components are input into the UV component coding (UV Encoder) network, and the UV Encoder network performs downsampling four times to obtain y_UV. (The UV Encoder network includes four downsampling steps.)

[0225] 11. The y_UV obtained in step 10 is processed through a UV component hyperplier coding network (UV Hyper Encoder Net), and then nearest neighbor rounding quantization is performed to obtain z_UV_hat.

[0226] 12. The z_UV_hat obtained in step 11 is processed through an entropy encoder (lossless encoder) to obtain a third bitstream denoted as bit_z_UV.

[0227] 13. The z_UV_hat obtained in step 11 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0228] 14. The z_UV_hat obtained in step 11 is processed through a UV component hyper-priority distributed decoding network (UV Hyper Scale Decoder) to obtain variance_UV.

[0229] 15. Subtract the mean_UV output in step 13 from the y_UV output in step 10 to obtain residual_UV, and perform nearest neighbor rounding quantization to obtain residual_UV_hat.

[0230] 16. Based on the variance_UV generated in step 14, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_UV is obtained, and the residual_UV_hat obtained in step 15 is processed through an entropy encoder (Lossless Encoder) to obtain a fourth bitrate denoted as bit_y_UV.

[0231] 17. Obtain the sum of the four bitstreams above as the final encoded bitstream.

[0232] Decoder side:

[0233] 1. The first bitstream bit_z_Y is processed through an entropy decoder (Lossless Decoder) to obtain z_Y_hat.

[0234] 2. The z_Y_hat obtained in step 1 is processed through the Y component hyperdecoder network (Y Hyper Decoder) to obtain mean_Y.

[0235] 3. The z_Y_hat obtained in step 1 is processed through the Y component hyper-priority distributed decoding network (Y Hyper Scale Decoder) to obtain variance_Y.

[0236] 4. Based on the variance_Y generated in step 2, a Gaussian probability distribution function with a mean of 0 and a standard deviation of variance_Y is obtained, and the second bitstream bit_y_Y is processed through an entropy decoder (Lossless Decoder) to obtain residual_Y_hat.

[0237] 5. Add residual_Y_hat obtained in step 4 and mean_Y obtained in step 2 to obtain y1_Y.

[0238] 6. The y1_Y obtained in step 5 is processed through a Y-component decoding (Y Decoder) network to obtain the Y component of the reconstructed image.

[0239] 7. The third bitstream, bit_z_UV, is processed through an entropy decoder (Lossless Decoder) to obtain z_UV_hat.

[0240] 8. The z_UV_hat obtained in step 7 is processed through the UV component hyperplier decoding network (UV Hyper Decoder) to obtain mean_UV.

[0241] 9. Process the obtained \(z_{\text{UV}}\hat{}\) in Step 7 through a UV component hyper-scale decoder to obtain \(\text{variance}_{\text{UV}}\).

[0242] 10. Based on the \(\text{variance}_{\text{UV}}\) generated in Step 9, obtain a Gaussian probability distribution function with a mean of 0 and a standard deviation of \(\text{variance}_{\text{UV}}\), and process the fourth bit stream \(\text{bit}_y_{\text{UV}}\) through an entropy decoder (Lossless Decoder) to obtain \(\text{residual}_{\text{UV}}\hat{}\).

[0243] 11. Add the obtained \(\text{residual}_{\text{UV}}\hat{}\) in Step 10 and the obtained \(\text{mean}_{\text{UV}}\) in Step 8 to obtain \(y1_{\text{UV}}\).

[0244] 12. Perform downsampling once on the obtained \(y1_Y\) in Step 5 to obtain \(y2_Y\), and perform channel concatenation on \(y2_Y\) and the obtained \(y1_{\text{UV}}\) in Step 11 to obtain \(y1\).

[0245] 13. Process the obtained \(y1\) in Step 12 through a convolutional layer to obtain \(y2\).

[0246] 14. Perform channel concatenation on the obtained \(y2\) in Step 13 and the obtained \(y1_{\text{UV}}\) in Step 11 to obtain \(y3\).

[0247] 15. The y3 obtained in step 14 is input to the UV Decoder network. The UV Decoder network performs upsampling four times, and then performs upsampling once on the output of the UV Decoder to obtain the UV component of the reconstructed image. (The UV Decoder network includes four upsampling steps.)

[0248] 16. Concatenate the obtained Y component in Step 6 and the obtained UV component in Step 15, and then perform color conversion to obtain a reconstructed RGB image.

[0249] The difference between this embodiment and the embodiment shown in Figure 8 lies in the underlined step 10 on the encoder side and the underlined step 15 on the decoder side. Specifically, on the encoder side, the number of downsampling cycles in the UV Encoder changes from 3 to 4, and on the decoder side, the number of upsampling cycles in the UV Decoder also changes from 3 to 4.

[0250] Figure 12 is a diagram showing the structure of an image decompression device 1200 according to an embodiment of this application. As shown in Figure 12, the image decompression device 1200 in this embodiment may be used on the decoder side. The image decompression device 1200 may include an acquisition module 1201, a processing module 1202, and a reconstruction module 1203.

[0251] The acquisition module 1201 is configured to acquire a first feature tensor and a second feature tensor, where the first feature tensor corresponds to the first component of the reconstructed image and the second feature tensor corresponds to the second component of the reconstructed image. The processing module 1202 is configured to perform concatenation and convolution on the first and second feature tensors to acquire a third feature tensor, and to acquire a fourth feature tensor based on the third and second feature tensors. The reconstruction module 1203 is configured to acquire a reconstructed image based on the first and fourth feature tensors.

[0252] In a possible implementation, the processing module 1202 is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain a fourth feature tensor.

[0253] In a possible implementation, the processing module 1202 is specifically configured to obtain a fourth feature tensor by adding the third feature tensor and the second feature tensor.

[0254] In a possible implementation, the processing module 1202 is specifically configured to perform channel extraction on the second feature tensor to obtain the fifth feature tensor, and then concatenate the third and fifth feature tensors to obtain the fourth feature tensor.

[0255] In a possible implementation, the processing module 1202 is specifically configured to perform channel extraction on the second feature tensor to obtain the fifth feature tensor, and then add the third and fifth feature tensors to obtain the fourth feature tensor.

[0256] In a possible implementation, the first component is the Y component, and the second component is the UV component.

[0257] The apparatus in this embodiment may be configured to perform the technical solution of the embodiment of the method shown in Figure 6. The implementation principle and technical effects are similar. Further details are not described herein.

[0258] In the implementation process, the steps in the embodiments of the above method may be performed by hardware integrated logic circuits within a processor or by using instructions in the form of software. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in embodiments of this application may be presented directly as being performed and completed by a hardware encoding processor, or as being performed and completed by a combination of hardware and software modules within the encoding processor. The software modules may reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory. The processor reads information from memory and, in combination with hardware within the processor, completes the steps of the above method.

[0259] The memory in the above embodiment may be volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) used as an external cache. Many forms of RAM may be used, not as an example but as an example, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus random access memory (direct rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described in this specification includes, but is not limited to, these and any other suitable types of memory.

[0260] Those skilled in the art will recognize, in combination with the examples described in the embodiments disclosed in this specification, that units and algorithmic steps may be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation methods should not be considered to exceed the scope of this application.

[0261] For the purpose of convenience and concise explanation, it will be readily apparent to those skilled in the art that the detailed operating processes of the above systems, apparatuses, and units are described by referring to the corresponding processes in the embodiments of the above methods. Further details are not described herein.

[0262] In some embodiments provided in this application, it should be understood that the disclosed systems, apparatuses and methods may be implemented in other ways. For example, the embodiments of the described apparatus are merely examples. For example, the division into units is merely a logical functional division, and other divisions may be used in actual implementations. For example, multiple units or components may be combined, integrated into other systems, or some features may be ignored or not performed. Furthermore, the mutual coupling, direct coupling, or communication connection indicated or discussed may be implemented by using some interfaces. Indirect coupling or communication connection between apparatuses or units may be implemented electronically, mechanically, or in other forms.

[0263] Units described as separate parts may or may not be physically separate, and parts shown as units may or may not be physical units, may be located in one place, or may be distributed across multiple network units. Some or all of the units may be selected based on the actual requirements to achieve the objectives of the solution of the embodiment.

[0264] Furthermore, the functional units in the embodiments of this application may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit.

[0265] When implemented in the form of a software function unit and sold or used as a standalone product, the functions may be stored on a computer-readable storage medium. Based on such understanding, the technical solutions of this application, or parts of the technical solutions that contribute to the prior art, may be implemented in the form of a software product. A computer software product is stored on a storage medium and includes several instructions for instructing a computer device (such as a personal computer, server, or network device) to perform all or part of the steps of the method in the embodiments of this application. The storage medium includes any medium capable of storing program code, such as a USB flash drive, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0266] The above description is merely a specific way of realizing this application and is not intended to limit the scope of protection of this application. Any modification or substitution that is readily conceivable by a person skilled in the art within the scope of the technical scope disclosed in this application shall fall within the scope of protection of this application. Accordingly, the scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. This is a method for decompressing images. The first step is to obtain a first feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image. The step is to obtain a second feature tensor, wherein the second feature tensor corresponds to a second component of the reconstructed image. The steps include performing concatenation and convolution on the first and second feature tensors to obtain a third feature tensor, A step of obtaining a fourth feature tensor based on the third feature tensor and the second feature tensor, The steps include obtaining the reconstructed image based on the first feature tensor and the fourth feature tensor, and A method that includes this.

2. The step of obtaining the fourth feature tensor based on the third feature tensor and the second feature tensor is: The method according to claim 1, comprising the step of concatenating the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

3. The step of obtaining the fourth feature tensor based on the third feature tensor and the second feature tensor is: The method according to claim 1, further comprising the step of adding the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

4. The step of obtaining the fourth feature tensor based on the third feature tensor and the second feature tensor is: The steps include performing channel extraction on the second feature tensor to obtain a fifth feature tensor, The steps include concatenating the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor, and The method according to any one of claims 1 to 3, including

5. The step of obtaining the fourth feature tensor based on the third feature tensor and the second feature tensor is: The steps include performing channel extraction on the second feature tensor to obtain a fifth feature tensor, The steps include adding the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor, and The method according to any one of claims 1 to 3, including

6. The method according to any one of claims 1 to 5, wherein the first component is a Y component and the second component is a UV component.

7. An image decompression device, An acquisition module configured to acquire a first feature tensor and a second feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image and the second feature tensor corresponds to a second component of the reconstructed image. A processing module configured to perform concatenation and convolution on the first and second feature tensors to obtain a third feature tensor, and to obtain a fourth feature tensor based on the third and second feature tensors, A reconstruction module configured to acquire the reconstructed image based on the first feature tensor and the fourth feature tensor, A device that includes this.

8. The apparatus according to claim 7, wherein the processing module is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

9. The apparatus according to claim 7, wherein the processing module is specifically configured to obtain the fourth feature tensor by adding the third feature tensor and the second feature tensor.

10. The apparatus according to any one of claims 7 to 9, wherein the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor, and to concatenate the third feature tensor and the fifth feature tensor to obtain a fourth feature tensor.

11. The apparatus according to any one of claims 7 to 9, wherein the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor, and to add the third feature tensor and the fifth feature tensor to obtain a fourth feature tensor.

12. The apparatus according to any one of claims 7 to 11, wherein the first component is the Y component and the second component is the UV component.

13. An electronic device comprising one or more processors and memory configured to store one or more programs, An electronic device in which, when the one or more programs described above are executed by the one or more processors, the one or more processors become capable of carrying out the method described in any one of claims 1 to 6.

14. A computer-readable storage medium containing a computer program, A computer-readable storage medium that, when the computer program is executed on the computer, enables the computer to perform the method according to any one of claims 1 to 6.

15. A computer program product, The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer becomes capable of performing the method according to any one of claims 1 to 6.

Images