Image decompression method and apparatus

MYPI2025006402A0Undetermined Publication Date: 2024-10-18HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
MYPI2025006402
Authority / Receiving Office
MY · MY
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-18
Filing Date
2024-01-30
Publication Date
2024-10-18

AI Technical Summary

Technical Problem

During the encoding and decoding process of the existing image/video compression algorithm, the information interaction between the Y component and the UV component is insufficient, resulting in insufficient compression performance. Especially under the lightweight target, the computational complexity is reduced but the performance drops sharply.

Method used

By obtaining the feature tensors of the Y component and UV component of the image for splicing and convolution, the fusion structure is optimized so that the number of channels of the fourth feature tensor is less than, equal to, or greater than the channels of the first feature tensor and the second feature tensor. The sum of the numbers can save or increase the number of channels, and improve the convolution operation to splicing or point addition operation to improve performance.

Benefits of technology

The performance improvement of the image/video compression algorithm has been achieved, which can not only save computing power while reducing the weight, but also greatly improve the performance and improve the image decoding quality without reducing the number of channels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure PI2025006402_DRAWING_001
    Figure PI2025006402_DRAWING_001
  • Figure PI2025006402_DRAWING_002
    Figure PI2025006402_DRAWING_002
  • Figure PI2025006402_DRAWING_003
    Figure PI2025006402_DRAWING_003
Patent Text Reader

Abstract

This application provides an image decompression method and apparatus. The image decompression method in this application includes: obtaining a first feature tensor, where the first feature tensor corresponds to a first component of a reconstructed image; obtaining a second feature tensor, where the second feature tensor corresponds to a second component of the reconstructed image; performing concatenation and convolution on the first feature tensor and the second feature tensor to obtain a third feature tensor; obtaining a fourth feature tensor based on the third feature tensor and the second feature tensor; and obtaining the reconstructed image based on the first feature tensor and the fourth feature tensor. In embodiments of this application, a structure for fusing a Y component and a UV component is optimized. This can reduce a quantity of channels, to save computing power and implement lightweight. Even if the quantity of channels is not reduced, performance can still be greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Image decompression method and device Technical Field

[0001] The present application relates to image processing technology, and in particular to an image decompression method and device. Background Art

[0002] As convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks like image recognition and object detection, a growing number of researchers are exploring deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms. For example, these algorithms combine modules such as the encoding network, entropy estimation network, entropy encoding engine, entropy decoding engine, and decoding network into a single entity, or combine modules such as the prediction module and residual compression module into a single entity. The encoding network and decoding network, also known as the transform module and inverse transform module, are generally composed of convolutional layers and nonlinear transformation units.

[0003] The Joint Photographic Experts Group (JPEG) Artificial Intelligence (AI) standard is the first international standard for AI image compression. In its typical structure, on the encoding side, the image is decomposed into Y and UV components through RGB-YUV color space conversion, and then encoded and transmitted separately; on the decoding side, the Y and UV components are restored separately, and then the YUV-RGB color space conversion is performed to obtain an RGB image. In the aforementioned process, there is also a certain amount of information interaction between the Y and UV components, which further improves the compression effect. Information interaction between different components is particularly important for performance optimization.

[0004] Summary of the Invention

[0005] The present application provides an image decompression method and apparatus that can improve performance.

[0006] In a first aspect, the present application provides an image decompression method, comprising: obtaining a first feature tensor, the first feature tensor corresponding to a first component of a reconstructed image; obtaining a second feature tensor, the second feature tensor corresponding to a second component of the reconstructed image; concatenating and convolving the first feature tensor and the second feature tensor to obtain a third feature tensor; obtaining a fourth feature tensor based on the third feature tensor and the second feature tensor; and obtaining the reconstructed image based on the first feature tensor and the fourth feature tensor.

[0007] In an embodiment of the present application, the fusion structure of the Y component and the UV component is optimized. First, the fourth feature tensor (corresponding to the UV component of the reconstructed image) is obtained based on the third feature tensor (i.e., the output tensor after convolution of the first feature tensor (corresponding to the Y component of the reconstructed image) and the second feature tensor (corresponding to the UV component of the reconstructed image)) and the second feature tensor, and then the reconstructed image is obtained based on the first feature tensor and the fourth feature tensor. There are three cases in which the number of channels of the aforementioned fourth feature tensor is less than, greater than, or equal to the sum of the number of channels of the first feature tensor and the second feature tensor. When the case of being less than occurs, the number of channels of the fourth feature tensor can be reduced, which can save computing power and achieve lightweighting; when the case of being greater than or equal to occurs, even if the number of channels of the fourth feature tensor is not reduced, the third feature tensor can obtain the fourth eigenvector only by operating with the second feature tensor, which can greatly improve performance.

[0008] Optionally, the first component may be a Y component of an image represented in a YUV color space, and the second component may be a UV component of an image represented in a YUV color space.

[0009] It should be noted that, depending on the image representation method, the first component and the second component may also be other components, and the embodiments of the present application do not specifically limit this.

[0010] As mentioned above, in the JPEG AI structure, at the encoder end, the image is decomposed into Y and UV components, which are then encoded and transmitted separately. At the decoder end, the Y and UV components are decoded and restored separately. It can be seen that the Y component (i.e., the first component) and the UV component (i.e., the second component) are processed separately, while at the decoder end, the Y and UV components are merged in the UV component processing path.

[0011] Concatenating the first feature tensor and the second feature tensor (Concat) can be channel concatenation of the two. For example, if the scale of the first feature tensor is [128, H, W] and the scale of the second feature tensor is [64, H, W], then the scale of the concatenated feature tensor is [192, H, W], that is, the width and height remain unchanged, and the number of channels is the sum of the number of channels of the two. The concatenated feature tensor is then convolved (Conv). The convolution process can increase / decrease / remain unchanged the number of channels. Correspondingly, the number of channels of the output third feature tensor increases / decreases / remains unchanged. For example, the scale of the third feature tensor is [64, H, W], [128, H, W], or [192, H, W]. The first element in the aforementioned scale represents the number of channels of the feature tensor.

[0012] Optionally, the third feature tensor and the second feature tensor may be concatenated to obtain a fourth feature tensor.

[0013] Splicing is channel splicing, that is, channel splicing of the third feature tensor and the second feature tensor.

[0014] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. When the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first feature tensor and the second feature tensor, the number of channels can be reduced, thereby saving computing power and achieving lightweighting; even if the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first feature tensor and the second feature tensor, after the convolution operation, the point-by-point addition operation is changed to a splicing operation, and the output of the convolution operation is changed from a point-by-point addition operation with the fused feature tensor of the Y component and the UV component to a splicing operation only with the UV component, which can greatly improve performance.

[0015] Optionally, channel extraction may be performed on the second feature tensor to obtain a fifth feature tensor; and the third feature tensor and the fifth feature tensor may be concatenated to obtain a fourth feature tensor.

[0016] The second feature tensor can be subjected to channel reduction processing to obtain the fifth feature tensor, and then the two aforementioned tensors can be concatenated.

[0017] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. When the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first feature tensor and the second feature tensor, the number of channels can be reduced, thereby saving computing power and achieving lightweighting; even if the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first feature tensor and the second feature tensor, after the convolution operation, the point-by-point addition operation is changed to a splicing operation, and the output of the convolution operation is changed from a point-by-point addition operation with the fused feature tensor of the Y component and the UV component to a splicing operation only with the UV component, which can greatly improve performance.

[0018] Optionally, the third eigentensor and the second eigentensor may be added together to obtain a fourth eigentensor.

[0019] When the number of channels of the third feature tensor and the second feature tensor is equal, the addition of the two can be point-by-point addition; when the number of channels of the third feature tensor and the second feature tensor is not equal, the addition of the two can be point-by-point addition of the feature tensor with a smaller channel number and part of the channels of the feature tensor with a larger channel number. For example, if the number of channels of the third feature tensor is 64 and the number of channels of the second feature tensor is 32, then the 32 channels of the second feature tensor can be added point-by-point to the first 32 channels of the third feature tensor; or, the 32 channels of the second feature tensor can be added point-by-point to the last 32 channels of the third feature tensor, and so on. The embodiment of the present application does not specifically limit the addition method in this case.

[0020] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. When the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first feature tensor and the second feature tensor, the number of channels can be reduced, thereby saving computing power and achieving lightweighting; even if the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first feature tensor and the second feature tensor, after the convolution operation, the output of the convolution operation is changed from a dot addition operation with the fused feature tensor of the Y component and the UV component to a dot addition operation only with the UV component, which can greatly improve performance.

[0021] Optionally, channel extraction may be performed on the second feature tensor to obtain a fifth feature tensor; and the third feature tensor and the fifth feature tensor may be added to obtain a fourth feature tensor.

[0022] When the number of channels of the third feature tensor is smaller than that of the second feature tensor, the second feature tensor can be subjected to channel reduction processing to obtain the fifth feature tensor, so that the number of channels of the fifth feature tensor is equal to that of the second feature tensor, and then the two are added point by point.

[0023] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. In the aforementioned case, when the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first feature tensor and the second feature tensor, the number of channels can be reduced, thereby saving computing power and achieving lightweighting; even if the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first feature tensor and the second feature tensor, after the convolution operation, the output of the convolution operation is changed from a dot addition operation with the fused feature tensor of the Y component and the UV component to a dot addition operation only with the UV component, which can greatly improve performance.

[0024] The fourth eigentensor obtained after the fusion process can be further processed (see the embodiment below) to obtain the UV component of the reconstructed image, which is then combined with the Y component of the reconstructed image to form a complete reconstructed image.

[0025] In second aspect, the present application provides an image decompression device, comprising: an acquisition module for acquiring a first feature tensor, the first feature tensor corresponding to a first component of a reconstructed image; acquiring a second feature tensor, the second feature tensor corresponding to a second component of the reconstructed image; a processing module for splicing and convolving the first feature tensor and the second feature tensor to obtain a third feature tensor; obtaining a fourth feature tensor based on the third feature tensor and the second feature tensor; and a reconstruction module for obtaining the reconstructed image based on the first feature tensor and the fourth feature tensor.

[0026] In a possible implementation, the processing module is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

[0027] In a possible implementation, the processing module is specifically configured to add the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

[0028] In a possible implementation, the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and concatenate the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

[0029] In a possible implementation, the processing module is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and add the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

[0030] In a possible implementation, the first component is a Y component, and the second component is a UV component.

[0031] In a third aspect, the present application provides an electronic device comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the above-mentioned first aspects.

[0032] In a fourth aspect, the present application provides a computer-readable storage medium comprising a computer program, wherein when the computer program is executed on a computer, the computer executes the method according to any one of the above-mentioned first aspects.

[0033] In a fifth aspect, the present application provides a computer program, which, when executed by a computer, is used to perform any of the methods described in the first aspect.

[0034] In a sixth aspect, the present application also provides a computer program product, which includes computer program code. When the computer program code runs on a computer, it enables the computer to execute the operations and / or processing performed by the electronic device in any of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is an example diagram of an end-to-end deep learning image encoding and decoding framework;

[0036] Figure 2 is an example diagram of an end-to-end deep learning video encoding and decoding framework;

[0037] FIG3 is an example diagram of an application scenario of an embodiment of the present application;

[0038] FIG4 is an example diagram of an application scenario of an embodiment of the present application;

[0039] FIG5 a is a schematic diagram of a typical structure in the JPEG AI standard;

[0040] FIG5 b is a schematic diagram of the structure within the bold frame in FIG5 a ;

[0041] FIG6 is a flowchart of a process 600 of an image decompression method according to an embodiment of the present application;

[0042] Figure 7a is a schematic diagram of the fusion structure;

[0043] Figure 7b is a schematic diagram of the fusion structure;

[0044] Figure 7c is a schematic diagram of the fusion structure;

[0045] Figure 7d is a schematic diagram of the fusion structure;

[0046] FIG8 is a schematic diagram of the structure of JPEG AI using a fusion structure;

[0047] FIG9 is a schematic diagram of the structure of JPEG AI using a fusion structure;

[0048] FIG10 is a schematic diagram of the structure of JPEG AI using a fusion structure;

[0049] FIG11 is a schematic diagram of the structure of JPEG AI using a fusion structure;

[0050] FIG12 is a schematic structural diagram of an image decompression device 1200 according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0052] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0054] Figure 1 is an example diagram of an end-to-end deep learning image encoding and decoding framework. As shown in Figure 1, the image encoding and decoding framework includes: encoding side: encoding network (Encoder), quantization module, hyper priori encoding network (Hyper Encoder), hyper priori decoding network, entropy estimation, entropy encoding engine (commonly used entropy encoding engines include arithmetic encoding (Arithmetic Encoder, AE), asymmetric numeral system (Asymmetric Numeral System, ANS)), etc.); decoding side: hyper priori decoding network (Hyper Decoder), decoding network (Decoder), entropy estimation and entropy decoding engine (commonly used entropy decoding engines include arithmetic decoding (Arithmetic Decoder, AD), ANS, etc.).

[0055] On the encoding side, the original image is transformed from the image domain to the feature domain through the encoding network. The transformed image features are then processed by the quantization module and the entropy coding engine and encoded into a bitstream for transmission or storage. On the decoding side, the bitstream is decoded into image features through the entropy decoding engine. The image features are then transformed from the feature domain to the image domain through the decoding network to produce a reconstructed image. The entropy estimation network estimates the image features and derives the estimated probability value for each feature element for processing by the entropy coding and decoding engines.

[0056] In this embodiment, both the encoding network (Encoder) and the decoding network (Decoder) have nonlinear transformation units.

[0057] FIG2 is an example diagram of an end-to-end deep learning video codec framework. As shown in FIG2 , the video codec framework includes a prediction module (predict model) and a residual compression module (residual compress).

[0058] The prediction module uses the reconstructed image of the previous frame to predict the current frame and generate a predicted image. The residual compression module compresses the residual between the original image and the predicted image of the current frame and decompresses it to obtain a reconstructed residual. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame. Both the encoding and decoding subnetworks in the prediction and residual compression modules have nonlinear transformation units.

[0059] In this embodiment, both the prediction module (predict model) and the residual compression module (residual compress) have nonlinear transformation units.

[0060] Figure 3 is an example diagram of an application scenario of an embodiment of the present application. As shown in Figure 3, the application scenario can be a terminal, cloud server, or video surveillance service involving image / video acquisition, storage, or transmission, such as terminal photo / video recording, photo album, cloud photo album, video surveillance, etc.

[0061] Encoding: The camera captures images / videos. The artificial intelligence (AI) image / video encoding network extracts features from the images / video to obtain low-redundancy image features. It then performs entropy encoding based on these features to produce a bitstream / image file.

[0062] Decoding end: When images / videos need to be output, the AI ​​image / video decoding network performs entropy decoding on the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.

[0063] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits them (for example, cloud services, live broadcast technology, etc.).

[0064] Figure 4 is an example diagram of an application scenario of an embodiment of the present application. As shown in Figure 4, the application scenario can be a business involving image / video acquisition, storage or transmission in the cloud or video surveillance, such as cloud photo album, video surveillance, live broadcast, etc.

[0065] Encoding: The local computer obtains images / videos, encodes them to obtain compressed images / videos, and then sends the compressed images / videos to the cloud. The cloud decodes the compressed images / videos to obtain images / videos, then performs AI encoding on the images / videos to obtain and store the bitstream / image files.

[0066] Decoding: When the local device needs to obtain an image / video from the cloud, the cloud performs AI decoding on the bitstream / image file to obtain the image / video. The image / video is then encoded to obtain a compressed image / video, which is then sent to the local device. The local device decodes the compressed image / video to obtain the image / video. The structure of the cloud and the purpose of each module can be referred to in Figure 4 and will not be further described in detail in this embodiment of the present application.

[0067] Figure 5a shows a typical structure in the JPEG AI standard. As shown in Figure 5a, at the encoding end, the image is decomposed into Y and UV components through RGB-to-YUV color space conversion, and then encoded and transmitted separately. At the decoding end, the Y and UV components are restored separately, and then the YUV-to-RGB color space conversion is performed to obtain an RGB image. There is also a certain amount of information exchange between the Y and UV components at the encoding and decoding ends, which further improves the compression effect.

[0068] FIG5b is a schematic diagram of the structure within the thick line frame in FIG5a, as shown in FIG5b, wherein, The scale is [128,H,W], The scale is [64, H, W], and the width and height of the two are equal. The processing steps are as follows:

[0069] 1) After 1 downsampling, we get

[0070] 2) and Input concatenation module, the two are concatenated on the channel to obtain x1, the scale of x1 is [192, H, W];

[0071] 3) x1 is input to the convolution (Conv) module and outputs x2. The scale of x2 is [192, H, W]. The 192 in the convolution module represents the number of output channels of the convolution module, 3×3 represents the size of the convolution kernel of the convolution module, and S1 represents the stride of the convolution module.

[0072] 4) Add x1 and x2 point by point to get x3, where the scale of x3 is [192, H, W].

[0073] The structure within the bold box is a ResBlock structure, characterized by the fact that the input feature tensor of the convolution module has the same number of channels as the output feature tensor. This allows the feature tensors before and after convolution to be directly added together using skip connections. However, in the JPEG AI structure, the input feature tensor is a concatenation of the Y and UV components, often with a large number of channels. While reducing the number of channels for lightweighting reduces computational complexity, performance also decreases dramatically.

[0074] To solve the above technical problems, in the above encoding / decoding network and application scenarios, the embodiments of the present application provide an image decompression method to improve the performance of the image / video compression algorithm. It should be noted that the image in the embodiments of the present application can refer to a single image, such as a photo or picture, or an image frame in a video. That is, the method of the embodiments of the present application can be used to process a single image or a sequence of image frames in a video. Unless otherwise specified, videos / images are collectively referred to as images in this document.

[0075] FIG6 is a flow chart of process 600 of the image decompression method according to an embodiment of the present application. Process 600 may be performed by the decoding end in the above-described embodiment. While process 600 is described as a series of steps or operations, it should be understood that process 600 may be performed in various orders and / or concurrently, and is not limited to the order shown in FIG6 . Process 600 includes the following steps:

[0076] Step 601: Obtain a first feature tensor, where the first feature tensor corresponds to a first component of a reconstructed image.

[0077] Step 602: Obtain a second eigenvalue tensor, where the second eigenvalue tensor corresponds to a second component of the reconstructed image.

[0078] Optionally, the first component may be a Y component of an image represented in a YUV color space, and the second component may be a UV component of an image represented in a YUV color space.

[0079] It should be noted that, depending on the image representation method, the first component and the second component may also be other components, and the embodiments of the present application do not specifically limit this.

[0080] As described above, in the JPEG AI structure, at the encoding end, the image is decomposed into Y and UV components, which are then encoded and transmitted separately; at the decoding end, the Y and UV components are decoded and restored separately. It can be seen that the Y component (i.e., the first component) and the UV component (i.e., the second component) are processed separately. However, as shown in Figure 5a, at the decoding end, the Y and UV components are fused in the path for processing the UV component, which is the operation within the bold box shown in Figure 5b. Assuming that the structure within the bold box (used to achieve the fusion of the Y and UV components) is regarded as a black box, then the black box has two inputs, one is the first feature tensor (corresponding to the Y component, from the path for processing the Y component, and its specific acquisition process can be referred to in the embodiment below), and the other is the second feature tensor (corresponding to the UV component, from the path for processing the UV component, and its specific acquisition process can be referred to in the embodiment below); the black box has one output (the fourth feature tensor below).

[0081] The embodiment of the present application improves the structure for implementing the fusion function, and the following steps can be referred to.

[0082] Step 603: Concatenate and convolve the first feature tensor and the second feature tensor to obtain a third feature tensor.

[0083] Concatenating the first feature tensor and the second feature tensor (Concat) can be channel concatenation of the two. For example, the scale of the first feature tensor is [128, H, W] and the scale of the second feature tensor is [64, H, W]. Then the scale of the concatenated feature tensor is [192, H, W], that is, the width and height remain unchanged, and the number of channels is the sum of the number of channels of the two. The concatenated feature tensor is then convolved (Conv). The convolution process can increase / decrease / remain unchanged the number of channels. Correspondingly, the number of channels of the output third feature tensor increases / decreases / remains unchanged. For example, the scale of the third feature tensor is [64, H, W], [128, H, W], or [192, H, W].

[0084] Step 604: Obtain a fourth eigentensor according to the third eigentensor and the second eigentensor.

[0085] Optionally, the third feature tensor and the second feature tensor may be concatenated to obtain a fourth feature tensor.

[0086] Splicing is channel splicing, that is, channel splicing of the third feature tensor and the second feature tensor.

[0087] Figure 7a is a schematic diagram of the fusion structure. As shown in Figure 7a, the fusion structure first concatenates and convolves the first feature tensor and the second feature tensor to obtain a third feature tensor, and then concatenates the third feature tensor and the second feature tensor to obtain a fourth feature tensor. The steps are as follows: the first feature tensor The scale is [c Y ,H,W], the second eigentensor The scale is [c UV ,H,W], the width and height of the two are equal.

[0088] 1) and Input Concat module, splice on the channel to get x1, the scale of x1 is [c Y +c UV ,H,W];

[0089] 2) x1 is input into the Conv module and outputs the third feature tensor x2, the scale of x2 is [c′ UV ,H,W];

[0090] 3) And x2 input Concat module, splicing on the channel to get the fourth feature tensor x3, the scale of x3 is [c′ UV +c UV ,H,W].

[0091] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. For example, c′ UV +c UV Can be smaller than c Y +c UV , or, c′ UV +c UV Can be greater than c Y +c UV , or, c′ UV +c UV Can be smaller than c Y , and so on. In the aforementioned case, when the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first and second feature tensors, the number of channels can be reduced, thereby saving computing power and achieving lightweighting. Even when the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first and second feature tensors, compared to the embodiment shown in FIG5b , this embodiment changes the point-by-point addition operation to a concatenation operation after the convolution operation, and changes the point-by-point addition operation on the output of the convolution operation to a concatenation operation only with the UV component instead of the point-by-point addition operation with the fused feature tensor of the Y component and the UV component, which can greatly improve performance.

[0092] Optionally, channel extraction may be performed on the second feature tensor to obtain a fifth feature tensor; and the third feature tensor and the fifth feature tensor may be concatenated to obtain a fourth feature tensor.

[0093] The second feature tensor can be subjected to channel reduction processing to obtain the fifth feature tensor, and then the two aforementioned tensors can be concatenated.

[0094] Figure 7b is a schematic diagram of the fusion structure. As shown in Figure 7b, the fusion structure first concatenates and convolves the first feature tensor and the second feature tensor to obtain the third feature tensor, performs channel extraction on the second feature tensor to obtain the fifth feature tensor, and concatenates the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor. The steps are as follows: the first feature tensor The scale is [c Y ,H,W], the second eigentensor The scale is [c UV ,H,W], the width and height of the two are equal.

[0095] 1) and Input Concat module, splice on the channel to get x1, the scale of x1 is [c Y +c UV ,H,W];

[0096] 2) x1 is input into the Conv module and outputs the third feature tensor x2, the scale of x2 is [c′ UV ,H,W];

[0097] 3) According to the pre-set rules, Extract c″ UV The fifth feature tensor y′ is obtained through the channels UV , y′ UV The scale is [c UV ,H,W], where c″ UV and c′ UV They may be equal or unequal, and there is no specific limitation on this;

[0098] 4) y′ UV And x2 input Concat module, splicing on the channel to get the fourth feature tensor x3, the scale of x3 is [c′ UV +c″ UV ,H,W].

[0099] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. For example, c′ UV +c″ UV Can be smaller than c Y +cUV , or, c′ UV +c″ UV Can be greater than c Y +c UV , or, c′ UV +c″ UV Can be smaller than c Y or c UV , and so on. In the aforementioned case, when the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first and second feature tensors, the number of channels can be reduced, thereby saving computing power and achieving lightweighting. Even when the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first and second feature tensors, compared to the embodiment shown in FIG5b , this embodiment changes the point-by-point addition operation to a concatenation operation after the convolution operation, and changes the point-by-point addition operation on the output of the convolution operation to a concatenation operation only with the UV component instead of the point-by-point addition operation with the fused feature tensor of the Y component and the UV component, which can greatly improve performance.

[0100] Optionally, the third eigentensor and the second eigentensor may be added together to obtain a fourth eigentensor.

[0101] When the number of channels of the third feature tensor and the second feature tensor is equal, the addition of the two can be point-by-point addition; when the number of channels of the third feature tensor and the second feature tensor is not equal, the addition of the two can be point-by-point addition of the feature tensor with a smaller channel number and part of the channels of the feature tensor with a larger channel number. For example, if the number of channels of the third feature tensor is 64 and the number of channels of the second feature tensor is 32, then the 32 channels of the second feature tensor can be added point-by-point to the first 32 channels of the third feature tensor; or, the 32 channels of the second feature tensor can be added point-by-point to the last 32 channels of the third feature tensor, and so on. The embodiment of the present application does not specifically limit the addition method in this case.

[0102] Figure 7c is a schematic diagram of the fusion structure. As shown in Figure 7c, the fusion structure first concatenates and convolves the first feature tensor and the second feature tensor to obtain a third feature tensor, and then adds the third feature tensor and the second feature tensor to obtain a fourth feature tensor. The steps are as follows: the first feature tensor The scale is [c Y ,H,W], the second eigentensor The scale is [c UV ,H,W], the width and height of the two are equal.

[0103] 1) and Input Concat module, splice on the channel to get x1, the scale of x1 is [c Y +c UV ,H,W];

[0104] 2) x1 is input into the Conv module and outputs the third feature tensor x2, the scale of x2 is [c′ UV ,H,W];

[0105] 3) Added to x2, we get the fourth feature tensor x3, the scale of x3 is [c UV ,H,W],c″ UV It is c UV and c′ UV The larger of .

[0106] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. For example, c″ UV Can be smaller than c Y +c UV , or, c″ UV Can be greater than c Y +c UV , or, c″ UV Can be smaller than c Y , and so on. In the aforementioned case, when the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first and second feature tensors, the number of channels can be reduced, thereby saving computing power and achieving lightweighting. Even when the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first and second feature tensors, compared to the embodiment shown in FIG5b , after the convolution operation, this embodiment changes the point-wise addition operation on the output of the convolution operation to only the UV component instead of the point-wise addition operation with the fused feature tensor of the Y component and the UV component, which can greatly improve performance.

[0107] Optionally, channel extraction may be performed on the second feature tensor to obtain a fifth feature tensor; and the third feature tensor and the fifth feature tensor may be added to obtain a fourth feature tensor.

[0108] When the number of channels of the third feature tensor is smaller than that of the second feature tensor, the second feature tensor can be subjected to channel reduction processing to obtain the fifth feature tensor, so that the number of channels of the fifth feature tensor is equal to that of the second feature tensor, and then the two are added point by point.

[0109] Figure 7d is a schematic diagram of the fusion structure. As shown in Figure 7d, the fusion structure first concatenates and convolves the first feature tensor and the second feature tensor to obtain the third feature tensor, performs channel extraction on the second feature tensor to obtain the fifth feature tensor, and adds the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor. The steps are as follows: the first feature tensor The scale is [c Y ,H,W], the second eigentensor The scale is [c UV ,H,W], the width and height of the two are equal.

[0110] 1) and Input Concat module, splice on the channel to get x1, the scale of x1 is [c Y +c UV ,H,W];

[0111] 2) x1 is input into the Conv module and outputs the third feature tensor x2, the scale of x2 is [c′ UV ,H,W];

[0112] 3) According to the pre-set rules, Extract c′ UV The fifth feature tensor y′ is obtained through the channels UV , y′ UV The scale is [c′ UV ,H,W];

[0113] 4) y′ UV Added to x2, we get the fourth feature tensor x3, the scale of x3 is [c′ UV ,H,W].

[0114] It should be noted that the embodiment of the present application does not specifically limit the number of channels of the fourth feature tensor. For example, c′ UV Can be smaller than c Y +c UV , or, c′ UV Can be greater than c Y +c UV , or, c′ UV Can be smaller than c Y or c UV , and so on. In the aforementioned case, when the number of channels of the fourth feature tensor is less than the sum of the number of channels of the first and second feature tensors, the number of channels can be reduced, thereby saving computing power and achieving lightweighting. Even when the number of channels of the fourth feature tensor is greater than the sum of the number of channels of the first and second feature tensors, compared to the embodiment shown in FIG5b , after the convolution operation, this embodiment changes the point-wise addition operation on the output of the convolution operation to only the UV component instead of the point-wise addition operation with the fused feature tensor of the Y component and the UV component, which can greatly improve performance.

[0115] Step 605: Obtain a reconstructed image according to the first eigentensor and the fourth eigentensor.

[0116] The fourth eigentensor obtained after the fusion processing can be further processed (refer to the embodiment below) to obtain the UV component of the reconstructed image, which is then combined with the Y component of the reconstructed image obtained by processing the first eigentensor to form a complete reconstructed image.

[0117] In the embodiment of the present application, by optimizing the fusion structure of the Y component and the UV component, on the one hand, the number of channels can be reduced, computing power can be saved, and lightweight can be achieved; on the other hand, even if the number of channels is not reduced, the performance can still be greatly improved.

[0118] The technical solution of this application is described below through several specific embodiments.

[0119] FIG8 is a schematic diagram of the structure of a JPEG AI using a fusion structure. As shown in FIG8 , the JPEG AI uses the fusion structure shown in FIG7a or FIG7b , and the processing process is as follows:

[0120] Encoding side:

[0121] 1. The original image in RGB format is converted into a YUV444 format image through color space conversion;

[0122] 2. Decompose the YUV444 format image obtained in step 1 according to the channel to obtain the Y component and UV component;

[0123] 3. The Y component obtained in step 2 is input into the Y component encoding (Y Encoder) network and is downsampled four times to obtain y_Y;

[0124] 4. The y_Y obtained in step 3 is passed through the Y component hyper priori encoding network (Y Hyper Encoder Net), and then rounded and quantized to obtain z_Y_hat;

[0125] 5. The z_Y_hat obtained in step 4 is entropy encoded (Lossless Encoder) to obtain the first bit stream, recorded as bit_z_Y;

[0126] 6. z_Y_hat obtained in step 4 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0127] 7. The z_Y_hat obtained in step 4 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0128] 8. Subtract y_Y obtained in step 3 from mean_Y obtained in step 6 to obtain residual_Y, which is rounded and quantized to obtain residual_Y_hat;

[0129] 9. Obtain the Gaussian probability distribution function with zero mean and standard deviation of variance_Y according to the variance_Y generated in step 7. Perform entropy encoding (Lossless Encoder) on the residual_Y_hat obtained in step 8 to obtain the second bitstream, recorded as bit_y_Y.

[0130] 10. The Y component and UV component obtained in step 2 are downsampled once each and then input into the UV component encoding (UV Encoder) network. After three downsamplings in the UV Encoder network, y_UV is obtained. (The UV Encoder network includes three downsamplings)

[0131] 11. The y_UV obtained in step 10 is passed through the UV Hyper Encoder Net and then rounded and quantized to obtain z_UV_hat;

[0132] 12. The z_UV_hat obtained in step 11 is entropy encoded (Lossless Encoder) to obtain the third bit stream, recorded as bit_z_UV;

[0133] 13. The z_UV_hat obtained in step 11 is passed through the UV component hyper-decoder network to obtain mean_UV;

[0134] 14. The z_UV_hat obtained in step 11 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0135] 15. Subtract the y_UV output from step 10 from the mean_UV output from step 13 to get residual_UV, and round it off to get residual_UV_hat;

[0136] 16. Obtain a Gaussian probability distribution function with zero mean and standard deviation of variance_UV according to the variance_UV generated in step 14, and perform entropy encoding (Lossless Encoder) on the residual_UV_hat obtained in step 15 to obtain the fourth bit rate, recorded as bit_y_UV;

[0137] 17. The final encoded code stream is the sum of the above four code streams.

[0138] Decoding end:

[0139] 1. Perform entropy decoding (Lossless Decoder) on the first bitstream bit_z_Y to obtain z_Y_hat;

[0140] 2. z_Y_hat obtained in step 1 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0141] 3. The z_Y_hat obtained in step 1 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0142] 4. Based on the variance_Y generated in step 2, obtain the zero mean and variance_Y standard deviation Gaussian probability distribution function, perform entropy decoding (Lossless Decoder) on the second bit stream bit_y_Y to obtain residual_Y_hat;

[0143] 5. Add residual_Y_hat obtained in step 4 to mean_Y obtained in step 2 to get y1_Y;

[0144] 6. The y1_Y obtained in step 5 is passed through the Y component decoding (Y Decoder) network to obtain the Y component of the reconstructed image;

[0145] 7. Perform entropy decoding (Lossless Decoder) on the third bit stream bit_z_UV to obtain z_UV_hat;

[0146] 8. The z_UV_hat obtained in step 7 is passed through the UV component hyper priori decoding network (UV Hyper Decoder) to obtain mean_UV;

[0147] 9. The z_UV_hat obtained in step 7 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0148] 10. Based on the variance_UV generated in step 9, obtain the Gaussian probability distribution function with zero mean and variance_UV standard deviation, perform entropy decoding (Lossless Decoder) on the fourth bit stream bit_y_UV to obtain residual_UV_hat;

[0149] 11. Add the residual_UV_hat obtained in step 10 to the mean_UV obtained in step 8 to get y1_UV.

[0150] 12. Concat the y1_Y obtained in step 5 with the y1_UV obtained in step 11 on the channel to obtain y1;

[0151] 13. After y1 obtained in step 12 is subjected to a layer of convolution, y2 is obtained;

[0152] 14. Concat y2 obtained in step 13 with y1_UV obtained in step 11 on the channel to obtain y3;

[0153] 15. Input y3 obtained in step 14 into the UV component decoding (UV Decoder) network, and undergo three upsamplings in the UV Decoder network (optionally, the three upsamplings can be achieved by a deconvolution module to achieve one scale upsampling, and then by pixel shuffle to achieve two scale upsamplings). Then, the output of the UV Decoder is upsampled once (optionally, the one upsampling can be achieved by nearest neighbor interpolation) to obtain the UV component of the reconstructed image; (The UV Decoder network includes three upsamplings)

[0154] 16. Concat the Y component obtained in step 6 with the UV component obtained in step 15, and then perform color transformation to obtain a reconstructed RGB image.

[0155] FIG9 is a schematic diagram of the structure of a JPEG AI using a fusion structure. As shown in FIG9 , the JPEG AI uses the fusion structure shown in FIG7 c or FIG7 d , and the processing process is as follows:

[0156] Encoding side:

[0157] 1. The original image in RGB format is converted into a YUV444 format image through color space conversion;

[0158] 2. Decompose the YUV444 format image obtained in step 1 according to the channel to obtain the Y component and UV component;

[0159] 3. The Y component obtained in step 2 is input into the Y component encoding (Y Encoder) network and is downsampled four times to obtain y_Y;

[0160] 4. The y_Y obtained in step 3 is passed through the Y component hyper priori encoding network (Y Hyper Encoder Net), and then rounded and quantized to obtain z_Y_hat;

[0161] 5. The z_Y_hat obtained in step 4 is entropy encoded (Lossless Encoder) to obtain the first bit stream, recorded as bit_z_Y;

[0162] 6. z_Y_hat obtained in step 4 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0163] 7. The z_Y_hat obtained in step 4 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0164] 8. Subtract y_Y obtained in step 3 from mean_Y obtained in step 6 to obtain residual_Y, which is rounded and quantized to obtain residual_Y_hat;

[0165] 9. Obtain the Gaussian probability distribution function with zero mean and standard deviation of variance_Y according to the variance_Y generated in step 7. Perform entropy encoding (Lossless Encoder) on the residual_Y_hat obtained in step 8 to obtain the second bitstream, recorded as bit_y_Y.

[0166] 10. The Y component and UV component obtained in step 2 are downsampled once respectively and input into the UV component encoding (UV Encoder) network. After three downsamplings in the UV Encoder network, y_UV is obtained; (The UV Encoder network includes three downsamplings)

[0167] 11. The y_UV obtained in step 10 is passed through the UV Hyper Encoder Net and then rounded and quantized to obtain z_UV_hat;

[0168] 12. The z_UV_hat obtained in step 11 is entropy encoded (Lossless Encoder) to obtain the third bit stream, recorded as bit_z_UV;

[0169] 13. The z_UV_hat obtained in step 11 is passed through the UV component hyper-decoder network to obtain mean_UV;

[0170] 14. The z_UV_hat obtained in step 11 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0171] 15. Subtract the y_UV output from step 10 from the mean_UV output from step 13 to get residual_UV, and round it off to get residual_UV_hat;

[0172] 16. Obtain a Gaussian probability distribution function with zero mean and standard deviation of variance_UV according to the variance_UV generated in step 14, and perform entropy encoding (Lossless Encoder) on the residual_UV_hat obtained in step 15 to obtain the fourth bit rate, recorded as bit_y_UV;

[0173] 17. The final encoded code stream is the sum of the above four code streams.

[0174] Decoding end:

[0175] 1. Perform entropy decoding (Lossless Decoder) on the first bitstream bit_z_Y to obtain z_Y_hat;

[0176] 2. z_Y_hat obtained in step 1 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0177] 3. The z_Y_hat obtained in step 1 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0178] 4. Based on the variance_Y generated in step 2, obtain the zero mean and variance_Y standard deviation Gaussian probability distribution function, perform entropy decoding (Lossless Decoder) on the second bit stream bit_y_Y to obtain residual_Y_hat;

[0179] 5. Add residual_Y_hat obtained in step 4 to mean_Y obtained in step 2 to get y1_Y;

[0180] 6. The y1_Y obtained in step 5 is passed through the Y component decoding (Y Decoder) network to obtain the Y component of the reconstructed image;

[0181] 7. Perform entropy decoding (Lossless Decoder) on the third bit stream bit_z_UV to obtain z_UV_hat;

[0182] 8. The z_UV_hat obtained in step 7 is passed through the UV component hyper priori decoding network (UV Hyper Decoder) to obtain mean_UV;

[0183] 9. The z_UV_hat obtained in step 7 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0184] 10. Based on the variance_UV generated in step 9, obtain the Gaussian probability distribution function with zero mean and variance_UV standard deviation, perform entropy decoding (Lossless Decoder) on the fourth bit stream bit_y_UV to obtain residual_UV_hat;

[0185] 11. Add the residual_UV_hat obtained in step 10 to the mean_UV obtained in step 8 to get y1_UV.

[0186] 12. Concat the y1_Y obtained in step 5 with the y1_UV obtained in step 11 on the channel to obtain y1;

[0187] 13. After y1 obtained in step 12 is subjected to a layer of convolution, y2 is obtained;

[0188] 14. Add y2 obtained in step 13 and y1_UV obtained in step 11 to obtain y3;

[0189] 15. Input y3 obtained in step 14 into the UV component decoding (UV Decoder) network, undergo three upsampling in the UV Decoder network, and then upsample the output of the UV Decoder once to obtain the UV component of the reconstructed image; (The UV Decoder network includes three upsampling)

[0190] 16. Concat the Y component obtained in step 6 with the UV component obtained in step 15, and then perform color transformation to obtain a reconstructed RGB image.

[0191] The difference between this embodiment and the embodiment shown in FIG8 is that in the above, in the steps 12 to 14 in bold at the decoding end, this embodiment adopts a method of adding two feature tensors.

[0192] FIG10 is a schematic diagram of the structure of a JPEG AI using a fusion structure. As shown in FIG10 , the JPEG AI uses the fusion structure shown in FIG7 a or FIG7 b , and the processing process is as follows:

[0193] Encoding side:

[0194] 1. The original image in RGB format is converted into a YUV444 format image through color space conversion;

[0195] 2. Decompose the YUV444 format image obtained in step 1 according to the channel to obtain the Y component and UV component;

[0196] 3. The Y component obtained in step 2 is input into the Y component encoding (Y Encoder) network and is downsampled four times to obtain y_Y;

[0197] 4. The y_Y obtained in step 3 is passed through the Y component hyper priori encoding network (Y Hyper Encoder Net), and then rounded and quantized to obtain z_Y_hat;

[0198] 5. The z_Y_hat obtained in step 4 is entropy encoded (Lossless Encoder) to obtain the first bit stream, recorded as bit_z_Y;

[0199] 6. z_Y_hat obtained in step 4 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0200] 7. The z_Y_hat obtained in step 4 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0201] 8. Subtract y_Y obtained in step 3 from mean_Y obtained in step 6 to obtain residual_Y, which is rounded and quantized to obtain residual_Y_hat;

[0202] 9. Obtain the Gaussian probability distribution function with zero mean and standard deviation of variance_Y according to the variance_Y generated in step 7. Perform entropy encoding (Lossless Encoder) on the residual_Y_hat obtained in step 8 to obtain the second bitstream, recorded as bit_y_Y.

[0203] 10. The Y component and UV component obtained in step 2 are input into the UV component encoding (UV Encoder) network respectively, and are downsampled 4 times in the UV Encoder network to obtain y_UV; (The UV Encoder network includes 4 downsampling)

[0204] 11. The y_UV obtained in step 10 is passed through the UV Hyper Encoder Net and then rounded and quantized to obtain z_UV_hat;

[0205] 12. The z_UV_hat obtained in step 11 is entropy encoded (Lossless Encoder) to obtain the third bit stream, recorded as bit_z_UV;

[0206] 13. The z_UV_hat obtained in step 11 is passed through the UV component hyper-decoder network to obtain mean_UV;

[0207] 14. The z_UV_hat obtained in step 11 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0208] 15. Subtract the y_UV output from step 10 from the mean_UV output from step 13 to get residual_UV, and round it off to get residual_UV_hat;

[0209] 16. Obtain a Gaussian probability distribution function with zero mean and standard deviation of variance_UV according to the variance_UV generated in step 14, and perform entropy encoding (Lossless Encoder) on the residual_UV_hat obtained in step 15 to obtain the fourth bit rate, recorded as bit_y_UV;

[0210] 17. The final encoded code stream is the sum of the above four code streams.

[0211] Decoding end:

[0212] 1. Perform entropy decoding (Lossless Decoder) on the first bitstream bit_z_Y to obtain z_Y_hat;

[0213] 2. z_Y_hat obtained in step 1 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0214] 3. The z_Y_hat obtained in step 1 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0215] 4. Based on the variance_Y generated in step 2, obtain the zero mean and variance_Y standard deviation Gaussian probability distribution function, perform entropy decoding (Lossless Decoder) on the second bit stream bit_y_Y to obtain residual_Y_hat;

[0216] 5. Add residual_Y_hat obtained in step 4 to mean_Y obtained in step 2 to get y1_Y;

[0217] 6. The y1_Y obtained in step 5 is passed through the Y component decoding (Y Decoder) network to obtain the Y component of the reconstructed image;

[0218] 7. Perform entropy decoding (Lossless Decoder) on the third bit stream bit_z_UV to obtain z_UV_hat;

[0219] 8. The z_UV_hat obtained in step 7 is passed through the UV component hyper priori decoding network (UV Hyper Decoder) to obtain mean_UV;

[0220] 9. The z_UV_hat obtained in step 7 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0221] 10. Based on the variance_UV generated in step 9, obtain the Gaussian probability distribution function with zero mean and variance_UV standard deviation, perform entropy decoding (Lossless Decoder) on the fourth bit stream bit_y_UV to obtain residual_UV_hat;

[0222] 11. Add the residual_UV_hat obtained in step 10 to the mean_UV obtained in step 8 to get y1_UV.

[0223] 12. Concat the y1_Y obtained in step 5 with the y1_UV obtained in step 11 on the channel to obtain y1;

[0224] 13. After y1 obtained in step 12 is subjected to a layer of convolution, y2 is obtained;

[0225] 14. Concat y2 obtained in step 13 with y1_UV obtained in step 11 on the channel to obtain y3;

[0226] 15. Input y3 obtained in step 14 into the UV component decoding (UV Decoder) network, and undergo four upsamplings in the UV Decoder network (optionally, the above four upsamplings can be achieved by two deconvolution modules to achieve two scale upsamplings, and then by pixel shuffle to achieve two scale upsamplings) to obtain the UV component of the reconstructed image; (The UV Decoder network includes four upsamplings)

[0227] 16. Concat the Y component obtained in step 6 with the UV component obtained in step 15, and then perform color transformation to obtain a reconstructed RGB image.

[0228] The difference between this embodiment and the embodiment shown in FIG8 is that in the above, the bolded step 10 on the encoding end and the bolded step 15 on the decoding end, that is, the encoding end no longer downsamples the Y component and the UV component, and the downsampling in the UV Encoder is increased from three times to four times; the upsampling in the UV Decoder on the decoding end is also increased from three times to four times, and then its output is no longer upsampled.

[0229] FIG11 is a schematic diagram of the structure of a JPEG AI using a fusion structure. As shown in FIG11 , the JPEG AI uses the fusion structure shown in FIG7a or FIG7b , and the processing process is as follows:

[0230] Encoding side:

[0231] 1. The original image in RGB format is converted into a YUV444 format image through color space conversion;

[0232] 2. Decompose the YUV444 format image obtained in step 1 according to the channel to obtain the Y component and UV component;

[0233] 3. The Y component obtained in step 2 is input into the Y component encoding (Y Encoder) network and is downsampled four times to obtain y_Y;

[0234] 4. The y_Y obtained in step 3 is passed through the Y component hyper priori encoding network (Y Hyper Encoder Net), and then rounded and quantized to obtain z_Y_hat;

[0235] 5. The z_Y_hat obtained in step 4 is entropy encoded (Lossless Encoder) to obtain the first bit stream, recorded as bit_z_Y;

[0236] 6. z_Y_hat obtained in step 4 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0237] 7. The z_Y_hat obtained in step 4 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0238] 8. Subtract y_Y obtained in step 3 from mean_Y obtained in step 6 to obtain residual_Y, which is rounded and quantized to obtain residual_Y_hat;

[0239] 9. Obtain the Gaussian probability distribution function with zero mean and standard deviation of variance_Y according to the variance_Y generated in step 7. Perform entropy encoding (Lossless Encoder) on the residual_Y_hat obtained in step 8 to obtain the second bitstream, recorded as bit_y_Y.

[0240] 10. The Y component and UV component obtained in step 2 are downsampled once each and then input into the UV component encoding (UV Encoder) network. After 4 downsampling in the UV Encoder network, y_UV is obtained; (The UV Encoder network includes 4 downsampling)

[0241] 11. The y_UV obtained in step 10 is passed through the UV Hyper Encoder Net and then rounded and quantized to obtain z_UV_hat;

[0242] 12. The z_UV_hat obtained in step 11 is entropy encoded (Lossless Encoder) to obtain the third bit stream, recorded as bit_z_UV;

[0243] 13. The z_UV_hat obtained in step 11 is passed through the UV component hyper-decoder network to obtain mean_UV;

[0244] 14. The z_UV_hat obtained in step 11 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0245] 15. Subtract the y_UV output from step 10 from the mean_UV output from step 13 to get residual_UV, and round it off to get residual_UV_hat;

[0246] 16. Obtain a Gaussian probability distribution function with zero mean and standard deviation of variance_UV according to the variance_UV generated in step 14, and perform entropy encoding (Lossless Encoder) on the residual_UV_hat obtained in step 15 to obtain the fourth bit rate, recorded as bit_y_UV;

[0247] 17. The final encoded code stream is the sum of the above four code streams.

[0248] Decoding end:

[0249] 1. Perform entropy decoding (Lossless Decoder) on the first bitstream bit_z_Y to obtain z_Y_hat;

[0250] 2. z_Y_hat obtained in step 1 is passed through the Y component hyper priori decoding network (Y Hyper Decoder) to obtain mean_Y;

[0251] 3. The z_Y_hat obtained in step 1 is passed through the Y component hyper-prior variance decoding network (Y Hyper Scale Decoder) to obtain variance_Y;

[0252] 4. Based on the variance_Y generated in step 2, obtain the zero mean and variance_Y standard deviation Gaussian probability distribution function, perform entropy decoding (Lossless Decoder) on the second bit stream bit_y_Y to obtain residual_Y_hat;

[0253] 5. Add residual_Y_hat obtained in step 4 to mean_Y obtained in step 2 to get y1_Y;

[0254] 6. The y1_Y obtained in step 5 is passed through the Y component decoding (Y Decoder) network to obtain the Y component of the reconstructed image;

[0255] 7. Perform entropy decoding (Lossless Decoder) on the third bit stream bit_z_UV to obtain z_UV_hat;

[0256] 8. The z_UV_hat obtained in step 7 is passed through the UV component hyper priori decoding network (UV Hyper Decoder) to obtain mean_UV;

[0257] 9. The z_UV_hat obtained in step 7 is passed through the UV component hyper-scale decoder network to obtain variance_UV;

[0258] 10. Based on the variance_UV generated in step 9, obtain the Gaussian probability distribution function with zero mean and variance_UV standard deviation, perform entropy decoding (Lossless Decoder) on the fourth bit stream bit_y_UV to obtain residual_UV_hat;

[0259] 11. Add the residual_UV_hat obtained in step 10 to the mean_UV obtained in step 8 to get y1_UV.

[0260] 12. Downsample y1_Y obtained in step 5 once to obtain y2_Y, and concat y2_Y with y1_UV obtained in step 11 on the channel to obtain y1;

[0261] 13. After y1 obtained in step 12 is subjected to a layer of convolution, y2 is obtained;

[0262] 14. Concat y2 obtained in step 13 with y1_UV obtained in step 11 on the channel to obtain y3;

[0263] 15. Input y3 obtained in step 14 into the UV component decoding (UV Decoder) network, undergo 4 upsampling in the UV Decoder network, and then upsample the output of the UV Decoder once to obtain the UV component of the reconstructed image; (The UV Decoder network includes 4 upsampling)

[0264] 16. Concat the Y component obtained in step 6 with the UV component obtained in step 15, and then perform color transformation to obtain a reconstructed RGB image.

[0265] The difference between this embodiment and the embodiment shown in Figure 8 is that in the above, the bolded step 10 on the encoding end and the bolded step 15 on the decoding end, that is, the downsampling in the UV Encoder on the encoding end is changed from three times to four times; the upsampling in the UV Decoder on the decoding end is also changed from three times to four times.

[0266] FIG12 is a schematic diagram of the structure of an image decompression device 1200 according to an embodiment of the present application. As shown in FIG12 , the image decompression device 1200 according to the present embodiment can be applied to a decoding end. The image decompression device 1200 may include: an acquisition module 1201, a processing module 1202, and a reconstruction module 1203.

[0267] An acquisition module 1201 is used to acquire a first feature tensor, which corresponds to a first component of a reconstructed image; and to acquire a second feature tensor, which corresponds to a second component of the reconstructed image; a processing module 1202 is used to concatenate and convolve the first feature tensor and the second feature tensor to obtain a third feature tensor; and to acquire a fourth feature tensor based on the third feature tensor and the second feature tensor; and a reconstruction module 1203 is used to acquire the reconstructed image based on the first feature tensor and the fourth feature tensor.

[0268] In a possible implementation, the processing module 1202 is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

[0269] In a possible implementation, the processing module 1202 is specifically configured to add the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

[0270] In a possible implementation, the processing module 1202 is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and concatenate the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

[0271] In a possible implementation, the processing module 1202 is specifically configured to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and add the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

[0272] In a possible implementation, the first component is a Y component, and the second component is a UV component.

[0273] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 6. Its implementation principle and technical effects are similar and will not be repeated here.

[0274] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0275] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0276] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0277] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0278] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0279] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0280] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0281] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0282] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image decompression method, characterized in that: include: Acquire a first feature tensor, wherein the first feature tensor corresponds to a first component of the reconstructed image; Acquire a second feature tensor, wherein the second feature tensor corresponds to a second component of the reconstructed image; Concatenating and convolving the first feature tensor and the second feature tensor to obtain a third feature tensor; Obtaining a fourth feature tensor according to the third feature tensor and the second feature tensor; The reconstructed image is acquired according to the first feature tensor and the fourth feature tensor.

2. The method according to claim 1, characterized in that The obtaining a fourth feature tensor according to the third feature tensor and the second feature tensor includes: The third feature tensor and the second feature tensor are concatenated to obtain the fourth feature tensor.

3. The method according to claim 1, characterized in that The obtaining a fourth feature tensor according to the third feature tensor and the second feature tensor includes: The third eigentensor and the second eigentensor are added to obtain the fourth eigentensor.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining a fourth feature tensor according to the third feature tensor and the second feature tensor includes: Performing channel extraction on the second feature tensor to obtain a fifth feature tensor; The third feature tensor and the fifth feature tensor are concatenated to obtain the fourth feature tensor.

5. The method according to any one of claims 1 to 3, characterized in that: The obtaining a fourth feature tensor according to the third feature tensor and the second feature tensor includes: Performing channel extraction on the second feature tensor to obtain a fifth feature tensor; The third eigentensor and the fifth eigentensor are added to obtain the fourth eigentensor.

6. The method according to any one of claims 1 to 5, characterized in that The first component is a Y component, and the second component is a UV component.

7. An image decompression device, characterized in that: include: An acquisition module, configured to acquire a first feature tensor, the first feature tensor corresponding to a first component of a reconstructed image; and acquire a second feature tensor, the second feature tensor corresponding to a second component of the reconstructed image; A processing module, configured to concatenate and convolve the first feature tensor and the second feature tensor to obtain a third feature tensor; and obtain a fourth feature tensor according to the third feature tensor and the second feature tensor; A reconstruction module is used to obtain the reconstructed image according to the first feature tensor and the fourth feature tensor.

8. The device according to claim 7, characterized in that The processing module is specifically configured to concatenate the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

9. The device according to claim 7, characterized in that The processing module is specifically configured to add the third feature tensor and the second feature tensor to obtain the fourth feature tensor.

10. The device according to any one of claims 7 to 9, characterized in that: The processing module is specifically used to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and concatenate the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

11. The device according to any one of claims 7 to 9, characterized in that: The processing module is specifically used to perform channel extraction on the second feature tensor to obtain a fifth feature tensor; and add the third feature tensor and the fifth feature tensor to obtain the fourth feature tensor.

12. The device according to any one of claims 7 to 11, characterized in that The first component is a Y component, and the second component is a UV component.

13. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The invention comprises a computer program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.