Encoding and decoding method, system and equipment of lightweight structure and storage medium

Abstract

The invention discloses a coding and decoding method, system and device of a lightweight structure and a storage medium, transformation from space to depth is directly adopted during coding, and compared with common convolution downsampling, the computing power is greatly reduced, the feature resolution can be effectively reduced, and the computing power required by subsequent coding is reduced; in addition, a multi-code-rate operator for decoding is removed and is changed into a self-adaptive structure, so that the decoding speed is effectively increased; generally speaking, the computing power required by the scheme provided by the invention is significantly smaller than that of other existing schemes, and real-time and end-to-end encoding and decoding can be easily achieved on various front-end / rear-end general AI (artificial intelligence) acceleration chips on the premise of ensuring the performance, so that actual landing use can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of image coding and decoding technology, and in particular to a coding and decoding method, system, device and storage medium with a lightweight structure. Background Art

[0002] With the continuous development of digital image processing technology, image coding and decoding technology has become an indispensable part of multimedia communication, video surveillance, virtual reality and other fields. However, existing image coding and decoding technologies generally face a common problem - high computational complexity and demanding hardware computing power. This problem is particularly prominent on resource-constrained platforms, such as embedded devices and mobile terminals, which usually have low computing power and limited memory resources.

[0003] Although the current mainstream image coding and decoding standards have made significant progress in compression efficiency, a large number of mathematical operations need to be performed during the encoding and decoding process. These operations are not only time-consuming but also have high requirements on processor performance.

[0004] On platforms with low computing power, such as low-end smartphones and IoT devices, existing end-to-end image coding and decoding technologies often have difficulty achieving real-time processing due to hardware resource limitations. This not only affects the user experience, but also limits the widespread use of these devices in multimedia applications. Therefore, it is particularly important to develop an end-to-end image coding and decoding technology that can run efficiently on low computing power platforms.

[0005] In many application scenarios, such as video calls, online live broadcasts, and real-time monitoring, real-time performance is a key indicator. Existing image encoding and decoding technologies often perform poorly in these scenarios, especially when network bandwidth and computing resources are limited. In order to ensure the smoothness and timeliness of image transmission, a new encoding and decoding method is needed that can significantly reduce the computational complexity while maintaining high compression efficiency, so as to achieve efficient image encoding and decoding on platforms of various computing power levels.

[0006] In summary, existing image encoding and decoding technologies are difficult to ensure real-time performance on low-computing platforms, which has become a bottleneck restricting their widespread application.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a lightweight coding and decoding method, system, device and storage medium to improve the efficiency and real-time performance of image coding and decoding, and to be applicable to various resource-constrained platforms.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A lightweight encoding and decoding method, comprising:

[0011] Coding part: Perform space-to-depth transformation on the input image, obtain coding features through several convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and arithmetic coding module to obtain the coded bitstream;

[0012] Decoding part: The encoded bitstream is processed by the hyper-prior module and the arithmetic decoding module to obtain the decoding features, and then after several deconvolution operations, a depth-to-space transformation operation is performed to obtain a reconstructed image.

[0013] A lightweight coding and decoding system, comprising:

[0014] The coding network is used to perform a space-to-depth transformation operation on the input image, obtain coding features through a number of convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and the arithmetic coding module to obtain a coded bitstream;

[0015] A decoding network processes the encoded bitstream through a super priori module and an arithmetic decoding module to obtain decoding features, and then performs a depth-to-space transformation operation after several deconvolution operations to obtain a reconstructed image;

[0016] Among them, the super-prior module belongs to the common part of the encoding network and the decoding network.

[0017] A processing device, comprising: one or more processors; a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0020] It can be seen from the technical solution provided by the present invention that directly using space-to-depth transformation during encoding greatly reduces computing power compared to ordinary convolution downsampling, and can also effectively reduce feature resolution and reduce the computing power required for subsequent encoding; in addition, the multi-rate operator of decoding is removed and replaced with an adaptive structure to effectively increase the decoding speed; in general, the computing power required for the solution provided by the present invention is significantly less than other existing solutions, and can easily achieve real-time end-to-end encoding and decoding on various front-end / back-end general AI (artificial intelligence) acceleration chips while ensuring performance, so that end-to-end encoding and decoding can be put into practical use. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 A flowchart of a lightweight encoding and decoding method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a lightweight coding and decoding system provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0026] First, the terms that may be used in this article are explained as follows:

[0027] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0028] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0029] The following is a detailed description of a lightweight coding and decoding method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. The instruments used in the embodiments of the present invention, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0030] Embodiment 1

[0031] The embodiment of the present invention provides a lightweight structure encoding and decoding method, which mainly includes the following steps:

[0032] Coding part: Perform space-to-depth transformation on the input image, obtain coding features through several convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and arithmetic coding module to obtain the coded bitstream;

[0033] Decoding part: The encoded bitstream is processed by the hyper-prior module and the arithmetic decoding module to obtain the decoding features, and then after several deconvolution operations, a depth-to-space transformation operation is performed to obtain a reconstructed image.

[0034] For ease of understanding, the following is a detailed introduction to the encoding and decoding parts respectively.

[0035] 1. Coding part.

[0036] like Figure 1 As shown, an example of the encoding part is provided. The input image space size is recorded as (H, W, T), where H is the height of the input image, W is the width of the input image, and T is the number of channels. For example, if it is an RGB image, the number of channels T=3.

[0037] In the embodiment of the present invention, after performing a space-to-depth transformation operation (space2depth) on the input image, a feature with a spatial size of (H / / S, W / / S, T*S*S) is obtained, where H is the height of the input image, W is the width of the input image, S is the step size of the space-to-depth transformation operation, / / indicates downward division, Figure 1 An example with a stride of 3 is provided. In this case, the transformed features are (H / / 3, W / / 3, T*3*3).

[0038] In an embodiment of the present invention, the spatial size of a coding feature obtained through several convolution operations is expressed as (H / / K, W / / K, C), wherein K is the total downsampling ratio involved in the space-to-depth transformation operation and several convolution operations, and C is the number of channels of the coding feature.

[0039] Figure 1 An example of two convolution operations is provided, where conv represents a convolution operation (convolution downsampling), and (4,2) represents a convolution kernel of 4 and a stride of 2. In this example, after a spatial-to-depth transformation with a stride of 3, and then two convolution operations with a stride of 2, the spatial size of the output coding feature is (H / / 12, W / / 12, C); the number of channels C of the coding feature can be adjusted according to actual conditions, and the present invention does not impose any limitation.

[0040] In the encoding part, the corresponding multi-rate parameter (mul) will be applied to the feature according to the requirements to adjust the range of the feature. After that, the encoding bitstream (binary encoding file) is obtained through the super prior module and arithmetic coding module.

[0041] Considering that both the super-a priori module and the arithmetic coding module can be implemented by existing technologies, and the processing processes involved in the two also belong to existing technologies, they will not be described in detail.

[0042] 2. Decoding part.

[0043] The decoding part is the inverse process of the encoding part. First, the decoding features are obtained through the super-prior module and the arithmetic decoding module. The arithmetic decoding module involved here also belongs to the scope of the prior art, so it will not be described in detail; then, the reconstructed image is obtained through deconvolution and depth-to-space transformation operations.

[0044] Figure 1 An example of two deconvolution operations is provided, where conv represents the deconvolution operation (deconvolution upsampling), (4,2) indicates that the convolution kernel is 4 and the stride is 2. The decoded image is gradually restored through two deconvolutions, and then the upsampled output spatial size (H / / 3, W / / 3, 3*3*3) is directly restored to a reconstructed image of size (H, W, 3) through depth2space (depth-to-space transformation operation), which can greatly reduce the complexity of decoding.

[0045] Considering that the existing general hardware does not achieve the optimal matrix operation of dot multiplication / dot division, and some hardware may not even support it, in the embodiment of the present invention, no additional operation is performed during decoding (i.e., the multi-rate operator of decoding is removed), and the decoded features are directly deconvolved, and reconstructed adaptively to obtain a reconstructed image of the corresponding bit rate, which effectively increases the decoding speed. Adaptive here means that during decoding, there is no need to use the compressed bit rate parameters as shown in the encoding part, but the decoding of the corresponding bit rate will be performed according to the feature adaptive, that is, the features of high and low bit rates are decoded, so there is no need to perform additional feature transformation.

[0046] The above overall process of the present invention can be described as:

[0047] (1) Encoding process: select the bit rate to be compressed (i.e., the aforementioned multi-bit rate parameters) → input the image → compression encoding → arithmetic encoding to generate a .bin file (i.e., a binary bit rate file).

[0048] (2) Decoding process: arithmetic decoding of .bin file → compression decoding → image reconstruction. There is no need to specify the bit rate during decoding, and decoding is performed adaptively.

[0049] The above scheme provided by the embodiment of the present invention introduces the spatial shuffle operation (i.e. the aforementioned space-to-depth transformation operation and depth-to-space transformation operation), which significantly saves computing power. When encoding, it is combined with two subsequent convolution operations, which is equivalent to performing 3.5 downsampling operations, effectively retaining network performance. Compared with the existing scheme, up and down sampling (convolution and deconvolution) need to be performed 4 times. The present invention not only improves the overall performance but also saves computing power, making real-time encoding and decoding possible. The special design of network structure + multi-bit rate makes the computing power of the present invention significantly smaller than other structures. Under the premise of ensuring performance, it can easily achieve real-time on various front-end / back-end general AI acceleration chips, and end-to-end encoding and decoding can be put into practical use.

[0050] In general, the present invention includes the following three improvements: (1) A multi-rate structure that does not increase the decoding computing power at all. (2) The input and output directly use the spatial shuffle operator, which significantly reduces the computing power without affecting the performance. (3) A 3.5-times up- and down-sampling codec structure with a more balanced computing power and performance.

[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0052] Embodiment 2

[0053] The present invention also provides a lightweight coding system, which is mainly used to implement the method provided in the above embodiment, such as Figure 2 As shown, the system mainly includes:

[0054] The coding network is used to perform a space-to-depth transformation operation on the input image, obtain coding features through a number of convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and the arithmetic coding module to obtain a coded bitstream;

[0055] A decoding network processes the encoded bitstream through a super priori module and an arithmetic decoding module to obtain decoding features, and then performs a depth-to-space transformation operation after several deconvolution operations to obtain a reconstructed image;

[0056] Among them, the super-prior module belongs to the common part of the encoding network and the decoding network.

[0057] In an embodiment of the present invention, after performing a space-to-depth transformation operation on the input image, a feature with a spatial size of (H / / S, W / / S, T*S*S) is obtained, wherein H is the height of the input image, W is the width of the input image, T is the number of channels of the input image, S is the step size of the space-to-depth transformation operation, and / / indicates downward division.

[0058] In an embodiment of the present invention, the spatial size of the encoded feature obtained through several convolution operations is expressed as (H / / K, W / / K, C), wherein H is the height of the input image, W is the width of the input image, K is the total downsampling ratio involved in the space-to-depth transformation operation and several convolution operations, and C is the number of channels of the encoded feature.

[0059] In the embodiment of the present invention, applying the set multi-rate parameter to the coding feature includes: multiplying the set multi-rate parameter by a point to the coding feature.

[0060] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0061] Embodiment 3

[0062] The present invention also provides a processing device, such as Figure 3 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0063] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0064] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0065] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0066] The output device may be a display terminal;

[0067] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0068] Embodiment 4

[0069] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0070] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0071] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A lightweight coding and decoding method, characterized in that: include: Coding part: Perform space-to-depth transformation on the input image, obtain coding features through several convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and arithmetic coding module to obtain the coded bitstream; Decoding part: The encoded bitstream is processed by the hyper-prior module and the arithmetic decoding module to obtain the decoding features, and then after several deconvolution operations, a depth-to-space transformation operation is performed to obtain a reconstructed image.

2. The encoding and decoding method of a lightweight structure according to claim 1, characterized in that: After performing a space-to-depth transformation operation on the input image, a feature with a spatial size of (H / / S, W / / S, T*S*S) is obtained, where H is the height of the input image, W is the width of the input image, T is the number of channels of the input image, S is the step size of the space-to-depth transformation operation, and / / indicates downward division.

3. The encoding and decoding method of a lightweight structure according to claim 1, characterized in that: The spatial size of the encoded feature obtained through several convolution operations is expressed as (H / / K, W / / K, C), where H is the height of the input image, W is the width of the input image, K is the total downsampling ratio involved in the space-to-depth transformation operation and several convolution operations, and C is the number of channels of the encoded feature.

4. The encoding and decoding method of a lightweight structure according to claim 1, characterized in that: The applying the set multi-rate parameter to the coding feature includes: multiplying the set multi-rate parameter by a point to the coding feature.

5. A lightweight coding and decoding system, characterized in that: include: The coding network is used to perform a space-to-depth transformation operation on the input image, obtain coding features through a number of convolution operations, apply the set multi-rate parameters to the coding features, and then process them through the super-prior module and the arithmetic coding module to obtain a coded bitstream; A decoding network processes the encoded bitstream through a super priori module and an arithmetic decoding module to obtain decoding features, and then performs a depth-to-space transformation operation after several deconvolution operations to obtain a reconstructed image; Among them, the super-prior module belongs to the common part of the encoding network and the decoding network.

6. A lightweight coding and decoding system according to claim 5, characterized in that: After performing a space-to-depth transformation operation on the input image, a feature with a spatial size of (H / / S, W / / S, T*S*S) is obtained, where H is the height of the input image, W is the width of the input image, T is the number of channels of the input image, S is the step size of the space-to-depth transformation operation, and / / indicates downward division.

7. A lightweight coding and decoding system according to claim 5, characterized in that: The spatial size of the encoded feature obtained through several convolution operations is expressed as (H / / K, W / / K, C), where H is the height of the input image, W is the width of the input image, K is the total downsampling ratio involved in the space-to-depth transformation operation and several convolution operations, and C is the number of channels of the encoded feature.

8. The lightweight coding and decoding system according to claim 5, characterized in that: The applying the set multi-rate parameter to the coding feature includes: multiplying the set multi-rate parameter by a point to the coding feature.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images