Image encoding method, decoding method, device, readable medium and electronic device

Abstract

The present application belongs to the field of artificial intelligence technology, and specifically relates to an image encoding method, decoding method, device, readable medium and electronic device. The method includes obtaining an original image and performing block processing; calculating the gradient value of pixels in each block, and screening important area blocks according to the gradient value of the pixel; inputting the important area blocks and the position information of the important area blocks in the original image into a visual transformation model for encoding to generate a bit stream. In this way, the method selectively and controllably compresses different information areas of the compressed image, retains as much as possible and compresses as little as possible the key areas with dense information in the image; and compresses as much as possible the non-key areas with sparse information in the image, thereby improving the image compression efficiency and realizing flexible bit rate control under a unified scheme.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image encoding method, a decoding method, a device, a readable medium and an electronic device. Background Art

[0002] Traditional image / video coding is oriented towards human visual tasks and is mostly used for entertainment purposes, focusing on the fidelity, high frame rate, and clarity of video data signals. With the rapid development of 5G, big data, and artificial intelligence, in the context of image / video big data applications, media content such as images and videos are widely used in intelligent visual tasks such as target detection, target tracking, image classification, image segmentation, and pedestrian re-identification. These intelligent visual tasks are also called intelligent tasks for machine vision.

[0003] With the popularization of intelligent machine vision tasks, such as the rapid development of image classification, video target detection, target tracking, image segmentation, and pedestrian re-identification, if the current related technical solutions based on convolutional neural networks are used for image / video encoding and decoding, since this method uniformly encodes all areas of the entire image, it is not conducive to image encoding / decoding.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present application is to provide an image encoding method, a decoding method, a device, a readable medium and an electronic device, which can improve the image compression efficiency to a certain extent.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a method for encoding an image is provided, the method comprising:

[0008] Get the original image and process it in blocks;

[0009] Calculate the gradient value of the pixels in each block, and select the important area blocks according to the gradient value of the pixels;

[0010] The important area blocks and the position information of the important area blocks in the original image are input into a visual transformation model for encoding to generate a bit stream.

[0011] According to one aspect of an embodiment of the present application, there is provided an image encoding device, the image encoding device comprising:

[0012] The acquisition module is used to acquire the original image and perform block processing;

[0013] A calculation module, used to calculate the gradient value of the pixels in each block, and screen the important area blocks according to the gradient value of the pixels;

[0014] The encoding module is used to input the important area blocks and the position information of the important area blocks in the original image into the visual conversion model for encoding to generate a bit stream.

[0015] In some embodiments of the present application, based on the above technical solution, the calculation module is also used to calculate the gradient values ​​of pixels in each block, and calculate the gradient average of each block according to the gradient values ​​of the pixels; sort the blocks according to the gradient average, and discard the blocks whose gradient average is less than a preset value, and the remaining blocks are the important area blocks.

[0016] In some embodiments of the present application, based on the above technical solution, the encoding module is also used to input the important area blocks into the visual transformation model, and output encoded visible patches and mask tokens; generate image tokens according to the encoded visible patches, the mask tokens and the position information in the original image, and generate the bit stream according to the image tokens.

[0017] In some embodiments of the present application, based on the above technical solution, the compression ratio α of the image satisfies the formula:

[0018] Among them, n is the size of the original image, and p is the number of discarded blocks.

[0019] In some embodiments of the present application, based on the above technical solution, the acquisition module is further used to acquire an n×n original image;

[0020] The original image is divided into m×m blocks evenly according to the non-overlapping areas, and the size of each evenly divided block is

[0021] According to one aspect of an embodiment of the present application, there is provided an image decoding method for decoding the encoding performed by the above-mentioned image encoding method, the image decoding method comprising:

[0022] receiving a bit rate generated by encoding;

[0023] The bit stream is decoded, and the decoding result is processed by normalization, multi-head attention mechanism and multi-layer perceptron in turn to output a reconstructed image.

[0024] According to one aspect of an embodiment of the present application, an image decoding device is provided, the image decoding device comprising:

[0025] A receiving module, used for receiving a bit rate generated through encoding;

[0026] The decoding module is used to decode the bit stream, and process the decoding results in sequence through normalization, multi-head attention mechanism and multi-layer perceptron processing, and output a reconstructed image.

[0027] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image encoding method or the image decoding method in the above technical solution is implemented.

[0028] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute an image encoding method or an image decoding method as in the above technical solution by executing the executable instructions.

[0029] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the image encoding method or the image decoding method in the above technical solution.

[0030] In the technical solution provided in the embodiment of the present application, the original image is first divided into blocks, and the block region gradient calculation and the visual conversion model are combined. In this way, the method selectively and controllably compresses different information areas of the compressed image, retains as much as possible the key areas with dense information in the image / video, and compresses as little as possible; while the non-key areas with sparse information in the image are compressed as much as possible, thereby improving the image compression efficiency and realizing flexible bit rate control under a unified solution.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 The following schematically shows an exemplary image coding and decoding system architecture block diagram.

[0034] Figure 2 The exemplary system architecture block diagram applying the technical solution of the present application is schematically shown.

[0035] Figure 3 The step flow of the image encoding method provided by an embodiment of the present application is schematically shown.

[0036] Figure 4 A schematic diagram of image segmentation using the technical solution of the present application is shown schematically.

[0037] Figure 5 The schematic diagram of the gradient average value of each block using the technical solution of the present application is shown schematically.

[0038] Figure 6 The schematic diagram of the encoder module applying the technical solution of the present application is shown schematically.

[0039] Figure 7 The schematic diagram of a decoder module applying the technical solution of the present application is shown schematically.

[0040] Figure 8 The schematic diagram of the encoding and decoding process using the technical solution of the present application is shown schematically.

[0041] Fig. 9 The structural block diagram of the image encoding device provided in an embodiment of the present application is schematically shown.

[0042] Fig.10 The structure block diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0043] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0044] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0047] See also Figure 1 , Figure 1 The following schematically shows an exemplary image coding and decoding system architecture block diagram.

[0048] The system includes a data acquisition module 101, an encoder module 102, and a decoder module 103, wherein the data acquisition module 101 is used to collect images / videos and transmit them to the encoder module; in the encoder module 102, a convolutional neural network module is used as an encoder to encode the image / video into a bit stream, and the bit stream is transmitted to the decoder module 103 at the other end; in the decoder module 103, a convolutional neural network module is also used as a decoder to reconstruct the bit stream into an image / video; then the reconstructed image / video is used as the input of the human vision task 104; finally, the result is obtained after calculation of the human vision task.

[0049] The following technical problems exist in this method: the encoder and decoder encode all areas of the image / video uniformly, and cannot distinguish between the key areas and non-key areas of the image itself; after uniformly encoding all areas of the image, the key areas of the image are compressed more, and important information of the image will be lost; the current method cannot selectively compress each area of ​​the image. That is, the current image encoding / decoding method cannot discard non-important area image blocks in the image when encoding the image. Therefore, the current encoders designed based on deep convolutional neural network structures have inflexible compression ratio control. In addition, the current encoding / decoding system method is oriented to human vision tasks. When facing machine vision tasks, the system cannot complete machine vision intelligent analysis tasks well.

[0050] In order to solve the above problems, this application redesigns the encoder and decoder modules in the encoding system for machine vision intelligent analysis tasks. In the design of the encoder, a Transformer image encoder based on regional gradient information is proposed. In the decoder design, a decoder based on Transformer Block is proposed. Figure 2 , Figure 2 The block diagram of an exemplary system architecture for applying the technical solution of the present application is schematically shown. The system architecture includes a data collection module 201 for collecting images / videos to obtain the original image, and then the original image is input into the encoder module 202. The original image is sequentially processed by block processing, gradient calculation, important area calculation and visionTransformer encoding to output a bit stream. The bit stream output by the encoder module 202 is reconstructed into an image / video through the decoder module 203 of the Transformer Block, and the reconstructed image / video is used as the input of the machine vision task 204; finally, the result is obtained after the machine vision task calculation.

[0051] In order to achieve selective compression of different areas of the image / video, the encoder designed in this application is a method that combines regional gradient calculation and Transformer module when encoding image / video data, and the image decoder is designed only as Transformer module. When encoding the image, the picture is divided into blocks for calculation, and then the gradient values ​​are calculated for the image pixels in each block area, and the average value of the gradient calculation value of each area is calculated. The blocks are sorted according to the average value of the gradient calculation value, and the blocks with the lowest order are discarded. The blocks with the highest order are input into the subsequent Transformer module, and the pictures in other block areas are directly discarded, so as to flexibly control the compression rate by the proportion of discarded pictures.

[0052] The image encoding method, decoding method, device, readable medium and electronic device provided by the present application are described in detail below in conjunction with specific implementation methods.

[0053] See also Figure 3 , Figure 3 The image encoding method provided by an embodiment of the present application is schematically shown as a step flow chart. The image encoding method can be executed by a controller, and mainly includes the following steps S301 to S303.

[0054] Step S301, obtaining an original image and performing block processing.

[0055] The data acquisition module acquires the image / video to obtain the original image, and then divides the original image into blocks. For example, if the original image is n×n, the n×n image is evenly divided into m×m blocks according to the non-overlapping areas, and the size of each evenly divided block is See also Figure 4 , Figure 4 The schematic diagram of image segmentation using the technical solution of the present application is schematically shown. Taking a 28*28 image as an example, it is evenly segmented into 4*4 segments according to non-overlapping areas, and each segment is 7*7. In this way, by segmenting the original image, it is convenient to subsequently determine the important area segments.

[0056] Step S302, calculating the gradient value of the pixels in each block, and selecting the important area blocks according to the gradient value of the pixels.

[0057] By calculating the gradient value of the pixels in each block, it is helpful to screen the important area blocks according to the gradient value of the pixels. In this way, the different information areas of the compressed image can be compressed selectively and controllably, and the key areas with dense information in the image / video can be retained as much as possible and compressed as little as possible; while the non-key areas with sparse information in the image can be compressed as much as possible, thereby improving the image compression efficiency and realizing flexible bit rate control under a unified solution.

[0058] Step S303: input the important region blocks and the position information of the important region blocks in the original image into the visual transformation model for encoding to generate a bit stream.

[0059] In the technical solution provided in the embodiment of the present application, the original image is first divided into blocks, and the block region gradient calculation and the visual conversion model are combined. In this way, the method selectively and controllably compresses different information areas of the compressed image, retains as much as possible the key areas with dense information in the image / video, and compresses as little as possible; while the non-key areas with sparse information in the image are compressed as much as possible, thereby improving the image compression efficiency and realizing flexible bit rate control under a unified solution.

[0060] In one embodiment of the present application, the gradient value of the pixels in each block is calculated, and the important area blocks are screened according to the gradient value of the pixels, including:

[0061] Calculate the gradient value of the pixels in each block, and calculate the average gradient value of each block based on the gradient value of the pixels;

[0062] The blocks are sorted according to the average gradient value, and the blocks with average gradient value less than the preset value are discarded. The remaining blocks are important area blocks.

[0063] In this way, the important areas where information is focused are sorted and screened out according to the mean value calculated according to the gradient, and then the non-important area blocks in the image can be discarded to achieve image compression.

[0064] Specifically, when selecting important area blocks, first calculate the gradients in the x and y directions for each pixel (x, y) in the block. The gradient calculation in the x direction is shown in the following formula:

[0065]

[0066] The gradient in the y direction is calculated as shown in the following formula:

[0067]

[0068] The gradient value g of the pixel (x, y) in the x direction x and the gradient value g in the y direction y The value of is calculated as follows:

[0069]

[0070] Among them, g(x,y) is the calculated gradient value of (x,y). Then the calculated gradient values ​​of all pixels in the block area are calculated as follows:

[0071]

[0072] Where d(i,j) is the average gradient of all pixels in each block. The value range of i and j is 0 to m-1.

[0073] See also Figure 5 , Figure 5 The schematic diagram of the gradient average value of each block using the technical solution of the present application is shown schematically. All blocks are sorted from large to small according to the value of d(i,j), that is, {d(2,2), d(1,2), d(2,1), d(1,1), ...} in sequence. The p blocks with smaller d(i,j) values ​​at the end of the sort are discarded, and the number of remaining blocks is n×np, and these remaining blocks are used as important area blocks.

[0074] In this way, the method selectively and controllably compresses different information areas of the compressed image, retains as much as possible the key areas with dense information in the image / video and compresses them as little as possible; while compressing the non-key areas with sparse information in the image as much as possible, thereby improving the image compression efficiency and achieving flexible bit rate control under a unified scheme.

[0075] In one embodiment of the present application, the important region blocks and the position information of the important region blocks in the original image are input into the visual conversion model for encoding to generate a bit stream, including:

[0076] The important regions are divided into patches and fed into the visual transformation model, and the output encodes visible patches and mask tokens;

[0077] Image tokens are generated according to the encoded visible patches, mask tokens and position information in the original image, and a bitstream is generated according to the image tokens.

[0078] See also Figure 6 , Figure 6 The schematic diagram of the encoder module using the technical solution of the present application is schematically shown. The image blocks that are not discarded and their positional relationship in the original image are input into the Vision Transformer model. The Patch Embeddings and Positional Embeddings information of the image blocks that are not discarded are input into the Encoder module of the Transformer.

[0079] After calculation by the Transformer Encoder module, the same number of p block information and their position information will be obtained. At this time, they will be rearranged according to the position information to obtain d×d blocks of the same size as the original image. Among these d×d blocks, the image blocks that have not been discarded before are obtained after calculation by the Vision Transformer and are called Encoded Visible Patches. The rest are rearranged according to the position information and are called Mask Tokens.

[0080] Thus, the video coding system methods of the related technical solutions are all oriented to human vision tasks, and cannot well complete machine vision intelligent analysis tasks when oriented to machine vision tasks. The technical proposal of this embodiment is oriented to machine vision tasks, and can well complete machine vision intelligent analysis tasks.

[0081] In one embodiment of the present application, the method further includes that the compression ratio α of the image satisfies the formula:

[0082] Among them, n is the size of the original image, and p is the number of discarded blocks.

[0083] In this way, by controlling the ratio of discarded image blocks after image segmentation, the compression rate can be flexibly controlled.

[0084] According to one aspect of an embodiment of the present application, an image decoding method is provided to decode the encoding performed by the above-mentioned image encoding method, and the image decoding method includes:

[0085] receiving a bit rate generated by encoding;

[0086] The bit stream is decoded, and the decoding results are processed by normalization, multi-head attention mechanism and multi-layer perceptron in turn to output the reconstructed image.

[0087] See also Figure 7 , Figure 7 The schematic diagram of the decoder module using the technical solution of the present application is schematically shown. After obtaining the outputs of the Transformer encoder module based on regional gradient information, Encoded Visible Patches and MaskTokens, these two parts are combined with the positional embeddings of the original image, and the result of the addition is input into the decoder Decoder constructed with the Transformer Block module. In the decoder, the Transformer Block module is composed of Normalize, Multi-head Self Attention and MLP modules.

[0088] After the image block t is input into Multi-head Self Attention, the weight matrix W in Multi-head Self Attention is randomly initialized t , and the attention weight matrix for each head

[0089] Then each image block vector is multiplied by the attention weight matrix of each head To calculate the three matrices Q corresponding to the image block vector t ,K t ,V t , the calculation formula is as follows:

[0090]

[0091] Each image vector corresponds to the attention of each head, and the calculation is shown in the following formula:

[0092]

[0093]

[0094] In the formula, h t Divide the image into blocks t corresponding to the heads of each attention. is the matrix K g The dimension of δ(Q t ,K t ,V t ) is the function for calculating attention, is the Softmax logistic regression function.

[0095] In the formula, Represents the Concatenate function, W t e is the parameter matrix, and the calculation result r represents the value of the multi-head. The output of the decoder is the reconstructed graph.

[0096] In order to facilitate understanding of the technical solution of this application, see Figure 8 , Figure 8 The schematic diagram of the encoding and decoding process using the technical solution of the present application is shown schematically.

[0097] At the encoding end: Step S801, the original image is divided into blocks, and the n×n image is evenly divided into m×m blocks according to non-overlapping areas, namely, Non-Overlapping Patches;

[0098] Step S802: for each pixel (x, y) in the image block, calculate the gradient in the x direction and the gradient in the y direction respectively:

[0099]

[0100]

[0101] Step S803, calculate the gradient value of the pixel (x, y):

[0102]

[0103] Step S804, calculate the gradient average of all pixels in each block area:

[0104]

[0105] Step S805: sort all blocks according to the value of d(i,j). Discard the p blocks with smaller d(i,j) values ​​at the end of the sort. The calculation of the compression ratio α satisfies the formula:

[0106]

[0107] Step S806, generating Image Tokens, namely, Encoded Visible Patches and Mask Tokens, according to the result of the discarding operation;

[0108] Step S807: Generate a bit stream according to the Image Tokens.

[0109] Decoding side:

[0110] Step S808, obtaining Encoded Visible Patches, Mask Tokens and Positional Embeddings information according to the bit stream;

[0111] Step S809, normalizing the data;

[0112] Step S810, calculating Multi-head Self Attention;

[0113]

[0114]

[0115]

[0116] Step S811, data normalization processing;

[0117] Step S812, multi-layer perceptron calculation;

[0118] Step S813, output the reconstructed image.

[0119] In the image coding and decoding, the present application adopts a method based on regional gradient calculation to design an image codec, that is, to selectively compress the image content information; it proposes to calculate the gradient, gradient calculation value and gradient calculation mean of the block image, and screen out the important information blocks according to the gradient calculation mean; it proposes the idea of ​​calculating the key area according to the gradient calculation mean information, sorting and screening out the important areas of information focus, and then discarding the non-important areas in the image to achieve selective compression of the image; by controlling the ratio of discarding the image blocks after the image is divided into blocks, the compression rate can be flexibly controlled; in addition, the video coding system methods of the related technical solutions are all oriented to human vision tasks, and when facing machine vision tasks, they cannot complete the machine vision intelligent analysis tasks well. The system proposed in the present application is oriented to machine vision tasks and can better complete the machine vision intelligent analysis tasks.

[0120] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0121] The following introduces an apparatus embodiment of the present application, which can be used to execute the image encoding method or image decoding method in the above-mentioned embodiment of the present application. Fig. 9 The structure block diagram of the image encoding device provided by the embodiment of the present application is schematically shown. Fig. 9 As shown, an image encoding device, the image encoding device 900 includes:

[0122] The acquisition module 901 is used to acquire the original image and perform block processing;

[0123] A calculation module 902 is used to calculate the gradient value of the pixels in each block and select the important area blocks according to the gradient value of the pixels;

[0124] The encoding module 903 is used to input the important area blocks and the position information of the important area blocks in the original image into the visual conversion model for encoding to generate a bit stream.

[0125] In some embodiments of the present application, based on the above technical solution, the calculation module 902 is also used to calculate the gradient values ​​of pixels in each block, and calculate the gradient average of each block according to the gradient value of the pixel; sort the blocks according to the gradient average, and discard the blocks with a gradient average value less than a preset value, and the remaining blocks are important area blocks.

[0126] In some embodiments of the present application, based on the above technical solution, the encoding module 903 is also used to input the important area blocks into the visual transformation model, and output the encoded visible patches and mask tokens; generate image tokens according to the encoded visible patches, mask tokens and position information in the original image, and generate a bit stream according to the image tokens.

[0127] In some embodiments of the present application, based on the above technical solution, the compression ratio α of the image satisfies the formula:

[0128] Among them, n is the size of the original image, and p is the number of discarded blocks.

[0129] In some embodiments of the present application, based on the above technical solution, the acquisition module 901 is further used to acquire an n×n original image;

[0130] The original image is divided into m×m blocks evenly according to the non-overlapping areas, and the size of each evenly divided block is

[0131] According to one aspect of an embodiment of the present application, there is provided an image decoding device, the image decoding device comprising:

[0132] A receiving module, used for receiving a bit rate generated through encoding;

[0133] The decoding module is used to decode the bit stream and process the decoding results through normalization, multi-head attention mechanism and multi-layer perceptron in turn to output the reconstructed image.

[0134] The specific details of the image encoding device or image decoding device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0135] Fig.10 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0136] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Fig.10 As shown, the computer system 1000 includes a central processing unit 1001 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 (ROM) or the program loaded from the storage part 1008 to the random access memory 1003 (RAM). Various programs and data required for system operation are also stored in the random access memory 1003. The central processing unit 1001, the read-only memory 1002 and the random access memory 1003 are connected to each other through a bus 1004. The input / output interface 1005 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1004.

[0138] The following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0139] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processor 1001, various functions defined in the system of the present application are executed.

[0140] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0143] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0144] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0145] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image coding method, characterized in that: The image encoding method comprises: Get the original image and process it in blocks; Calculate the gradient value of the pixels in each block, and select the important area blocks according to the gradient value of the pixels; The important area blocks and the position information of the important area blocks in the original image are input into a visual transformation model for encoding to generate a bit stream, which includes: inputting the important area blocks into the visual transformation model, outputting encoded visible patches and mask tokens; generating image tokens according to the encoded visible patches, the mask tokens and the position information in the original image, and generating the bit stream according to the image tokens.

2. The image encoding method according to claim 1, characterized in that The step of calculating the gradient value of pixels in each block and selecting important area blocks according to the gradient value of the pixels includes: Calculate the gradient value of the pixels in each block, and calculate the gradient average value of each block according to the gradient value of the pixels; The blocks are sorted according to the gradient average value, and the blocks whose gradient average value is less than a preset value are discarded, and the remaining blocks are the important area blocks.

3. The image encoding method according to claim 2, characterized in that The method also includes that the compression ratio α of the image satisfies the formula: Among them, n is the size of the original image, and p is the number of discarded blocks.

4. The image encoding method according to claim 1, characterized in that: The obtaining of the original image and performing block processing includes: Get the original image of n×n; The original image is divided into m×m blocks evenly according to the non-overlapping areas, and the size of each evenly divided block is 5. An image decoding method, characterized in that: Decoding the encoding performed by the image encoding method according to any one of claims 1 to 4, the image decoding method comprising: Receiving a bit stream generated by encoding; The bit stream is decoded to obtain the position information of the coded visible patches, mask tokens and important area blocks in the original image, and a decoding result is generated according to the position information of the coded visible patches, the mask tokens and the important area blocks in the original image. The decoding result is processed by normalization, multi-head attention mechanism and multi-layer perceptron in sequence to output a reconstructed image.

6. An image encoding device, characterized in that: The image encoding device comprises: The acquisition module is used to acquire the original image and perform block processing; A calculation module, used to calculate the gradient value of the pixels in each block, and screen the important area blocks according to the gradient value of the pixels; The encoding module is used to input the important area blocks and the position information of the important area blocks in the original image into the visual transformation model for encoding to generate a bit stream, which includes: inputting the important area blocks into the visual transformation model, outputting encoded visible patches and mask tokens; generating image tokens according to the encoded visible patches, the mask tokens and the position information in the original image, and generating the bit stream according to the image tokens.

7. An image decoding device, characterized in that: The image decoding device is used to decode the encoding performed by the image encoding method according to any one of claims 1 to 4, and the image decoding device comprises: A receiving module, used for receiving a bit stream generated by encoding; A decoding module is used to decode the bit stream to obtain the position information of the coded visible patches, mask tokens and important area blocks in the original image, generate a decoding result according to the position information of the coded visible patches, the mask tokens and the important area blocks in the original image, and process the decoding result in turn through normalization, multi-head attention mechanism and multi-layer perceptron to output a reconstructed image.

8. A computer-readable medium, characterized in that The computer-readable medium stores a computer program, which, when executed by a processor, implements the image encoding method according to any one of claims 1 to 4, or the image decoding method according to claim 5.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the image encoding method according to any one of claims 1 to 4, or the image decoding method according to claim 5, by executing the executable instructions.

Images