Image encoding method, decoding method, apparatus, readable medium and electronic device

Abstract

The present disclosure belongs to the field of artificial intelligence technology, and specifically relates to an image encoding method, a decoding method, an apparatus, a readable medium, and an electronic device. The method includes obtaining an original image, performing a blocking process to obtain a plurality of image blocks, calculating a gradient value of pixels in each image block, and selecting an important region block based on the gradient value of the pixels, and inputting the important region block and position information of the important region block in the original image into a visual conversion model for encoding to generate a bitstream. In this way, this method can selectively and controllably compress different information regions of the compressed image, perform as much retention as possible for the key regions where the information in the image is concentrated, and compress as little as possible. On the other hand, for the non-key regions where the information in the image is sparse, compress as much as possible to improve the image compression efficiency and achieve flexible coding rate control under a unified solution.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This disclosure is based on and claims priority to a Chinese patent application with application number 202210837739.2, filing date July 15, 2022, and invention title "Image Encoding Method, Decoding Method, Apparatus, Readable Medium, and Electronic Device". All contents of the Chinese patent application are incorporated herein by reference.

[0002] This disclosure belongs to the field of artificial intelligence technology, and specifically relates to an image encoding method, a decoding method, an apparatus, a readable medium, and an electronic device.

Background Art

[0003] Conventional image / video encoding is targeted at human visual tasks and is mostly used for entertainment purposes, emphasizing the fidelity, high frame rate, sharpness, etc. of video data signals. With the rapid development of 5G, big data, and artificial intelligence, under the background of image / video big data applications, media contents, such as images and videos, are widely applied in fields such as intelligent visual tasks like target detection, target tracking, image classification, image segmentation, pedestrian re - identification, etc. These intelligent visual tasks are also called intelligent tasks for machine vision.

[0004] It should be noted that the information disclosed in the above background art part is only used to enhance the understanding of the background of this disclosure, and may include information that does not constitute prior art known to those skilled in the art.

Summary of the Invention

[0005] Other characteristics and advantages of this disclosure will become clear from the following detailed description or will be partially acquired through the practice of this disclosure.

[0006] One embodiment of the present disclosure provides an image encoding method, which includes acquiring an original image, performing a blocking process to acquire a plurality of image blocks, calculating the gradient value of pixels in each image block, selecting important region blocks from the plurality of image blocks based on the gradient value of the pixels, and inputting the important region blocks and the positional information of the important region blocks in the original image into a visual transformation model for encoding to generate a bitstream.

[0007] In one embodiment of the present disclosure, an image encoding device is provided, the image encoding device comprising an acquisition module, a calculation module, and an encoding module, wherein the acquisition module is configured to acquire an original image, perform a blocking process, and acquire a plurality of image blocks; the calculation module is configured to calculate the gradient value of pixels in each image block and select important region blocks from the plurality of image blocks based on the gradient value of the pixels; and the encoding module is configured to input the important region blocks and the positional information of the important region blocks in the original image into a visual transformation model, encode it, and generate a bitstream.

[0008] In some embodiments of the present disclosure, the calculation module is further configured to calculate the gradient value of a pixel in each image block, calculate the average gradient of each image block based on the gradient value of the pixel, sort the plurality of image blocks based on the average gradient, and determine the image blocks among the plurality of image blocks whose average gradient is greater than or equal to a predetermined value as the important region blocks.

[0009] In some embodiments of the present disclosure, the encoding module is further configured to input the critical region block into the visual transformation model, output encoded visible patches and mask tokens, generate image tokens based on the encoded visible patches, the mask tokens and the location information of the critical region block in the original image, and generate the bitstream based on the image tokens.

[0010] In some embodiments of the present disclosure, the acquisition module further acquires an n × n original image, where n is a positive integer, and blocks the n × n original image into m × m blocks evenly according to the non-overlapping region, with the size of each image block being

number

[0011] In some embodiments of the present disclosure, the calculation module further discards image blocks whose gradient mean value is less than a predetermined value, and sets the predetermined value so that the number of discarded image blocks and the preset compression ratio α of the image are equal.

number

[0012] One embodiment of the present disclosure provides an image decoding method for decoding an encoding by the above-described image encoding method, the image decoding method comprising receiving an encoded bitstream, decoding the bitstream, normalizing the decoding result, and obtaining a reconstructed image through a multi-head self-attention mechanism and multilayer perceptron processing.

[0013] One embodiment of the present disclosure provides an image decoding device comprising a receiving module and a decoding module, wherein the receiving module is configured to receive an encoded bitstream, and the decoding module is configured to decode the bitstream, normalize the decoding result, and obtain a reconstructed image through a multi-head self-attention mechanism and multilayer perceptron processing.

[0014] One embodiment of the present disclosure provides a computer-readable medium for storing a computer program that, when executed by a processor, implements the image encoding method or image decoding method described above.

[0015] One embodiment of the present disclosure provides an electronic device including a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to perform an image encoding method or an image decoding method in the present embodiment by executing the executable instructions.

[0016] In one embodiment of the present disclosure, a computer program product or computer program is provided which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, and the computer device executes an image encoding method or an image decoding method as described in the above-described technical proposal.

[0017] It should be understood that the above general explanation and the detailed explanation below are illustrative and interpretive and do not limit this disclosure. [Brief explanation of the drawing]

[0018] The drawings are incorporated herein by reference to embodiments conforming to the Disclosure and are used together with the Specification to illustrate the principles of the Disclosure. Clearly, the drawings in the following description are only a few embodiments of the Disclosure, and those skilled in the art can obtain other drawings based on these without any creative effort. [Figure 1] A schematic block diagram of an exemplary image coding and decoding system architecture is shown. [Figure 2] A schematic block diagram illustrating an exemplary system architecture applying the proposed technology of this disclosure is shown below. [Figure 3] A schematic step flow of an image encoding method in one embodiment of this disclosure is shown. [Figure 4] Schematically shows a schematic diagram of image blocking to which the technical proposal of the present disclosure is applied. [Figure 5] It is a schematic diagram schematically showing the gradient average of each block to which the technical proposal of the present disclosure is applied. [Figure 6] Schematically shows a schematic diagram of an encoder module to which the technical proposal of the present disclosure is applied. [Figure 7] Schematically shows a schematic diagram of a decoder module to which the technical proposal of the present disclosure is applied. [Figure 8] Schematically shows a schematic diagram of an encoding / decoding flow to which the technical proposal of the present disclosure is applied. [Figure 9] It is a diagram schematically showing a configuration block diagram of an image encoding device provided according to an embodiment of the present disclosure. [Figure 10] Schematically shows a computer system configuration block diagram of an electronic device suitable for use in implementing an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0019] Next, exemplary embodiments will be described in more detail with reference to the drawings. However, it should be understood that the exemplary embodiments can be implemented in various forms and are not limited to the examples described herein. In contrast, these embodiments are provided to make the present disclosure more comprehensive and complete and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0020] Furthermore, the described features, structures, or characteristics can be incorporated into one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will recognize that they can implement the technical approach of the present disclosure without one or more of the specific details, or can adopt other methods, components, devices, steps, etc. In other cases, well-known methods, devices, implementations, or operations cannot be illustrated or described in detail so as not to obscure the aspects of the present disclosure.

[0021] The block diagrams shown in the drawings represent only functional entities and do not need to correspond to physically independent entities. That is, these functional entities may be implemented in software form, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowchart shown in the diagram is merely illustrative and does not require the inclusion of all content and operations / steps, nor does it require execution in the order described. For example, some operations / steps can be broken down, while others can be merged or partially merged, so the actual order of execution may vary depending on the situation.

[0023] With the increasing prevalence of machine vision intelligence tasks such as image classification, video target detection, target tracking, image segmentation, and pedestrian re-identification, current related technological solutions employ image / video coding / decoding techniques based on convolutional neural networks. However, this method is disadvantageous for image coding / decoding because it uniformly encodes all regions of the entire image.

[0024] In response to this, the present disclosure provides an image encoding method, a decoding method, an apparatus, a readable medium, and an electronic device. In the technical solutions provided by the embodiments of the present disclosure, the original image is first divided into blocks and then combined using block region gradient calculation and a visual transformation model. In this way, the method selectively and controllably compresses different information regions of the compressed image, retaining as much information as possible and compressing as little as possible in importance regions where information in the image / video is dense, and compressing as much as possible in non-important regions where information in the image is sparse, thereby improving image compression efficiency and achieving flexible code rate control under a unified solution, thereby improving image compression efficiency to some extent.

[0025] Referring to Figure 1, Figure 1 schematically shows an exemplary image coding and decoding system architecture block diagram.

[0026] The system includes a data acquisition module 101, an encoder module 102, and a decoder module 103, where the data acquisition module 101 is used to acquire an image / video 1001 and transmit it to the encoder module 102, which may use a convolutional neural network to encode the image / video 1001 into a bitstream 1002 and transmit the bitstream 1002 to the decoder module 103 at the other end. The decoder module 103 can similarly use a convolutional neural network to reconstruct the bitstream into an image / video 1003, which is then input as a human vision task 104, and finally, a result 1004 is obtained after computation by the human vision task.

[0027] Adopting this method presents the following technical challenges: The encoder and decoder encode all areas of the image / video uniformly, failing to distinguish between important and unimportant regions of the image. After uniformly encoding all areas of the image, important regions are heavily compressed, resulting in a loss of crucial information. Current methods cannot selectively compress individual regions of an image. In other words, current image encoding / decoding methods cannot discard unimportant image blocks during encoding. Consequently, these encoders, designed based on deep convolutional neural network structures, lack flexibility in controlling the compression ratio. Furthermore, current encoding / decoding systems are designed for human vision tasks and therefore cannot effectively perform machine vision intelligence analysis tasks when applied to machine vision tasks.

[0028] To address the above issues, this disclosure redesigns encoder and decoder modules in an encoding system for machine vision intelligent analysis tasks. For the encoder design, a transformer-based image encoder is proposed. For the decoder design, a decoder based on a transformer module (Block) is proposed. Referring to Figure 2, Figure 2 schematically shows an exemplary system architecture block diagram applying the technical proposal of this disclosure. This system architecture includes a data acquisition module, which is used to acquire images / videos (S201) to obtain the original image 2001. The original image is then input to an encoder module 2002, which sequentially performs blocking (S202), gradient calculation (S203), important region calculation (S204), and a vision transformer to encode (S205) the original image and output a bitstream. The bitstream output by the encoder module 2002 is then reconstructed into an image / video (S206) via a decoder module 2003 using a converter module. The reconstructed image / video is used as input for the machine vision task, and finally, the machine vision task calculation (S207) is performed to obtain result 2004.

[0029] To achieve selective compression of different regions of an image / video, this disclosure describes a method in which, when encoding image / video data, the designed encoder combines region gradient calculation and a converter module, while the image decoder design consists only of the converter module. When encoding an image, the image is block-based, gradient values ​​are calculated for image pixels within each block region, the average of the gradient values ​​for each region is calculated, all image blocks are sorted based on the average gradient values, and the lower sorted blocks are discarded. The sorted upper image blocks are input to the subsequent converter module, and the images of other image block regions are discarded as is. The compression ratio can be flexibly controlled by controlling the percentage of images to be discarded.

[0030] The image encoding method, decoding method, apparatus, readable medium, and electronic device provided in this disclosure will be described in detail below based on specific embodiments.

[0031] Referring to Figure 3, Figure 3 schematically shows the step flow of an image encoding method in one embodiment of the present disclosure. The image encoding method can be performed by a controller and mainly includes the following steps S301 to S303.

[0032] In step S301, the original image is obtained, a blocking process is performed, and multiple image blocks are obtained.

[0033] In some embodiments, an image / video is acquired using a data acquisition module to obtain the original image, and the original image is then processed into blocks. For example, if the size of the original image is n×n, the n×n image is evenly divided into m×m image blocks according to the non-overlapping area, and the size of each image block is...

number

[0034] In step S302, the gradient value of the pixels within each image block is calculated, and important region blocks are selected from multiple image blocks based on the gradient value of the pixels.

[0035] In some embodiments, calculating the gradient value of pixels within each block is advantageous for selecting important region blocks based on the pixel gradient values. This allows for selective and controllable compression of different information regions of the compressed image, retaining as much information as possible and compressing as little as possible in importance regions where information is dense in the image / video, while compressing as much as possible in non-important regions where information is sparse in the image, thereby increasing image compression efficiency and achieving flexible code rate control under a unified scheme.

[0036] In step S303, the important region block and the positional information of the important region block within the original image are input to the visual transformation model, encoded, and a bitstream is generated.

[0037] In the technical solutions provided by the embodiments of this disclosure, the original image is first divided into blocks, and then combined using block region gradient calculation and a visual transformation model. In this way, this method selectively and controllably compresses different information regions of the compressed image, retaining as much information as possible and compressing as little as possible in importance regions where information in the image / video is dense, while compressing as much as possible in non-important regions where information in the image is sparse, thereby increasing image compression efficiency and achieving flexible code rate control under a unified solution.

[0038] In some embodiments of the present disclosure, calculating the gradient value of pixels within each image block and selecting important region blocks from a plurality of image blocks based on the pixel gradient values ​​includes calculating the gradient value of pixels within each image block, calculating the average gradient value of each image block based on the pixel gradient values, sorting the plurality of image blocks based on the average gradient value, and determining the image blocks among the plurality of image blocks whose average gradient value is greater than or equal to a predetermined value as important region blocks.

[0039] In this way, important regions of information focus can be sorted and selected based on the average gradient calculation, non-important blocks of the image can be discarded, and image compression can be achieved.

[0040] In some embodiments, when selecting important region blocks, the gradients in the x and y directions are calculated for each pixel (x,y) in each image block. The calculation of the gradient in the x direction is as shown in equation (1) below.

[0041]

number

[0042] The gradient in the y-direction is calculated as shown in equation (2) below.

[0043]

number

[0044] The gradient value g of the pixel (x,y) in the x direction. x and the gradient value g in the y direction y The value of is calculated as follows: (3)

[0045]

number

[0046] Here, g(x,y) is the calculated gradient of (x,y). Next, the calculated gradient of all pixels in the block region is calculated as shown in equation (4) below.

[0047]

number

[0048] Here, d(i,j) is the average gradient of all pixels within each block. The range of values ​​for both i and j is 0 to m-1.

[0049] Referring to Figure 5, Figure 5 schematically shows a schematic diagram of the average gradient of each image block to which the technical method of this disclosure is applied. The original image 5002, which has a size of n×n, is evenly divided into m×m504 blocks according to the non-overlapping region (S502), and the size of each image block is

number

number

[0050] Thus, this method selectively and controllly compresses different information regions of a compressed image, retaining as much information as possible and compressing as little as possible in importance regions where information is dense in the image / video, while compressing as much as possible in non-important regions where information is sparse, thereby improving image compression efficiency and achieving flexible code rate control under a unified scheme.

[0051] In one embodiment of the present disclosure, inputting a critical region block and the positional information of the critical region block in the original image into a visual transformation model for encoding and generating a bitstream includes inputting the critical region block into the visual transformation model and outputting an encoded visible patch and a mask token, and generating an image token based on the encoded visible patch, the mask token and the positional information of the critical region block in the original image, and generating a bitstream based on the image token.

[0052] Referring to Figure 6, Figure 6 schematically shows a schematic diagram of an encoder module to which the proposed technology of this disclosure is applied. After obtaining the non-overlapping region segmentation result 6002 of the original image, gradient calculation (S602) is performed, followed by critical region calculation (S604) to determine critical region blocks. The critical region blocks that have not been discarded, along with their positional information in the original image, are input to the Vision Transformer model 6004. Here, the patch embeddings and positional embeddings 60042 information of the critical region blocks are input to the encoder 60044 module of the Vision Transformer model 6004.

[0053] When the 6004 visual transformation model uses multiple encoder modules to perform calculations, it obtains p patch information and their positional information, the same number as the input. These are then re-sorted based on their positional information to obtain d×d image blocks, the same number as the original image size. Of these d×d image blocks, the patch information of important region blocks that have not been discarded so far is calculated by the 6004 visual transformation model and is called Encoded Visible Patches. The remaining blocks, sorted based on their positional information, are called Mask Tokens.

[0054] Thus, the video coding system methods of related technical proposals are directed towards human vision tasks, and when directed towards machine vision tasks, they cannot successfully complete machine vision intelligence analysis tasks. The technical proposal of this embodiment is directed towards machine vision tasks and can perform machine vision intelligence analysis tasks more effectively.

[0055] In some embodiments of this disclosure, the number of image blocks to be discarded and the preset image compression ratio α can be set to predetermined values ​​such that they satisfy equation (5).

[0056]

number

[0057] Here, p is the number of image blocks that were discarded.

[0058] In this way, the compression ratio can be flexibly controlled by controlling the ratio of image blocks to discard after the image has been block-compressed.

[0059] One embodiment of the present disclosure provides an image decoding method for decoding an encoding performed by the above-described image encoding method, the method comprising: receiving a bitstream generated by encoding; decoding the bitstream; normalizing the decoding result; and outputting a reconstructed image after processing with a multi-head self-attention mechanism and a multilayer perceptron.

[0060] Referring to Figure 7, Figure 7 schematically shows a schematic diagram of a decoder module to which the proposed technology of this disclosure is applied. Referring to Figure 6, after obtaining the output coded visible patch 70022 and mask token 70024 of the visual transformation model based on region gradient information, these two parts are combined with the position information of the original image and embedded in position (S702) and added together, and the result of the addition is input to a decoder constructed by the converter module 7004 and decoded (S704). In the decoder, the converter module 7004 consists of normalize layers 70042 and 70046, a multi-head self-attention layer 70044, and a multi-layer neural network (also called a multi-layer perceptron, MLP) module 70048.

[0061] After the image block information of the vector t output by the normalization layer 70042 is input to the multi-head self-attention layer 70044, the weight matrix W in the multi-head self-attention layer 70044 is then input. t , and the attention weight matrix for each head

number

[0062] Next, the attention weight matrix for each header is assigned to the image block vector t.

number

number

[0063]

number

[0064] Each image block vector t is calculated corresponding to the attention given to each header, as shown in the following equations (7) and (8):

[0065]

number

[0066]

number

[0067] In the above equation, h t Here, the image block vector t is the header corresponding to each attention,

number

number

number

[0068] In equation (8),

number

number

[0069] To facilitate understanding of the proposed technology described herein, please refer to Figure 8, which schematically shows a codec flow to which the proposed technology described herein is applied.

[0070] On the encoding side, in step S801, the original image (or video) 8002 is block-based, and the n×n image is evenly blocked into m×m blocks according to the non-overlapping regions.

[0071] In step S802, for each pixel (x,y) in the image block, the gradient in the x-direction and the gradient in the y-direction are calculated using equations (1) and (2).

[0072] In step S803, the gradient value of the pixel (x,y) is calculated using equation (3).

[0073] In step S804, the mean gradient of all pixels in each image block is calculated using equation (4).

[0074] In step S805, all image blocks are sorted according to the value of d(i,j). The p blocks with the smallest sorted d(i,j) values ​​are discarded, and the calculation of the compression ratio α satisfies equation (5).

[0075] In step S806, an image token is generated based on the result of the discard operation, and for example, an encoded visible patch is included as a mask token.

[0076] In step S807, a bitstream 8004 is generated based on the image token.

[0077] On the decryption side, in step S808, the encoded visible patch, mask token, and position embedding information are obtained based on the bitstream 8006.

[0078] In step S809, the data with the encoded visible patch and mask token embedded in its position is normalized. In step S810, multi-head self-attention is calculated using (6) to (8).

[0079] In step S811, the multi-head self-attention calculation results are normalized.

[0080] In step S812, the normalized results are computed using a multilayer perceptron.

[0081] In step S813, the reconstructed image / video 8008 is output.

[0082] This disclosure proposes an image codec design using a method based on region gradient calculation in image coding and decoding, that is, selectively compressing image content information, calculating gradients, calculated gradient values, and average calculated gradient values ​​for block images, selecting important information blocks based on the average calculated gradient values, and calculating key regions based on the average calculated gradient information. This sorts and selects important regions of information focus, discards non-important regions in the image, and achieves selective image compression. By controlling the ratio of images to be blocked after the image has been blocked, the compression ratio can be flexibly controlled. Furthermore, all related technical proposals for video coding systems are for human vision tasks, and those for machine vision tasks cannot perform machine vision intelligent analysis tasks well. The system proposed in this disclosure is for machine vision tasks and can better achieve machine vision intelligent analysis tasks.

[0083] While the accompanying drawings illustrate the steps of the method in this disclosure in a specific order, this does not require or imply that these steps must be performed in that specific order, or that the desired results cannot be achieved unless all illustrated steps are performed. Additionally or alternatively, some steps may be omitted, multiple steps may be merged into a single step execution, or a single step may be broken down into multiple step executions.

[0084] Embodiments of the apparatus of the present disclosure are described below and can be used to perform the image coding method or image decoding method in the above-described embodiments of the present disclosure. Figure 9 is a schematic block diagram of the configuration of the image coding apparatus provided by the embodiments of the present disclosure. As shown in Figure 9, the image coding apparatus 900 may include an acquisition module 901, a calculation module 902, and an encoding module 903.

[0085] The acquisition module 901 is configured to acquire the original image, perform a blocking process, and acquire multiple image blocks. The calculation module 902 is configured to calculate the gradient value of pixels within each image block and to select important region blocks from multiple image blocks based on the pixel gradient values. The encoding module 903 is configured to input important region blocks and their positional information within the original image into a visual transformation model, encode them, and generate a bitstream.

[0086] In some embodiments of the present disclosure, the computation module 902 is further configured to calculate the gradient value of pixels in each image block, calculate the average gradient of each image block based on the pixel gradient values, sort the multiple image blocks based on the average gradient, and determine the image blocks among the multiple image blocks whose average gradient is greater than or equal to a predetermined value as important region blocks.

[0087] In some embodiments of the present disclosure, the encoding module 903 is further configured to input important region blocks into a visual transformation model, output encoded visible patches and mask tokens, generate image tokens based on the location information of the encoded visible patches, mask tokens and important region blocks in the original image, and generate a bitstream based on the image tokens.

[0088] In some embodiments of the present disclosure, the acquisition module 901 further acquires an n × n original image, where n is a positive integer, and blocks the n × n original image into m × m blocks evenly according to the non-overlapping region, with the size of each image block being

number

[0089] In some embodiments of this disclosure, the calculation module is further configured to discard image blocks whose gradient mean value is less than a predetermined value, where the number of image blocks to be discarded and the preset compression ratio α of the images are determined by setting the predetermined value.

[0090]

number

[0091] One embodiment of the present disclosure provides an image decoding device comprising a receiving module and a decoding module, wherein the receiving module is configured to receive an encoded bitstream, and the decoding module is configured to decode the bitstream, normalize the decoding result, and output a reconstructed image through a multi-head self-attention mechanism and multilayer perceptron processing.

[0092] Specific details of the image encoding or decoding apparatus provided in each embodiment of this disclosure are described in detail in the corresponding method embodiments not further described herein.

[0093] Figure 10 schematically shows a computer system configuration block diagram of an electronic device for realizing an embodiment of the present disclosure.

[0094] Note that the computer system 1000 of the electronic device shown in Figure 10 is just one example and does not impose any limitations on the functions and scope of use of the embodiments of this disclosure.

[0095] As shown in Figure 10, the computer system 1000 includes a central processor 1001 (Central Processing Unit, CPU) capable of performing various appropriate operations and processes according to programs stored in read-only memory 1002 (ROM) or programs loaded from memory unit 1008 into random access memory 1003 (RAM). The random access memory 1003 also stores various programs and data necessary for the operation of the system. The central processor 1001, read-only memory 1002, and random access memory 1003 are connected to each other via a bus 1004. An input / output interface 1005 (i.e., I / O interface) is also connected to the bus 1004.

[0096] The input / output interface 1005 includes an input unit 1006, which includes a keyboard and mouse; an output unit 1007, which includes a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage unit 1008, which includes a hard disk; and a communication unit 1009, which includes a network interface card, such as a local network card or modem. The communication unit 1009 performs communication processing via a network such as the Internet. A driver 1010 is also connected to the input / output interface 1005 as needed. Removable media 1011, such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memory, are installed on the drive 1010 as needed, and easily readable computer programs are installed on the storage unit 1008 as needed.

[0097] In particular, according to embodiments of the present disclosure, processes described in various method flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product that includes a computer program carried on a computer-readable medium containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication unit 1009 and / or installed from removable media 1011. When this computer program is executed by the central processor 1001, various functions specific to the system of the present disclosure are performed.

[0098] The computer-readable medium shown in the embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of more than one of these. More specific examples of computer-readable storage mediums include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that is instructed to run or be used in combination with a system, apparatus, or device. On the other hand, in this disclosure, a computer-readable signaling medium may include data signals propagating in or as part of a baseband carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination described above. The computer-readable signaling medium may be any computer-readable medium other than a computer-readable storage medium capable of transmitting, propagating, or transmitting programs used by or in combination with systems, apparatus, or devices. Program code contained in a computer-readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, or any suitable combination described above.

[0099] Flowcharts and block diagrams in the drawings illustrating the feasible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code containing one or more executable instructions for performing a given logical function. It should also be noted that in alternative implementations, the functions shown in the blocks may occur in an order different from the order shown in the drawings. For example, two consecutively shown blocks may actually be executed essentially in parallel, or in reverse order depending on the related functions. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented in a dedicated hardware-based system that performs a given function or operation, or in a combination of dedicated hardware and computer instructions.

[0100] While the detailed description above refers to several modules or units of the device for performing operations, such division is not mandatory. In practice, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in a single module or unit. Conversely, the features and functions of a single module or unit described above can be further divided and embodied in multiple modules or units.

[0101] As will be readily apparent to those skilled in the art from the above description of embodiments, the exemplary embodiments described herein can be implemented by software, and can also be implemented by combining the necessary hardware with the software. Accordingly, the technical aspects according to the embodiments of this disclosure may be embodied in the form of a software product which may be stored on a single non-volatile storage medium (which may be a CD-ROM, USB disk, mobile hard disk, etc.) and include several instructions for performing the methods according to the embodiments of this disclosure.

[0102] A person skilled in the art will readily come up with other embodiments of the Disclosure after considering the specification and practicing the inventions disclosed herein. This Disclosure aims to cover any variations, uses, or adaptations of the Disclosure, including common or conventional techniques known in the Art but not disclosed herein, in accordance with the general principles of the Disclosure.

[0103] This disclosure is not limited to the exact structure shown in the drawings above, and it should be understood that various modifications and changes are possible without departing from its scope. The scope of this disclosure is limited only to the attached claims.

Claims

1. An image encoding method, The process involves obtaining the original image, performing a block-forming process, and obtaining multiple image blocks. The process involves calculating the gradient value of pixels within each image block, and selecting important region blocks from the plurality of image blocks based on the gradient value of the pixels. This includes inputting the important region block and the positional information of the important region block within the original image into a visual transformation model, encoding it, and generating a bitstream. An image encoding method characterized by the following.

2. Calculating the gradient value of pixels within each of the aforementioned image blocks, and selecting important region blocks from the plurality of image blocks based on the gradient value of the pixels, The process involves calculating the gradient value of each pixel within each image block, and then calculating the average gradient value of each image block based on the gradient value of the pixels. This includes sorting the plurality of image blocks based on the average gradient value, and determining the image blocks whose average gradient value is greater than or equal to a predetermined value as the important region blocks. The image coding method according to feature 1.

3. Inputting the aforementioned important region block and the positional information of the aforementioned important region block within the original image into the visual transformation model, encoding it, and generating a bitstream is: The aforementioned important region block is input to the visual transformation model, and an encoded visible patch and a mask token are output. This includes generating image tokens based on the encoded visible patch, the mask token, and the positional information of the important region block in the original image, and generating the bitstream based on the image tokens. The image coding method according to feature 1.

4. Inputting the aforementioned important region block and the positional information of the aforementioned important region block within the original image into the visual transformation model, encoding it, and generating a bitstream is: The aforementioned important region block is input to the visual transformation model, and an encoded visible patch and a mask token are output. This includes generating image tokens based on the encoded visible patch, the mask token, and the positional information of the important region block in the original image, and generating the bitstream based on the image tokens. The image coding method according to feature 2.

5. Obtaining the original image, performing a blocking process, and obtaining multiple image blocks means obtaining an n × n original image, where n is a positive integer. The original n×n image is evenly divided into m×m blocks according to the non-overlapping regions, and the size of each image block is... [Math 1] The image encoding method according to claim 2, characterized in that m is a positive integer and n > m.

6. Calculating the gradient value of pixels within each of the aforementioned image blocks and selecting important region blocks from the plurality of image blocks based on the gradient value of the pixels further involves, This includes discarding image blocks whose average gradient value among the plurality of image blocks is less than a predetermined value. By setting the predetermined value, the number of discarded image blocks and the preset image compression ratio α are determined. teeth, [Math 2] The image encoding method according to claim 5, characterized in that the condition is met and p is the number of discarded image blocks.

7. An image decoding method for decoding a bitstream obtained by encoding using the image encoding method described in any one of claims 1 to 6, Receiving an encoded and generated bitstream, The process includes: decoding the bitstream to obtain the positional information of encoded visible patches, mask tokens, and important region blocks in the original image; generating a decoding result based on the positional information of encoded visible patches, mask tokens, and important region blocks in the original image; and obtaining a reconstructed image by normalizing the decoding result and processing it through a multi-head self-attention mechanism and a multilayer perceptron. An image decoding method characterized by the following:

8. An image encoding device comprising an acquisition module, a calculation module, and an encoding module, The acquisition module is configured to acquire the original image, perform a blocking process, and acquire multiple image blocks. The calculation module is configured to calculate the gradient value of pixels within each image block and to select important region blocks from the plurality of image blocks based on the gradient value of the pixels. The encoding module is configured to input the important region block and the positional information of the important region block within the original image into a visual transformation model, encode it, and generate a bitstream. An image coding device characterized by the following:

9. An image decoding device for decoding a bitstream obtained by encoding using the image encoding method described in any one of Claims 1 to 6, It includes a receiving module and a decoding module, The receiving module is configured to receive the encoded and generated bitstream, The decoding module is configured to decode the bitstream to obtain the positional information of encoded visible patches, mask tokens, and important region blocks in the original image, generate a decoding result based on the positional information of the encoded visible patches, mask tokens, and important region blocks in the original image, and obtain a reconstructed image by normalizing the decoding result and processing it with a multi-head self-attention mechanism and a multilayer perceptron. An image decoding device characterized by the following features.

10. A computer program that, when executed by a processor, realizes the image encoding method described in any one of claims 1 to 6.

11. A computer program that, when executed by a processor, realizes the image decoding method described in claim 7.

12. An electronic device including a processor and memory for storing executable instructions of the processor, The processor is configured to execute the image encoding method described in any one of claims 1 to 6 by executing the executable instructions. An electronic device characterized by the following features.

13. An electronic device including a processor and memory for storing executable instructions of the processor, The processor is configured to execute the image decoding method described in claim 7 by executing the executable instructions. An electronic device characterized by the following features.

Images