Image processing method, device, system and equipment, storage medium and program product

Abstract

The invention discloses an image processing method, device, system and equipment, a storage medium and a program product, and the method comprises the steps: carrying out the mask processing of a part of image blocks in a plurality of image blocks of an original image according to the edge position of an object in the original image, and obtaining a mask image block and a visible image block; partial image blocks do not contain the edge positions, and the visible image blocks are image blocks which are not subjected to mask processing; a bitstream is generated based on the masked image block and the visible image block. Considering that the edge of an object belongs to a key feature in an image and is a key influencing image reconstruction, the edge position of the object in the original image is used for selectively performing mask processing on image blocks segmented from the original image, so that the content of unimportant image blocks is covered and the content of important image blocks is reserved to be visible; therefore, the amount of data needing to be compressed is reduced, the length of the generated bit stream is reduced, and the requirement for low-bit-rate compression can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image compression, in particular to an image processing method, device, system, equipment, storage medium and program product. BACKGROUND

[0002] With the development of deep learning algorithms, artificial intelligence technology has been widely applied in the field of image compression. Using artificial intelligence technology for image encoding and decoding can not only enhance the perceptual quality of compressed data, but also show significant performance in restoring high-frequency details of images, especially in enhancing image texture and details.

[0003] However, the current image coding and decoding related technical solutions using artificial intelligence mainly focus on improving visual perceptual quality, while relatively ignoring the importance of image compression under low bit rate conditions. Especially under extremely low bit rate conditions, the image reconstruction effect of the existing related technical solutions is not good, thereby reducing the quality of the recovered image. SUMMARY

[0004] The purpose of the present application is to propose an image processing method, device, system, equipment, storage medium and program product to solve the above problems of the prior art. The purpose is achieved by the following technical solutions.

[0005] The first aspect of the present application proposes an image processing method, comprising:

[0006] According to the edge position of the object in the original image, the mask processing is performed on part of the image blocks obtained by cutting the original image, to obtain mask image blocks and visible image blocks; the part of the image blocks does not contain the edge position, and the visible image blocks are image blocks without mask processing;

[0007] Based on the mask image blocks and the visible image blocks, a bit stream is generated.

[0008] Based on the first aspect of the above image processing method, the present application has at least the following beneficial effects or advantages:

[0009] Considering that the edge of the object is a key feature in the image and is a key factor affecting image reconstruction, the mask processing is selectively performed on the image blocks of the original image by using the edge position of the object in the original image, so as to cover the contents of unimportant image blocks and keep the contents of important image blocks visible, and then a bit stream is generated based on the mask image blocks and the visible image blocks. Since the mask image blocks are covered with image information and the image information of the visible image blocks is kept, the amount of data to be compressed is reduced, thereby reducing the length of the generated bit stream, which can meet the low bit rate compression demand.

[0010] The second aspect of the present application provides an image processing method, which is used for processing the bitstream obtained by the first aspect, and the method comprises the following steps of:

[0011] reconstructing the bitstream to obtain an initial reconstructed image, wherein the initial reconstructed image is obtained by splicing the reconstructed visible image blocks and the reconstructed mask image blocks;

[0012] dividing the initial reconstructed image into a plurality of regions, and dividing each of the regions into a plurality of sub-blocks;

[0013] extracting one target sub-block from each of the regions each time, and calling a preset visual conversion model to recover the region belonging to the reconstructed mask image blocks by using the content belonging to the reconstructed visible image blocks in the extracted target sub-block, until the sub-blocks in each of the regions are extracted;

[0014] splicing the recovered sub-blocks output by each calling of the visual conversion model to obtain a target reconstructed image.

[0015] Based on the image processing method of the second aspect, the present application has at least the following beneficial effects or advantages:

[0016] After the initial reconstructed image obtained by splicing the reconstructed visible image blocks and the reconstructed mask image blocks is decoded from the bitstream, in order to recover the missing image information in the image, the initial reconstructed image is divided into a plurality of regions, and each of the regions is divided into a plurality of sub-blocks, then one sub-block is extracted from each of the regions each time, and the visual conversion model is called to recover the region belonging to the reconstructed mask image blocks by using the content belonging to the reconstructed visible image blocks in the extracted sub-block, until the sub-blocks in each of the regions are extracted, thereby realizing multi-stage recovery reconstruction, each stage considering the spatial context relationship between the sub-blocks, so as to use the content belonging to the reconstructed visible image blocks to recover the region belonging to the reconstructed mask image blocks in the sub-blocks, and each stage extracts one sub-block from each of the regions to recover and reconstruct, which avoids the appearance of large continuous reconstructed mask image blocks, reduces the difficulty of recovery reconstruction, and effectively realizes the recovery reconstruction even in the case of extremely low bit rate. Finally, the recovered sub-blocks output by each calling of the visual conversion model are spliced to obtain a target reconstructed image with high quality.

[0017] The third aspect of the present application provides an image processing system, which comprises:

[0018] The sending end is configured to perform mask processing on part of the image blocks obtained by cutting the original image according to the edge position of the object in the original image, to obtain mask image blocks and visible image blocks; the part of the image blocks does not contain the edge position, and the visible image blocks are image blocks that are not subjected to mask processing; and the sending end is further configured to generate a bitstream based on the mask image blocks and the visible image blocks and send the bitstream to the receiving end.

[0019] The receiving end is configured to perform image reconstruction on the bitstream to obtain an initial reconstructed image; the initial reconstructed image is obtained by splicing reconstructed visible image blocks and reconstructed mask image blocks; the initial reconstructed image is divided into a plurality of regions, and each of the regions is divided into a plurality of blocks; each time, one target block is extracted from each of the regions, and a preset visual conversion model is called to use the content of the target block belonging to the reconstructed visible image blocks to recover the region belonging to the reconstructed mask image blocks, to output a corresponding recovered block of the target block, until the blocks in each of the regions are extracted; and the recovered blocks output by each time calling the visual conversion model are spliced to obtain a target reconstructed image.

[0020] A fourth aspect of the present application provides an image processing device, which comprises:

[0021] A mask module is configured to perform mask processing on part of a plurality of image blocks of an original image according to an edge position of an object in the original image, to obtain mask image blocks and visible image blocks; the part of the image blocks does not contain the edge position, and the visible image blocks are image blocks that are not subjected to mask processing.

[0022] A compression module is configured to generate a bitstream based on the mask image blocks and the visible image blocks.

[0023] A fifth aspect of the present application provides an image processing device, which comprises:

[0024] A reconstruction module is configured to perform image reconstruction on a received bitstream to obtain an initial reconstructed image; the initial reconstructed image is obtained by splicing reconstructed visible image blocks and reconstructed mask image blocks.

[0025] A segmentation module is configured to divide the initial reconstructed image into a plurality of regions, and divide each of the regions into a plurality of blocks.

[0026] A recovery module is configured to extract one target block from each of the regions each time, and call a preset visual conversion model to use the content of the target block belonging to the reconstructed visible image blocks to recover the region belonging to the reconstructed mask image blocks, to output a corresponding recovered block of the target block, until the blocks in each of the regions are extracted.

[0027] The splicing module is configured to splice the recovered patches output by the visual conversion model in each call to obtain a target reconstructed image.

[0028] The sixth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to the first aspect or the second aspect.

[0029] The seventh aspect of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the method according to the first aspect or the second aspect.

[0030] The eighth aspect of the present application provides a computer program product, comprising a computer program executable by a processor to implement the method according to the first aspect or the second aspect.

[0031] The above description is only a summary of the technical solutions of the present application. In order to better understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0032] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0033] Figure 1 The figure is a schematic diagram of an image processing system according to an exemplary embodiment;

[0034] Figure 2 The figure is an embodiment flow chart of an image processing method according to an exemplary embodiment;

[0035] Figure 3 The figure is a mask processing schematic diagram according to an exemplary embodiment;

[0036] Figure 4 The figure is a structure schematic diagram of an encoding model according to an exemplary embodiment;

[0037] Figure 5 The figure is an embodiment flow chart of another image processing method according to an exemplary embodiment;

[0038] Figure 6A The figure is a division schematic diagram of an initial reconstructed image according to an exemplary embodiment;

[0039] Figure 6B This is a schematic diagram illustrating the structure of a visual conversion model according to an exemplary embodiment;

[0040] Figure 7 This is a schematic diagram illustrating an overall process of image processing according to an exemplary embodiment;

[0041] Figure 8 This is a schematic diagram illustrating the entire process of image processing according to an exemplary embodiment;

[0042] Figure 9 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment;

[0043] Figure 10 This is a schematic diagram illustrating the structure of another image processing apparatus according to an exemplary embodiment;

[0044] Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment;

[0045] Figure 12 This is a schematic diagram illustrating the structure of a storage medium according to an exemplary embodiment. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0049] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0050] Currently, image encoding and decoding technologies using artificial intelligence are designed for compressing complete images without the process of image generation. In cases of low bit rate compression, important information in the image cannot be effectively preserved, resulting in relatively low quality of the reconstructed image.

[0051] To address the aforementioned technical problems, this application proposes an image processing method focused on improving image restoration quality under low bit-rate compression. In the encoding process, the original image is divided into multiple image blocks. Considering that object edges are the most important factor affecting image reconstruction results, selective masking is performed on multiple image blocks based on the edge positions of objects in the original image, resulting in masked image blocks and visible image blocks. This masks the content of unimportant image blocks while retaining the content of important image blocks. A bitstream is then generated based on the masked image blocks and visible image blocks. Since the image information of the masked image blocks is masked while the image information of the visible image blocks is preserved, the amount of data requiring compression is reduced, thereby decreasing the length of the generated bitstream and meeting the requirements for low bit-rate compression.

[0052] During the decoding process, the decoded bitstream obtains an initial reconstructed image obtained by stitching together reconstructed visible image blocks and reconstructed mask image blocks. To recover the missing image information from the mask image blocks, the initial reconstructed image is divided into multiple regions, and each region is further divided into multiple blocks. Then, one block is extracted from each region at a time, and the visual transformation model is invoked to use the content of the extracted block belonging to the reconstructed visible image block to reconstruct the region belonging to the reconstructed mask image block. This process continues until all blocks in each region have been extracted, thus achieving multi-stage reconstruction. Each stage considers the spatial context between blocks to utilize the content of the reconstructed visible image block to reconstruct the region belonging to the reconstructed mask image block within the block. Furthermore, each stage extracts one block from each region for reconstruction, avoiding large, continuous reconstructed mask image blocks and reducing the difficulty of reconstruction. Even under extremely low bitrate conditions, reconstruction can be effectively achieved. Finally, the recovered blocks output from each call to the visual transformation model are stitched together to obtain a high-quality target reconstructed image.

[0053] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0054] See Figure 1 , Figure 1This specification illustrates an architecture diagram of an image processing system according to an embodiment of the present specification. The image processing system may include a receiver 100 and a transmitter 200. This application embodiment does not limit the number of receivers 100, and the number of receivers 100 may be one or more. The transmitter 200 may be implemented as a distributed transmitter cluster composed of multiple transmitters, or as a single transmitter. The transmitter may also be a transmitter of a distributed system, or a transmitter combined with blockchain. The sending end can also be a cloud sending end of basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing sending end or intelligent cloud host with artificial intelligence technology. This application embodiment does not limit this. The receiving end 100 can be a browser, APP (Application), or web application such as H5 (Hypertext Markup Language 5), or a lightweight application (also known as a mini-program), or a cloud application, but is not limited to these. The sending end 200 and the receiving end 100 can establish a direct communication connection through wired communication or an indirect communication connection through wireless communication. This application embodiment does not limit this.

[0055] exist Figure 1 In the image processing system shown, the transmitting end 200 compresses and encodes the original image and transmits the generated bit stream to the receiving end 100. The receiving end 100 performs the decoding process of the bit stream, thereby enabling intelligent analysis tasks for machine vision, such as target detection, target tracking, image classification, image segmentation, and pedestrian re-identification, on the decoded and reconstructed image.

[0056] The transmitting end 200 is used to perform masking processing on a portion of the multiple image blocks obtained by segmenting the original image according to the edge position of the object in the original image, to obtain masked image blocks and visible image blocks; the portion of the image blocks does not contain the edge position, and the visible image blocks are image blocks that have not been masked; based on the masked image blocks and the visible image blocks, a bit stream is generated and the bit stream is sent to the receiving end 100;

[0057] The receiving end 100 is used to perform image reconstruction on the bitstream to obtain an initial reconstructed image. The initial reconstructed image is obtained by stitching together reconstructed visible image blocks and reconstructed mask image blocks. The initial reconstructed image is divided into multiple regions, and each region is further divided into multiple blocks. The initial reconstructed image is obtained by stitching together reconstructed visible image blocks and reconstructed mask image blocks. Each time, a target block is extracted from each region, and a preset visual transformation model is called to use the content of the reconstructed visible image blocks in the extracted target block to restore the region belonging to the reconstructed mask image block, so as to output the corresponding restored block of the target block, until all blocks in each region are extracted. The restored blocks output by each call to the visual transformation model are stitched together to obtain the target reconstructed image.

[0058] The scheme implemented in this specification involves image segmentation during encoding, with partial image blocks masked based on the edges of objects. Since the masked image blocks obscure the image information, while the important image information of the remaining image blocks is preserved, the amount of data to be compressed is reduced, thereby lowering the length of the generated bitstream and meeting the requirements for low bit rate compression.

[0059] During decoding, in order to recover the missing image information of the mask image block, a multi-stage recovery and reconstruction is performed using a visual transformation model. Each stage considers the spatial context relationship between blocks and uses the content of the reconstructed mask image block to recover the region of the reconstructed mask image block in the block. Moreover, each stage extracts a block from each region for recovery and reconstruction. This avoids the occurrence of large continuous reconstructed mask image blocks, reduces the difficulty of recovery and reconstruction, and can effectively achieve recovery and reconstruction even under extremely low bit rate conditions.

[0060] It is understood that the image processing system described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0061] Example 1

[0062] Figure 2 This is a flowchart illustrating an embodiment of an image processing method according to an exemplary embodiment. This embodiment is used to explain the image encoding process, such as... Figure 2 As shown, it includes the following steps 201-202:

[0063] Step 201: Based on the edge position of the object in the original image, perform masking processing on a portion of the multiple image blocks obtained by segmenting the original image to obtain masked image blocks and visible image blocks. The portion of the image blocks does not contain edge positions, and the visible image blocks are the image blocks that have not been masked.

[0064] Step 202: Generate a bitstream based on the masked image block and the visible image block.

[0065] The original image can be understood as the image that needs to be compressed and encoded. The original image is divided into multiple small image blocks so that subsequent processing can be performed on a block-by-block basis. The original image can be divided into uniform blocks, that is, divided according to a fixed block size. This application does not specify the fixed block size, for example, it can be 16*16, 14*14, 4*4, etc.

[0066] For example, assuming the original image size is 16*16, the original image can be divided into (16 / 4)*(16 / 4) = 16 image blocks according to the fixed block size of 4*4.

[0067] Edge location can be understood as the positional information of the object's edge contour in the original image. Its data representation can be expressed by the set of pixel coordinates (x, y) of the edge contour. The object's edge contour reflects the object's shape and is a salient feature in the image. Experimental studies have shown that the object's edge contour has the greatest impact on the image reconstruction result. Therefore, selective image block masking is performed with reference to the object's edge location to ensure that the object's edge contour is not lost.

[0068] A masked image block refers to an image block that has undergone masking processing. Since masking processing fills the pixels of the image block with preset values, the original pixel information of the image block is lost. For example, the preset value of the filled pixels can be 0 or 1.

[0069] Visible image patches refer to image patches that have not undergone masking and retain their original pixel information. Visible image patches can be considered more important image patches in the image, while masked image patches can be considered less important. Since the pixel information in masked image patches is identical, the amount of data to be compressed is negligible.

[0070] Bitstream can be understood as a way of transmitting or storing image data in the form of bits; it is a digital representation of an image.

[0071] For example, suppose a 16*16 original image is divided into 16 image blocks. Based on the edge positions of objects in the image, the four image blocks that do not contain edge positions are masked to obtain four masked image blocks and 14 visible image blocks. Then, compression encoding is performed on these four masked image blocks and 14 visible image blocks to generate the bitstream of the original image.

[0072] This completes the above. Figure 2 The image processing flow shown considers that object edges are key features in an image and significantly impact image reconstruction. Therefore, by using the edge locations of objects in the original image, selective masking is applied to image patches. This masks less important patches while preserving the content of important patches. A bitstream is then generated based on the masked and visible image patches. Because the masked patches obscure image information while the visible patches retain it, the amount of data requiring compression is reduced, thus decreasing the length of the generated bitstream and meeting the requirements for low bitrate compression.

[0073] The image processing methods provided in this application can be applied, but are not limited to, to image processing application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, product image processing scenarios in e-commerce services, video image processing scenarios in education services, and evidence image processing scenarios in legal services are not limited here.

[0074] In one optional embodiment of this application, step 201 above, which involves masking a portion of the image blocks obtained by segmenting the original image based on the edge positions of objects in the original image, includes the following specific steps:

[0075] Edge detection is performed on the original image to obtain the edge positions of objects in the original image;

[0076] Filter candidate image patches from multiple image patches that do not contain edge locations;

[0077] Based on network bandwidth, select the target image block that needs to be masked from the candidate image blocks;

[0078] Masking is performed on the target image patch.

[0079] Edge detection can be understood as an image processing technique that identifies the location of object boundaries in an image. Commonly used edge detection algorithms include the Sobel operator, Canny edge detection, and Laplacian operator.

[0080] Candidate image patches can be understood as image patches whose pixel information can be masked; that is, these image patches can be discarded as image data at low bit rates. Since candidate image patches do not contain edge locations, important regions can still be compressed completely even at low bit rates.

[0081] Network bandwidth refers to the ability or rate at which a network can transmit image data within a certain time. It can reflect the network's transmission rate. When the network bandwidth decreases, the transmission rate will decrease. By increasing the number of image blocks that are masked, the transmission rate can be adapted to the lower transmission rate. When the network bandwidth is sufficient, the number of image blocks that are masked can be reduced to provide higher quality image reconstruction. Therefore, the number of image blocks that are masked can be dynamically selected based on the network bandwidth.

[0082] The target image block is masked by means of: constructing a mask image of the same size as the image block; masking the target image block according to the mask image; the pixel value of each pixel in the mask image can be 0; after masking the target image block with the mask image, the pixel value of each pixel in the target image block is set to 0, thereby achieving the effect of masking.

[0083] For example, such as Figure 3 As shown, among the 24 image blocks that are divided, 5 image blocks are masked by path masking to obtain masked image blocks, while the remaining 19 image blocks are all visible image blocks.

[0084] In this embodiment, edge detection is performed on the original image to discover key features in the image, namely the edge positions of objects. Candidate image blocks that do not contain edge positions are filtered out, and target image blocks that need to be masked are selected according to the current network bandwidth for masking processing. This ensures that important information in the image can be completely compressed while adapting to the current network bandwidth limitations.

[0085] In one optional embodiment of this application, the process of selecting the target image block to be masked from candidate image blocks based on network bandwidth includes the following specific steps:

[0086] The target compression ratio is adjusted according to changes in network bandwidth; the target compression ratio is negatively correlated with the length of the bitstream.

[0087] The target number of image blocks that need to be masked is determined based on the adjusted target compression ratio and the number of candidate image blocks.

[0088] Select the target number of target image blocks from the candidate image blocks.

[0089] The target compression ratio can be understood as an indicator used to measure image compression efficiency, reflecting the current network bandwidth. In this embodiment, the target compression ratio is dynamically adjusted as the network bandwidth changes. A higher target compression ratio indicates a lower image compression bitrate, which in turn requires masking of more image targets. Therefore, the target compression ratio is negatively correlated with the length of the final compressed bitstream.

[0090] The target compression ratio can be viewed as a mask ratio, and thus the target compression ratio can be directly multiplied by the number of candidate image blocks to obtain the target number. It is understood that the method for determining the target number given above is merely an illustrative example and does not limit the scope of protection of this application. For example, a functional relationship can also be used to convert the target compression ratio into a mask ratio to determine the target number.

[0091] It should be noted that the number of targets is less than or equal to the number of candidate image patches.

[0092] The target image blocks of the specified number are selected from the candidate image blocks. Specifically, the target image blocks can be selected randomly from the candidate image blocks or non-contiguous target image blocks can be selected. This application does not impose any specific limitations on this.

[0093] In this embodiment, by adjusting the appropriate target compression ratio according to changes in network bandwidth, the target image blocks are selected by combining the adjusted target compression ratio with the number of candidate image blocks. This ensures that the original image information is fully compressed as much as possible while meeting network bandwidth limitations, thereby reducing the difficulty of subsequent recovery work.

[0094] In one optional embodiment of this application, the process of adjusting the target compression ratio according to changes in network bandwidth includes the following specific steps:

[0095] If the difference between the currently acquired network bandwidth and the previously acquired network bandwidth exceeds a threshold, the target compression ratio is updated based on the currently acquired network bandwidth; if the difference between the currently acquired network bandwidth and the previously acquired network bandwidth does not exceed the threshold, the current target compression ratio remains unchanged.

[0096] When network bandwidth varies significantly, the target compression ratio needs to be dynamically adjusted to adapt to bandwidth limitations.

[0097] The target compression ratio is updated based on the currently acquired network bandwidth. Specifically, a pre-trained adaptive adjustment model is used to determine a compression ratio suitable for the current network conditions based on the input network bandwidth, and this compression ratio is then used to update the target compression ratio.

[0098] It should be noted that when network bandwidth is reduced, resulting in a limited transmission rate, the required transmission rate is reduced by increasing the target compression ratio. When network bandwidth is increased, the target compression ratio is reduced to reduce the pixel data processed by the mask, thereby reducing the amount of content that needs to be reconstructed during the decoding process.

[0099] In this embodiment, in order to cope with different bandwidth conditions, when the network bandwidth changes beyond a certain threshold, the target compression ratio is adjusted according to the current network bandwidth to ensure a better compression effect.

[0100] In one optional embodiment of this application, step 202 above, which generates a bitstream based on masked image blocks and visible image blocks, includes the following specific steps:

[0101] Feature extraction is performed on the masked image block and the visible image block to obtain the first feature vector of the masked image block and the second feature vector of the visible image block;

[0102] Quantize the first and second eigenvectors;

[0103] A bitstream is generated based on the first eigenvector and the second eigenvector after quantization.

[0104] The first feature vector can be understood as the feature representation of the masked image patch, that is, the pixel data of the masked image patch is converted into a latent variable representation.

[0105] The second feature vector can be understood as the feature representation of the visible image patch, that is, converting the pixel data of the visible image patch into a latent variable representation.

[0106] Feature extraction is performed on the masked image patch and the visible image patch. Specifically, this involves using a pre-defined encoding model to extract features from both. The encoding model extracts features by performing a series of convolutional (CONV) layer operations. Furthermore, the encoding model also includes batch normalization (BN) or group normalization (GN) layers, as well as ReLU (Rectified Linear Unit) layers for normalization.

[0107] For example, such as Figure 4 As shown, the encoding model consists of four convolutional layers. In the first three convolutional layers, each convolutional layer is followed by a normalization layer, and finally, the fourth convolutional layer outputs the feature representation. After the mask image patch and the visible image patch are input into the encoding model, they undergo three convolutional operations and normalization processing, and then one more convolutional operation before outputting the feature representation.

[0108] It should be noted that since the pixels on the mask image patch are all set to preset values, boundary effects will occur when performing convolutional layer operations on the mask image patch, resulting in discontinuous or unnatural results. Therefore, before each convolutional layer operation on the mask image patch, padding is added to the boundary of the mask image patch. Specifically, the padding method can be any of zero padding or effective padding, and this application does not impose any specific limitations on it.

[0109] Eigenvector quantization refers to converting continuous or high-dimensional eigenvectors into discrete representations, that is, mapping continuous latent variables into a finite codebook.

[0110] In this embodiment, after obtaining the feature representations of the masked image block and the visible image block, these two feature representations are quantized and converted into discrete representations respectively to reduce the data representation complexity and make it more suitable for subsequent processing.

[0111] In one optional embodiment of this application, a bitstream is generated based on the quantized first feature vector and the quantized second feature vector, including the following specific steps:

[0112] The preset entropy model is invoked to estimate the probability distribution using the quantized first feature vector and the quantized second feature vector. This probability distribution characterizes the statistical properties of the first feature vector and the second feature vector.

[0113] The second eigenvector after quantization is entropy encoded according to the probability distribution to obtain the bit stream.

[0114] An entropy model can be understood as a prior probability model that performs probability statistics on quantified latent variables through prior distributions, and its output is the probability distribution of the latent variables.

[0115] This probability distribution can be understood as the entropy estimation result of the first and second eigenvectors after quantization. It represents the statistical characteristics of the first and second eigenvectors. In the entropy coding process, the probability distribution can be used to determine how to minimize the bitstream length. That is, variable values ​​with high probability can be represented by shorter codes, while variable values ​​with low probability can be represented by longer codes, which is beneficial for achieving lossless compression.

[0116] Entropy coding refers to selecting an appropriate coding scheme based on a probability distribution to effectively compress the quantized second feature vector into a compact bitstream. Examples of entropy coding include arithmetic coding, Shannon coding, and Huffman coding.

[0117] In this embodiment, the possible values ​​of the quantized first feature vector and the quantized second feature vector are estimated by the entropy model to obtain the probability distribution. Then, the probability distribution provided by the entropy model is used to entropy encode the quantized second feature vector. That is, the image blocks that have been masked are discarded and the image blocks that retain the original image information are encoded to minimize the bit stream length and meet the low bit rate requirement.

[0118] It is understandable that since the pixels of the masked image block are filled with the same preset value, the feature representation of the masked image block is also a single discrete symbol representation after quantization, and its data volume is very small. Even if the masked image block is not discarded during the entropy coding process, and the first feature vector and the second feature vector after quantization are entropy encoded together, the bit stream length can be reduced to meet the low bit rate requirement.

[0119] Furthermore, the entropy encoding given above is only a preferred example. Other encoding methods can also be used to generate bitstreams, and this application does not make any specific limitations on this.

[0120] Example 2

[0121] Based on the above Figure 2 Based on the image processing method shown, Figure 5 This is a flowchart illustrating an embodiment of another image processing method according to an exemplary embodiment. This embodiment is used to explain the image decoding process, and the bitstream involved in the decoding process is the one described above. Figure 2 The embodiment shown is obtained as follows: Figure 5 As shown, it includes the following steps 501-504:

[0122] Step 501: Perform image reconstruction on the bitstream to obtain an initial reconstructed image, which is obtained by stitching together the reconstructed visible image blocks and the reconstructed mask image blocks;

[0123] Step 502: Divide the initial reconstructed image into multiple regions, and divide each region into multiple blocks;

[0124] Step 503: Extract a target block from each region each time, and call the preset visual transformation model to use the content of the visible image block belonging to the reconstruction in the extracted target block to restore the region belonging to the reconstructed mask image block, so as to output the corresponding restored block of the target block, until the blocks in each region are extracted.

[0125] Step 504: Stitch together the restored blocks output by each call to the visual transformation model to obtain the target reconstructed image.

[0126] The initial reconstructed image can be understood as the image obtained by directly decoding the bitstream. This initial reconstructed image is not a complete representation of the original image. It contains regions with missing image information, namely the reconstructed mask image blocks. Therefore, the initial reconstructed image is obtained by splicing the reconstructed visible image blocks and the reconstructed mask image blocks.

[0127] The bitstream decoding process specifically includes: performing entropy decoding on the bitstream according to the inverse process of compression to obtain feature vectors; dequantizing the feature vectors; and then converting the dequantized feature vectors into an initial reconstructed image represented by pixel data.

[0128] Entropy decoding is the inverse process of entropy encoding. Entropy encoding encodes data based on the probability distribution provided by the entropy model, generating a compact bitstream. Entropy decoding initializes the bitstream using the same entropy model as encoding, reconstructing the original feature vectors from the bitstream. This entropy model provides the probability distribution used for the feature vectors during encoding.

[0129] It should be noted that after the initial reconstructed image is divided into multiple regions, the number of blocks in each region is the same. This ensures that the blocks in each region are restored simultaneously during multi-stage restoration and reconstruction.

[0130] The Vision Transformer (ViT) model can be understood as capturing the long-range dependencies of an image to understand its global information and local details, thereby helping to recover lost image content. Therefore, each call to the Vision Transformer model can restore the regions in the extracted blocks that belong to the reconstructed mask image blocks.

[0131] Each time a target block is extracted from each region, it can be extracted randomly or in a certain order; this application does not impose any restrictions on this.

[0132] For example, such as Figure 6A As shown, the initial reconstructed image is divided into four regions, each of which is further divided into four blocks, and these four blocks are labeled with 0, 1, 2, and 3 respectively. Figure 6A It can be seen that some blocks contain both the reconstructed mask image block area and the reconstructed visible image block area. The reconstructed mask image block is an area that lacks the original image information and needs to be reconstructed and restored with the help of the surrounding image information.

[0133] In the multi-stage reconstruction process, firstly, blocks marked as 0 are extracted from each of the four regions. A visual transformation model is then invoked to reconstruct the regions belonging to the reconstructed mask image blocks from the visible image blocks within the four blocks marked as 0. These four reconstructed blocks serve as the reconstruction results for the four blocks marked as 0. Then, blocks marked as 1 are extracted from each of the four regions. The same visual transformation model is invoked to reconstruct the regions belonging to the reconstructed mask image blocks from the visible image blocks within the four blocks marked as 1. These four reconstructed blocks serve as the reconstruction results for the four blocks marked as 1. Then, blocks labeled 2 are extracted from these four regions respectively. The visual transformation model is called to restore the regions of the reconstructed mask image blocks in the four blocks labeled 2, and four restored blocks are obtained as the reconstruction restoration results of the four blocks labeled 2. Finally, blocks labeled 3 are extracted from these four regions respectively. The visual transformation model is called to restore the regions of the reconstructed mask image blocks in the four blocks labeled 3, and four restored blocks output by the visual transformation model are obtained as the reconstruction restoration results of the four blocks labeled 3.

[0134] As can be seen, the above-mentioned restoration and reconstruction is implemented in four stages. Each stage considers the spatial context relationship between blocks, uses the content of the visible image blocks belonging to the reconstruction to restore the regions of the mask image blocks belonging to the reconstruction, and each stage extracts a block from each region for restoration and reconstruction. This can avoid the occurrence of large areas of continuous reconstructed mask image blocks, reduce the difficulty of restoration and reconstruction, and effectively achieve restoration and reconstruction even under extremely low bit rate conditions.

[0135] It is understood that the above division of the initial reconstructed image is merely an illustrative example and does not constitute a limitation on the scope of protection of this application.

[0136] This completes the above. Figure 5The image processing flow shown involves decoding the bitstream to obtain an initial reconstructed image, obtained by stitching together reconstructed visible image blocks and reconstructed mask image blocks. To recover missing image information, the initial reconstructed image is divided into multiple regions, and each region is further divided into multiple blocks. Then, one block is extracted from each region at a time, and a visual transformation model is invoked to use the content of the extracted block belonging to the reconstructed visible image block to reconstruct the region belonging to the reconstructed mask image block. This process continues until all blocks in each region have been extracted, achieving multi-stage reconstruction. Each stage considers the spatial context between blocks, utilizing the content of the reconstructed visible image block to reconstruct the region belonging to the reconstructed mask image block within that block. Since each stage extracts one block from each region for reconstruction, large, continuous reconstructed mask image blocks are avoided, reducing the difficulty of reconstruction. This allows for effective reconstruction even at extremely low bit rates. Finally, the recovered blocks output from each call to the visual transformation model are stitched together to obtain a high-quality target reconstructed image.

[0137] In one optional embodiment of this application, step 503 above, which calls a preset visual transformation model to restore the region belonging to the reconstructed mask image block using the content of the visible image block belonging to the reconstructed image block in the extracted target block, includes the following specific steps:

[0138] When the target block is extracted for the first time, the visual transformation model is invoked to perform attention processing on the target block in order to output the recovered block corresponding to the target block;

[0139] If the target block is not extracted for the first time, the visual transformation model is invoked to perform attention processing on the target block based on the recovered block output by the previous call to the visual transformation model, so as to output the recovered block corresponding to the target block.

[0140] It should be noted that for the first call to the visual transformation model, since there is no output from the previous call, the input to the first call to the visual transformation model is only the extracted blocks. For subsequent calls, the extracted blocks can be processed by referring to the recovered blocks from the previous call output, thereby enhancing the model's understanding ability.

[0141] Attention processing captures the dependencies between input data to understand the global information and local details of the data, thereby enhancing the understanding of spatial context. This helps to reasonably predict and fill in areas lacking information, thus greatly assisting in the recovery of lost image content.

[0142] Based on the above example, in the multi-stage reconstruction process, firstly, blocks marked as 0 are extracted from each of the four regions to form the input sequence of the visual transformation model. The visual transformation model is then invoked to perform attention processing on the four blocks marked as 0 to recover the regions belonging to the reconstructed mask image blocks within the blocks. The four recovered blocks output by the visual transformation model are obtained as the reconstruction results of the four blocks marked as 0. Then, blocks marked as 1 are extracted from each of the four regions and combined with the four recovered blocks output previously to form the input sequence of the visual transformation model. The visual transformation model is then invoked to perform attention processing on the four blocks marked as 1 based on the four recovered blocks output previously. The four recovered blocks output by the visual transformation model are obtained as the reconstruction results of the four blocks marked as 1. The reconstruction and restoration results of the blocks are obtained. Then, blocks labeled as 2 are extracted from these four regions and combined with the four restored blocks output from the previous time to form the input sequence of the visual transformation model. The visual transformation model is called to perform attention processing on the four restored blocks output from the previous time based on the four restored blocks output from the previous time, and obtains the four restored blocks output by the visual transformation model as the reconstruction and restoration results of the four blocks labeled as 2. Finally, blocks labeled as 3 are extracted from these four regions and combined with the four restored blocks output from the previous time to form the input sequence of the visual transformation model. The visual transformation model is called to perform attention processing on the four restored blocks output from the previous time based on the four restored blocks output from the previous time, and obtains the four restored blocks output by the visual transformation model as the reconstruction and restoration results of the four blocks labeled as 3.

[0143] As can be seen, the above restoration and reconstruction is implemented in four stages. In each stage, in addition to considering the spatial context relationship between the blocks in this stage, the spatial context relationship of the blocks in the previous stage is also referenced. This can enhance the model's understanding of spatial context relationships and better restore the content of the mask image blocks that belong to the reconstruction in the current stage.

[0144] In this embodiment, the visual transformation model performs attention processing on the extracted blocks based on the restored blocks output by the last call to the visual transformation model. Since it considers not only the spatial context relationship between blocks in the current stage, but also the spatial context relationship of blocks in the previous stage, it can enhance the model's understanding of spatial context relationships and better restore the regions in the current stage blocks that belong to the reconstructed mask image blocks.

[0145] In one optional embodiment of this application, the above-mentioned invocation of the visual transformation model performs attention processing on the target block based on the recovered block output by the previous invocation of the visual transformation model, including the following specific steps:

[0146] The embedding unit in the visual transformation model is called to convert both the target block and the restored block output by the previous visual transformation model into vector sequences. The vector sequence of the target block is the vector representation of the target block, and the vector sequence of the restored block is the vector representation of the restored block.

[0147] The transformation coding unit in the visual transformation model is invoked to extract features from the vector sequence of the target block based on the vector sequence of the recovered block;

[0148] The attention unit in the visual conversion model is invoked to perform attention calculation based on the feature extraction results of the target block, thereby obtaining the recovery block corresponding to the target block.

[0149] like Figure 6B The structure of the visual transformation model shown includes an embedding unit, a transform encoder unit, and a multi-head attention unit. The embedding unit converts the input image patch into a vector sequence suitable for model processing; that is, it maps a single image patch to a one-dimensional vector, corresponding to a token. The vector sequence contains the positional encoding information and feature representation of the image patch. The transform encoder unit extracts global features from the input vector sequence and contains multiple encoder layers. The feature representation of the image patch is iteratively updated through these multiple encoder layers. The attention unit performs attention calculations to obtain the final result for the image patch.

[0150] Suppose there are 4 extracted blocks labeled 1, and the previous output is 4 recovered blocks. In the embedding unit, each block is converted into a one-dimensional vector of dimension 3072, resulting in an 8*3072 vector sequence. In the transformation and encoding unit, the 8*3072 vector sequence is processed, and the feature extraction results corresponding to the 4 blocks labeled 1 are output as a 4*3072 vector sequence. In the attention unit, attention is calculated on the 4*3072 vector sequence, and the recovered blocks corresponding to the 4 blocks labeled 1 are output.

[0151] In this embodiment, the image data is converted into a vector representation suitable for model processing by the embedding unit in the visual transformation model, and the feature extraction is performed by the transformation coding unit in the visual transformation model according to the vector representation provided by the embedding unit to extract the global features of the image. Finally, the attention unit in the visual transformation model performs attention calculation on the feature extraction results to obtain the final reconstruction and restoration result.

[0152] It is worth noting that for the first extracted target block, the processing flow of the visual transformation model is the same as the above-mentioned processing flow. The target block is converted into a vector sequence in the embedding unit, the vector sequence of the target block is extracted in the transformation encoding unit, and the attention is calculated on the feature extraction result in the attention unit.

[0153] In one optional embodiment of this application, the attention unit in the visual transformation model is invoked to perform attention calculation based on the feature extraction results of the target block, specifically including:

[0154] The attention unit calculates the query matrix, key matrix, value matrix, and mask matrix based on the feature extraction results. It then uses the query matrix, key matrix, value matrix, and mask matrix to perform attention calculations. The mask matrix is ​​used to indicate the position of the mask image block belonging to the reconstruction in the target block.

[0155] The query matrix, key matrix, value matrix, and mask matrix are four matrices obtained by linearly transforming the feature extraction results. These four matrices serve as the initial data for attention calculation, and their sizes are consistent with the size of the feature extraction results.

[0156] Compared to existing attention calculations, this application adds a mask matrix to the query matrix, key matrix, and value matrix to calculate attention values, thereby improving the model's inference ability. This mask matrix indicates the location of the reconstructed mask image patch within the target block. For example, the mask matrix can be a matrix composed of 0s and 1s, where elements represented by 0s represent the location of the reconstructed mask image patch, and elements represented by 1s represent the location of the reconstructed visible image patch.

[0157] Furthermore, attention is calculated using the query matrix, key matrix, value matrix, and mask matrix. The specific calculation formula is as follows:

[0158]

[0159] In the above formula, MA represents the mask matrix, Q represents the query matrix, and K... T V represents the transpose of the key matrix, Softmax() represents the normalization function, and d = h × w × c, where h, w, and c represent the height, width, and number of channels of the mask matrix, query matrix, key matrix, and value matrix, respectively. These four matrices are of the same size.

[0160] Based on the above Figure 2 and Figure 5 Based on the illustrated embodiment, the overall image processing flow includes an image encoding flow and an image decoding flow, see [link to example]. Figure 7 and Figure 8 The process includes the following steps:

[0161] Step 1: Edge Detection

[0162] Edge detection is performed on the original image to obtain the edge positions of objects in the image. Using the edge positions as key features in the image helps subsequent steps focus on compressing the salient parts of the image and reduces the amount of data.

[0163] Step 2: Masking

[0164] The original image is divided into multiple image blocks. Candidate image blocks without edge positions are selected from the multiple image blocks to be unimportant regions in the image, so as to ensure that important regions can be compressed completely. Then, according to the current target compression ratio, the target image block is selected from the candidate image blocks and masked to cover up the image information of the target image block, resulting in a masked image block. The image blocks that have not been masked are taken as visible image blocks.

[0165] In the subsequent multi-stage reconstruction and restoration, the visual conversion model uses the image information that has not been covered in the image to predict the covered part. Therefore, the more content is covered, the fewer the effective area that can be used, and the more difficult the restoration work becomes. This application uses a dynamically adjusted target compression ratio to determine the target image block that needs to be masked.

[0166] Step 3: Image Encoding

[0167] The masked image block and the visible image block are used to extract features using an encoding model, resulting in a first feature vector of the masked image block and a second feature vector of the visible image block. The first and second feature vectors are then quantized for subsequent entropy encoding.

[0168] Step 4: Entropy Coding

[0169] The probability distribution is estimated using the quantized first and second eigenvectors through an entropy model. Then, arithmetic coding is used to compress the second eigenvector into a bitstream based on the probability distribution provided by the entropy model.

[0170] Step 5: Entropy Decoding

[0171] Using the same entropy model provided by the probability distribution during encoding, the bitstream is restored to the original feature vector using arithmetic decoding, and the feature vector is dequantized.

[0172] Step 6: Image Decoding

[0173] The dequantized feature vectors are converted into an initial reconstructed image represented by pixel data using a decoding model corresponding to the encoding model. The initial reconstructed image includes reconstructed visible image patches and reconstructed mask image patches.

[0174] Step 7: Image Restoration

[0175] To identify spatial context relationships in an image, a visual transformation model is introduced for image restoration. To enhance the model's understanding of spatial context relationships, a multi-stage restoration and reconstruction process is employed. This involves dividing the initial reconstructed image into multiple regions, further subdividing each region into multiple blocks, and then extracting one block from each region at a time. The visual transformation model is then invoked to restore the regions belonging to the reconstructed mask image blocks within the extracted blocks, continuing until all blocks in each region have been extracted. This multi-stage restoration and reconstruction process is then completed. Finally, the restored blocks output from each call to the visual transformation model are stitched together to obtain the target reconstructed image.

[0176] Corresponding to the embodiments of the aforementioned image processing methods, this application also provides embodiments of an image processing apparatus.

[0177] Figure 9 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment. The apparatus is used to perform the image processing method provided in Embodiment 1 above, such as... Figure 9 As shown, the image processing apparatus includes:

[0178] The masking module 910 is used to perform masking processing on a portion of the image blocks among multiple image blocks of the original image based on the edge position of the object in the original image, to obtain masked image blocks and visible image blocks; the portion of the image blocks does not contain the edge position, and the visible image blocks are image blocks that have not been masked.

[0179] Compression module 920 is used to generate a bitstream based on the mask image block and the visible image block.

[0180] In an optional implementation, the masking module 910 is specifically used to perform edge detection on the original image to obtain the edge position of the object in the original image; filter candidate image blocks that do not contain the edge position from the plurality of image blocks; select a target image block that needs to be masked from the candidate image blocks according to the network bandwidth; and perform masking processing on the target image block.

[0181] In an optional implementation, the masking module 910 is specifically used to adjust the target compression ratio according to changes in the network bandwidth during the process of selecting target image blocks to be masked from the candidate image blocks based on the network bandwidth; the target compression ratio is negatively correlated with the length of the bitstream; the target number of image blocks to be masked is determined based on the adjusted target compression ratio and the number of candidate image blocks; and the target number of target image blocks is selected from the candidate image blocks.

[0182] In an optional implementation, the compression module 920 is specifically used to extract features from the masked image block and the visible image block to obtain a first feature vector of the masked image block and a second feature vector of the visible image block; to quantize the first feature vector and the second feature vector; and to generate a bitstream based on the quantized first feature vector and the quantized second feature vector.

[0183] In an optional implementation, the compression module 920 is specifically used to, during the process of generating a bitstream based on the quantized first feature vector and the quantized second feature vector, call a preset entropy model to estimate a probability distribution using the quantized first feature vector and the quantized second feature vector, wherein the probability distribution characterizes the statistical properties of the first feature vector and the second feature vector; and perform entropy encoding on the quantized second feature vector according to the probability distribution to obtain the bitstream.

[0184] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0185] Figure 10 This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment. This apparatus is used to perform the image processing method provided in Embodiment 2 above, such as... Figure 10 As shown, the image processing apparatus includes:

[0186] The reconstruction module 1010 is used to perform image reconstruction on the received bitstream to obtain an initial reconstructed image; the initial reconstructed image is obtained by stitching together the reconstructed visible image blocks and the reconstructed mask image blocks;

[0187] The segmentation module 1020 is used to divide the initial reconstructed image into multiple regions, and to divide each region into multiple blocks;

[0188] The recovery module 1030 is used to extract a target block from each of the regions at a time, and call a preset visual transformation model to use the content of the visible image block belonging to the reconstruction in the extracted target block to recover the region belonging to the reconstructed mask image block, so as to output the corresponding recovered block of the target block, until all the blocks in each of the regions are extracted.

[0189] The stitching module 1040 is used to stitch together the restored blocks output by each call to the visual transformation model to obtain the target reconstructed image.

[0190] In an optional implementation, the recovery module 1030 is specifically used to, during the process of recovering the region belonging to the reconstructed mask image block by using the content of the visible image block belonging to the reconstructed image block in the extracted target block using a preset visual transformation model, when the extracted target block is being extracted for the first time, call the visual transformation model to perform attention processing on the target block to output the recovery block corresponding to the target block; when the extracted target block is not being extracted for the first time, call the visual transformation model to perform attention processing on the target block according to the recovery block output by the previous call to the visual transformation model to output the recovery block corresponding to the target block.

[0191] In an optional implementation, the recovery module 1030 is specifically configured to, during the process of calling the visual transformation model to perform attention processing on the target block based on the recovery block output by the previous call to the visual transformation model, call the embedding unit in the visual transformation model to convert both the extracted block and the recovery block output by the previous call to the visual transformation model into vector sequences, wherein the vector sequence of the target block is the vector representation of the target block, and the vector sequence of the recovery block is the vector representation of the recovery block; call the transformation encoding unit in the visual transformation model to perform feature extraction on the vector sequence of the target block based on the vector sequence of the recovery block; and call the attention unit in the visual transformation model to perform attention calculation based on the feature extraction result of the target block to obtain the recovery block corresponding to the target block.

[0192] In an optional implementation, the recovery module 1030 is specifically used to perform attention calculations during the process of calling the attention unit in the visual transformation model to perform attention calculations based on the feature extraction results of the target block. The attention unit calculates a query matrix, a key matrix, a value matrix, and a mask matrix based on the feature extraction results, and performs attention calculations using the query matrix, the key matrix, the value matrix, and the mask matrix. The mask matrix is ​​used to indicate the position of the mask image block belonging to the reconstruction in the target block.

[0193] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0194] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0195] This application also provides an electronic device corresponding to the image processing method provided in the foregoing embodiments, for executing the above-described image processing method.

[0196] Figure 11 The present invention illustrates a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes a communication interface 601, a processor 602, a memory 603, and a bus 604. The communication interface 601, processor 602, and memory 603 communicate with each other via the bus 604. The processor 602 can execute the image processing method described above by reading and executing machine-executable instructions corresponding to the control logic of the image processing method stored in the memory 603. The specific content of this method is described in the above embodiment and will not be repeated here.

[0197] The memory 603 mentioned in this application can be any electronic, magnetic, optical, or other physical storage system, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof. Communication between this system network element and at least one other network element is achieved through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0198] Bus 604 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0199] Processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 602 or by instructions in software form. The processor 602 can be a general-purpose processor, including a network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.

[0200] The electronic device provided in this application embodiment and the image processing method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0201] This application also provides a computer-readable storage medium corresponding to the image processing method provided in the foregoing embodiments. Please refer to... Figure 12 As shown, the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image processing method provided in any of the foregoing embodiments.

[0202] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0203] The computer-readable storage medium provided in the above embodiments of this application and the image processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0204] This application also provides a computer program product corresponding to the image processing method provided in the foregoing embodiments. The computer program product includes a computer program that is executed by a processor to implement the image processing method provided in the foregoing embodiments.

[0205] The computer program products provided in the above embodiments of this application and the image processing methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0206] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0207] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0208] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, The method comprises: According to the edge position of the object in the original image, a part of the image blocks in the original image is masked to obtain a masked image block and a visible image block; the part of the image blocks does not contain the edge position, and the visible image block is an image block which is not masked; Based on the masked image block and the visible image block, a bit stream is generated.

2. The method of claim 1, wherein, According to the edge position of the object in the original image, a part of the image blocks in the original image is masked to obtain a masked image block and a visible image block; the part of the image blocks does not contain the edge position, and the visible image block is an image block which is not masked; The original image is edge detected to obtain the edge position of the object in the original image; From the plurality of image blocks, candidate image blocks not containing the edge position are screened; According to the network bandwidth, target image blocks requiring masking are selected from the candidate image blocks; The target image blocks are masked.

3. The method of claim 2, wherein, According to the network bandwidth, target image blocks requiring masking are selected from the candidate image blocks, comprising: According to the change of the network bandwidth, a target compression ratio is adjusted; the size of the target compression ratio is negatively correlated with the length of the bit stream; According to the adjusted target compression ratio and the number of the candidate image blocks, a target number of image blocks requiring masking is determined; The target number of target image blocks is selected from the candidate image blocks.

4. The method of claim 1, wherein, Based on the masked image block and the visible image block, a bit stream is generated, comprising: Feature extraction is performed on the masked image block and the visible image block to obtain a first feature vector of the masked image block and a second feature vector of the visible image block; The first feature vector and the second feature vector are quantized; According to the quantized first feature vector and the quantized second feature vector, a bit stream is generated; According to the quantized first feature vector and the quantized second feature vector, a bit stream is generated, comprising: A preset entropy model is called to estimate a probability distribution using the quantized first feature vector and the quantized second feature vector; the probability distribution represents statistical characteristics of the first feature vector and the second feature vector; According to the probability distribution, the quantized second feature vector is entropy encoded to obtain the bit stream.

5. An image processing method characterized by, The method is used for processing the bit stream obtained by any one of claims 1-4, and the method comprises: An initial reconstructed image is obtained by image reconstruction of the bit stream; the initial reconstructed image is obtained by splicing a reconstructed visible image block and a reconstructed masked image block; The initial reconstructed image is divided into a plurality of regions, and each region is divided into a plurality of subblocks; Each time, a target subblock is extracted from each region, and a preset visual conversion model is called to use the content of the reconstructed visible image block in the extracted target subblock to restore the region of the reconstructed masked image block, so as to output a restored subblock corresponding to the target subblock, until the subblocks in each region are extracted; The restored subblocks output by each time calling the visual conversion model are spliced to obtain a target reconstructed image.

6. The method of claim 5, wherein, The calling preset visual conversion model uses the content of the extracted target sub-block belonging to the reconstructed visible image block to recover the area belonging to the reconstructed mask image block, including: In the case of first extraction of the extracted target sub-block, the visual conversion model is called to perform attention processing on the target sub-block to output a recovery sub-block corresponding to the target sub-block; In the case of non-first extraction of the extracted target sub-block, the visual conversion model is called to perform attention processing on the target sub-block according to the recovery sub-block output by the last calling of the visual conversion model to output a recovery sub-block corresponding to the target sub-block.

7. The method of claim 6, wherein, The calling of the visual conversion model according to the recovery sub-block output by the last calling of the visual conversion model to perform attention processing on the target sub-block includes: The embedding unit in the visual conversion model is called to convert the target sub-block and the recovery sub-block output by the last calling of the visual conversion model into vector sequences, the vector sequence of the target sub-block being a vector representation of the target sub-block, and the vector sequence of the recovery sub-block being a vector representation of the recovery sub-block; The conversion encoding unit in the visual conversion model is called to perform feature extraction on the vector sequence of the target sub-block according to the vector sequence of the recovery sub-block; The attention unit in the visual conversion model is called to perform attention calculation according to the feature extraction result of the target sub-block to obtain a recovery sub-block corresponding to the target sub-block.

8. The method of claim 7, wherein, The calling of the attention unit in the visual conversion model according to the feature extraction result of the target sub-block to perform attention calculation includes: The attention unit calculates a query matrix, a key matrix, a value matrix, and a mask matrix according to the feature extraction result, and performs attention calculation using the query matrix, the key matrix, the value matrix, and the mask matrix, the mask matrix being used to indicate the position of the target sub-block belonging to the reconstructed mask image block.

9. An image processing system, characterized by The system includes: The sending end is configured to perform mask processing on part of the image blocks in the original image based on the edge position of the object in the original image to obtain mask image blocks and visible image blocks; the part of the image blocks does not include the edge position, and the visible image blocks are image blocks that are not subjected to mask processing; a bitstream is generated based on the mask image blocks and the visible image blocks, and the bitstream is sent to the receiving end; The receiving end is configured to perform image reconstruction on the bitstream to obtain an initial reconstructed image; the initial reconstructed image is obtained by splicing reconstructed visible image blocks and reconstructed mask image blocks; each of the regions is divided into a plurality of sub-blocks; each time, one target sub-block is extracted from each of the regions, and a preset visual conversion model is called to use the content of the extracted target sub-block belonging to the reconstructed visible image block to recover the area belonging to the reconstructed mask image block to output a corresponding recovery sub-block of the target sub-block until the sub-blocks in each of the regions are extracted; and the recovery sub-blocks output by each calling of the visual conversion model are spliced to obtain a target reconstructed image.

10. An image processing apparatus characterized by comprising: The device includes: The device comprises: a compression module configured to generate a bitstream based on the masked image block and the visible image block.

11. An image processing apparatus characterized by comprising: The device comprises: a reconstruction module configured to perform image reconstruction on the received bitstream to obtain an initial reconstructed image, wherein the initial reconstructed image is obtained by splicing a reconstructed visible image block and a reconstructed masked image block; a segmentation module configured to divide the initial reconstructed image into a plurality of regions, and divide each of the regions into a plurality of sub-blocks; a recovery module configured to extract one target sub-block from each of the regions at a time, and call a preset visual conversion model to use the content of the extracted target sub-block belonging to the reconstructed visible image block to recover the region belonging to the reconstructed masked image block, to output a corresponding recovered sub-block of the target sub-block, until the sub-blocks in each of the regions are extracted; a splicing module configured to splice the recovered sub-blocks output by each call of the visual conversion model to obtain a target reconstructed image.

12. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1-8.

13. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-8.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-8.

Images