Image processing method and device, image coding and decoding system, equipment and storage medium

Abstract

The invention discloses an image processing method and device, an image encoding and decoding system, equipment and a storage medium. The image processing method comprises the following steps: decoding a text code stream from a coding end through a first decoder to obtain semantic information of an original image; decoding a feature code stream from a coding end through a second decoder to obtain feature decoding information of the original image; processing the feature decoding information through a texture information acquisition module to obtain texture information of the original image; processing the semantic information and the texture information based on a hidden layer representation diffusion model to obtain fusion information; and processing the fusion information through an image generation module to obtain a decoded image. According to the image processing method provided by the embodiment of the invention, the semantic information and the texture information are processed based on the hidden layer representation diffusion model to obtain the fusion information, and the fusion information is processed through the image generation module to obtain the decoded image, so that a relatively clear image can be obtained, the condition of image blurring is greatly reduced, and the processing effect is relatively good.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically to an image processing method, apparatus, image encoding and decoding system, equipment and storage medium. Background Art

[0002] With the explosive growth of image content in the era of big data, there is an urgent need to design an efficient coding scheme. With the adoption of more and more efficient adaptive coding tools, the coding efficiency of current traditional coding standards continues to improve. However, with the emergence of boundary effects, it is increasingly difficult to gain coding performance gains by simply relying on the increase in coding modes and the expansion of parameter search ranges. Traditional coding methods based on signal processing encounter performance improvement bottlenecks. The image processing methods in related technologies decode and reconstruct the encoded images, but it is difficult to obtain clear images, and image blur is prone to occur, resulting in poor processing effects.

[0003] The above statements are only used to provide background information related to the present application and do not necessarily constitute prior art. Summary of the invention

[0004] The purpose of the present application is to provide an image processing method, apparatus, image encoding and decoding system, device and storage medium. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, which is applied to a decoding end, wherein the decoding end includes a first decoder, a second decoder, and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model, and an image generation module connected in sequence; the method includes: Decoding the text code stream from the encoding end by a first decoder to obtain semantic information of the original image; Decoding the feature code stream from the encoding end by a second decoder to obtain feature decoding information of the original image; Processing the feature decoding information through the texture information acquisition module to obtain texture information of the original image; Processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; The fusion information is processed by the image generation module to obtain a decoded image.

[0006] In some embodiments of the present application, the texture information acquisition module includes a latent space representation module, an image reconstruction module and a feature extractor connected in sequence; the processing of the feature decoding information by the texture information acquisition module to obtain the texture information of the original image includes: Inputting the feature decoding information into the latent space representation module for processing to obtain latent space representation information; Processing the latent space representation information through the image reconstruction module to obtain image reconstruction information; The image reconstruction information is processed by the feature extractor to obtain texture information of the original image.

[0007] In some embodiments of the present application, the processing of the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information includes: The semantic information and the texture information are fused through the hidden layer representation diffusion model based on the visual knowledge of the hidden layer representation diffusion model to obtain fused information.

[0008] In some embodiments of the present application, the image generation module includes an image generator and an enhancement network module connected in sequence; the step of processing the fusion information through the image generation module to obtain a decoded image includes: Processing the fusion information by the image generator to obtain an initial generated image; The enhancement network module removes the block effect of the initially generated image to obtain a decoded image.

[0009] In some embodiments of the present application, the encoding end includes a graph-generated text model, a latent space representation extraction model, a first encoder connected to the graph-generated text model, and a second encoder connected to the latent space representation extraction model; The text code stream is obtained by encoding first semantic information by the first encoder; the first semantic information is semantic information of the original image extracted by the image-to-text model.

[0010] According to another aspect of an embodiment of the present application, there is provided an image processing method, which is applied to an encoding end, wherein the encoding end includes a graph-to-text model, a latent space representation extraction model, a first encoder connected to the graph-to-text model, and a second encoder connected to the latent space representation extraction model, the method comprising: Processing the original image through the image-to-text model to obtain semantic information; Processing the original image through the latent space representation extraction model to obtain texture information; Encoding the semantic information using the first encoder to obtain a text code stream; Encoding the texture information using the second encoder to obtain a feature code stream; The text code stream and the feature code stream are both sent to a decoding end, so that the decoding end executes the image processing method applied to the decoding end described in any embodiment of the present application.

[0011] According to another aspect of an embodiment of the present application, an image processing device is provided, characterized in that it is applied to a decoding end, the decoding end includes a first decoder, a second decoder and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model and an image generation module connected in sequence; the device includes: A first decoding module, used for decoding the text code stream from the encoding end through a first decoder to obtain semantic information of the original image; A second decoding module, configured to decode the feature code stream from the encoding end through a second decoder to obtain feature decoding information of the original image; A texture information acquisition module, used to process the feature decoding information through the texture information acquisition module to obtain texture information of the original image; A fusion information acquisition module, used for processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; The image generation module is used to process the fusion information through the image generation module to obtain a decoded image.

[0012] According to another aspect of an embodiment of the present application, there is provided an image processing device, which is applied to an encoding end, the encoding end comprising an image-generated text model, a latent space representation extraction model, a first encoder connected to the image-generated text model, and a second encoder connected to the latent space representation extraction model, the device comprising: A semantic information acquisition module, used to process the original image through the image-to-text model to obtain semantic information; A texture information acquisition module, used to process the original image through the latent space representation extraction model to obtain texture information; A text code stream acquisition module, used for encoding the semantic information by using the first encoder to obtain a text code stream; A feature code stream acquisition module, used to encode the texture information using the second encoder to obtain a feature code stream; The sending module is used to send both the text code stream and the feature code stream to a decoding end, so that the decoding end executes the image processing method applied to the decoding end according to any embodiment of the present application.

[0013] According to another aspect of an embodiment of the present application, an image encoding and decoding system is provided, including an encoder and a decoder, wherein the decoder is used to execute the image processing method applied to a decoding end of any embodiment of the present application, and the encoder is used to execute the image processing method applied to an encoding end of any embodiment of the present application.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any embodiment of the present application.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to implement the method described in any embodiment of the present application.

[0016] The technical solution provided by one aspect of the embodiments of the present application may have the following beneficial effects: The image processing method provided by the embodiment of the present application decodes the text code stream from the encoding end through a first decoder to obtain the semantic information of the original image, and decodes the feature code stream from the encoding end through a second decoder to obtain the feature decoding information of the original image, processes the feature decoding information through a texture information acquisition module to obtain the texture information of the original image, processes the semantic information and texture information based on a hidden layer representation diffusion model to obtain fusion information, and processes the fusion information through an image generation module to obtain a decoded image, which can obtain a clearer image, greatly reduces the image blur, and has a good processing effect.

[0017] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart of an image processing method according to an embodiment of the present application is shown.

[0020] Figure 2A flow chart of processing feature decoding information by a texture information acquisition module in one embodiment of the present application is shown.

[0021] Figure 3 A flow chart of an image processing method according to another embodiment of the present application is shown.

[0022] Figure 4 A schematic diagram of the generative encoding and decoding framework structure in one embodiment of the present application is shown.

[0023] Figure 5 A schematic diagram of the encoder framework structure in one embodiment of the present application is shown.

[0024] Figure 6 A schematic diagram of the decoder framework structure in one embodiment of the present application is shown.

[0025] Figure 7 A visualization diagram showing comparison results of an image processing method according to an embodiment of the present application and processing results of other methods is shown.

[0026] Figure 8 A structural block diagram of an image processing device according to an embodiment of the present application is shown.

[0027] Fig. 9 A structural block diagram of an image processing device according to another embodiment of the present application is shown.

[0028] Fig.10 A structural block diagram of an electronic device according to an embodiment of the present application is shown.

[0029] Fig.11 A schematic diagram of a computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0031] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0032] With the development of deep learning, end-to-end coding schemes based on neural networks are becoming increasingly mature. Significant performance improvements have been achieved through the optimization of nonlinear transformations and entropy models. However, as the compression ratio increases, the loss of data signals will lead to a significant decrease in the quality of image and video. Especially under the condition of compression ratios of thousands, the reconstructed image and video will be drastically distorted in visual characteristics such as edges, colors, and textures, and may cause the loss of key visual information such as concepts and semantics. Since the compression domain representation based on signal transformation fails to explicitly model and retain more critical visual information, the code stream still needs to transmit more feature information to ensure the reconstruction quality at the decoding end, which limits the compression efficiency of the encoding. In order to further improve the compression efficiency, it is necessary to go beyond the traditional optimization goals and develop new image and video coding methods with higher compression efficiency to meet the growing data processing needs and diversified application scenarios. The mainstream compression method in the related art reconstructs the image through the statistical characteristics of the data. Under high compression ratios, it is difficult to reconstruct images that meet the subjective effects of the human eye, and artifacts such as blur and block effects are prone to occur.

[0033] In view of the problems existing in the related art, the embodiment of the present application provides an image processing method, which decodes the text code stream from the encoding end through a first decoder to obtain the semantic information of the original image, decodes the feature code stream from the encoding end through a second decoder to obtain the feature decoding information of the original image, processes the feature decoding information through a texture information acquisition module to obtain the texture information of the original image, processes the semantic information and texture information based on a hidden layer representation diffusion model to obtain fusion information, and processes the fusion information through an image generation module to obtain a decoded image. In this way, a clearer image can be obtained, the image blur is greatly reduced, and the processing effect is better.

[0034] The following describes an image processing method, apparatus, image encoding and decoding system, device and storage medium proposed according to an embodiment of the present application in conjunction with the accompanying drawings.

[0035] refer to Figure 1 As shown, an embodiment of the present application provides an image processing method, which is applied to a decoding end, wherein the decoding end includes a first decoder, a second decoder, and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model, and an image generation module connected in sequence; the image processing method may include steps S10-S50: S10. Decoding the text code stream from the encoding end through a first decoder to obtain semantic information of the original image.

[0036] Exemplarily, the encoder converts the input image or image-related information (such as the visual features of the image, semantic labels, etc.) into a compact text code stream or hidden layer representation. This text code stream may be data after some form of encoding (such as natural language description, symbol encoding or vector representation), representing the high-level semantic information of the image. The decoder is a component that converts this compact text code stream back to structural or semantic information in a human-understandable form. The decoder may contain multiple decoding layers and rely on some form of neural network (such as Transformer, LSTM, etc.) to gradually convert high-dimensional information back to the semantic representation of the image. The first decoder recovers the semantic information of the image by gradually decoding the code stream. This semantic information may include the category, position, relationship, scene description, etc. of objects in the image. For example, whether the image contains a cat, whether the cat is on the left, whether the cat is sitting or standing, etc.

[0037] The decoder can use the Attention Mechanism, which helps to pay more attention to the important parts of the bitstream during decoding. During the decoding process, the decoder can learn the complex semantic structure of the image based on the context information. For example, semantic information may not only be limited to the recognition of a single object, but also involve the relationship between objects or the understanding of the background scene.

[0038] Each decoding step in the decoding process may generate an intermediate representation that represents a part of the semantic information of the image. For example, through the first decoder, the rough structural information of the image (such as the location and category of the main objects) may be decoded first, and then further refined to decode more specific details (such as the spatial relationship between objects or more contextual information).

[0039] Finally, the semantic information of the original image decoded by the first decoder is usually expressed in a structured form, which may be a vector, matrix or text description containing a label, scene description, object location or attribute of the image content. This information provides the basis for subsequent tasks such as image reconstruction, image generation or classification.

[0040] The first decoder decodes the text code stream from the encoder to obtain the semantic information of the original image, which specifically includes receiving and decoding the encoded text stream, extracting high-level semantic information of the image, and using context association and attention mechanisms to generate accurate semantic representations, laying the foundation for subsequent processing or tasks.

[0041] S20. Decode the feature code stream from the encoding end by a second decoder to obtain feature decoding information of the original image.

[0042] Exemplarily, the feature code stream from the encoder contains high-level semantic feature information of the image, such as edges, shapes, textures, etc. If the encoder performs quantization or encoding processing, the decoder will first perform inverse quantization or decoding operations. Inverse quantization is the process of restoring the quantized discrete values ​​to close to the original features, usually restored to higher precision floating values. If the feature code stream undergoes other transformations (such as wavelet transform, Fourier transform, etc.), the decoder will perform the corresponding inverse transform operations to restore a spatial representation that is closer to the original features. The decoder reorganizes the decoded features into the spatial information of the original image or its corresponding feature maps, which represent the local structure, texture, edges, etc. of the image in the spatial domain.

[0043] The decoder gradually reconstructs the detailed features of the original image (such as color, brightness, shape, etc.), and gradually fills in the missing information using previously extracted features and restored high-level semantic information to restore the integrity of the image. The decoder also predicts the specific pixel values ​​of the image based on the restored feature information, and gradually refines the image restoration process. The predicted pixel values ​​are processed and optimized multiple times between decoding layers, and eventually approach the original image. After the above steps, the decoder will eventually output the feature decoding information of the image, which usually includes the key detailed features, structure, and texture of the image.

[0044] By using the first decoder and the second decoder to decode the text code stream and the feature code stream respectively, the semantic information and feature information of the original image can be decoded independently. This helps to retain more image details and semantic structures, making the decoded image more expressive and effectively avoiding information loss.

[0045] S30. Process the feature decoding information through the texture information acquisition module to obtain texture information of the original image.

[0046] The texture information acquisition module processes the feature decoding information to obtain the texture information of the image, which helps to improve the detail expression of the image and makes the decoded image more realistic and detailed visually. Especially in low-resolution image decoding, the enhancement of texture information can supplement and improve the quality of the image.

[0047] In some embodiments, the texture information acquisition module includes a latent space representation module, an image reconstruction module and a feature extractor connected in sequence; Figure 2 As shown, the texture information acquisition module processes the feature decoding information to obtain the texture information of the original image, which may include steps S301-S303: S301: Input the feature decoding information into the latent space representation module for processing to obtain latent space representation information.

[0048] By processing feature decoding information through the latent space representation module, the potential features of the image can be efficiently represented in a low-dimensional latent space. This helps to reduce data redundancy while retaining the main information of the image, thereby improving computational efficiency.

[0049] S302: Process the latent space representation information through the image reconstruction module to obtain image reconstruction information.

[0050] By converting the latent space representation information into image reconstruction information through the image reconstruction module, the structure and details of the image can be effectively restored, thereby enhancing the image information through the learned image features and improving the clarity and accuracy of the image during the reconstruction process.

[0051] S303: Process the image reconstruction information through the feature extractor to obtain texture information of the original image.

[0052] Texture information is detailed information in an image and has strong visual recognition. Through step S303, texture features in an image can be effectively extracted and restored, thereby providing a more accurate basis for subsequent image processing, recognition or analysis.

[0053] Through the three-step process of S301-S303, the accuracy of texture information extraction can be effectively improved and the detail loss rate in the image reconstruction process can be reduced. The feature extraction module can pay more attention to details, which helps to improve the texture restoration quality of the image, especially for complex texture image processing. The latent space representation module is used to map image information to a lower dimensional space, which effectively reduces the amount of calculation. The step-by-step processing of the image reconstruction and feature extraction modules also helps to optimize computing resources, improve operating efficiency, and reduce unnecessary computing overhead.

[0054] S40: Process the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information.

[0055] The fusion of semantic information and texture information based on the latent representation diffusion model can efficiently combine information in the latent space of the model, thereby capturing higher-level image features and improving the image generation effect. For example, the latent representation diffusion model can effectively process complex image features and details and generate smoother and more natural images.

[0056] In some embodiments, processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fused information may include: fusing the semantic information and the texture information based on the visual knowledge of the hidden layer representation diffusion model through the hidden layer representation diffusion model to obtain fused information. In this way, different types of information (such as visual knowledge, semantic information, and texture information) are integrated into the same representation space through the hidden layer representation diffusion model. This multimodal fusion method can provide a richer and more comprehensive input for the model, enhance the model's ability to understand the image and its semantic hierarchy, and thus improve the accuracy and reliability of the fused information.

[0057] S50: Process the fusion information through the image generation module to obtain a decoded image.

[0058] Through the image generation module, the fused information is further processed to generate the decoded image. By utilizing the fused semantic information and texture information, the decoded image can be optimized in terms of semantic consistency and visual details.

[0059] In some embodiments, the image generation module includes an image generator and an enhancement network module connected in sequence; and processing the fusion information by the image generation module to obtain a decoded image may include: S501, processing the fusion information by the image generator to obtain an initial generated image; S502: Remove the block effect of the initial generated image through the enhancement network module to obtain a decoded image.

[0060] In the image generation process, block effects (also called mosaic effects) usually appear in the image decoding process, especially in the restoration of low-resolution images or high-compression ratio images. By using the enhancement network module to remove the block effects of the initial generated image, this artifact can be effectively eliminated, making the final image smoother and more natural. After removing the block effects, the visual quality of the image will be significantly improved, especially in the details, and more texture and structural information can be retained, making the decoded image clearer and more realistic.

[0061] In some embodiments, the encoding end includes a large image-generated text model, a latent space representation extraction model, a first encoder connected to the large image-generated text model, and a second encoder connected to the latent space representation extraction model; the text code stream is obtained by encoding first semantic information by the first encoder; the first semantic information is the semantic information of the original image extracted by the large image-generated text model.

[0062] The image processing method of this embodiment can effectively reduce information loss during the decoding process through a multi-stage processing method, and can effectively improve the decoded image quality, detail expression and visual effect, so that the final decoded image is closer to the original image.

[0063] The image processing method provided in the embodiment of the present application can obtain a relatively clear image. Under the condition of a high compression ratio, it can reconstruct an image that meets the subjective effect of the human eye, greatly reducing the probability of artifacts such as image blur and block effect, and the processing effect is good.

[0064] Another embodiment of the present application provides an image processing method, applied to an encoding end, the encoding end comprising an image-generated text model, a latent space representation extraction model, a first encoder connected to the image-generated text model, and a second encoder connected to the latent space representation extraction model, referring to Figure 3 As shown, the method may include steps 1 to 5: Step 1: Process the original image through the image-to-text model to obtain semantic information.

[0065] The graph-to-text model is a model that converts images into text or semantic descriptions. It is used to extract semantic information (such as objects, scenes, actions, etc.) from images through image analysis. The original image is first input into the graph-to-text model, which identifies various objects, relationships, and contexts in the original image and converts this information into semantic information that can be further processed. This semantic information is a high-level understanding of the original image, including information such as the relationship between the elements in the image, object categories, and scene types.

[0066] Step 2: Process the original image through the latent space representation extraction model to obtain texture information.

[0067] The latent space representation extraction model can extract low-level texture patterns, colors, shapes, details and other feature information from the original image through a deep learning model (such as a convolutional neural network, etc.). These feature information does not directly express the object content of the image (unlike the semantic information in step 1), but expresses the texture characteristics of the original image, such as the texture pattern, detail structure, color distribution, etc.

[0068] Step 3: Encode the semantic information using the first encoder to obtain a text code stream.

[0069] The text code stream can be a format that converts semantic information into a format that is easy to transmit and process through a coding process. This is usually a compressed symbol sequence or feature vector that contains a simplified representation of all semantic information in the image.

[0070] Step 4: Encode the texture information using the second encoder to obtain a feature code stream.

[0071] The second encoder is used to encode the texture information extracted in step 2, which can convert visual details and texture features into a feature code stream. This process compresses the detail information and converts it into another format suitable for storage and processing.

[0072] Step 5: Send both the text code stream and the feature code stream to the decoding end, so that the decoding end executes the image processing method applied to the decoding end in any of the above implementation modes.

[0073] The text code stream and feature code stream are sent to the decoding end, so that the decoding end can decode the two code streams respectively and restore the complete information of the image. The decoding process will use the strategies of semantic decoding and texture decoding respectively, combine the semantic and texture information, and obtain the decoded image.

[0074] Exemplarily, for an input image, it can be first divided into blocks to obtain multiple block images (for example, an input image can be divided into 4 block images), each block image is input into the encoding end for processing, and finally the decoded images corresponding to each separated image output at the decoding end are spliced ​​together to obtain a complete image corresponding to the input image.

[0075] Another embodiment of the present application provides an image processing method, which generates semantic information of an image using a large graph-text model, models texture information of the image through a latent space representation of the image, and fuses the semantic information and texture information of the image based on the visual knowledge of the generated diffusion model, thereby achieving efficient compression of the image.

[0076] The generative encoding and decoding framework of this embodiment is as follows Figure 4 As shown. The generative encoding and decoding framework includes a large graph-generated text model, a latent space representation extraction model, two encoders, two decoders and a generative diffusion model. The input image is input into the large graph-generated text model and the latent space representation extraction model respectively. The large graph-generated text model and the latent space representation extraction model are connected to an encoder respectively. The encoder connected to the large graph-generated text model outputs a text code stream, which is input into a decoder for decoding to obtain semantic information. The encoder connected to the latent space representation extraction model outputs a feature code stream, which is input into another decoder for decoding to obtain texture information. Both semantic information and texture information are input into the generative diffusion model for processing to obtain a reconstructed image.

[0077] The encoding end framework of this embodiment is as follows Figure 5As shown in the figure, the encoding end framework includes a graph-to-text model, a latent space representation generation module and two AEs (Auto-Encoders, AE for short). The graph-to-text model is connected to one AE, and the latent space representation generation module is connected to another AE.

[0078] The decoding framework is as follows Figure 6 As shown in FIG. 1 , the decoding end framework includes two ADs (Auto Decoders), an image reconstruction module, and a hidden layer representation diffusion model, an image generator, and an enhancement module connected in sequence. The input end of the hidden layer representation diffusion model is connected to a feature extractor, and the image reconstruction module is connected to the hidden layer representation diffusion model through another feature extractor. The semantic information of the decoded text is obtained by decoding the text code stream through an AD, and the semantic information is input into the hidden layer representation diffusion model; the latent space representation is obtained by decoding the feature code stream through another AD, and then the latent space representation is input into the image reconstruction module for processing. The information output by the image reconstruction module is input into the feature extractor connected to the image reconstruction module for processing, and the output texture information is input into the hidden layer representation diffusion model; another feature extractor inputs the visual information into the hidden layer representation diffusion model; the hidden layer representation diffusion model processes the received semantic information, visual information, and texture information, and inputs the output processing result into the image generator for processing. The image generator inputs the output processing result into the enhancement module for processing, and the enhancement module outputs the decoded image.

[0079] Exemplarily, the text and latent space representation are first decoded from the bitstream, and then the latent space representation is used to reconstruct the low-quality image using the feature decoder. The enhanced image and text are then input into LAControlNet with pre-trained SD2.1 to generate the denoised latent representation and original pixels in turn. During the training process, LAControlNet is trained and other parameters are fixed. Finally, the two representations are input into the diffusion model, which iteratively denoises the noisy image content. The image generator uses DiffBIR, a blind image restoration model built based on Stable Diffusion2.1.

[0080] In order to obtain compact and accurate text, the method of the embodiment of the present application uses the 1.5 versions of mPLUG-Owl and StableDiffusion as the image-to-text and text-to-image models respectively. The decoding end splices the divided images to generate a complete image, and uses an enhancement network to enhance the image quality and remove the block effect.

[0081] Four image datasets are selected for training and testing, including OpenImages, DiffusionDB, Kodak and AVS datasets. OpenImages is the training dataset, DiffusionDB, Kodak and AVS datasets are the test datasets. DiffusionDB includes 100 AIGC images. AVS is obtained by taking the first frame of each pass sequence. The bit depth of the datasets is 8 bits. The detailed introduction is shown in the following table.

[0082]

[0083] Experimental comparisons were conducted on the above datasets. The methods compared with the method of this embodiment include JPEG, HEVC (HM-16.0) and VVC (VTM-11.0). The visualization of the comparison results is shown in the figure below. Figure 7 As shown, Figure 4 NIGM in represents the curve corresponding to the method of this embodiment. Figure 7 The results shown in the figure show that the method proposed in this embodiment has better image processing effect than the other three methods. The method proposed in this embodiment is far superior to other encoding methods in terms of subjective indicators of human eyes. This embodiment adopts generative coding for image data, and realizes efficient encoding of images by using compact representation of text and images generated by the large model of graph-to-text and rich visual information generated by the diffusion model.

[0084] This embodiment uses text representation to provide semantic information and latent space feature representation to provide texture structure information, and adopts a diffusion model based on text-latent space representation to achieve efficient image compression. This embodiment uses the compact representation of text and images generated by the large model of graph-text and the visual information capability of the generation extension model to achieve efficient encoding of images, which exceeds the effect of related technologies in terms of subjective effect of the human eye, which provides strong support for the applicability of generative coding.

[0085] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0086] refer to Figure 8 As shown, another embodiment of the present application provides an image processing device, which is applied to a decoding end, wherein the decoding end includes a first decoder, a second decoder, and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model, and an image generation module connected in sequence; the device includes: A first decoding module, used for decoding the text code stream from the encoding end through a first decoder to obtain semantic information of the original image; A second decoding module, configured to decode the feature code stream from the encoding end through a second decoder to obtain feature decoding information of the original image; A texture information acquisition module, used to process the feature decoding information through the texture information acquisition module to obtain texture information of the original image; A fusion information acquisition module, used for processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; The image generation module is used to process the fusion information through the image generation module to obtain a decoded image.

[0087] Exemplarily, the texture information acquisition module includes a latent space representation module, an image reconstruction module and a feature extractor connected in sequence; the texture information acquisition module includes: A latent space representation information acquisition unit, used for inputting the feature decoding information into the latent space representation module for processing to obtain latent space representation information; An image reconstruction information acquisition unit, configured to process the latent space representation information through the image reconstruction module to obtain image reconstruction information; The texture information acquisition unit is used to process the image reconstruction information through the feature extractor to obtain the texture information of the original image.

[0088] Exemplarily, the fusion information acquisition module is further specifically used to fuse the semantic information and the texture information through the hidden layer representation diffusion model based on the visual knowledge of the hidden layer representation diffusion model to obtain the fusion information.

[0089] Exemplarily, the image generation module includes an image generator and an enhancement network module connected in sequence; the step of processing the fusion information through the image generation module to obtain a decoded image includes: Processing the fusion information by the image generator to obtain an initial generated image; The enhancement network module removes the block effect of the initially generated image to obtain a decoded image.

[0090] Exemplarily, the encoding end includes a large image-generated text model, a latent space representation extraction model, a first encoder connected to the large image-generated text model, and a second encoder connected to the latent space representation extraction model; the text code stream is obtained by encoding first semantic information by the first encoder; the first semantic information is the semantic information of the original image extracted by the large image-generated text model.

[0091] The image processing device provided by the embodiment of the present application decodes the text code stream from the encoding end through a first decoder to obtain the semantic information of the original image, and decodes the feature code stream from the encoding end through a second decoder to obtain the feature decoding information of the original image, processes the feature decoding information through a texture information acquisition module to obtain the texture information of the original image, processes the semantic information and texture information based on a hidden layer representation diffusion model to obtain fusion information, and processes the fusion information through an image generation module to obtain a decoded image, which can obtain a clearer image, greatly reduces the image blur, and has a good processing effect.

[0092] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0093] refer to Fig. 9 As shown, another embodiment of the present application provides an image processing device, which is applied to an encoding end, wherein the encoding end includes a graph-to-text model, a latent space representation extraction model, a first encoder connected to the graph-to-text model, and a second encoder connected to the latent space representation extraction model, wherein the device includes: A semantic information acquisition module, used to process the original image through the image-to-text model to obtain semantic information; A texture information acquisition module, used to process the original image through the latent space representation extraction model to obtain texture information; A text code stream acquisition module, used for encoding the semantic information by using the first encoder to obtain a text code stream; A feature code stream acquisition module, used to encode the texture information using the second encoder to obtain a feature code stream; The sending module is used to send both the text code stream and the feature code stream to a decoding end, so that the decoding end executes the image processing method applied to the decoding end in any embodiment of the present application.

[0094] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0095] Another embodiment of the present application provides an image encoding and decoding system, including an encoder and a decoder, wherein the encoder is used to execute the method applied to the encoder in any embodiment of the present application, and the decoder is used to execute the image processing method applied to the decoding end in any embodiment of the present application.

[0096] The decoding end includes a first decoder, a second decoder and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model and an image generation module connected in sequence; the image processing method applied to the decoding end includes: decoding the text code stream from the encoding end by the first decoder to obtain the semantic information of the original image; decoding the feature code stream from the encoding end by the second decoder to obtain the feature decoding information of the original image; processing the feature decoding information by the texture information acquisition module to obtain the texture information of the original image; processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; processing the fusion information by the image generation module to obtain the decoded image.

[0097] The encoding end includes a graph-to-text model, a latent space representation extraction model, a first encoder connected to the graph-to-text model, and a second encoder connected to the latent space representation extraction model. The image processing method applied to the encoding end includes: processing the original image through the graph-to-text model to obtain semantic information; processing the original image through the latent space representation extraction model to obtain texture information; encoding the semantic information using the first encoder to obtain a text code stream; encoding the texture information using the second encoder to obtain a feature code stream; and sending both the text code stream and the feature code stream to a decoding end, so that the decoding end executes the image processing method applied to the decoding end of any embodiment of the present application.

[0098] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0099] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method of any of the above-mentioned embodiments.

[0100] refer to Fig.10 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected via the bus 102; the memory 101 stores a computer program that can be run on the processor 100, and when the processor 100 runs the computer program, it executes the method provided in any of the aforementioned embodiments of the present application.

[0101] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0102] The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store a program, and the processor 100 executes the program after receiving an execution instruction. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 100, or implemented by the processor 100.

[0103] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 100 or an instruction in the form of software. The above processor 100 may be a general-purpose processor, which may include a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the above method in combination with its hardware.

[0104] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0105] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the image processing method of any of the above embodiments. Fig.11 As shown, the computer-readable storage medium shown is a CD 20 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided by any of the aforementioned embodiments will be executed.

[0106] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0107] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0108] It should be noted that: The term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There may or may not be clear boundaries between different modules.

[0109] The algorithm and display provided herein are not inherently related to any specific computer, virtual device or other equipment. Various general devices can also be used together with examples based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description made to specific languages ​​above is for the purpose of disclosing the best mode of implementation of the application.

[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0111] The above embodiments only express the implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: Applied to a decoding end, the decoding end includes a first decoder, a second decoder and a generation diffusion model; the generation diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model and an image generation module connected in sequence; the method includes: Decoding the text code stream from the encoding end by a first decoder to obtain semantic information of the original image; Decoding the feature code stream from the encoding end by a second decoder to obtain feature decoding information of the original image; Processing the feature decoding information through the texture information acquisition module to obtain texture information of the original image; Processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; The fusion information is processed by the image generation module to obtain a decoded image.

2. The method according to claim 1, characterized in that The texture information acquisition module includes a latent space representation module, an image reconstruction module and a feature extractor connected in sequence; the feature decoding information is processed by the texture information acquisition module to obtain the texture information of the original image, including: Inputting the feature decoding information into the latent space representation module for processing to obtain latent space representation information; Processing the latent space representation information through the image reconstruction module to obtain image reconstruction information; The image reconstruction information is processed by the feature extractor to obtain texture information of the original image.

3. The method according to claim 1, characterized in that The processing of the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information includes: The semantic information and the texture information are fused through the hidden layer representation diffusion model based on the visual knowledge of the hidden layer representation diffusion model to obtain fused information.

4. The method according to claim 1, characterized in that: The image generation module includes an image generator and an enhancement network module connected in sequence; the image generation module processes the fusion information to obtain a decoded image, including: Processing the fusion information by the image generator to obtain an initial generated image; The enhancement network module removes the block effect of the initially generated image to obtain a decoded image.

5. The method according to claim 1, characterized in that The encoding end includes a large model of graph-generated text, a latent space representation extraction model, a first encoder connected to the large model of graph-generated text, and a second encoder connected to the latent space representation extraction model; The text code stream is obtained by encoding first semantic information by the first encoder; the first semantic information is semantic information of the original image extracted by the image-to-text model.

6. An image processing method, characterized in that: Applied to an encoding end, the encoding end includes a graph-to-text model, a latent space representation extraction model, a first encoder connected to the graph-to-text model, and a second encoder connected to the latent space representation extraction model, the method includes: Processing the original image through the image-to-text model to obtain semantic information; Processing the original image through the latent space representation extraction model to obtain texture information; Encoding the semantic information using the first encoder to obtain a text code stream; Encoding the texture information using the second encoder to obtain a feature code stream; The text code stream and the feature code stream are both sent to a decoding end, so that the decoding end executes the image processing method described in any one of claims 1-5.

7. An image processing device, characterized in that: Applied to a decoding end, the decoding end includes a first decoder, a second decoder and a generative diffusion model; the generative diffusion model includes a texture information acquisition module, a hidden layer representation diffusion model and an image generation module connected in sequence; the device includes: A first decoding module, used for decoding the text code stream from the encoding end through a first decoder to obtain semantic information of the original image; A second decoding module, configured to decode the feature code stream from the encoding end through a second decoder to obtain feature decoding information of the original image; A texture information acquisition module, used to process the feature decoding information through the texture information acquisition module to obtain texture information of the original image; A fusion information acquisition module, used for processing the semantic information and the texture information based on the hidden layer representation diffusion model to obtain fusion information; The image generation module is used to process the fusion information through the image generation module to obtain a decoded image.

8. An image processing device, characterized in that: Applied to an encoding end, the encoding end includes a graph-to-text model, a latent space representation extraction model, a first encoder connected to the graph-to-text model, and a second encoder connected to the latent space representation extraction model, the device includes: A semantic information acquisition module, used to process the original image through the image-to-text model to obtain semantic information; A texture information acquisition module, used to process the original image through the latent space representation extraction model to obtain texture information; A text code stream acquisition module, used for encoding the semantic information by using the first encoder to obtain a text code stream; A feature code stream acquisition module, used to encode the texture information using the second encoder to obtain a feature code stream; A sending module is used to send both the text code stream and the feature code stream to a decoding end, so that the decoding end executes the image processing method according to any one of claims 1 to 5.

9. An image encoding and decoding system, characterized in that: The method comprises an encoder and a decoder, wherein the decoder is used to execute the image processing method according to any one of claims 1 to 5, and the encoder is used to execute the image processing method according to claim 6.

10. An electronic device, characterized in that: The device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images