Image processing method and apparatus, electronic device and storage medium

Abstract

An image processing method and apparatus, an electronic device and a storage medium are provided.SOLUTION: An image processing method acquires input content adapted to an image processing task. The input content includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence. The method also performs cross-modal semantic modeling on the input content to obtain a unified feature representation that includes multi-modal semantic information. The multi-modal semantic information is used to indicate a semantic association relationship between different modalities of the input content. The method further generates output content adapted for the image processing task based on the consolidated feature representation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of artificial intelligence technologies, particularly to the field of technologies such as computer vision, deep learning, and large-scale models, and in particular to image processing methods, apparatus, electronic devices, and storage media. [Background technology]

[0002] With the development of artificial intelligence and computer vision technology, image processing tasks are no longer limited to traditional single-vision perceptron tasks such as image classification, target detection, and image segmentation, but are gradually expanding into more complex advanced application fields such as image understanding, fine image editing, and high-quality image generation. In this context, how to efficiently and accurately process images has become an important issue for improving the quality of vision content generation and interaction experience, and has significant research value and application significance. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides an image processing method, apparatus, electronic device, and storage medium. [Means for solving the problem]

[0004] According to one aspect of the present disclosure, there is provided an image processing method, the method comprising: obtaining input content suitable for an image processing task, the input content including at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multi-modal semantic information, the multi-modal semantic information being used to indicate semantic association relationships between different modalities of the input content; and generating output content adapted to the image processing task based on the integrated feature representation.

[0005] According to another aspect of the present disclosure, there is provided an image processing device, the device comprising: an acquisition module for acquiring input content suitable for an image processing task, the input content including at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence; a processing module for performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, the multimodal semantic information being used to indicate semantic association relationships between different modalities of the input content; and and a generation module for generating output content adapted to the image processing task based on the integrated feature representation.

[0006] According to another aspect of the present disclosure, there is provided an electronic device, the electronic device comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a method as provided by one of the above aspects of the present disclosure.

[0007] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing the computer to perform a method as set forth in the above aspect of the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a computer program which, when executed by a processor, implements the method set out in the above aspect of the present disclosure.

[0009] It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following description. [Brief explanation of the drawings]

[0010] The drawings are used for better understanding of the present technical solution and are not intended to limit the present disclosure. [Figure 1] 1 is a schematic flowchart of an image processing method provided by Example 1 of the present disclosure. [Figure 2] 10 is a schematic flowchart of an image processing method provided by Example 2 of the present disclosure. [Figure 3] 10 is a schematic flowchart of an image processing method provided by Example 3 of the present disclosure. [Figure 4] 10 is a schematic flowchart of an image processing method provided by Example 4 of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of the training principle of a token extraction model provided by an embodiment of the present disclosure. [Figure 6] 10 is a schematic flowchart of an image processing method provided by Example 5 of the present disclosure. [Figure 7] 1 is a schematic diagram of the principle of image generation provided by an embodiment of the present disclosure; [Figure 8] 1 is a schematic diagram of the principle of image understanding provided by an embodiment of the present disclosure; [Figure 9] FIG. 13 is a schematic configuration diagram of an image processing device provided by a sixth embodiment of the present disclosure. [Figure 10] FIG. 1 is a schematic block diagram of an exemplary electronic device for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011]

[0033] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the following description will omit descriptions of well-known functions and structures.

[0012] In addition, in the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, etc. of relevant user personal information will all be carried out with the user's consent, will comply with the provisions of relevant laws and regulations, and will not violate public order and morals.

[0013] Transformer-based multimodal large-scale models have been highly successful in image understanding and generation tasks. These models simultaneously process text and image inputs and output corresponding text or image responses in response to user queries. Unlike text tokens obtained directly by a tokenizer, image data typically relies on external encoders and decoders to obtain corresponding token sequences through feature extraction and quantization. During image data encoding, the original image is divided into multiple regions, a vector matrix is ​​first obtained by the encoder, and then the vectors of each region are quantized into discrete tokens. However, while image token sequences obtained in this way perform well in image generation tasks, they fail to achieve sufficient application effectiveness in image understanding.

[0014] For example, in the image encoding stage, an image is first segmented into multiple regions. Then, through the image encoding stage, which uses a convolutional neural network (CNN) as its core architecture, image feature representations corresponding to each region are obtained. These image feature representations are then quantized to obtain image tokens. However, this process typically focuses only on the pixel features of the image, resulting in insufficient performance in image understanding, especially in form and document understanding tasks. To address this issue, current multimodal large-scale modeling approaches typically fall into two categories: one directly separates the two models and uses separate encoders for image generation and image understanding. Image generation uses an image encoder with the aforementioned CNN as its core architecture, while image understanding uses an image encoder with a Transformer core architecture that focuses on semantic information modeling. The other approach introduces semantic information distillation into the training process of the CNN image encoder to impart a certain degree of semantic perceptron capabilities.

[0015] The use of separate encoders for image generation and image understanding essentially treats image generation and image understanding as two different modal tasks. Complex image generation tasks often lack sufficient semantic understanding of the preamble of the image, resulting in insufficient performance in tasks such as data generation where images and text intersect or semantic editing based on the generated images. Introducing semantic information distillation into the training process of a CNN image encoder can usually improve the semantic perceptron capabilities of the image encoder to a certain extent, and this method can achieve significant performance improvements in image question answering tasks. However, such high-level semantic information is still insufficient for form and document understanding tasks, which rely on character recognition as their core.

[0016] In response to at least one of the above problems, the present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0017] Hereinafter, an image processing method, an apparatus, an electronic device, and a storage medium according to embodiments of the present disclosure will be described with reference to the drawings.

[0018] FIG. 1 is a schematic flowchart of an image processing method provided by a first embodiment of the present disclosure.

[0019] The embodiments of the present disclosure are described as an example in which the image processing method is configured in an image processing device, and the image processing device can be applied to any electronic device such that the electronic device can perform image processing functions.

[0020] The electronic device may be any device with computing capabilities, such as a computer, a mobile terminal, a server, etc., and the mobile terminal may be a hardware device with various operating systems, touchscreens and / or displays, such as an in-vehicle device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0021] As shown in FIG. 1, this image processing method may include the following steps 101 to 103.

[0022] In step 101, input content suitable for an image processing task is obtained.

[0023] The input content includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence.

[0024] In an embodiment of the present disclosure, input data is acquired based on a specific image processing task (e.g., image understanding, image generation, image editing, etc.). These inputs can include at least one of three forms: a first text token sequence, a first image token sequence, and an image-text fusion sequence. The first text token sequence refers to text information that linguistically describes the content of an image. For example, the input text is "A yellow dog is running on the grass." The text is divided into tokens to obtain a first text token sequence. The first image token sequence refers to a set of discretized representation units with both semantic and visual characteristics extracted from the original input image. The first image token sequence is a set of discretized representation units with both semantic and visual characteristics extracted from the input image. The image-text fusion sequence is generated from fusion features obtained by fusing high-level semantic features (e.g., meanings of objects, scenes, etc.) and low-level pixel features (e.g., details such as color and pattern). For example, the fusion features are sequentially downsampled and quantized to obtain a first image token sequence. The first image token sequence includes a plurality of image tokens, each of which is a basic semantic unit that is structured and discretized to represent the image context, and can effectively represent local or global semantic information of the image. The image-text fusion sequence is generated by fusing the text token sequence of the input text and the image token sequence of the input image.

[0025] In step 102, cross-modal semantic modeling is performed on the input content to obtain a unified feature representation containing multi-modal semantic information.

[0026] Multimodal semantic information is used to indicate semantic association relationships between different modalities of input content.

[0027] To enable the model to understand the semantic relationships between different modalities, one possible implementation is to use a self-attention or cross-attention mechanism to capture the semantic associations between different modalities in the input content, thereby generating a unified feature representation that includes multimodal information. For example, if the input includes a first sequence of image tokens and a corresponding first sequence of text tokens describing "the cat is sleeping on the sofa," a cross-modal attention mechanism can recognize the regions in the image where "cat" and "sofa" are located and establish semantic associations between the corresponding words in the text, thereby achieving cross-modal alignment and fusion.

[0028] In step 103, output content suitable for the image processing task is generated based on the integrated feature representation.

[0029] In the embodiments of the present disclosure, after the integrated features fusing multimodal semantic information are obtained, corresponding output content can be generated according to the needs of specific image processing tasks. For example, for an image understanding task, the output can be a natural language sentence; for example, for an image generation task, the output can be a new image.

[0030] In this way, input content suitable for image processing tasks is obtained, and the input includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence, thereby effectively improving the flexibility and expressiveness of the input format. Furthermore, cross-modal semantic modeling is performed on the input content, which can deeply explore the semantic association relationships between different modalities and generate an integrated feature representation containing multi-modal semantic information. This integrated feature not only preserves the semantic structure of each modality, but also effectively captures the alignment, complementarity, and interaction relationships between the modalities, thereby helping to construct a unified semantic space, thereby significantly improving the model's context understanding and cross-modal reasoning capabilities. Finally, output content suitable for image processing tasks is generated based on this integrated feature representation, ensuring that the output result is highly consistent with the input at a semantic level, further improving the accuracy and intelligence of task execution.

[0031] In the above embodiment, to clearly explain how to perform cross-modal semantic modeling on input content to obtain a unified feature representation containing multi-modal semantic information, the present disclosure provides another image processing method.

[0032] FIG. 2 is a schematic flowchart of an image processing method provided by the second embodiment of the present disclosure.

[0033] As shown in FIG. 2, this image processing method may include the following steps 201 to 206.

[0034] In step 201, an input content suitable for an image processing task is obtained, where the image processing task includes an image understanding task or an image editing task, and the input content includes a first sequence of text tokens and a first sequence of image tokens.

[0035] In one possible implementation, if the image task is an image understanding task (e.g., image classification, visual question answering) or an image editing task (e.g., image redaction), the input content can include a first sequence of text tokens and a first sequence of image tokens, i.e., the model can receive and process input information from text and images simultaneously.

[0036] In step 202, semantic coding is performed on the first text token sequence based on the context information of the first text token sequence to obtain first text semantic features in the text space.

[0037] In one possible implementation, a semantic encoder (e.g., the text encoder in Transformer) is used to process the text token sequence, capturing the semantic relationships and contextual dependencies between words, and obtaining a high-dimensional vector sequence, called the first text semantic feature, which represents the meaning of each token in the text in the semantic space. For example, for the text "The cat is sleeping on the sofa," the semantic relationships between the keywords "cat," "sofa," and "sleeping" can be recognized.

[0038] In step 203, feature extraction is performed on the first image token sequence to obtain first image semantic features in the image space.

[0039] In an embodiment of the present disclosure, the first image token sequence is a basic semantic unit extracted from an image, which has both semantic and visual characteristics, and each token can be understood as an abstract representation of a meaningful region in the image (e.g., an object, a structure, etc.), reflecting the semantic content and visual features of this region. Furthermore, a feature encoding network (e.g., an encoder in a Transformer or other feature expansion module) extracts features from the first image token sequence and converts them into a high-dimensional semantic representation that is more suitable for multimodal interaction. For example, a region in the input image corresponding to the first image token sequence is recognized as a "cat," and another region is recognized as a "sofa."

[0040] In step 204, a first semantic association relationship is established between the first text semantic feature and the first image semantic feature.

[0041] In order to improve the model's ability to understand and represent complex meanings in image processing tasks, one possible implementation is to perform semantic alignment between the first text semantic feature and the first image semantic feature, i.e., to realize that each token in the text focuses on the most relevant region in the image, for example, to semantically associate the token "cat" with the region in the image where "cat" exists.

[0042] In step 205, a fusion operation is performed on the first text semantic feature and the first image semantic feature based on the first semantic association relationship to generate a unified feature representation containing multimodal semantic information.

[0043] Furthermore, based on the first semantic association relationship, the first text semantic feature and the first image semantic feature are fused to obtain an integrated feature representation, which not only includes the semantic information of the first text semantic feature and the first image feature, but also fuses the interactions and associations between modals.

[0044] In step 206, output content suitable for the image processing task is generated based on the integrated feature representation.

[0045] To improve the quality of image processing, one possible implementation is to process the integrated feature representation using an autoregressive mechanism to obtain output content adapted to the image processing task.

[0046] As an example, in response to the image processing task including an image understanding task, multiple iterations are performed on the integrated feature representation based on an autoregressive mechanism to obtain a second sequence of text tokens, and image understanding text is generated based on the second sequence of text tokens.

[0047] That is, in order to improve the quality of image understanding and explanation, if the image processing task currently to be performed is an image understanding task, such as visual question answering, image context summarization, image explanation, etc., it is necessary to output natural language text to express the understanding of the image context. Therefore, in the embodiments of the present disclosure, based on the already generated content, each token in the image understanding text is sequentially generated to obtain a second text token sequence, i.e., using the integrated feature representation as the initial context, one text token sequence is generated through multiple iterations (i.e., stepwise token prediction), and finally, these tokens are spliced ​​or decoded into natural language text, i.e., image understanding text, which may be an image explanation, an answer to a question, or any character expression related to the content of the image, etc.

[0048] As another example, in response to the image processing task including an image editing task, performing at least one edit on a first image token sequence based on an autoregressive mechanism and based on the integrated feature representation to obtain an edited image token sequence, and generating a first target image based on the edited image token sequence.

[0049] That is, in order to improve the quality and automation level of image editing, if the image processing task is an image editing task, based on an integrated feature representation that combines the original image information and the user's editing intention, sequential editing is performed on the first image token sequence according to an autoregressive mechanism, and an edited image token sequence is generated by adjusting the image tokens one by one. Then, based on this edited image token sequence, a corresponding complete image, i.e., a first target image, is generated by a pixel decoder, thereby realizing semantic-level editing of the input image.

[0050] As described above, by performing semantic encoding on the first text token sequence based on the context information of the first text token sequence, first text semantic features in the text space can be obtained, and the first text semantic features can effectively represent the linguistic semantic information contained in the text. At the same time, feature extraction is performed on the first image token sequence to obtain first image semantic features in the image space, and the first image semantic features can preserve the spatial structure and visual meaning of the image context. Furthermore, to achieve effective cross-modal information interaction, a semantic association relationship is established between the first text semantic features and the first image semantic features, thereby dynamically recognizing and enhancing highly correlated parts in different modalities. Furthermore, based on the semantic association relationship, a fusion operation is performed on the first text semantic features and the first image semantic features to generate an integrated feature representation containing multimodal semantic information. This integrated feature representation not only combines the visual meaning of the image context but also integrates the semantic intent of the text description, thereby providing stronger expressive power and semantic completeness. This can provide richer and more accurate contextual support for subsequent tasks (e.g., image understanding, image editing), and improve the comprehension ability and execution accuracy of complex image processing tasks.

[0051] In the above embodiment, to clearly explain how to perform cross-modal semantic modeling on input content to obtain a unified feature representation containing multi-modal semantic information, the present disclosure provides another image processing method.

[0052] FIG. 3 is a schematic flowchart of an image processing method provided by a third embodiment of the present disclosure.

[0053] As shown in FIG. 3, the image processing method may include the following steps 301 to 304.

[0054] In step 301, input content suitable for an image processing task is obtained, where the image processing task comprises an image understanding task and the input content comprises a first image token sequence.

[0055] In one possible implementation, when the image task is an image understanding task (e.g., image classification, visual question answering), the input content may include a first text token sequence, where the first image token sequence refers to basic semantic units of a discretized representation having both semantic and visual characteristics extracted from the original input image, and the first image token sequence is generated based on fusion features obtained by fusing high-level semantic features of the image (e.g., meanings of objects, scenarios, etc.) and low-level pixel features (e.g., details such as color and pattern), and illustratively, the fusion features are sequentially subjected to downsampling and quantization processes to obtain the first image token sequence.

[0056] In step 302, feature extraction is performed on the first image token sequence based on the context information of the first image token sequence to obtain second image semantic features in the image space.

[0057] To better characterize the spatial structure and semantic content of images, one possible implementation is to use context modeling (e.g., Transformer, CNN, etc.) to extract higher-level visual features from the first image token sequence to obtain second image semantic features in image space.

[0058] In step 303, cross-modal semantic modeling is performed on the second image semantic features to obtain second text semantic features in the text space.

[0059] To map image information into a textually understandable format, embodiments of the present disclosure may map image features into a text semantic space to generate second text semantic features that are semantically aligned therewith.

[0060] In step 304, an integrated feature representation is generated based on the second image semantic features and the second text semantic features.

[0061] To enhance the model's ability to understand and express complex semantics and improve its versatility, one possible implementation is to fuse the second image semantic features and the second text semantic features to generate a unified feature vector, i.e., an integrated feature representation, that includes the semantic information of both.

[0062] In another possible implementation form, when the image processing task is an image understanding task, the input content may include an image-text fusion sequence, where the image-text fusion sequence is obtained by fusing a text token sequence of the input text and an image token sequence of the input image, extracting third text semantic features in the text space and third image semantic features in the image space from the image-text fusion sequence, respectively, establishing a second semantic association relationship between the third text semantic features and the third image semantic features, and fusing the third text semantic features and the third image semantic features based on the second semantic association relationship to generate an integrated feature representation including multimodal semantic information.

[0063] That is, in a scenario where the image processing task is an image understanding task, an image-text fusion sequence can be used as input content, and the image-text fusion sequence is a unified representation generated by cross-modal fusion processing of the text token sequence of the input text and the image token sequence of the input image. The image-text fusion sequence includes semantic information from the text and visual information from the image. Third text semantic features in the text space and third image semantic features in the image space are further extracted from the image-text fusion sequence to realize fine-grained modeling of cross-modal information. A second semantic relationship is established between the third text semantic features and the third image semantic features based on an attention mechanism or a semantic similarity calculation method, thereby clarifying the correspondence and interaction between the text description and the image context. Finally, the third text semantic features and the third image semantic features are fused based on the second semantic relationship to generate an integrated feature representation that can simultaneously reflect the text meaning and the image meaning, thereby enhancing the multimodal semantic representation ability in image understanding tasks and supporting subsequent image understanding tasks such as image description generation, text-image matching, and visual question answering.

[0064] In step 305, output content suitable for the image processing task is generated based on the integrated feature representation.

[0065] As described above, the input content suitable for the image processing task is obtained, and the input content includes a first image token sequence, which is a set of discretized visual units generated by an encoder from the original input image, and can effectively preserve the spatial structure and local details of the image. Based on the context information of the first image token sequence, feature extraction is performed on it to obtain a second image semantic feature in the image space. This process enhances the overall semantic perceptron ability of the image through the context modeling mechanism, and can more accurately capture the relationship between objects in the image, the scenario layout, and the visual context information. In order to further improve the language explainability of the image context, cross-modal semantic modeling is performed on the second image semantic features and mapped to the text semantic space to generate second text semantic features in the text space, thereby realizing the conversion from image information to natural language meaning and providing basic support for subsequent text-image interaction. Furthermore, an integrated feature representation containing multimodal semantic information is generated based on the second image semantic features and the second text semantic features. This integrated feature representation not only aligns the visual meaning of the image context, but also introduces language-level abstraction, resulting in stronger semantic representation and context understanding capabilities.

[0066] To clearly explain how to generate the first image token sequence in the above embodiment, the present disclosure puts forward another image processing method.

[0067] FIG. 4 is a schematic flowchart of an image processing method provided by a fourth embodiment of the present disclosure.

[0068] As shown in FIG. 4, this image processing method may include the following steps 401 to 404.

[0069] In step 401, a semantic feature extraction is performed on an input image using a semantic coding network in a token extraction model to obtain a first semantic feature.

[0070] To accurately capture the image's theme content and key object information, a semantic-rich feature basis is provided for subsequent multimodal understanding and image tokenization processing. In one possible implementation, a semantic encoding network in the token extraction model is used to input a neural network of high-level semantic information contained in the image, and the semantic encoding network extracts feature vectors that can represent high-level concepts such as image context, image theme, and object category, which are referred to as first semantic features.

[0071] The token extraction model is obtained by training using the following steps.

[0072] (1) Perform semantic feature extraction on the sample image using a semantic coding network to obtain a second semantic feature, and perform pixel feature extraction on the sample image using a pixel coding network to obtain a second pixel feature.

[0073] In an embodiment of the present disclosure, a semantic coding network (e.g., a high-level semantic extractor based on Transformer or CNN) is used to extract high-level semantic information from the sample image to obtain second semantic features, which are used to indicate abstract semantic content such as the theme, object category, or scenario of the image; and a pixel coding network (e.g., a CNN structure) is used to extract low-level visual features of the image to obtain second pixel features, in order to preserve the local structure and visual details of the image (e.g., detailed information such as edges, colors, and textures).

[0074] (2) Perform semantic restoration and image restoration on the sample image based on the second semantic features and the second pixel features to obtain restored semantic features and restored images. To improve the semantic consistency, detail retention, and overall representation quality of image token sequences, one possible implementation method is to introduce a feature fusion network, a semantic decoding network, and a pixel decoding network, so that the token extraction model can achieve bidirectional modeling and reconstruction of image content at two levels: semantic and visual.

[0075] As an example, the token extraction model further includes a feature fusion network, a semantic decoding network, and a pixel decoding network, where the feature fusion network is used to fuse a second semantic feature and a second pixel feature to obtain a second fused feature, the semantic decoding network is used to perform semantic restoration on the sample image based on the second fused feature to obtain a restored semantic feature, and the pixel decoding network is used to perform image restoration on the sample image based on the second fused feature to obtain a restored image.

[0076] That is, the token extraction model further includes a feature fusion network, a semantic decoding network, and a pixel decoding network. The feature fusion network is used to fuse semantic features and pixel features, the semantic decoding network is used to recover semantic information from the fused features, and the pixel decoding network is used to reconstruct the original image from the fused features. The feature fusion network combines the second semantic feature and the second pixel feature to generate a unified feature representation, referred to as the second fused feature. The second fused feature not only contains the thematic content of the image (e.g., "A dog is running on the grass") but also preserves the visual details of the image (e.g., color, texture, shape, etc.), providing a stronger feature representation ability for subsequent tasks. The semantic information of the original image is further reconstructed from the fused features using the semantic decoding network to obtain the recovered semantic feature, which is used to measure whether the model accurately captures the semantic content of the image. At the same time, the pixel decoding network is used to reconstruct a sample image from the second fused features to obtain the recovered image, which is used to evaluate whether the model has retained sufficient visual details.

[0077] The token extraction model further includes a multi-layer perceptron network and a quantization network, wherein the multi-layer perceptron network is used to downsample the second fusion feature to obtain a second sampled feature, the quantization network is used to perform quantization on the second sampled feature to obtain a sample image token sequence, and the semantic decoding network is used to perform semantic restoration on the sample image according to the sample image token sequence to obtain a restored semantic feature. At the same time, the pixel decoding network is used to reconstruct the sample image according to the sample image token sequence to obtain a restored image.

[0078] Specifically, a multilayer perceptron network (MLP) is used to downsample the second fused feature to reduce the feature dimensionality and extract more representative high-level semantic information to obtain the second sampled feature. A quantization network then discretizes the second sampled feature and maps it to a discrete token space to generate a sample image token sequence, achieving a compact representation of the image context. Finally, a semantic decoding network is used to reconstruct the image semantic information based on the image token sequence and output the corresponding restored semantic feature, which is then used to evaluate the semantic preservation ability of the model. By using a multilayer perceptron network to downsample the feature, we effectively compress the spatial dimension of the image feature and improve computational efficiency. At the same time, by converting continuous features into a discrete image token sequence through the quantization network, we achieve efficient encoding and semantic abstraction of image content. Furthermore, semantic reconstruction of the sample token sequence through the semantic decoding network can be performed to verify whether the extracted tokens retain sufficient semantic information. Image reconstruction of the sample token sequence through the pixel decoding network is then performed to evaluate whether the model retains sufficient visual details.

[0079] (3) Training a token extraction model based on the difference between the restored semantic features and the second semantic features and the difference between the restored image and the sample image.

[0080] To improve the model's ability to understand high-level image semantics and recover low-level visual details, one possible implementation is to perform joint training on the token extraction model by combining the semantic recovery loss and the image reconstruction loss.

[0081] As an example, a first loss function value is generated based on the difference between the restored semantic features and the second semantic features, a second loss function value is generated based on the difference between the restored image and the sample image, and a token extraction model is trained based on the first loss function value and the second loss function value.

[0082] For example, based on the first loss function value, a semantic coding network in the image processing model is trained, the semantic coding network is frozen, and based on the second loss function value, another network other than the semantic coding network in the token extraction model is trained to obtain a trained token extraction model.

[0083] That is, to improve the stability and efficiency of model training and enhance the semantic accuracy and visual authenticity of the final generated image token sequence, one possible implementation is to use a step-by-step training strategy to ensure the quality of semantic encoding and then gradually optimize other components of the token extraction model.

[0084] As an example, a semantic coding network can be trained based on the difference between the restored semantic features and the second semantic features and the generated first loss function value. After the initial training of the semantic coding network is completed, its parameters can be frozen (not updated). The second loss function value (usually derived from an image reconstruction task) generated based on the difference between the restored image and the sample image can be used to train the remaining parts of the extraction model, such as the pixel coding network, feature fusion network, and decoding network, thereby ensuring that subsequent network modules can better adapt to the stable semantic features while focusing on modeling and reconstructing image details.

[0085] For example, as shown in Figure 5, the token extraction model includes two branches: a semantic branch based on a Vision Transformer (ViT) and a pixel branch based on a Convolutional Neural Network (CNN). After the features of the semantic branch and the pixel branch are fused, a quantization operation is performed. Based on the quantized features, a pixel decoder performs image restoration and calculates the reconstruction loss between the restored image and the original image (sample image). At the same time, the quantized features are also semantically restored by a semantic decoder, and the semantic restoration loss between the restored features and the ViT output is calculated. The training flow includes two parts: (1) independently pre-training the ViT; and (2) fixing the ViT and training the remaining components.

[0086] In this method, by combining the semantic restoration loss and the image reconstruction loss, the token extraction model is jointly trained, which simultaneously improves the model's ability to understand high-level image semantics and restore low-level visual details, thereby generating image token sequences with consistent meaning and clear structure.

[0087] In step 402, pixel feature extraction is performed on the input image using a pixel encoding network in a token extraction model to obtain a first pixel feature.

[0088] To further improve the fine-grained structural representation of the image and provide basic support for subsequent feature fusion, one possible implementation is to use a pixel coding network to extract low-level visual features of the image.

[0089] As an example, a pixel coding network in the token extraction model is used to extract low-level visual features such as edges, colors, and textures from the original pixel values ​​of the input image; these low-level features help to preserve the detailed information of the image and are called first pixel features.

[0090] In step 403, the first semantic feature and the first pixel feature are fused to obtain a first fused feature.

[0091] To create a comprehensive feature representation that reflects the macroscopic semantics of an image while preserving the microscopic details, an embodiment of the present disclosure fuses a first semantic feature and a first pixel feature to obtain a first fused feature, and the fusion can be achieved in various ways, such as a simple connection operation or a more complex interactive fusion method.

[0092] In step 404, a first sequence of image tokens is generated based on the first fusion feature.

[0093] In order to improve image comprehension, one possible implementation is to use the above-obtained features to convert them into a series of discrete tokens, which can be regarded as a new image representation format, where each image token represents an abstract semantic representation of a specific local area in the image, and the entire image token sequence constitutes a structured and semantic high-level generalization of the entire image context, which not only retains the essential visual and semantic information of the image, but also has compactness and modelability, making it applicable to multimodal tasks such as image generation, text-image fusion modeling, etc., i.e., the first image token sequence can be applied to the image processing method of any of the above embodiments.

[0094] To further achieve efficient discrete modeling of image context, one possible implementation is to perform a dimensionality reduction process on the fused features using a multi-layer perceptron network and map it to a discrete image token sequence in conjunction with a quantization network.

[0095] As an example, a multi-layer perceptron network in the token extraction model is used to downsample the first fusion feature to obtain a first sampled feature, and a quantization network in the token extraction model is used to perform a quantization process on the first sampled feature to generate a first image token sequence.

[0096] That is, a multi-layer perceptron (MLP) network is used to perform downsampling on the first fused feature to compress the dimension of the feature, thereby obtaining the first sampled feature. This process significantly improves the compactness and computational efficiency of the feature representation while preserving the essential meaning and visual information of the original fused feature. Then, a quantization network is used to map the continuous-value vectors in the first sampled feature into a set of pre-defined discrete codebook spaces. Specifically, each sampled feature is replaced with the closest vector in the codebook and the corresponding index value is recorded. This index value is the image token. Furthermore, the first image token sequence is formed by arranging multiple image tokens in order.

[0097] As described above, a semantic coding network in the token extraction model is used to extract semantic features from the input image, thereby obtaining first semantic features that can effectively characterize high-level semantic information, such as thematic content of the image, object categories, and scenario information; a pixel coding network is then used to extract pixel-level features from the input image, thereby obtaining first pixel features that retain low-level visual details, such as image edges, colors, and textures, and are helpful in improving the accuracy of image reconstruction and understanding; the first semantic features and the first pixel features are fused to generate first fused features containing multi-layer information, which not only inherit the high-level semantic representation of the image, but also retain essential visual details, and have stronger representation capabilities and context perceptron capabilities; and finally, a first image token sequence is generated based on the first fused features, which is used for subsequent image understanding or editing tasks. This token sequence retains image semantic consistency while also having good structural integrity and visual authenticity, and can be widely applied to image processing tasks based on the Transformer architecture.

[0098] In the above embodiment, to clearly explain how to perform cross-modal semantic modeling on input content to obtain a unified feature representation containing multi-modal semantic information, the present disclosure provides another image processing method.

[0099] FIG. 6 is a schematic flowchart of an image processing method provided by the fifth embodiment of the present disclosure.

[0100] As shown in FIG. 6, this image processing method may include the following steps 601 to 605.

[0101] In step 601, input content that is compatible with an image processing task is obtained, where the image processing task comprises an image generation task and the input content comprises a first sequence of text tokens.

[0102] In an embodiment of the present disclosure, if the current image processing task is an image generation task, the input content is a first sequence of text tokens, ie, a textual description or instructions for the image to be generated provided by the user.

[0103] In step 602, semantic coding is performed on the first text token sequence based on the context information of the first text token sequence to obtain a fourth text semantic feature of the text space.

[0104] To accurately capture user intent and improve control over image generation, one possible implementation is to use a semantic encoder (e.g., Transformer, BERT, etc.) to analyze the semantic relationships between text tokens and extract a high-dimensional vector representation, called the fourth text semantic feature, which is used to characterize the deep linguistic meaning of the text corresponding to the first text token sequence.

[0105] In step 603, cross-modal semantic modeling is performed on the fourth text semantic feature to obtain a fourth image semantic feature in the image space.

[0106] To achieve text-image semantic alignment and improve the consistency of generation, embodiments of the present disclosure can map text semantic features to an image semantic space to obtain a fourth image semantic feature, i.e., a visual semantic representation that can be used for image generation.

[0107] In step 604, an integrated feature representation is generated based on the fourth text semantic feature and the fourth image semantic feature.

[0108] To further enhance the semantic representation capability of images, the embodiments of the present disclosure combine text semantic features and image semantic features to form a unified integrated feature representation, which simultaneously possesses the abstractness of linguistic description and the concreteness of image structure, and is the driving factor for subsequent image generation.

[0109] In step 605, output content suitable for the image processing task is generated based on the integrated feature representation.

[0110] To improve the image generation quality and semantic consistency with the input text, the embodiment of the present disclosure converts the integrated features into a pixel image.

[0111] As an example, based on an autoregressive mechanism, multiple iterations are performed on the integrated feature representation to obtain a second image token sequence corresponding to at least one resolution, and a second target image is generated based on the second image token sequence corresponding to the at least one resolution.

[0112] That is, to improve the quality of image generation, the integrated feature representation is gradually subdivided by an autoregressive method to generate image token sequences at different resolutions (i.e., second image token sequences), and in each iteration, further optimization is performed based on the previous results to ensure that the final generated image meets the requirements of the input content and has high quality and details.

[0113] During generation of a second target image based on second image token sequences corresponding to at least one resolution, a semantic hierarchy of any second image token sequence is determined based on semantic features of the any second image token sequence, and based on the semantic hierarchy, any second image token sequence is upsampled to obtain a third image token sequence, and each third image token sequence is fused to obtain the second target image.

[0114] That is, to improve the image generation quality, based on the semantic features of each second image token sequence, the information type or theme (i.e., semantic features) contained in each second image token sequence is evaluated, and they are classified into different semantic levels based on the importance or degree of abstraction of this information, for example, low levels may include basic colors and shapes, while high levels relate to specific objects or scenarios. Based on the determined semantic hierarchy, each second image token sequence is subjected to an enlargement process (i.e., upsampling) to improve its resolution and details, thereby obtaining new and clearer image parts (i.e., third image token sequences), and the final step is to merge (i.e., fuse) the image parts after the upsampling process to form one complete and high-resolution second target image.

[0115] In this way, the context information in the first text token sequence is analyzed, and semantic coding is performed on the first text token sequence to convert it into fourth text semantic features that can be expressed in text space. This process can effectively capture the core meaning and key details in the text information and ensure accurate understanding of the user's intention. Furthermore, the fourth text semantic features are further mapped to the image space to generate corresponding fourth image semantic features, thereby achieving effective conversion from text description to image meaning and providing feasibility for subsequent deep fusion of text-image information. Furthermore, the extracted Based on the text semantic features and image semantic features, a comprehensive integrated feature representation is generated. This integrated feature representation not only fully retains the essential semantic information of the original text, but also integrates the visual structure and style features required for the image, providing a unified and semantically consistent representation basis for high-quality image generation. Finally, based on this integrated feature representation, output content suitable for image processing tasks is generated. The generated images accurately reflect the explanatory content of the input text and achieve high standards in visual quality, detail expression, and overall composition, greatly improving the accuracy of image generation tasks and the user experience.

[0116] For example, as shown in FIG. 7, taking the image processing task as an image generation task as an example, the image processing method of the embodiment of the present disclosure may include the following steps:

[0117] (1) When a user inputs text (e.g., "TTT"), the text is converted into a series of text tokens, i.e., a first text token sequence.

[0118] (2) These text tokens are processed by the Transformer layer, which captures the contextual information and core meaning in the text and performs semantic encoding on the text tokens to obtain third text semantic features. The third text semantic features are then input to the Image Decoder, which converts these text features into image features, thereby achieving cross-modal semantic modeling from the text space to the image space, and thereby obtaining third image semantic features in the image space.

[0119] (3) Based on the fourth text semantic feature and the fourth image semantic feature, the system generates an integrated feature representation, which not only contains all the essential information of the original text, but also integrates the visual characteristics that the target image should have.

[0120] (4) Based on an autoregressive mechanism, multiple iterations are performed on the integrated feature representation to obtain a second image token sequence corresponding to multiple resolutions (e.g., Scale1, Scale2, and Scale3).

[0121] (5) Based on the pixel decoder of the trained token extraction model, image restoration is performed on the second image token sequence corresponding to multiple resolutions to generate a second target image.

[0122] For example, as shown in FIG. 8, taking the image processing task as an image understanding task as an example, the image processing method can include the following steps:

[0123] (1) Text input and encoding When a user inputs text (e.g., "TTT"), the text is converted into a series of text tokens, i.e., a first text token sequence, which is processed by a Transformer layer, which performs semantic encoding on the first text token sequence to capture contextual information and core meaning in the text and obtain first text semantic features.

[0124] (2) An image is input, and the image is converted into an image token sequence, i.e., a first image token sequence, by the ViT (Vision Transformer) and CNN (Convolutional Neural Network) in the trained word extraction model. The first image token sequence is similarly processed by the Transformer layer to extract image feature information and obtain the first image semantic feature.

[0125] (3) Both text tokens and image tokens are input into a Transformer layer for integrated modeling. Because there is no ordering relationship within the image modal, a bidirectional modeling approach is used to enable the model to more effectively capture the relationship between images and text. In the Transformer layer, the first text token sequence and the first image token sequence interact to generate an integrated feature representation that combines text and image information.

[0126] (4) Based on the integrated feature representation, operations such as recognition and understanding are performed on the image, and text content that understands and explains the image is output.

[0127] Corresponding to the image processing method provided by the examples of Figures 1 to 8 above, the present disclosure further provides an image processing device, and since the image processing device provided by the examples of the present disclosure corresponds to the image processing method by the examples of Figures 1 to 8 above, the embodiments of the image processing method are also applied to the image processing device provided by the examples of the present disclosure, and detailed descriptions are omitted in the examples of the present disclosure.

[0128] FIG. 9 is a schematic configuration diagram of an image processing device provided according to a sixth embodiment of the present disclosure.

[0129] As shown in FIG. 9, the image processing device includes an acquisition module 910, a processing module 920 and a generation module 930.

[0130] The acquisition module 910 is used to acquire input content suitable for the image processing task, where the input content includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence. The processing module 920 is used to perform cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, where the multimodal semantic information is used to indicate semantic association relationships between different modalities of the input content. The generation module 930 is used to generate output content suitable for the image processing task based on the unified feature representation.

[0131] In one possible implementation, the image processing task includes an image understanding task or an image editing task, the input content includes a first text token sequence and a first image token sequence, and the processing module 920 is used to perform semantic encoding on the first text token sequence based on context information of the first text token sequence to obtain first text semantic features in a text space, perform feature extraction on the first image token sequence to obtain first image semantic features in an image space, establish a first semantic association relationship between the first text semantic features and the first image semantic features, and perform a fusion operation on the first text semantic features and the first image semantic features based on the first semantic association relationship to generate an integrated feature representation including multimodal semantic information.

[0132] In one possible implementation form, the generation module 930 is used to, in response to the image processing task including an image understanding task, perform multiple iterations on the integrated feature representation based on an autoregressive mechanism to obtain a second text token sequence, and generate image understanding text based on the second text token sequence.

[0133] In one possible implementation form, the generation module 930 is used, in response to the image processing task including the image editing task, to perform at least one edit on the first image token sequence based on an autoregressive mechanism and based on the integrated feature representation to obtain an edited image token sequence, and to generate a first target image based on the edited image token sequence.

[0134] In one possible implementation, the image processing task includes an image understanding task, the input content includes a first image token sequence, and the processing module 920 performs feature extraction on the first image token sequence based on context information of the first image token sequence to obtain second image semantic features in an image space, performs cross-modal semantic modeling on the second image semantic features to obtain second text semantic features in a text space, and generates an integrated feature representation based on the second image semantic features and the second text semantic features.

[0135] In one possible implementation, a first image token sequence is obtained by processing using a first extraction module, a fusion module and a decision module.

[0136] The first extraction module is used to perform semantic feature extraction on an input image using a semantic coding network in a token extraction model to obtain first semantic features, and to perform pixel feature extraction on the input image using a pixel coding network in the token extraction model to obtain first pixel features; the fusion module is used to fuse the first semantic features and the first pixel features to obtain first fused features; and the decision module is used to generate a first image token sequence based on the first fused features.

[0137] In one possible implementation, the decision module is used to downsample the first fusion feature using a multi-layer perceptron network in the token extraction model to obtain a first sampled feature, and to perform a quantization process on the first sampled feature using a quantization network in the token extraction model to generate a first image token sequence.

[0138] In one possible implementation, the token extraction model is obtained by training using a second extraction module, a restoration module and a training module.

[0139] The second extraction module is used to perform semantic feature extraction on the sample image using a semantic coding network to obtain second semantic features and to perform pixel feature extraction on the sample image using a pixel coding network to obtain second pixel features; the restoration module is used to perform semantic restoration and image restoration on the sample image based on the second semantic features and the second pixel features to obtain restored semantic features and restored images; and the training module is used to train a token extraction model based on the differences between the restored semantic features and the second semantic features and the differences between the restored images and the sample images.

[0140] In one possible implementation, the training module generates a first loss function value based on a difference between the restored semantic features and a second semantic feature, generates a second loss function value based on a difference between the restored image and the sample image, and is used to train a token extraction model based on the first loss function value and the second loss function value.

[0141] In one possible implementation form, the training module is used to train a semantic coding network in the image processing model based on the first loss function value, freeze the semantic coding network, and train another network other than the semantic coding network in the token extraction model based on the second loss function value to obtain a trained token extraction model.

[0142] In one possible implementation, the token extraction model further includes a feature fusion network, a semantic decoding network, and a pixel decoding network, and the restoration module is used to use the feature fusion network to fuse the second semantic feature and the second pixel feature to obtain a second fused feature, use the semantic decoding network to perform semantic restoration on the sample image based on the second fused feature to obtain a restored semantic feature, and use the pixel decoding network to perform image restoration on the sample image based on the second fused feature to obtain a restored image.

[0143] In one possible implementation, the token extraction model further includes a multi-layer perceptron network and a quantization network, and the restoration module is used to downsample the second fusion feature using the multi-layer perceptron network to obtain a second sampled feature, perform quantization processing on the second sampled feature using the quantization network to obtain a sample image token sequence, and use a semantic decoding network to perform semantic restoration on the sample image word sequence and the sample image to obtain a restored semantic feature.

[0144] In one possible implementation, the image processing task includes an image generation task or an image editing task, the input content includes an image-text fusion sequence, and the processing module 920 is used to respectively extract third text semantic features in the text space and third image semantic features in the image space from the image-text fusion sequence, establish a second semantic association relationship between the third text semantic features and the third image semantic features, and fuse the third text semantic features and the third image semantic features based on the second semantic association relationship to generate an integrated feature representation including multimodal semantic information.

[0145] In one possible implementation, the image processing task includes an image generation task, the input content includes a first text token sequence, and the processing module 920 is used to perform semantic encoding on the first text token sequence based on context information of the first text token sequence to obtain fourth text semantic features in a text space, perform cross-modal semantic modeling on the fourth text semantic features to obtain fourth image semantic features in an image space, and generate an integrated feature representation based on the fourth text semantic features and the third image semantic features.

[0146] In one possible implementation, the generation module 930 performs multiple iterations on the integrated feature representation based on an autoregressive mechanism to obtain a second image token sequence corresponding to at least one resolution, and is used to generate a second target image based on the second image token sequence corresponding to the at least one resolution.

[0147] In one possible implementation, the generation module 930 is used to determine a semantic hierarchy of any second image token sequence based on the semantic features of the any second image token sequence, upsample the any second image token sequence based on the semantic hierarchy to obtain a third image token sequence, and fuse each third image token sequence to obtain a second target image.

[0148] An image processing device according to an embodiment of the present disclosure obtains input content suitable for an image processing task, where the input includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence, thereby effectively improving the flexibility and expressiveness of the input format; further performs cross-modal semantic modeling on the input content, thereby deeply exploring the semantic association relationships between different modalities; and generates an integrated feature representation containing multi-modal semantic information. This integrated feature not only preserves the semantic structure of each modality, but also effectively captures the alignment, complementarity, and interaction relationships between the modalities, thereby helping to construct a unified semantic space, thereby significantly improving the model's context understanding and cross-modal reasoning capabilities; and finally generates output content suitable for the image processing task based on this integrated feature representation, thereby ensuring that the output result is highly consistent with the input at a semantic level, further improving the accuracy and intelligence of task execution.

[0149] To achieve the above embodiments, the present disclosure further provides an electronic device, which may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor such that the at least one processor can perform the image processing method as provided by any of the above embodiments of the present disclosure.

[0150] To realize the above embodiments, the present disclosure further provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions causing a computer to perform an image processing method as provided by any of the above embodiments of the present disclosure.

[0151] To realize the above embodiments, the present disclosure further provides a computer program, which, when executed by a processor, realizes the image processing method provided by any of the above embodiments of the present disclosure.

[0152] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0153] 10 shows a schematic block diagram of an exemplary electronic device for implementing embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the description herein and / or the practice of the present disclosure as sought.

[0154] 10, the electronic device 1000 includes a computing unit 1001 that performs various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 1002 or loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data necessary for the operation of the electronic device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to one another via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0155] The components of the electronic device 1000 are connected to an I / O interface 1005, which includes input units 1006 such as a keyboard, a mouse, etc., output units 1007 such as various types of displays, speakers, etc., storage units 1008 such as a magnetic disk, an optical disk, etc., and communication units 1009 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 enables the electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 1001 may be various general-purpose and / or specialized processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various machine driving learning model algorithm computing units, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes the methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium such as the storage unit 1008. In some embodiments, some or all of the computer program may be loaded and / or installed into the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, it may perform one or more steps of the image processing method described above. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., via firmware) to perform the image processing method described above.

[0157] Various implementations of the systems and techniques described herein above may be realized in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that can be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, and that can receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0158] Program code for carrying out the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine, partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or can store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, a CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can also provide interaction with a user; for example, the feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and can receive input from the user in any form (including acoustic, speech, or tactile input).

[0161] The systems and techniques described herein may be implemented on a computing system including a back-end component (e.g., a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or on a computing system including any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0162] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by computer programs running on corresponding computers. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a host product in a cloud computing service system that solves the problems of traditional physical hosts and VPS (Virtual Private Server) services, such as difficulty in management and poor business scalability. The server may be a server in a distributed system or a server incorporating blockchain.

[0163] Artificial intelligence is a field that studies how computers can simulate certain human thought processes and intelligent behaviors (learning, reasoning, thinking, planning, etc.), and it includes both hardware-level and software-level technologies. AI hardware technology generally includes sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, etc. AI software technology mainly includes several aspects such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0164] According to the technical solution of the embodiments of the present disclosure, input content suitable for image processing tasks is obtained, where the input includes at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence, thereby effectively improving the flexibility and expressiveness of the input format; further, cross-modal semantic modeling is performed on the input content, which can deeply explore the semantic association relationships between different modalities, and further generates an integrated feature representation containing multi-modal semantic information; this integrated feature not only preserves the semantic structure of each modality itself, but also effectively captures the alignment, complementarity, and interaction relationships between the modalities, thereby helping to construct a unified semantic space, thereby significantly improving the context understanding and cross-modal reasoning capabilities of the model; finally, output content suitable for image processing tasks is generated based on this integrated feature representation, ensuring that the output result is highly consistent with the input at a semantic level, further improving the accuracy and intelligence level of task execution.

[0165] It should be understood that steps can be rearranged, added, or deleted using the various types of flows shown above. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, but this specification is not limited thereto as long as the technical solution proposed by the present disclosure can achieve the desired results.

[0166] The above specific embodiments do not limit the scope of protection of the present disclosure. It should be understood that those skilled in the art can make various modifications, combinations, subcombinations, and substitutions according to design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. 1. An image processing method, comprising: obtaining input content suitable for an image processing task, the input content including at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multi-modal semantic information, the multi-modal semantic information being used to indicate semantic association relationships between different modalities of the input content; generating output content adapted to the image processing task based on the integrated feature representation; Including, An image processing method comprising:

2. the image processing task comprises an image understanding task or an image editing task, and the input content comprises a first sequence of text tokens and a first sequence of image tokens; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, performing semantic coding on the first sequence of text tokens based on context information of the first sequence of text tokens to obtain first text semantic features in a text space; performing feature extraction on the first sequence of image tokens to obtain first image semantic features in image space; establishing a first semantic association relationship between the first text semantic feature and the first image semantic feature; fusing the first text semantic features and the first image semantic features based on the first semantic association relationship to generate an integrated feature representation including multimodal semantic information; Including, 2. The image processing method according to claim 1.

3. generating output content suited to the image processing task based on the integrated feature representation, responsive to the image processing task including the image understanding task, performing a plurality of iterations on the integrated feature representation based on an autoregressive mechanism to obtain a second sequence of text tokens; generating an image understanding text based on the second text token sequence; Including, 3. The image processing method according to claim 2.

4. generating output content suited to the image processing task based on the integrated feature representation, responsive to the image processing task including the image editing task, performing at least one edit on the first image token sequence using the integrated feature representation based on an autoregressive mechanism to obtain an edited image token sequence; generating a first target image based on the edited sequence of image tokens; Including, 3. The image processing method according to claim 2.

5. the image processing task comprises an image understanding task, and the input content comprises a first sequence of image tokens; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, performing feature extraction on the first image token sequence based on context information of the first image token sequence to obtain second image semantic features in image space; performing cross-modal semantic modeling on the second image semantic features to obtain second text semantic features in a text space; generating the integrated feature representation based on the second image semantic features and the second text semantic features; Including, 2. The image processing method according to claim 1.

6. The first sequence of image tokens comprises: performing semantic feature extraction on the input image using a semantic encoding network in a token extraction model to obtain first semantic features; performing pixel feature extraction on the input image using a pixel encoding network in the token extraction model to obtain first pixel features; fusing the first semantic feature and the first pixel feature to obtain a first fused feature; and generating the first image token sequence based on the first fusion feature.

2. The image processing method according to claim 1.

7. generating the first image token sequence based on the first fusion feature, downsampling the first fused feature using a multi-layer perceptron network in the token extraction model to obtain a first sampled feature; performing a quantization process on the first sampled features using a quantization network in the token extraction model to generate a first sequence of image tokens; Including, 7. The image processing method according to claim 6.

8. The token extraction model is performing semantic feature extraction on the sample image using the semantic coding network to obtain second semantic features, and performing pixel feature extraction on the sample image using the pixel coding network to obtain second pixel features; performing semantic restoration and image restoration on the sample image based on the second semantic features and the second pixel features to obtain restored semantic features and a restored image; training the token extraction model based on the differences between the restored semantic features and the second semantic features and the differences between the restored image and the sample image; 7. The image processing method according to claim 6.

9. training the token extraction model based on differences between the restored semantic features and the second semantic features and differences between the restored image and the sample image, generating a first loss function value based on a difference between the recovered semantic features and the second semantic features; generating a second loss function value based on the difference between the restored image and the sample image; training the token extraction model based on the first loss function value and the second loss function value; Including, 9. The image processing method according to claim 8.

10. training the token extraction model based on the first loss function value and the second loss function value, training a semantic coding network in the image processing model based on the first loss function value; freezing the semantic coding network, and training another network other than the semantic coding network in the token extraction model based on the second loss function value to obtain a trained token extraction model; Including, 10. The image processing method according to claim 9.

11. the token extraction model further includes a feature fusion network, a semantic decoding network, and a pixel decoding network; performing semantic restoration and image restoration on the sample image based on the second semantic features and the second pixel features to obtain restored semantic features and a restored image, fusing the second semantic feature and the second pixel feature using the feature fusion network to obtain a second fused feature; performing semantic reconstruction on the sample image based on the second fusion feature using the semantic decoding network to obtain a reconstructed semantic feature; performing image restoration on the sample image using the pixel decoding network based on the second fusion feature to obtain the restored image; Including, 9. The image processing method according to claim 8.

12. the token extraction model further includes a multi-layer perceptron network and a quantization network; performing semantic reconstruction on the sample image based on the second fusion feature using the semantic decoding network to obtain a reconstructed semantic feature, down-sampling the second fused feature using the multilayer perceptron network to obtain second sampled features; performing a quantization process on the second sampled features using the quantization network to obtain a sample image token sequence; performing semantic reconstruction on the sample images based on the sample image token sequences using the semantic decoding network to obtain reconstructed semantic features; Including, 12. The image processing method according to claim 11.

13. the image processing task comprises an image generation task or an image editing task, and the input content comprises the image-text fusion sequence; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, extracting third text semantic features in text space and third image semantic features in image space from the image-text fusion sequence respectively; establishing a second semantic association relationship between the third text semantic feature and the third image semantic feature; fusing the third text semantic features and the third image semantic features based on the second semantic association relation to generate an integrated feature representation including multimodal semantic information; Including, 2. The image processing method according to claim 1.

14. the image processing task comprises an image generation task, and the input content comprises a first sequence of text tokens; performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, performing semantic coding on the first sequence of text tokens based on context information of the first sequence of text tokens to obtain fourth text semantic features in a text space; performing cross-modal semantic modeling on the fourth text semantic features to obtain fourth image semantic features in image space; generating the integrated feature representation based on the fourth text semantic feature and the fourth image semantic feature; Including, 2. The image processing method according to claim 1.

15. generating output content suited to the image processing task based on the integrated feature representation, performing multiple iterations on the integrated feature representation based on an autoregressive mechanism to obtain a second sequence of image tokens corresponding to at least one resolution; generating a second target image based on a second sequence of image tokens corresponding to the at least one resolution; Including, 15. The image processing method according to claim 14.

16. generating a second target image based on a second sequence of image tokens corresponding to the at least one resolution, determining a semantic hierarchy of any second image token sequence based on semantic features of said second image token sequence; upsampling the any second sequence of image tokens based on the semantic hierarchy to obtain a third sequence of image tokens; fusing each of the third image token sequences to obtain the second target image; Including, 16. The image processing method according to claim 15.

17. An image processing device, an acquisition module for acquiring input content suitable for an image processing task, the input content including at least one of a first text token sequence, a first image token sequence, and an image-text fusion sequence; a processing module for performing cross-modal semantic modeling on the input content to obtain a unified feature representation including multimodal semantic information, the multimodal semantic information being used to indicate semantic association relationships between different modalities of the input content; a generation module for generating output content adapted to the image processing task based on the integrated feature representation; Including, 1. An image processing device comprising:

18. 1. An electronic device comprising: at least one processor; a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor performs a method according to any one of claims 1 to 16; An electronic device characterized by:

19. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions cause a computer to perform the method according to any one of claims 1 to 16. A non-transitory computer-readable storage medium comprising:

20. A computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

1. A computer program product comprising:

Images