Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the scheme, the method comprises the steps of obtaining input content matched with an image processing task; wherein the input content comprises at least one of a first text lexical sequence, a first image lexical sequence and an image-text fusion sequence; performing cross-modal semantic modeling on the input content to obtain joint feature representation containing multi-modal semantic information; wherein the multi-mode semantic information is used for indicating a semantic association relationship of the input content among different modes; based on the joint feature representation, output content adapted to the image processing task is generated.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of computer vision, deep learning, and large models, and particularly to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of artificial intelligence and computer vision technologies, image processing tasks are no longer limited to traditional single visual perception tasks, such as image classification, object detection, and image segmentation. Instead, they are gradually expanding to more complex and advanced application areas such as image understanding, fine-grained image editing, and high-quality image generation. Against this backdrop, how to perform image processing efficiently and accurately has become a key issue in improving the quality of visual content generation and interactive experience, possessing significant research value and application implications. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, an image processing method is provided, comprising:

[0005] Obtain input content adapted to the image processing task; wherein, the input content includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence;

[0006] Cross-modal semantic modeling is performed on the input content to obtain a joint feature representation containing multimodal semantic information; wherein, the multimodal semantic information is used to indicate the semantic association relationship between the input content in different modalities;

[0007] Based on the joint feature representation, output content adapted to the image processing task is generated.

[0008] According to another aspect of this disclosure, an image processing apparatus is provided, comprising:

[0009] An acquisition module is used to acquire input content adapted to the image processing task; wherein, the input content includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence;

[0010] The processing module is used to perform cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information; wherein the multimodal semantic information is used to indicate the semantic association relationship between the input content in different modalities;

[0011] The generation module is used to generate output content adapted to the image processing task based on the joint feature representation.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method proposed in one aspect of this disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in the foregoing aspect of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method proposed in the foregoing aspect of this disclosure.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a schematic flowchart of the image processing method provided in Embodiment 1 of this disclosure;

[0021] Figure 2 This is a schematic flowchart of the image processing method provided in Embodiment 2 of this disclosure;

[0022] Figure 3 This is a schematic flowchart of the image processing method provided in Embodiment 3 of this disclosure;

[0023] Figure 4 This is a schematic flowchart of the image processing method provided in Embodiment 4 of this disclosure;

[0024] Figure 5 A schematic diagram illustrating the training principle of the lexical extraction model provided in this embodiment of the disclosure;

[0025] Figure 6 This is a schematic flowchart of the image processing method provided in Embodiment 5 of this disclosure;

[0026] Figure 7This is a schematic diagram illustrating the principle of image generation provided in the embodiments of this disclosure;

[0027] Figure 8 A schematic diagram illustrating the principle of image understanding provided in the embodiments of this disclosure;

[0028] Figure 9 This is a schematic diagram of the image processing apparatus provided in Embodiment Six of this disclosure;

[0029] Figure 10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0032] Transformer-based multimodal large models have achieved great success in image understanding and generation tasks. These models process both text and image inputs simultaneously and output corresponding text or image responses based on user questions. Unlike methods that directly obtain text tokens through a tokenizer, image data typically relies on external encoders and decoders for feature extraction and quantization to obtain the corresponding token sequences. In the image data encoding process, the original image is segmented into multiple regions. After the encoder obtains a vector matrix, the vector of each region is quantized into a discrete token. However, although the image token sequences obtained by this method perform well in image generation tasks, they are not well applied in image understanding.

[0033] For example, in the image encoding stage, the image is first segmented into multiple regions. After passing through the image encoding stage with a Convolutional Neural Network (CNN) as its core architecture, each region obtains a corresponding image feature representation. Subsequently, this image feature representation undergoes a quantization stage to obtain an image term. However, this process typically only focuses on the pixel features of the image, resulting in poor performance in image understanding, especially in form and document understanding tasks. To address this issue, current processing methods on multimodal large models generally fall into two categories: one is to directly split the model into two parts, using different encoders for image generation and image understanding. Image generation uses the aforementioned CNN-based image encoder, while image understanding uses a Transformer-based image encoder that focuses on semantic information modeling; the other is to incorporate semantic information distillation during the training process of the CNN image encoder, enabling it to possess a certain degree of semantic perception capability.

[0034] Image generation and image understanding use different encoders. This approach essentially treats image generation and image understanding as two different modalities. In complex image generation tasks, it cannot achieve sufficient semantic understanding of the preceding image. Therefore, it performs poorly in tasks such as image-text interleaved data generation or semantic editing based on generated images. Adding semantic information distillation during the training of CNN image encoders can usually improve the semantic perception ability of image encoders to a certain extent. Although this type of method can achieve considerable performance improvement in image question answering tasks, this high-level semantic information is still not suitable for form and document understanding tasks with text recognition as the core.

[0035] In view of at least one of the above-mentioned problems, this disclosure proposes an image processing method, apparatus, electronic device and storage medium.

[0036] The image processing method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic flowchart of the image processing method provided in Embodiment 1 of this disclosure.

[0038] This disclosure illustrates an example where the video processing method is configured in an image processing apparatus, which can be applied to any electronic device to enable the electronic device to perform image processing functions.

[0039] Among them, electronic devices can be any device with computing capabilities, such as computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0040] like Figure 1 As shown, the image processing method may include the following steps:

[0041] Step 101: Obtain input content that is suitable for the image processing task.

[0042] The input content includes at least one of the following: a first text word sequence, a first image word sequence, and an image-text fusion sequence.

[0043] In this embodiment of the disclosure, corresponding input data is obtained according to the specific image processing task (such as image understanding, image generation, image editing, etc.). These inputs can include at least one of the following three forms: a first text token sequence, a first image token sequence, and an image-text fusion sequence. The first text token sequence refers to textual information that describes the image content in language. For example, if the input text is "a yellow dog is running on the grass," this text is segmented into individual tokens to obtain the first text token sequence. The first image token sequence refers to a set of discrete representation units with both semantic and visual characteristics extracted from the original input image. The first image token sequence is generated by fusing high-level semantic features (such as the meaning of objects and scenes) and low-level pixel features (such as color and texture details) of the input image. For example, the fusion features are downsampled and quantized sequentially to obtain the first image token sequence. The first image token sequence includes multiple image tokens, each of which is a basic semantic unit for structurally and discretizing the image content, effectively expressing the local or global semantic information of the image. The image-text fusion sequence is generated by fusing the text token sequence of the input text and the image token sequence of the input image.

[0044] Step 102: Perform cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information.

[0045] Among them, multimodal semantic information is used to indicate the semantic association between different modalities of the input content.

[0046] To enable the model to understand the semantic relationships between different modalities, one possible approach is to employ self-attention or cross-attention mechanisms to capture the semantic associations between the input content and different modalities, thereby generating a joint feature representation containing multimodal information. For example, when the input includes a first image word sequence and a corresponding first text word sequence describing "a cat sleeping on a sofa," a cross-modal attention mechanism can be used to identify the regions in the image containing "cat" and "sofa" and establish semantic connections between them and the corresponding words in the text, achieving alignment and fusion between modalities.

[0047] Step 103: Based on the joint feature representation, generate output content adapted to the image processing task.

[0048] In this embodiment of the disclosure, after obtaining the joint features that integrate multimodal semantic information, the corresponding output content can be generated according to the needs of the specific image processing task. For example, if it is an image understanding task, the output may be a natural language sentence; or if it is an image generation task, the output may be a new image.

[0049] In summary, by acquiring input content adapted to the image processing task, including at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence, the flexibility and expressiveness of the input format are effectively improved. Furthermore, cross-modal semantic modeling of the input content can deeply explore the semantic relationships between different modalities, thereby generating a joint feature representation containing multimodal semantic information. This joint feature not only preserves the semantic structure of each modality itself, but also effectively captures the alignment, complementarity, and interaction relationships between modalities, which helps to construct a unified semantic space, thus significantly improving the model's ability in contextual understanding and cross-modal reasoning. Finally, based on this joint feature representation, output content adapted to the image processing task is generated, ensuring that the output results are highly consistent with the input at the semantic level, further enhancing the accuracy and intelligence level of task execution.

[0050] To clearly illustrate how cross-modal semantic modeling of the input content is performed in the above embodiments to obtain a joint feature representation containing multimodal semantic information, this disclosure proposes another image processing method.

[0051] Figure 2 This is a schematic flowchart of the image processing method provided in Embodiment 2 of this disclosure.

[0052] like Figure 2 As shown, the image processing method may include the following steps:

[0053] Step 201: Obtain input content adapted to the image processing task, wherein the image processing task includes an image understanding task or an image editing task, and the input content includes a first text word sequence and a first image word sequence.

[0054] As one possible implementation, when the image task is an image understanding task (such as image classification, visual question answering) or an image editing task (such as image modification), the input content may include a first text word sequence and a first image word sequence, that is, the model can simultaneously receive and process input information from text and images.

[0055] Step 202: Based on the contextual information of the first text word sequence, perform semantic encoding on the first text word sequence to obtain the first text semantic features of the text space.

[0056] As one possible implementation, a semantic encoder (such as the text encoder in Transformer) is used to process the text word sequence, capture the semantic relationships and contextual dependencies between words, and obtain a high-dimensional vector sequence called the first text semantic feature. It represents the meaning of each word in the text in the semantic space. For example, for the text "A cat is sleeping on the sofa", the semantic relationships between keywords such as "cat", "sofa", and "sleeping" can be identified.

[0057] Step 203: Extract features from the first image word sequence to obtain the first image semantic features in the image space.

[0058] In this embodiment, the first image lexical sequence is a set of basic semantic units with both semantic and visual characteristics extracted from an image. Each lexical unit can be understood as an abstract representation of a meaningful region (such as an object or structure) in the image, reflecting the semantic meaning and visual features of that region. Furthermore, the first image lexical sequence is feature-extracted through a feature encoding network (such as the encoder in a Transformer or other feature enhancement modules), transforming it into a high-dimensional semantic representation more suitable for cross-modal interaction. For example, in the input image corresponding to the first image lexical sequence, one region is identified as "cat," and another region is identified as "sofa."

[0059] Step 204: Establish a first semantic association between the first text semantic features and the first image semantic features.

[0060] To improve the model's ability to understand and express complex semantics in image processing tasks, one possible approach is to perform semantic alignment between the first text semantic features and the first image semantic features. This means that each word in the text focuses on the most relevant region in the image. For example, the word "cat" can be semantically associated with the region in the image where the "cat" is located.

[0061] Step 205: Based on the first semantic association, perform a fusion operation on the first text semantic features and the first image semantic features to generate a joint feature representation containing multimodal semantic information.

[0062] Furthermore, based on the first semantic association, the first text semantic features and the first image semantic features are fused to obtain a joint feature representation. This joint feature representation not only includes the semantic information of the first text semantic features and the first image features respectively, but also integrates the interaction and association between modalities.

[0063] Step 206: Generate output content adapted to the image processing task based on joint feature representation.

[0064] To improve the quality of image processing, one possible approach is to use an autoregressive mechanism to process the joint feature representation in order to obtain output content that is adapted to the image processing task.

[0065] As an example, in response to image processing tasks, including image understanding tasks, a second text word sequence is obtained by iterating the joint feature representation multiple times based on an autoregressive mechanism; and image understanding text is generated based on the second text word sequence.

[0066] In other words, in order to improve the quality of image understanding and description, when the current image processing task to be performed is an image understanding task, such as visual question answering, image content summarization, image description, etc., it is necessary to output a piece of natural language text to express the understanding of the image content. Therefore, in this embodiment of the disclosure, each word in the image understanding text is generated step by step according to the generated content to obtain a second text word sequence. That is, with the joint feature representation as the initial context, a text word sequence is generated through multiple rounds of iteration (i.e., predicting words step by step). Finally, these words are concatenated or decoded into natural language text, i.e., image understanding text. The image understanding text can be an image description, an answer to a question, or any textual expression related to the image content.

[0067] As another example, in response to an image processing task including an image editing task, based on an autoregressive mechanism, at least one round of editing is performed on a first image word sequence according to the joint feature representation to obtain an edited image word sequence; based on the edited image word sequence, a first target image is generated.

[0068] In other words, in order to improve the quality and automation level of image editing, when the image processing task is an image editing task, the first image word sequence is sequentially edited according to the joint feature representation that integrates the original image information and the user's editing intention, and the image words are adjusted one by one to generate the edited image word sequence. Then, based on the edited image word sequence, the corresponding complete image, i.e. the first target image, is generated by the pixel decoder, thereby realizing the semantic-level editing of the input image.

[0069] In summary, by semantically encoding the first text semantic features based on the contextual information of the first text word sequence, a first text semantic feature in the text space is obtained. This first text semantic feature can effectively represent the linguistic semantic information contained in the text. Simultaneously, feature extraction is performed on the first image word sequence to obtain a first image semantic feature in the image space. This first image semantic feature retains the spatial structure and visual semantics of the image content. Furthermore, to achieve effective interaction of cross-modal information, a semantic association is established between the first text semantic feature and the first image semantic feature. This can dynamically identify and enhance highly correlated parts in different modalities. Then, based on the semantic association, the first text semantic feature and the first image semantic feature are fused to generate a joint feature representation containing multimodal semantic information. This joint feature representation not only integrates the visual semantics of the image content but also integrates the semantic intent of the text description, possessing stronger expressive power and semantic integrity. It can provide richer and more accurate contextual support for subsequent tasks (such as image understanding and image editing), improving the understanding and execution accuracy of complex image processing tasks.

[0070] To clearly illustrate how cross-modal semantic modeling of the input content is performed in the above embodiments to obtain a joint feature representation containing multimodal semantic information, this disclosure proposes another image processing method.

[0071] Figure 3 This is a schematic flowchart of the image processing method provided in Embodiment 3 of this disclosure.

[0072] like Figure 3 As shown, the image processing method may include the following steps:

[0073] Step 301: Obtain input content adapted to the image processing task; wherein, the image processing task includes an image understanding task, and the input content includes a first image word sequence.

[0074] As one possible implementation, when the image task is an image understanding task (such as image classification or visual question answering), the input content may include a first text word sequence. The first image word sequence refers to a set of discrete representations based on semantic units with both semantic and visual characteristics extracted from the original input image. The first image word sequence is generated based on the fused features obtained by fusing high-level semantic features (such as the meaning of objects and scenes) and low-level pixel features (such as details such as color and texture). For example, the fused features are downsampled and quantized sequentially to obtain the first image word sequence.

[0075] Step 302: Based on the contextual information of the first image word sequence, feature extraction is performed on the first image word sequence to obtain the second image semantic features in the image space.

[0076] To better characterize the spatial structure and semantic content of images, as a possible approach, context modeling (such as Transformer, CNN, etc.) can be used to extract higher-level visual features from the first image word sequence, thereby obtaining the second image semantic features in the image space.

[0077] Step 303: Perform cross-modal semantic modeling on the second image semantic features to obtain the second text semantic features in the text space.

[0078] In order to map image information into a text-understandable form, in this embodiment of the disclosure, image features can be mapped to a text semantic space to generate a second text semantic feature that is semantically aligned with it.

[0079] Step 304: Generate a joint feature representation based on the second image semantic features and the second text semantic features.

[0080] To enhance the model's ability to understand and express complex semantics and improve its versatility, one possible approach is to fuse the second image semantic features with the second text semantic features to generate a unified feature vector containing the semantic information of both, i.e., a joint feature representation.

[0081] As another possible implementation, when the image processing task is an image understanding task, the input content may include an image-text fusion sequence. The image-text fusion sequence is obtained by fusing the text word sequence of the input text and the image word sequence of the input image. The third text semantic features in the text space and the third image semantic features in the image space are extracted from the image-text fusion sequence, respectively. A second semantic association relationship is established between the third text semantic features and the third image semantic features. Based on the second semantic association relationship, the third text semantic features and the third image semantic features are fused to generate a joint feature representation containing multimodal semantic information.

[0082] In other words, in image processing tasks that are essentially image understanding tasks, a text-image fusion sequence can be used as input. This sequence is a unified representation generated by cross-modal fusion processing of the text word sequence of the input text and the image word sequence of the input image. This fusion sequence contains both semantic information from the text and visual information from the image. Furthermore, third-dimensional text semantic features in the text space and third-dimensional image semantic features in the image space are extracted from the fusion sequence to achieve refined modeling of different modalities. Based on attention mechanisms or semantic similarity calculation methods, a second semantic association is established between the third-dimensional text semantic features and the third-dimensional image semantic features, thereby clarifying the correspondence and interaction between the text description and the image content. Finally, based on the second semantic association, the third-dimensional text semantic features and the third-dimensional image semantic features are fused to generate a joint feature representation that simultaneously reflects both text and image semantics. This enhances the multimodal semantic expression capability in image understanding tasks, supporting subsequent image understanding tasks such as image description generation, text-image matching, and visual question answering.

[0083] Step 305: Based on the joint feature representation, generate output content adapted to the image processing task.

[0084] In summary, by acquiring input content adapted to the current image processing task, including a first image word sequence (a set of discrete visual units generated from the original input image by an encoder, effectively preserving the spatial structure and local details of the image), and extracting features from the contextual information of the first image word sequence, a second image semantic feature in the image space is obtained. This process enhances the overall semantic perception capability of the image through a contextual modeling mechanism, enabling more accurate capture of the relationships between objects, scene layout, and visual context information in the image. To further improve the interpretability of the image content, cross-modal semantic modeling is performed on the second image semantic feature, mapping it to the text semantic space to generate a second text semantic feature in the text space. This realizes the conversion of image information to natural language semantics, providing basic support for subsequent image-text interaction. Furthermore, based on the second image semantic feature and the second text semantic feature, a joint feature representation containing multimodal semantic information is generated. This joint feature representation not only integrates the visual semantics of the image content but also introduces abstract expressions at the language level, possessing stronger semantic expression and contextual understanding capabilities.

[0085] To clearly illustrate how the first image word sequence is generated in the above embodiments, this disclosure proposes another image processing method.

[0086] Figure 4This is a schematic flowchart of the image processing method provided in Embodiment 4 of this disclosure.

[0087] like Figure 4 As shown, the image processing method may include the following steps:

[0088] Step 401: Use the semantic encoding network in the word extraction model to extract semantic features from the input image to obtain the first semantic feature.

[0089] In order to accurately capture the subject content and key object information of an image and provide a semantically rich feature foundation for subsequent multimodal understanding and image lexicalization, as a possible implementation method, a neural network containing high-level semantic information in the input image is used in the semantic encoding network of the lexical extraction model. The semantic encoding network analyzes the image content and extracts feature vectors that can represent high-level concepts such as image subject and object category, which are called the first semantic features.

[0090] It should be noted that the word extraction model was trained using the following steps:

[0091] (1) A semantic coding network is used to extract semantic features from the sample image to obtain the second semantic features, and a pixel coding network is used to extract pixel features from the sample image to obtain the second pixel features;

[0092] In this embodiment of the disclosure, a semantic coding network (e.g., a high-level semantic extractor based on Transformer or CNN) is used to extract high-level semantic information from the sample image to obtain a second semantic feature, wherein the second semantic feature is used to indicate the abstract semantic content of the image such as the subject, object category, and scene; at the same time, in order to preserve the local structure and visual details of the image (e.g., details such as edges, colors, and textures), a pixel coding network (e.g., a CNN structure) is used to extract low-level visual features of the image to obtain a second pixel feature.

[0093] (2) Based on the second semantic features and the second pixel features, perform semantic restoration and image restoration on the sample image to obtain restored semantic features and restored image;

[0094] To improve the semantic consistency, detail preservation, and overall representation quality of image lexical sequences, as a possible approach, a feature fusion network, a semantic decoding network, and a pixel decoding network are introduced to enable the lexical extraction model to achieve bidirectional modeling and reconstruction of image content at both the semantic and visual levels.

[0095] As an example, the lexical extraction model also includes: a feature fusion network, a semantic decoding network, and a pixel decoding network. The feature fusion network fuses the second semantic feature and the second pixel feature to obtain the second fused feature; the semantic decoding network performs semantic restoration on the sample image based on the second fused feature to obtain the restored semantic feature; and the pixel decoding network performs image restoration on the sample image based on the second fused feature to obtain the restored image.

[0096] In other words, the word extraction model also includes: a feature fusion network, a semantic decoding network, and a pixel decoding network. The feature fusion network fuses semantic features and pixel features; the semantic decoding network recovers semantic information from the fused features; and the pixel decoding network reconstructs the original image from the fused features. The feature fusion network combines the second semantic feature and the second pixel feature to generate a unified feature representation, called the second fused feature. This second fused feature includes both the image's main content (e.g., "a dog is running on the grass") and preserves the image's visual details (e.g., color, texture, shape), providing stronger representation capabilities for subsequent tasks. Then, the semantic decoding network reconstructs the semantic information of the original image from the fused features, obtaining the restored semantic feature, which is used to measure whether the model accurately captures the image's semantic content. Simultaneously, the pixel decoding network reconstructs the sample image from the second fused features, obtaining the restored image, which is used to evaluate whether the model retains sufficient visual detail information.

[0097] It should be noted that the word extraction model also includes: a multilayer perceptron network and a quantization network. The multilayer perceptron network downsamples the second fused feature to obtain the second sampled feature; the quantization network quantizes the second sampled feature to obtain the word sequence of the sample image; and a semantic decoding network performs semantic restoration of the sample image based on the word sequence of the sample image to obtain the restored semantic features. Simultaneously, the pixel decoding network reconstructs the sample image based on the word sequence of the sample image to obtain the restored image.

[0098] In other words, a multilayer perceptron (MLP) is used to downsample the second fused features to reduce the feature dimensionality and extract more representative high-level semantic information, resulting in the second sampled features. Subsequently, a quantization network is used to discretize the second sampled features, mapping them to a discrete word space to generate a sample image word sequence, achieving a compact representation of the image content. Finally, a semantic decoding network is used to reconstruct the semantic information of the image based on this image word sequence, outputting the corresponding restored semantic features to evaluate the model's semantic preservation capability. Thus, by introducing a multilayer perceptron for feature downsampling, the spatial dimensionality of image features is effectively compressed, improving computational efficiency. Simultaneously, the quantization network converts continuous features into discrete image word sequences, achieving efficient encoding and semantic abstraction of image content. Furthermore, the semantic decoding network performs semantic restoration on the sample word sequence, verifying whether the extracted words retain sufficient semantic information, while the pixel decoding network performs image restoration on the sample word sequence to evaluate whether the model retains sufficient visual detail information.

[0099] (3) The word extraction model is trained based on the difference between the restored semantic features and the second semantic features, as well as the difference between the restored image and the sample image.

[0100] To improve the model's ability to understand high-level semantics of images and restore low-level visual details, one possible approach is to jointly train the lexical extraction model by combining semantic restoration loss and image reconstruction loss.

[0101] As an example, a first loss function value is generated based on the difference between the restored semantic features and the second semantic features; a second loss function value is generated based on the difference between the restored image and the sample image; and the word extraction model is trained based on the first and second loss function values.

[0102] For example, the semantic coding network in the image processing model is trained based on the first loss function value; the semantic coding network is frozen, and other networks in the word extraction model other than the semantic coding network are trained based on the second loss function value to obtain the trained word extraction model.

[0103] In other words, in order to improve the stability and efficiency of model training, and at the same time enhance the semantic accuracy and visual realism of the final generated image lexical sequence, as a possible approach, a phased training strategy can be adopted to gradually optimize other components of the lexical extraction model while ensuring the quality of semantic encoding.

[0104] As an example, the semantic encoding network can be trained based on the first loss function value generated by the difference between the restored semantic features and the second semantic features. After the initial training of the semantic encoding network is completed, its parameters are frozen (no longer updated). The remaining parts of the word extraction model, including the pixel encoding network, feature fusion network, decoding network, etc., are trained using the second loss function value generated by the difference between the restored image and the sample image (usually from the image reconstruction task). This ensures that subsequent network modules can better adapt to the stable semantic features, while focusing on the modeling and reconstruction of image details.

[0105] For example, such as Figure 5 As shown, the word extraction model includes two branches: a semantic branch based on VisionTransformer (ViT) and a pixel branch based on Convolutional Neural Network (CNN). The features of the semantic branch and the pixel branch are fused and then quantized. Based on the quantized features, the pixel decoder performs image reconstruction and calculates the reconstruction loss between the reconstructed image and the original image (sample image). At the same time, the quantized features are also semantically reconstructed by the semantic decoder and the semantic reconstruction loss between the reconstructed features and the ViT output is calculated. The training process includes two parts: (1) independent pre-training of ViT; (2) fixed training of the remaining components of ViT.

[0106] This approach combines semantic restoration loss and image reconstruction loss to jointly train the lexical extraction model, which can simultaneously improve the model's ability to understand high-level semantics of images and restore low-level visual details, thereby generating semantically consistent and structurally clear image lexical sequences.

[0107] Step 402: The pixel encoding network in the word extraction model is used to extract pixel features from the input image to obtain the first pixel features.

[0108] To further improve the fine-scale representation of images and provide foundational support for subsequent feature fusion, one possible approach is to extract low-level visual features of images using a pixel coding network.

[0109] As an example, a pixel encoding network in a lexical extraction model is used to extract low-level visual features, such as edges, colors, and textures, from the raw pixel values ​​of the input image. These low-level features help preserve the detailed information of the image and are called first pixel features.

[0110] Step 403: Fuse the first semantic feature and the first pixel feature to obtain the first fused feature.

[0111] In order to create a comprehensive feature representation that can reflect both the macroscopic meaning of an image and preserve its microscopic details, in this embodiment of the disclosure, the first semantic feature and the first pixel feature are fused together to obtain the first fused feature. The fusion can be achieved in various ways, such as a simple connection operation or a more complex interactive fusion method.

[0112] Step 404: Generate a first image word sequence based on the first fusion feature.

[0113] To improve image understanding, as a possible approach, the obtained fusion features are transformed into a series of discrete tokens. These tokens can be considered a new form of image representation. Each image token represents an abstract semantic representation of a local region in the image, and the entire image token sequence constitutes a high-level structured and semantic generalization of the entire image content. This representation not only preserves the key visual and semantic information of the image but also possesses compactness and modelability, making it suitable for multimodal tasks such as image generation and image-text fusion modeling. In other words, the first image token sequence can be used in the image processing method of any of the above embodiments.

[0114] To further achieve efficient discrete modeling of image content, as a possible approach, a multilayer perceptual network is used to reduce the dimensionality of the fused features, and a quantization network is combined to map them into discrete image word sequences.

[0115] As an example, a multilayer perceptual network in the word extraction model is used to downsample the first fused feature to obtain the first sampled feature; a quantization network in the word extraction model is used to quantize the first sampled feature to generate the first image word sequence.

[0116] In other words, a multi-layer perceptron (MLP) is used to downsample the first fused feature to compress its feature dimension, thereby obtaining the first sampled feature. This process significantly improves the compactness and computational efficiency of the feature representation while preserving the key semantic and visual information in the original fused feature. Subsequently, a quantization network maps the continuous value vector in the first sampled feature to a set of predefined discrete codebook spaces. Specifically, each sampled feature is replaced with its closest vector in the codebook, and the corresponding index value is recorded. This index value is the image word. Then, multiple image words are arranged in sequence to form the first image word sequence.

[0117] In summary, the semantic encoding network in the lexical extraction model is used to extract semantic features from the input image, thereby obtaining the first semantic feature. This feature can effectively represent high-level semantic information such as the image's subject content, object category, and scene information. Further, a pixel encoding network is used to extract pixel-level features from the input image, obtaining the first pixel feature. This feature preserves low-level visual details such as image edges, colors, and textures, which helps improve the accuracy of image reconstruction and understanding. The first semantic feature and the first pixel feature are fused to generate a first fused feature containing multi-level information. This fused feature not only inherits the high-level semantic expression of the image but also retains key visual detail information, possessing stronger expressive power and context awareness. Finally, based on the first fused feature, a first image lexical sequence is generated for subsequent image understanding or editing tasks. This lexical sequence maintains the semantic consistency of the image while also possessing good structural integrity and visual realism, and can be widely applied to image processing tasks based on the Transformer architecture.

[0118] To clearly illustrate how cross-modal semantic modeling of the input content is performed in the above embodiments to obtain a joint feature representation containing multimodal semantic information, this disclosure proposes another image processing method.

[0119] Figure 6 This is a schematic flowchart of the image processing method provided in Embodiment 5 of this disclosure.

[0120] like Figure 6 As shown, the image processing method may include the following steps:

[0121] Step 601: Obtain input content adapted to the image processing task; wherein, the image processing task includes an image generation task, and the input content includes a first text word sequence.

[0122] In this embodiment of the disclosure, when the current image processing task is an image generation task, the input content is a first text word sequence, that is, a text description or instruction provided by the user for the image to be generated.

[0123] Step 602: Based on the contextual information of the first text word sequence, perform semantic encoding on the first text word sequence to obtain the fourth text semantic feature of the text space.

[0124] In order to accurately capture user intent and enhance the controllability of image generation, as a possible approach, a semantic encoder (such as Transformer, BERT, etc.) is used to analyze the semantic relationships between text words and extract a high-dimensional vector representation, called the fourth text semantic feature, which is used to characterize the deep language semantics of the text corresponding to the first text word sequence.

[0125] Step 603: Perform cross-modal semantic modeling on the fourth text semantic features to obtain the fourth image semantic features in the image space.

[0126] In order to achieve semantic alignment between text and images and improve generation consistency, in this embodiment of the disclosure, text semantic features can be mapped to image semantic space to obtain a fourth image semantic feature, that is, a visual semantic representation that can be used for image generation.

[0127] Step 604: Generate a joint feature representation based on the fourth text semantic features and the fourth image semantic features.

[0128] To further enhance the semantic expressive power of images, in this embodiment of the disclosure, text semantic features and image semantic features are fused to form a unified joint feature representation. This joint feature representation has both the abstractness of language description and the concreteness of image structure, and is the driving factor for subsequent image generation.

[0129] Step 605: Based on the joint feature representation, generate output content adapted to the image processing task.

[0130] In order to improve the quality of the generated image and its semantic consistency with the input text, in this embodiment of the disclosure, the joint features are converted into pixel images.

[0131] As an example, based on an autoregressive mechanism, the joint feature representation is iterated multiple times to obtain a second image word sequence corresponding to at least one resolution; and a second target image is generated based on the second image word sequence corresponding to at least one resolution.

[0132] In other words, in order to improve the quality of image generation, the joint feature representation is gradually refined through autoregression to generate image word sequences (i.e., second image word sequences) at different resolutions. Each iteration further optimizes the results of the previous iteration to ensure that the final generated image not only meets the requirements of the input content but also has high quality and detail.

[0133] In the process of generating the second target image based on the second image word sequence corresponding to at least one resolution, the semantic level of any second image word sequence is determined based on the semantic features of any second image word sequence; based on the semantic level, any second image word sequence is upsampled to obtain a third image word sequence; and the third image word sequences are fused to obtain the second target image.

[0134] In other words, to improve the quality of image generation, based on the semantic features of each second image word sequence, the information type or topic (i.e., semantic features) contained in each second image word sequence is evaluated, and these information are classified into different semantic levels according to their importance or degree of abstraction. For example, lower levels may include basic colors and shapes, while higher levels involve specific objects or scenes. Based on the determined semantic level, each second image word sequence is magnified (i.e., upsampled) to increase its resolution and detail, thereby obtaining new and clearer image parts (i.e., third image word sequences). The final step is to merge all the upsampled image parts (i.e., fuse) to form a complete, high-resolution second target image.

[0135] In summary, by analyzing the contextual information in the first text word sequence, semantic encoding is performed on the first text word sequence, converting it into a fourth text semantic feature that can be expressed in the text space. This process effectively captures the core semantics and key details in the text information, ensuring an accurate understanding of the user's intent. Furthermore, the fourth text semantic feature is mapped to the image space to generate the corresponding fourth image semantic feature, thereby achieving an effective transformation from text description to image semantics and providing feasibility for the deep fusion of subsequent text and image information. Further, based on the extracted text semantic features and image semantic features, a comprehensive joint feature representation is generated. This joint feature representation not only fully preserves the key semantic information of the original text but also integrates the visual structure and style features required by the image, providing a unified and semantically consistent representation basis for high-quality image generation. Finally, output content adapted to the image processing task is generated based on this joint feature representation. The generated image can accurately reflect the description content of the input text, while achieving high standards in visual quality, detail representation, and overall composition, significantly improving the accuracy of the image generation task and the user experience.

[0136] For example, such as Figure 7 As shown, taking image generation as an example, the image processing method of this disclosure embodiment may include the following steps:

[0137] (1) The user inputs a piece of text (such as “TTT”), which is converted into a series of text tokens, i.e. the first text token sequence;

[0138] (2) These text words are processed through the Transformer layer; the Transformer layer captures the context information and core meaning in the text, performs semantic encoding on the text words, and obtains the third text semantic features; the third text semantic features are input into the image decoder, which converts these text features into image features, realizes cross-modal semantic modeling from text space to image space, and thus obtains the third image semantic features in the image space;

[0139] (3) Based on the fourth text semantic features and the fourth image semantic features, the system generates a comprehensive joint feature representation. This joint feature not only contains all the key information of the original text, but also integrates the visual characteristics that the target image should have;

[0140] (4) Based on the autoregressive mechanism, the joint feature representation is iterated in multiple rounds to obtain the second image word sequence corresponding to multiple resolutions (e.g., Scale1, Scale2 and Scale3);

[0141] (5) Based on the pixel decoder of the trained word extraction model, the second image word sequence corresponding to multiple resolutions is restored to generate the second target image.

[0142] For example, such as Figure 8 As shown, taking image processing task as an example of image understanding task, the image processing method may include the following steps:

[0143] (1) Text Input and Encoding

[0144] The user inputs a piece of text (such as "TTT"), which is converted into a series of text tokens, i.e., the first text token sequence. The first text token sequence is processed by the Transformer layer. The Transformer layer performs semantic encoding on the first text token sequence to capture the contextual information and core meaning in the text, and obtains the first text semantic features.

[0145] (2) Input an image and convert it into an image token sequence, i.e., the first image token sequence, through the ViT (Vision Transformer) and CNN (Convolutional Neural Network) in the trained word extraction model. The first image token sequence is also processed through the Transformer layer to extract the feature information of the image and obtain the first image semantic features.

[0146] (3) Input text terms and image terms together into the Transformer layer for joint modeling; it should be noted that since there is no order relationship within the image modality, a bidirectional modeling approach is adopted so that the model can better capture the relationship between the image and the text; in the Transformer layer, the first text term sequence and the first image term sequence will interact to generate a joint feature representation that integrates text and image information;

[0147] (4) Based on joint feature representation, perform operations such as image recognition and understanding, and output the text content of the image for understanding and interpretation.

[0148] With the above Figures 1 to 8 Corresponding to the image processing method provided in the embodiments, this disclosure also provides an image processing apparatus. Since the image processing apparatus provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 8 The image processing method provided in the embodiments corresponds to the image processing apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0149] Figure 9 This is a schematic diagram of the image processing apparatus provided in Embodiment Six of this disclosure.

[0150] like Figure 9 As shown, the image processing apparatus includes: an acquisition module 910, a processing module 920, and a generation module 930.

[0151] The acquisition module 910 is used to acquire input content adapted to the image processing task; wherein the input content includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence; the processing module 920 is used to perform cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information; wherein the multimodal semantic information is used to indicate the semantic association between different modalities of the input content; and the generation module 930 is used to generate output content adapted to the image processing task based on the joint feature representation.

[0152] As one possible implementation, the image processing task includes an image understanding task or an image editing task. The input includes a first text word sequence and a first image word sequence. The processing module 920 is used to perform semantic encoding on the first text word sequence based on the context information of the first text word sequence to obtain the first text semantic features in the text space; to extract features from the first image word sequence to obtain the first image semantic features in the image space; to establish a first semantic association between the first text semantic features and the first image semantic features; and to perform a fusion operation on the first text semantic features and the first image semantic features according to the first semantic association to generate a joint feature representation containing multimodal semantic information.

[0153] As one possible implementation, the generation module 930 is used to respond to image processing tasks, including image understanding tasks, by performing multiple iterations on the joint feature representation based on an autoregressive mechanism to obtain a second text word sequence; and to generate image understanding text based on the second text word sequence.

[0154] As one possible implementation, the generation module 930 is configured to, in response to an image processing task including the image editing task, perform at least one round of editing on a first image lexical sequence based on an autoregressive mechanism and according to joint feature representation, to obtain an edited image lexical sequence; and generate a first target image based on the edited image lexical sequence.

[0155] As one possible implementation, the image processing task includes an image understanding task. The input content includes a first image word sequence. The processing module 920 is used to extract features from the first image word sequence based on the context information of the first image word sequence to obtain a second image semantic feature in the image space; to perform cross-modal semantic modeling on the second image semantic feature to obtain a second text semantic feature in the text space; and to generate a joint feature representation based on the second image semantic feature and the second text semantic feature.

[0156] As one possible implementation, the first image word sequence is obtained by processing the following modules: a first extraction module, a fusion module, and a determination module.

[0157] The first extraction module is used to extract semantic features from the input image using the semantic encoding network in the word extraction model to obtain a first semantic feature; and to extract pixel features from the input image using the pixel encoding network in the word extraction model to obtain a first pixel feature; the fusion module is used to fuse the first semantic feature and the first pixel feature to obtain a first fused feature; and the determination module is used to generate a first image word sequence based on the first fused feature.

[0158] As one possible implementation, a determination module is used to downsample the first fused features using a multilayer perceptual network in the word extraction model to obtain the first sampled features; and to quantize the first sampled features using a quantization network in the word extraction model to generate the first image word sequence.

[0159] As one possible implementation, the lexical extraction model is trained using the following modules: a second extraction module, a restoration module, and a training module.

[0160] The second extraction module is used to extract semantic features from the sample image using a semantic encoding network to obtain second semantic features, and to extract pixel features from the sample image using a pixel encoding network to obtain second pixel features; the restoration module is used to perform semantic restoration and image restoration on the sample image based on the second semantic features and the second pixel features to obtain restored semantic features and restored image; the training module is used to train the word extraction model based on the difference between the restored semantic features and the second semantic features, as well as the difference between the restored image and the sample image.

[0161] As one possible implementation, the training module is used to generate a first loss function value based on the difference between the restored semantic features and the second semantic features; generate a second loss function value based on the difference between the restored image and the sample image; and train the word extraction model based on the first loss function value and the second loss function value.

[0162] As one possible implementation, a training module is used to train the semantic coding network in the image processing model based on the first loss function value; the frozen semantic coding network is used to train other networks in the word extraction model other than the semantic coding network based on the second loss function value, so as to obtain the trained word extraction model.

[0163] As one possible implementation, the lexical extraction model also includes: a feature fusion network, a semantic decoding network, and a pixel decoding network, and a restoration module, which is used to fuse the second semantic feature and the second pixel feature using the feature fusion network to obtain the second fused feature; to use the semantic decoding network to perform semantic restoration on the sample image based on the second fused feature to obtain the restored semantic feature; and to use the pixel decoding network to perform image restoration on the sample image based on the second fused feature to obtain the restored image.

[0164] As one possible implementation, the lexical extraction model also includes: a multilayer perceptron network and a quantization network, a restoration module, which is used to downsample the second fused feature using the multilayer perceptron network to obtain the second sampled feature; to quantize the second sampled feature using the quantization network to obtain the lexical sequence of the sample image; and to use a semantic decoding network to perform semantic restoration of the sample image based on the word sequence of the sample image to obtain the restored semantic feature.

[0165] As one possible implementation, the image processing task includes an image generation task or an image editing task. The input content includes an image-text fusion sequence. The processing module 920 is used to extract the third text semantic features in the text space and the third image semantic features in the image space from the image-text fusion sequence, respectively; establish a second semantic association between the third text semantic features and the third image semantic features; and fuse the third text semantic features and the third image semantic features according to the second semantic association to generate a joint feature representation containing multimodal semantic information.

[0166] As one possible implementation, the image processing task includes an image generation task. The input content includes a first text word sequence. The processing module 920 is used to perform semantic encoding on the first text word sequence based on the context information of the first text word sequence to obtain a fourth text semantic feature in the text space; to perform cross-modal semantic modeling on the fourth text semantic feature to obtain a fourth image semantic feature in the image space; and to generate a joint feature representation based on the fourth text semantic feature and the third image semantic feature.

[0167] As one possible implementation, the generation module 930 is used to perform multiple iterations on the joint feature representation based on an autoregressive mechanism to obtain a second image word sequence corresponding to at least one resolution; and to generate a second target image based on the second image word sequence corresponding to at least one resolution.

[0168] As one possible implementation, the generation module 930 is used to determine the semantic level of any second image word sequence based on the semantic features of any second image word sequence; based on the semantic level, upsample any second image word sequence to obtain a third image word sequence; and fuse each third image word sequence to obtain a second target image.

[0169] The image processing apparatus of this disclosure improves the flexibility and expressiveness of the input format by acquiring input content adapted to the image processing task, wherein the input includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence. Furthermore, cross-modal semantic modeling of the input content can deeply explore the semantic relationships between different modalities, thereby generating a joint feature representation containing multimodal semantic information. This joint feature not only preserves the semantic structure of each modality itself, but also effectively captures the alignment, complementarity, and interaction relationships between modalities, which helps to construct a unified semantic space, thereby significantly improving the model's ability in context understanding and cross-modal reasoning. Finally, output content adapted to the image processing task is generated based on the joint feature representation, ensuring that the output result is highly consistent with the input at the semantic level, further enhancing the accuracy and intelligence level of task execution.

[0170] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image processing method proposed in any of the above embodiments of this disclosure.

[0171] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the image processing method proposed in any of the above embodiments of this disclosure.

[0172] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the image processing method proposed in any of the above embodiments of this disclosure.

[0173] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0174] Figure 10 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in ROM (Read-Only Memory) 1002 or loaded from storage unit 1008 into RAM (Random Access Memory) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. An I / O (Input / Output) interface 1005 is also connected to bus 1004.

[0176] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the image processing methods described above. For example, in some embodiments, the image processing methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the above-described image processing method by any other suitable means (e.g., by means of firmware).

[0178] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0183] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0184] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0185] According to the technical solution of this disclosure, by acquiring input content adapted to the image processing task, wherein the input includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence, the flexibility and expressiveness of the input form are effectively improved. Furthermore, cross-modal semantic modeling of the input content can deeply explore the semantic relationships between different modalities, thereby generating a joint feature representation containing multimodal semantic information. This joint feature not only preserves the semantic structure of each modality itself, but also effectively captures the alignment, complementarity, and interaction relationships between modalities, which helps to construct a unified semantic space, thereby significantly improving the model's ability in context understanding and cross-modal reasoning. Finally, output content adapted to the image processing task is generated based on this joint feature representation, ensuring that the output result is highly consistent with the input at the semantic level, further enhancing the accuracy and intelligence level of task execution.

[0186] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution proposed in this disclosure can be achieved, and this is not limited herein.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: Obtain input content adapted to the image processing task; wherein, the input content includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence; Cross-modal semantic modeling is performed on the input content to obtain a joint feature representation containing multimodal semantic information; wherein, the multimodal semantic information is used to indicate the semantic association relationship between the input content in different modalities; Based on the joint feature representation, output content adapted to the image processing task is generated.

2. The method according to claim 1, wherein, The image processing task includes an image understanding task or an image editing task. The input content includes a first text word sequence and a first image word sequence. The cross-modal semantic modeling of the input content to obtain a joint feature representation containing multimodal semantic information includes: Based on the contextual information of the first text word sequence, the first text word sequence is semantically encoded to obtain the first text semantic features of the text space; Feature extraction is performed on the first image word sequence to obtain the first image semantic features in the image space; Establish a first semantic association relationship between the first text semantic features and the first image semantic features; Based on the first semantic association, the first text semantic features and the first image semantic features are fused to generate a joint feature representation containing multimodal semantic information.

3. The method according to claim 2, wherein, The step of generating output content adapted to the image processing task based on the joint feature representation includes: In response to the image processing task including the image understanding task, the joint feature representation is iterated multiple times based on an autoregressive mechanism to obtain a second text word sequence; Based on the second text word sequence, generate image-understanding text.

4. The method according to claim 2, wherein, The step of generating output content adapted to the image processing task based on the joint feature representation includes: In response to the image processing task including the image editing task, based on the autoregressive mechanism and according to the joint feature representation, at least one round of editing is performed on the first image word sequence to obtain the edited image word sequence; Based on the edited image lexical sequence, a first target image is generated.

5. The method according to claim 1, wherein, The image processing task includes an image understanding task, and the input content includes a first image word sequence. The step of performing cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information includes: Based on the contextual information of the first image word sequence, feature extraction is performed on the first image word sequence to obtain the second image semantic features in the image space; Cross-modal semantic modeling is performed on the second image semantic features to obtain the second text semantic features in the text space; The joint feature representation is generated based on the second image semantic features and the second text semantic features.

6. The method according to any one of claims 1-5, wherein, The first image word sequence is obtained using the following steps: The semantic encoding network in the word extraction model is used to extract semantic features from the input image to obtain the first semantic feature; The pixel encoding network in the lexical extraction model is used to extract pixel features from the input image to obtain the first pixel feature; The first semantic feature and the first pixel feature are fused together to obtain the first fused feature; Based on the first fusion feature, the first image word sequence is generated.

7. The method according to claim 6, wherein, The step of generating the first image word sequence based on the first fusion feature includes: The first fused feature is downsampled using the multilayer perceptron in the lexical extraction model to obtain the first sampled feature; The first sampled feature is quantized using the quantization network in the lexical extraction model to generate the first image lexical sequence.

8. The method according to claim 6, wherein, The word extraction model is trained using the following steps: The semantic coding network is used to extract semantic features from the sample image to obtain second semantic features, and the pixel coding network is used to extract pixel features from the sample image to obtain second pixel features; Based on the second semantic feature and the second pixel feature, semantic restoration and image restoration are performed on the sample image to obtain restored semantic features and restored image; The word extraction model is trained based on the differences between the restored semantic features and the second semantic features, as well as the differences between the restored image and the sample image.

9. The method according to claim 8, wherein, The step of training the lexical extraction model based on the difference between the restored semantic features and the second semantic features, and the difference between the restored image and the sample image, includes: Based on the difference between the restored semantic features and the second semantic features, a first loss function value is generated; A second loss function value is generated based on the difference between the restored image and the sample image; The word extraction model is trained based on the first loss function value and the second loss function value.

10. The method according to claim 9, wherein, The step of training the word extraction model based on the first loss function value and the second loss function value includes: Based on the first loss function value, the semantic coding network in the image processing model is trained; Freeze the semantic encoding network, and based on the second loss function value, train the other networks in the word extraction model other than the semantic encoding network to obtain the trained word extraction model.

11. The method according to claim 8, wherein, The lexical extraction model also includes: a feature fusion network, a semantic decoding network, and a pixel decoding network. The step of performing semantic restoration and image restoration on the sample image based on the second semantic feature and the second pixel feature to obtain restored semantic features and restored image includes: The feature fusion network is used to fuse the second semantic feature and the second pixel feature to obtain the second fused feature; Using the semantic decoding network, based on the second fusion feature, the sample image is semantically restored to obtain restored semantic features; Using the pixel decoding network, based on the second fusion feature, the sample image is restored to obtain the restored image.

12. The method according to claim 11, wherein, The word extraction model also includes: a multilayer perceptron network and a quantization network. The step of using the semantic decoding network to perform semantic restoration on the sample image based on the second fusion feature to obtain restored semantic features includes: The second fused feature is downsampled using the multilayer perceptron to obtain the second sampled feature; The second sampling feature is quantized using the quantization network to obtain a sequence of word terms in the sample image; Using the semantic decoding network, the semantic features of the sample image are restored based on the word sequence of the sample image.

13. The method according to claim 1, wherein, The image processing task includes an image generation task or an image editing task, the input content includes the image-text fusion sequence, and the cross-modal semantic modeling of the input content to obtain a joint feature representation containing multimodal semantic information includes: The third text semantic features in the text space and the third image semantic features in the image space are extracted from the image-text fusion sequence, respectively. A second semantic association is established between the third text semantic features and the third image semantic features; Based on the second semantic association, the third text semantic features and the third image semantic features are fused to generate a joint feature representation containing multimodal semantic information.

14. The method according to claim 1, wherein, The image processing task includes an image generation task, and the input content includes a first text word sequence. The step of performing cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information includes: Based on the contextual information of the first text word sequence, the first text word sequence is semantically encoded to obtain the fourth text semantic feature of the text space; Cross-modal semantic modeling is performed on the fourth text semantic features to obtain the fourth image semantic features in the image space; The joint feature representation is generated based on the fourth text semantic features and the fourth image semantic features.

15. The method according to claim 14, characterized in that, The step of generating output content adapted to the image processing task based on the joint feature representation includes: Based on the autoregressive mechanism, the joint feature representation is iterated multiple times to obtain at least one second image word sequence corresponding to a resolution; A second target image is generated based on the second image word sequence corresponding to the at least one resolution.

16. The method according to claim 15, characterized in that, The step of generating a second target image based on the second image word sequence corresponding to the at least one resolution includes: Based on the semantic features of any second image word sequence, determine the semantic level of the any second image word sequence; Based on the semantic hierarchy, any second image word sequence is upsampled to obtain a third image word sequence; The third image word sequences are fused to obtain the second target image.

17. An image processing apparatus, comprising: An acquisition module is used to acquire input content adapted to the image processing task; wherein, the input content includes at least one of a first text word sequence, a first image word sequence, and an image-text fusion sequence; The processing module is used to perform cross-modal semantic modeling on the input content to obtain a joint feature representation containing multimodal semantic information; wherein the multimodal semantic information is used to indicate the semantic association relationship between the input content in different modalities; The generation module is used to generate output content adapted to the image processing task based on the joint feature representation.

18. An electronic device, wherein, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-16.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-16.

20. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-16.

Images