Visual generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a visual generation method and device, equipment, a storage medium and a program product. The method includes, in response to receiving a content generation request including a first image, extracting a semantic feature representation from the first image using a trained semantic encoder, and extracting a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on the first sample image and a sample text, the sample text describing semantic information of the first sample image; generating a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation; and using the trained content generation model to determine response content to the content generation request based at least on the target feature representation.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly to methods, apparatuses, devices, computer-readable storage media, and computer program products for visual generation. Background Art

[0002] Visual generation technology has made rapid progress recently, achieving high-quality, high-resolution image and video synthesis. However, there is still a lack of effective technology in multimodal visual generation. How to make both the input and output of the model multimodal is an urgent problem to be solved. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for visual generation is provided. The method includes: in response to receiving a content generation request including a first image, extracting a semantic feature representation from the first image using a trained semantic encoder, and extracting a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, and the sample text describes the semantic information of the first sample image; generating a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation; and determining, using the trained content generation model, a response content to the content generation request based at least on the target feature representation.

[0004] In a second aspect of the present disclosure, a device for visual generation is provided. The device includes: an extraction module configured to extract a semantic feature representation from the first image using a trained semantic encoder in response to receiving a content generation request including a first image, and to extract a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, and the sample text describes the semantic information of the first sample image; a generation module configured to generate a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation; and a determination module configured to determine the response content to the content generation request based at least on the target feature representation using a trained content generation model.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A schematic diagram showing an example architecture for visual generation at the inference stage according to some embodiments of the present disclosure;

[0012] Figure 3 Some embodiments of the present disclosure are shown Figure 2 Schematic diagram of an example structure of a semantic discrete encoder in ;

[0013] Figure 4 Some embodiments of the present disclosure are shown Figure 3 Schematic diagram of the training architecture of the semantic encoder in the training phase;

[0014] Figure 5 A schematic diagram of a training architecture for a semantic discrete encoder according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A flowchart showing a method of visual generation according to some embodiments of the present disclosure is shown;

[0016] Figure 7 An exemplary structural block diagram of an apparatus for visual generation according to some embodiments of the present disclosure is shown; and

[0017] Figure 8 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0028] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, an electronic device 110 applies a visual generation model 105 to perform visual generation. The visual generation model 105 is configured to generate a target image 114. The visual generation model 105 is configured to process a text prompt word 112 input by a user to generate the target image 114. In some embodiments, the text prompt word 112 is used to guide the visual generation model 105 to generate an image of a specific object, for example, the text prompt word 112 may be "please generate an image of a flower".

[0029] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of interface for the user (such as "wearable" circuit, etc.). Visual generation model 105 can be implemented, for example, in various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0030] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0031] With the development of visual generation technology, a unified model for visual generation and understanding has begun to be explored. The architecture of most multimodal understanding models usually includes a visual encoder and a language model, which is called a visual language model (VLM). In order to enhance the visual perception ability of the visual language model, a visual model is usually used as a visual encoder. The visual features extracted from these pre-trained encoders have been aligned with the text features and focus on extracting high-level semantic information. In this way, it becomes more direct and effective to align the visual features of the pre-trained image encoder with the input embedding space of the language model compared to the unaligned features in the multimodal model.

[0032] However, current visual word segmenters usually only consider low-level information, which makes it difficult to align with image semantic features. This leads to high training complexity and requires a large amount of training data to achieve optimal performance. In addition, their performance is still far lower than that of specialized understanding models.

[0033] When building a language model (LM), it is common practice to use a text tokenizer to discretize text data into a series of discrete text feature units (which can be represented by the ID of the corresponding text feature unit in the vocabulary). Subsequently, a Decoder-only Transformer (a specific form of the Transformer architecture) can be used to perform prediction training for the next text token (the basic unit of text, token, also referred to as a feature unit in this article). After extensive training on large-scale datasets, the model has acquired the ability to predict the next appropriate text token based on the input text prompt.

[0034] Current image quantization methods fail to fully consider the semantic information of the image during the discretization process. These methods often capture too much low-level information, making language models ineffective for multimodal understanding tasks. In addition, there are significant differences between the representations required for multimodal understanding and generation tasks. There have been research works using these quantization methods to extend language models to visual language models (VLMs), but they face difficulties in effectively aligning the basic units (tokens) of visual data (such as images, videos, etc.) with the text words of the language model. This misalignment leads to high training complexity and has a negative impact on visual understanding performance.

[0035] In order to solve at least some of the above problems, in an embodiment of the present disclosure, a visual generation scheme is proposed. In this scheme, in response to receiving a content generation request including a first image, a semantic feature representation is extracted from the first image using a trained semantic encoder, and a visual feature representation is extracted from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, and the sample text describes the semantic information of the first sample image. By aggregating the semantic feature representation and the visual feature representation, a target feature representation corresponding to the first image is generated. Furthermore, using the trained content generation model, the response content to the content generation request is determined based at least on the target feature representation.

[0036] According to the scheme disclosed in the present invention, semantic discrete coding is implemented using a trained semantic encoder and a trained visual encoder. The semantic discrete coding method can be used for image reconstruction and generation, so that semantic information is taken into account in the image discretization process, meeting the requirements of visual understanding and generation tasks. The image is discretized into visual coding through the effective integration of semantic features. This method allows seamless adaptation to any pre-trained language model without modifying the model structure, thereby facilitating the joint training of visual understanding and generation tasks. By adding semantic constraints to the visual encoder, the information of the visual feature unit and the language feature unit is effectively aligned. This greatly reduces the difficulty of training and improves the performance of the unified model. Such a model realizes the integration of visual understanding and generation tasks in a unified multimodal model, which can outperform a dedicated understanding model in performance.

[0037] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0038] Figure 2 FIG. 2 is a schematic diagram of an example architecture 200 for visual generation in the inference phase according to some embodiments of the present disclosure. For ease of discussion, reference will be made to Figure 1 These embodiments are described with reference to the environment 100 of FIG. Figure 1 The architecture 200 may be implemented in an electronic device 110. Figure 1 environment 100. Figure 2 As shown, the architecture 200 includes at least a content generation model 210 and a semantic discrete encoder 220. The content generation model 210 and the semantic discrete encoder 220 may be part of the visual generation model 105 in the environment 100.

[0039] In some embodiments, the content generation model 210 can be used at least for visual content generation of images, videos, etc. The content generation model 210 can be configured to receive discrete feature units of input images and texts, and output corresponding image response feature units and text response feature units. As an example, the content generation model 210 can be constructed using at least a Transformer model structure. It should be understood that the content generation model 210 can also be constructed using any other appropriate one or more machine learning model structures that can at least be used for visual content generation. For the semantic discrete encoder 220, the semantic discrete encoder 220 can be combined with the semantic discrete encoder 220 to generate the semantic discrete encoder 220. Figure 3 Let’s discuss its structure and working methods in detail.

[0040] Figure 3 Some embodiments of the present disclosure are shown Figure 2 Schematic diagram of an example structure of the semantic discrete encoder 220 in FIG. Figure 3 The structure of is used to describe the reasoning stage of the semantic discrete encoder 220. Figure 3 As shown, the semantic discrete encoder 220 includes at least a semantic encoder 310 and a visual encoder 320 .

[0041] refer to Figure 2 or Figure 3 , the electronic device 110 may receive a content generation request including the first image 202. Here, the content generation request includes at least a request for image generation such as the first image 202. It should be understood that the content generation request may also include a request for video generation having multiple frames of images. In some cases, the content generation request may also include a request for text generation such as the text description 204.

[0042] refer to Figure 3 , if a content generation request including the first image 202 is received, the electronic device 110 extracts a semantic feature representation 315 from the first image 202 using the trained semantic encoder 310, and extracts a visual feature representation 325 from the first image 202 using the trained visual encoder 320. The semantic feature representation 315 indicates semantic information of the first image 202. The visual feature representation 325 indicates visual information of the first image 202. The semantic encoder 310 is trained based on the first sample image and the sample text, and the sample text describes the semantic information of the first sample image. Figure 4 Let's discuss in detail how the semantic encoder 310 is trained to extract the semantic feature representation 315 from the first image 202.

[0043] Furthermore, the electronic device 110 generates a target feature representation 225 corresponding to the first image 202 by aggregating the semantic feature representation 315 and the visual feature representation 325. The target feature representation 225 may be Figure 2 The sequence of visual feature units (or visual word units) output by the semantic discrete encoder 220 in .

[0044] In some embodiments, when generating the target feature representation 225 corresponding to the first image 202, the electronic device 110 may aggregate the semantic feature representation 315 and the visual feature representation 325 to obtain a fused feature representation 330. Thus, through feature fusion, the fused feature representation 330 may indicate the semantic information and visual information of the first image 202.

[0045] In some embodiments, the electronic device 110 may generate a plurality of discrete feature units by performing vector quantization on the fused feature representation 330. Figure 2 or Figure 3As shown, the target feature representation 225 may include multiple discrete feature units. In such an embodiment, the fused feature representation 330 may be subjected to feature extraction via MLP 340 (Multilayer Perceptron), and then vector quantization is performed by vector quantizer (VQ) 350 to generate multiple discrete feature units. MLP 340 can automatically extract useful features from the fused feature representation 330 to help identify visual elements and scene understanding in the first image 202. Vector quantizer 350 can be a tool for signal processing and data compression. Specifically, it can be used to divide the input vector space into several regions, each region is represented by a representative vector (codeword), and when a new vector is input, it is assigned to the region where the most similar representative vector is located.

[0046] In order to ensure that the discrete codes generated by the semantic discrete encoder 220 contain semantic information and are more closely aligned with the language feature units, a semantic decoder and a semantic loss are introduced for training the semantic discrete encoder 220. The training process of the semantic discrete encoder 220 will be referred to below. Figure 5 Discuss in detail.

[0047] Assume that a discrete codebook is used from learning The feature representation z q To represent the image x (such as the first image 202). First, the image is encoded as Then, by performing element-wise quantization q(·) on each spatial feature z, we obtain the quantized vector z q , which is quantized based on its most recent codebook entry. Finally, the decoder obtains the quantized vector z q Reconstruct the image. The training objectives include image reconstruction loss, vector quantization (VQ) loss, discriminator loss, and perceptual loss.

[0048] Specifically, for an image (such as the first image 202) Where H and W represent the height and width dimensions respectively, the semantic encoder 310 generates a feature representation z=Enc(x). Here, z∈R h×w×d , where h and w are the downsampled sizes and d is the dimension of the codebook vector. Subsequently, the feature representation z is converted into a quantized code q∈Z through a quantization operation h×w and quantized embedding z q ∈R h×w×d .

[0049] In the embodiment of the present disclosure, the semantic encoder 310 can generate a semantic feature representation T corresponding to the input image. The model can be trained on a large number of image-text pair datasets and effectively aligned with the language. The training process of the semantic encoder 310 will be referred to in Figure 4 In order to further enhance the semantics of discrete coding, the semantic feature representation T is fused with the visual feature representation z during the encoding process, and then the combined feature representation is quantized, as shown below:

[0050] z′=Quant(T+z)

[0051] refer to Figure 2 Further, the electronic device 110 uses the trained content generation model 210 to determine the response content for the content generation request based on at least the target feature representation 225. In some embodiments, the response content for the content generation request may include, for example, the image content of the second image 206 and / or the text content such as the response text 208. It should be understood that the response content includes the image content and / or the text content may be dependent on the content generation request.

[0052] In some embodiments, the content generation request may further include a text description 204. When determining the response content, the electronic device 110 may utilize the trained text encoder 230 to extract a text feature representation 235 from the text description 204. The text feature representation 235 may include a plurality of discrete text feature units.

[0053] Furthermore, in some embodiments, the electronic device 110 may generate response content using the content generation model 210 based at least on the target feature representation 225 corresponding to the first image 202 and the text feature representation 235 corresponding to the text description 204. Both the target feature representation 225 and the text feature representation 235 may be part of the input of the content generation model 210.

[0054] by Figure 2Take an example to illustrate generating image content and / or text content for different content generation requests. Assuming that a first image 202 and a text description 204 are given, and the specific content of the text description 204 is "A flamingo is standing on the grass.", then according to the text prompt word 112 input by the user, a response text 208 with the content "It is searching for food in the pond and walking towards the nest with three eggs inside." and a second image 206 corresponding to the response text can be obtained.

[0055] It should be understood that Figure 2 The specific contents of the first image 202 and the text description 204 included in the content generation request shown in the figure are only examples, which are illustrated for the purpose of facilitating understanding, and are not intended to be limiting. Accordingly, the specific contents of the second image 206 and / or the response text 208 included in the response content are also only examples, which are illustrated for the purpose of facilitating understanding, and are not intended to be limiting.

[0056] In some embodiments, when determining the response content, the electronic device 110 may utilize the trained content generation model 210 to generate a response sequence based at least on the target feature representation 225, and the response sequence may include multiple response feature units. If the multiple response feature units include a start unit 252 and an end unit 259 indicating a visual feature unit, the electronic device 110 may extract a set of response feature units 255 between the start unit 252 and the end unit 259. In such an embodiment, the start unit 252 and the end unit 259 may be a start symbol and an end symbol specifically for the response feature unit in the image content being responded to. As an example, the start unit 252 may be represented by an SOI (Start of Image) identifier, and correspondingly, the end unit 259 may be represented by an EOI (End of Image) identifier.

[0057] Furthermore, in some embodiments, the electronic device 110 can use the trained visual decoder 250 to decode the second image 206 from a set of response feature units 255 as at least a part of the response content. In other words, by identifying the start unit 252 and the end unit 259 in the response sequence generated by the content generation model 210, a set of response feature units 255 that can decode the image content can be determined.

[0058] In some embodiments, when determining the response content, if the response sequence includes at least one additional response feature unit 245 (which may be referred to as text response feature unit 245 for ease of discussion) in addition to a set of response feature units 255, the electronic device 110 may utilize the trained text decoder 240 to decode the response text 208 from the at least one additional response feature unit 245 as at least a part of the response content.

[0059] In such an embodiment, by identifying the response characteristic units other than the start unit 252 and the end unit 259 , at least one additional response characteristic unit 245 other than the group of response characteristic units 255 can be determined.

[0060] In addition, in some embodiments, the response sequence may further include a sequence end unit 219. As an example, the sequence end unit 219 may be represented by an EOS (End of Sequence) identifier.

[0061] It should be noted that the content generation model 210 may be an autoregressive model. When determining the response content, the content generation model 210 may generate each response feature unit in sequence. For the currently generated response feature unit, it will regress to the input of the content generation model 210 to generate the next response feature unit. Figure 2 For example, the text response feature unit 245, the start unit 252, a group of response feature units 255 and the end unit 259 generated in sequence by the content generation model 210 can be sequentially returned to the input of the content generation model 210 as a partial input response sequence 215.

[0062] Therefore, according to the working mode of the content generation model 210 , a group of response feature units 255 and at least one additional response feature unit 245 can be determined through the start unit 252 and the end unit 259 .

[0063] Thus, the image is preprocessed into visual feature units {q1, q2, ..., qi} of length (h×w) by the semantic discrete encoder 220, while the text data is converted by the text encoder 230. In order to achieve joint modeling of language and vision, the embedding layer of the existing language model is extended to include the newly added visual feature unit ID. This modification enables multimodal input to be seamlessly integrated in the model architecture. In order to distinguish the visual feature units, two special units are added <soi>and <eoi>, marking the beginning and end of the visual feature unit respectively.

[0064] The content generation model 210 of the disclosed embodiment can be a simple and unified autoregressive transformer, applicable to both visual and language modalities. In this model, visual and language data can be input and output in the same discrete encoding format. For the language modality, through the text encoder 230 and the language model, these models are extensively trained on a large amount of text data. The semantic discrete encoding proposed in this paper serves as a word segmenter for the visual modality to generate visual feature units that are well aligned with the language. Based on this, a unified visual-language model MUSE-VL (which may include a semantic discrete encoder 220, a text encoder 230, a visual decoder 250, and a text decoder 240) is proposed, which can handle models of mixed visual and language tags and support visual understanding and generation tasks. A unified multimodal model that can generate and reason about mixed text and image content sequences in arbitrary interleaved configurations.

[0065] As mentioned above, in the embodiment of the present disclosure, the semantic encoder 310 is trained to be able to extract feature representations that can characterize the semantic information of the image from the image. Figure 4 Some embodiments of the present disclosure are shown Figure 3 Schematic diagram of the training architecture 400 of the semantic encoder 310 in the process 300. The training architecture 400 can be implemented at the electronic device 110 or can be implemented outside of a separate device / system. The trained semantic encoder 310 obtained by training the architecture 400 can be used in the semantic discrete encoder 220 of the process 300 and the architecture 200.

[0066] The training data of the semantic encoder 310 includes a sample image 402 (sometimes referred to herein as a “first sample image”) and a sample text 404 corresponding to the sample image 402, and the sample text 404 describes the semantic information in the sample image 402. For example, if the sample image includes an image corresponding to a bird foraging, the sample text may include “a bird foraging”.

[0067] During the training process of the semantic encoder 310, a trained text encoder 410 is also introduced to assist the training of the semantic encoder 310. The text encoder 410 can be Figure 2 The text encoder 410 may be the same as the text encoder 230 in , or may be any other suitable text encoder. The text encoder 410 has been trained to extract text feature representations that can accurately represent the text semantics from the input text.

[0068] During training, the semantic encoder 310 to be trained is used to extract the predicted semantic feature representation 425 from the sample image 402, and the trained text encoder 410 is used to extract the sample text feature representation 415 from the sample text 404. The training goal of the semantic encoder 310 is to make the semantic feature representation output by the semantic encoder 310 as close as possible to the text feature representation output by the text encoder 410. At this time, the text feature representation output by the text encoder 410 can be considered as the true value of the output of the semantic encoder 310. Therefore, during training, the training loss 430 can be defined based on the difference between the predicted semantic feature representation 425 and the sample text feature representation 415. The difference between the predicted semantic feature representation 425 and the sample text feature representation 415 can be based on the cross entropy loss or any other loss function for judging the difference between two feature representations. The semantic encoder 310 can be trained based on the difference between the predicted semantic feature representation 425 and the sample text feature representation 415.

[0069] The training goal of the semantic encoder 310 can be to minimize or reduce the difference between the predicted semantic feature representation 425 output by the semantic encoder 310 and the sample text feature representation 415 output by the text encoder 410 to meet the predetermined goal. In this case, it can be considered that the semantic feature representation output by the semantic encoder 310 can indicate the semantic information of the input image. In the training of the semantic encoder 310, a large number of training pairs consisting of sample images and sample texts can be used to train the semantic encoder 310. During the training process, the model parameter values ​​of the semantic encoder 310 are continuously updated until the training goal is met. In the training of the semantic encoder 310, the model parameter values ​​of the text encoder 410 can remain unchanged, because the introduction of the text encoder 410 is mainly to provide the true value of the output of the semantic encoder 310.

[0070] After the semantic encoder 310 is obtained through training, other components (including the visual encoder 320) in the semantic discrete encoder 220 can be further trained to determine the parameter values ​​of these components. During the training process of the visual encoder 320, the model parameters of the semantic encoder 310 remain unchanged. Figure 5 A schematic diagram of a training architecture 500 for a semantic discrete encoder 220 according to some embodiments of the present disclosure is shown. The training architecture 500 may be implemented at the electronic device 110 or may be implemented outside of an independent device / system. The trained semantic discrete encoder 220 and the trained visual decoder 250 obtained by training the architecture 500 may be used in the architecture 200.

[0071] In some embodiments, the semantic decoder 510 and / or the visual decoder 250 may be introduced during the training process of the visual encoder 320 for joint training. The training data of the visual encoder 320 includes a plurality of sample images.

[0072] During joint training, the trained semantic encoder 310 may be used to extract a semantic feature representation 515 from the sample image 505 (sometimes also referred to as the second sample image), which is represented as T. The visual encoder 320 to be trained extracts a predicted visual feature representation 525 from the sample image 505. In the semantic discrete encoder 220, the predicted semantic feature representation 515 and the predicted visual feature representation 525 may be fused to generate a predicted target feature representation 535 corresponding to the sample image 505, which is represented as z. q . Predict the target feature representation z q The generation process of 535 follows the above reference Figure 3 Specifically, the predicted semantic feature representation 515 and the predicted visual feature representation 525 are fused to obtain a fused feature representation 530. The fused feature representation 530 is processed by the MLP 340 and then quantized by the vector quantizer 350 to obtain a plurality of discrete feature units, which are used to form the predicted target feature representation z. q 535.

[0073] In some embodiments, a semantic decoder 545 is used to predict the target feature representation z q 535 decodes the predicted semantic feature representation 545. The semantic decoder 545 can be represented as Dec s , represents the predicted target feature z q The decoding process of 535 can be represented as Dec s (z q ).

[0074] The electronic device 110 may represent z based on at least the predicted semantic feature q The difference between 535 and the semantic feature representation 545 decoded by the semantic encoder 310 is s (z q ) (also called semantic loss 510) to train the visual encoder 320 and the semantic decoder 510. That is, during the training process, the model parameters of the visual encoder 320, the visual decoder 250 and the semantic decoder 510 can be updated simultaneously. The semantic loss 510 can be expressed as the following loss function L sem :

[0075] L sem =1-cos(z s , T) = 1-cos(Dec s (z q ),T)

[0076] where z s represents the semantic feature representation obtained by decoding by the semantic decoder 510, that is, z s =Dec s (z q ). cos() identifies the cosine similarity between two feature representations. The above formula indicates that by minimizing the loss function L sem , which can ensure Dec s (z q ) and the semantic feature representation 515T. Through the loss function L sem , the model parameter values ​​of the visual encoder 320 (and the model parameter values ​​in the discrete semantic encoder 220) can be constrained so that the processing of the visual features can be updated to align with the semantic feature representation as much as possible. In some embodiments, although the semantic decoder 510 is also updated during the training process, the semantic decoder 510 may not be applied to the architecture 200 after the training is completed.

[0077] In some embodiments, in order to ensure that the visual generation capability is maintained after visual encoding, a separate visual decoder 250 is also used to reconstruct the image based on discrete visual feature units. The visual encoder 320 is also jointly trained with the visual decoder 250. During joint training, the predicted image 555 can be decoded from the predicted target feature representation 535 using the visual decoder 250 to be trained. In this way, in addition to the semantic loss 510, the visual encoder 320, the visual decoder 250 and the semantic decoder 510 are jointly trained based on the difference between the predicted image 555 and the sample image 505. The difference between the predicted image 555 and the sample image 505 can be represented as a reconstruction loss 520. The reconstruction loss 520 is used to measure the difference between the two images.

[0078] In some embodiments, the reconstruction loss 520 may include, but is not limited to, the following loss function:

[0079]

[0080] Wherein, l2 is the L2 reconstruction loss between the predicted image 555 and the sample image 505 (which can be the difference at the pixel level of the image), LP(·) refers to the perceptual loss between the predicted image 555 and the sample image 505, and LG(·) is the adversarial loss. z represents the sample image, the second term of Lvq is the commitment loss, and β is its weight. It can be understood that the above instruction gives an example loss function for measuring the difference between two images, and any other type of loss function can actually be configured as needed.

[0081] In some embodiments, the visual encoder 320, the visual decoder 250, the semantic decoder 510, etc. may be trained together based on the semantic loss 510 and the reconstruction loss 520. The overall loss function may be expressed as follows:

[0082] L total =L sem +L img +L vq

[0083] The training objectives of the visual encoder 320, the visual decoder 250, and the semantic decoder 510 are to make the overall loss function L total Minimize or reduce to the expected target. In training, a large number of sample images can be used to train the visual encoder 320, the visual decoder 250 and the semantic decoder 510. During the training process, the model parameter values ​​of the visual encoder 320, the visual decoder 250 and the semantic decoder 510 are continuously updated until the training target is met. The model parameter values ​​of the semantic encoder 310 can remain unchanged.

[0084] The above describes the training process of the semantic encoder 310 , the visual encoder 320 , and the visual decoder 250 .

[0085] Through the embodiments of the present disclosure, the visual understanding ability of the unified multimodal model is enhanced by using semantic discrete coding, which is a semantic-aware visual word segmenter based on VQ that can discretize images into visual coding. Semantic information is taken into account in the image discretization process to meet the requirements of visual understanding and generation tasks. Based on the semantic-aware visual word segmenter, MUSE-VL is introduced. MUSE-VL is pre-trained on diverse multimodal data, including image-text pairs, video-text pairs, interleaved image-text data, and plain text data. The model is then fine-tuned using high-quality multimodal instruction following data. The training goal is to predict the next tag using the standard cross entropy loss.

[0086] Return to reference Figure 2 For the overall architecture 200, the training target of the content generation model 210 can be an autoregressive task. In some embodiments, the training of the content generation model 210 can be performed after the semantic discrete encoder 220, the text encoder 230, the text decoder 240, and the visual decoder 250 are completed. The training of the content generation model 210 can include a pre-training phase and an instruction tuning phase.

[0087] In the pre-training stage, a diverse dataset can be used as a pre-training dataset, which includes only images, only text, and pairs of images and text, for forming model inputs. In the pre-training stage, the loss of all feature units (text or visual feature units) generated by the content generation model 210 to be trained for the input can be calculated to optimize the model parameters using the next feature unit prediction target. The main goal of this stage is to allow the content generation model 210 to effectively learn a robust feature representation of the visual feature unit, align it with the text feature unit, and establish the model's ability to accurately predict image feature generation.

[0088] In the instruction tuning stage, for image understanding tasks, the input is usually an image and text, and the output is to express the understanding of the input image in text form. The training of the content generation model 210 is to use the visual instruction tuning data and the image description. These data are organized in the following format, where the input of the content generation model 210 includes the visual feature unit of the image object and the text feature unit corresponding to the text, and the real output of the real label is the response text.

[0089] Such an input-output sample pair is represented as:

[0090] Model input: {text feature unit} <soi>{Visual Feature Unit} <eoi>

[0091] The actual output of the tag: {response}

[0092] For the image understanding task, the training of the content generation model 210 can be based on the calculation of the loss between the output predicted by the model and the true output of the true label.

[0093] For image generation tasks, the input is usually a text description of the image to be generated, and the output is the image generated based on the text description. Therefore, in the training samples, the order of the image and text in the input is reversed, allowing the content generation model 210 to generate visual feature units based on the description. Such an input-output sample pair is represented as:

[0094] Model input: {system text} {text describing the image}

[0095] Target: <soi>{Visual Feature Unit} <eoi>

[0096] In some embodiments, the system text is randomly extracted from a set of image generation instructions, such as "please generate an image", "show me a photo", etc. In the inference stage, the user provides a prompt to generate an image, and the content generation model 210 can predict the corresponding visual feature unit. Then, the predicted visual feature unit is converted into a generated image through the image decoder 250.

[0097] According to the scheme disclosed in the present invention, semantic discrete coding is implemented using a trained semantic encoder and a trained visual encoder. The semantic discrete coding method can be used for image reconstruction and generation, so that semantic information is taken into account in the image discretization process, meeting the requirements of visual understanding and generation tasks. The image is discretized into visual coding through the effective integration of semantic features. This method allows seamless adaptation to any pre-trained language model without modifying the model structure, thereby facilitating the joint training of visual understanding and generation tasks. By adding semantic constraints to the visual encoder, the information of the visual feature unit and the language feature unit is effectively aligned. This greatly reduces the difficulty of training and improves the performance of the unified model. Such a model realizes the integration of visual understanding and generation tasks in a unified multimodal model, which can outperform a dedicated understanding model in performance.

[0098] Figure 6 600 according to some embodiments of the present disclosure. The method 600 may be implemented in Figure 1 The computing device 110 will refer to Figure 1 The method 600 is described with reference to the environment 100 of FIG.

[0099] In box 610, in response to receiving a content generation request including a first image, a semantic feature representation is extracted from the first image using a trained semantic encoder, and a visual feature representation is extracted from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, and the sample text describes the semantic information of the first sample image.

[0100] In block 620 , a target feature representation corresponding to the first image is generated by aggregating the semantic feature representation and the visual feature representation.

[0101] At block 630 , a response content to the content generation request is determined based on at least the target feature representation using the trained content generation model.

[0102] In some embodiments, generating a target feature representation corresponding to the first image includes: aggregating a semantic feature representation and a visual feature representation to obtain a fused feature representation; generating multiple discrete feature units by performing vector quantization on the fused feature representation, and the target feature representation includes multiple discrete feature units.

[0103] In some embodiments, the semantic encoder is trained by: extracting a predicted semantic feature representation from a first sample image using a semantic encoder to be trained; extracting a sample text feature representation from a sample text using a trained text encoder; and training the semantic encoder based on the difference between the predicted semantic feature representation and the sample text feature representation.

[0104] In some embodiments, during the training of the visual encoder, the model parameters of the semantic encoder remain unchanged.

[0105] In some embodiments, the visual encoder and the semantic decoder are jointly trained, wherein the joint training includes: extracting a second semantic feature representation from a second sample image using the trained semantic encoder, and extracting a predicted visual feature representation from the second sample image using the visual encoder to be trained; generating a predicted target feature representation corresponding to the second sample image by fusing the second semantic feature representation and the predicted visual feature representation; decoding a predicted semantic feature representation from the predicted target feature representation using the semantic decoder to be trained; and training the visual encoder and the semantic decoder based at least on the difference between the predicted semantic feature representation and the second semantic feature representation extracted by the semantic encoder.

[0106] In some embodiments, the visual encoder is also jointly trained with the visual decoder, wherein the joint training further includes: decoding a predicted image from a predicted target feature representation using the visual decoder to be trained; and also training the visual encoder, the visual decoder, and the semantic decoder based on the difference between the predicted image and a second sample image.

[0107] In some embodiments, determining the response content includes: using a trained content generation model to generate a response sequence based at least on a target feature representation, the response sequence including a plurality of response feature units; in response to the plurality of response feature units including a start unit and an end unit indicating a visual feature unit, extracting a group of response feature units between the start unit and the end unit; and using a trained visual decoder to decode a second image from the group of response feature units as at least a part of the response content.

[0108] In some embodiments, determining the response content includes: in response to the response sequence including at least one additional response feature unit other than a set of response feature units, using a trained text decoder to decode a response text from the at least one additional response feature unit as at least a part of the response content.

[0109] In some embodiments, the content generation request also includes a text description, and determining the response content includes: extracting a text feature representation from the text description using a trained text encoder; and generating the response content using a content generation model based at least on a target feature representation corresponding to the first image and a text feature representation corresponding to the text description.

[0110] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 7 An exemplary structural block diagram of an apparatus 700 for visual generation according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0111] like Figure 7 As shown, the device 700 includes an extraction module 710, which is configured to extract a semantic feature representation from the first image using a trained semantic encoder in response to receiving a content generation request including a first image, and to extract a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, and the sample text describes the semantic information of the first sample image. The device 700 also includes a generation module 720, which is configured to generate a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation. The device 700 also includes a determination module 730, which is configured to determine the response content to the content generation request based on at least the target feature representation using a trained content generation model.

[0112] In some embodiments, the generation module 720 is further configured to aggregate the semantic feature representation and the visual feature representation to obtain a fused feature representation; generate multiple discrete feature units by performing vector quantization on the fused feature representation, and the target feature representation includes multiple discrete feature units.

[0113] In some embodiments, the device 700 also includes a training module configured to extract a predicted semantic feature representation from a first sample image using a semantic encoder to be trained; extract a sample text feature representation from a sample text using a trained text encoder; and train the semantic encoder based on the difference between the predicted semantic feature representation and the sample text feature representation.

[0114] In some embodiments, during the training of the visual encoder, the model parameters of the semantic encoder remain unchanged.

[0115] In some embodiments, the visual encoder and the semantic decoder are jointly trained, and the device 700 also includes a joint training module, which is configured to extract a second semantic feature representation from the second sample image using the trained semantic encoder, and extract a predicted visual feature representation from the second sample image using the visual encoder to be trained; generate a predicted target feature representation corresponding to the second sample image by fusing the second semantic feature representation and the predicted visual feature representation; decode the predicted semantic feature representation from the predicted target feature representation using the semantic decoder to be trained; and train the visual encoder and the semantic decoder based at least on the difference between the predicted semantic feature representation and the second semantic feature representation extracted by the semantic encoder.

[0116] In some embodiments, the visual encoder is also jointly trained with the visual decoder, and the joint training module is further configured to decode a predicted image from the predicted target feature representation using the visual decoder to be trained; and to train the visual encoder, the visual decoder, and the semantic decoder based on the difference between the predicted image and the second sample image.

[0117] In some embodiments, the determination module 730 is further configured to generate a response sequence based at least on the target feature representation using a trained content generation model, the response sequence comprising a plurality of response feature units; in response to the plurality of response feature units comprising a start unit and an end unit indicating a visual feature unit, extract a group of response feature units between the start unit and the end unit; and, using a trained visual decoder, decode a second image from the group of response feature units as at least a part of the response content.

[0118] In some embodiments, the determination module 730 is further configured to, in response to the response sequence including at least one additional response feature unit in addition to the set of response feature units, decode the response text from the at least one additional response feature unit using a trained text decoder as at least a part of the response content.

[0119] In some embodiments, the content generation request also includes a text description, and the determination module 730 is further configured to extract a text feature representation from the text description using a trained text encoder; and generate response content using a content generation model based at least on the target feature representation corresponding to the first image and the text feature representation corresponding to the text description.

[0120] The units and / or modules included in the device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 700 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 The computing device 110 in FIG.

[0122] Figure 8 8 is a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 Electronic device 110 or Figure 7 Device 700.

[0123] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0124] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0125] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0126] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0127] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0128] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0129] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0130] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0131] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0132] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0133] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.< / eoi> < / soi> < / eoi> < / soi> < / eoi> < / soi>

Claims

1. A method for visual generation, comprising: In response to receiving a content generation request including a first image, extracting a semantic feature representation from the first image using a trained semantic encoder and extracting a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and sample text, the sample text describing semantic information of the first sample image; Generate a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation; as well as The trained content generation model is used to determine response content to the content generation request based at least on the target feature representation.

2. The method according to claim 1, wherein generating a target feature representation corresponding to the first image comprises: Aggregating the semantic feature representation and the visual feature representation to obtain a fused feature representation; A plurality of discrete feature units are generated by performing vector quantization on the fused feature representation, and the target feature representation includes the plurality of discrete feature units.

3. The method according to claim 1, wherein the semantic encoder is trained by: extracting a predicted semantic feature representation from the first sample image using the semantic encoder to be trained; Extracting sample text feature representation from the sample text using the trained text encoder; as well as The semantic encoder is trained based on the difference between the predicted semantic feature representation and the sample text feature representation.

4. The method according to claim 1, wherein during the training process of the visual encoder, the model parameters of the semantic encoder remain unchanged.

5. The method of claim 1, wherein the visual encoder is jointly trained with a semantic decoder, wherein the joint training comprises: Extracting a second semantic feature representation from a second sample image using the trained semantic encoder, and extracting a predicted visual feature representation from the second sample image using the visual encoder to be trained; Generate a predicted target feature representation corresponding to the second sample image by fusing the second semantic feature representation and the predicted visual feature representation; Decoding a predicted semantic feature representation from the predicted target feature representation using a semantic decoder to be trained; as well as The visual encoder and the semantic decoder are trained based at least on a difference between the predicted semantic feature representation and the second semantic feature representation extracted by the semantic encoder.

6. The method of claim 5, wherein the visual encoder is further jointly trained with a visual decoder, wherein the joint training further comprises: Decoding a predicted image from the predicted target feature representation using a visual decoder to be trained; as well as The visual encoder, the visual decoder, and the semantic decoder are also trained based on a difference between the predicted image and the second sample image.

7. The method of claim 1, wherein determining the response content comprises: generating a response sequence based at least on the target feature representation using the trained content generation model, the response sequence comprising a plurality of response feature units; In response to the plurality of response feature units including a start unit and an end unit indicating a visual feature unit, extracting a group of response feature units between the start unit and the end unit; as well as Using the trained visual decoder, a second image is decoded from the set of response feature units to serve as at least a part of the response content.

8. The method of claim 7, wherein determining the response content comprises: In response to the response sequence including at least one additional response feature unit other than the set of response feature units, a response text is decoded from the at least one additional response feature unit using a trained text decoder as at least a part of the response content.

9. The method of claim 1, wherein the content generation request further includes a text description, and wherein determining the response content comprises: Extracting text feature representation from the text description using a trained text encoder; The response content is generated by using the content generation model based at least on the target feature representation corresponding to the first image and the text feature representation corresponding to the text description.

10. A device for visual generation, comprising: an extraction module configured to, in response to receiving a content generation request including a first image, extract a semantic feature representation from the first image using a trained semantic encoder, and extract a visual feature representation from the first image using a trained visual encoder, wherein the semantic feature representation indicates semantic information of the first image and the visual feature representation indicates visual information of the first image, and wherein the semantic encoder is trained based on a first sample image and a sample text, the sample text describing the semantic information of the first sample image; A generating module, configured to generate a target feature representation corresponding to the first image by aggregating the semantic feature representation and the visual feature representation; as well as The determination module is configured to determine the response content to the content generation request based on at least the target feature representation by using the trained content generation model.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images