Image processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, equipment and a storage medium. The method includes receiving a text input for an initial image, the text input describing a visual effect for the initial image. Based on the text input and the initial image, a fused feature is generated for the text input and the initial image. And generating a target image corresponding to the initial image based on the first image feature of the initial image and the fusion feature, the target image having visual elements related to the visual effect. The fusion of the text and the image can better express an expected visual effect. In this way, the quality of the generated image may be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatuses, devices, and computer-readable storage media for image processing. Background Technology

[0002] In the field of computer vision (CV), various machine learning-based image processing techniques have seen significant development and widespread application. For example, in many application scenarios such as social media, gaming, and image editing, there is a desire to generate and use images with certain visual effects (e.g., special effects, filters). Machine learning-based image processing techniques can be used in such applications to improve the user experience. In some example application scenarios, the goal is to generate images that match the user's input, such as textual descriptions. Summary of the Invention

[0003] In a first aspect of this disclosure, an image processing method is provided. The method includes: receiving text input for an initial image, the text input describing a visual effect for the initial image; generating fusion features for the text input and the initial image based on the text input and the initial image; and generating a target image corresponding to the initial image based on a first image feature and the fusion features, the target image having visual elements related to the visual effect.

[0004] In a second aspect of this disclosure, an apparatus for image processing is provided. The apparatus includes: a receiving module configured to receive text input for an initial image, the text input describing a visual effect for the initial image; a first generating module configured to generate fusion features for the text input and the initial image based on the text input and the initial image; and a second generating module configured to generate a target image corresponding to the initial image based on a first image feature and the fusion features, the target image having visual elements related to the visual effect.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A schematic diagram of an example architecture of an image processing system according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram showing an example of initial images according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram of an example architecture of a multimodal model according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic diagram illustrating an example of a target image according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A flowchart illustrating an image processing procedure according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A block diagram of an apparatus for image processing according to some embodiments of the present disclosure is shown; and

[0016] Figure 8 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0019] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0025] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0027] As used herein, a “unit,” “operation unit,” or “subunit” can consist of any suitable machine learning model or network. As used herein, a set of elements or similar expressions can include one or more such elements. For example, “a set of convolutional units” can include one or more convolutional units.

[0028] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, environment 100 may include electronic device 130.

[0029] Electronic device 130 can perform image editing operations on initial image 120 based on user-provided text input 110, thereby generating a target image 150 that meets user requirements. In some embodiments, initial image 120 can be an image input by the user or an image provided by electronic device 130 to the user. In some embodiments, initial image 120 can be any one or more frames from a video. Electronic device 130 can adjust image attributes (e.g., contrast, brightness) or add, remove, or modify image elements of initial image 120 according to user requirements. In some embodiments, during the editing of initial image 120, electronic device 130 first determines the elements in initial image 120 indicated by text input 110, and then changes the corresponding elements in initial image 120 according to the instructions of text input 110. For example, for initial image 120 input by the user, if text input 110 is to change the background of initial image 120, then the background in initial image 120 is replaced with the background corresponding to text input 110. If text input 110 is to add elements to the image, then the elements to be added are first obtained, and then the elements are added to the specified positions in initial image 120.

[0030] In some embodiments, electronic device 130 may utilize a trained machine learning model 140 to perform image processing tasks. Machine learning model 140 may be, for example, any suitable model including but not limited to, Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), etc. Machine learning model 140 may be a model native to electronic device 130 or a model installed on another electronic device 130 (e.g., installed on a remote device).

[0031] Electronic device 130 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side equipment may include computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0032] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0033] As briefly mentioned earlier, image processing technology has been applied to various image processing tasks. With the development of image processing technology, there is a wide demand for image processing tasks in various fields. Users can obtain images from content-sharing applications and enter corresponding text to give instructions for processing the image. For example, changing the image's tone or replacing any object in the image. Such images may be generated by terminal devices (e.g., mobile devices) with content-sharing applications installed.

[0034] Taking image editing as an example, most current image editing technologies are based on diffusion models. They utilize codecs to process text input and the initial image to obtain the target image corresponding to the user's instructions. However, the initial image often contains many elements, making it difficult to accurately edit the target image. For instance, if the initial image contains a face, the text input might instruct the user to add sunglasses to that face. Because the target to be edited cannot be accurately determined during image editing, information from the initial image may be lost.

[0035] Embodiments of this disclosure propose a scheme for processing images. According to various embodiments of this disclosure, text input describing a visual effect for an initial image is received. Based on the text input and the initial image, fusion features for the text input and the initial image are generated. Based on a first image feature of the initial image and the fusion features, a target image corresponding to the initial image is generated, the target image having visual elements related to the visual effect.

[0036] In the embodiments of this disclosure, the text input and the initial image are first fused, and then a target image matching the text input is generated based on the image features of the initial image itself and the fused features. Utilizing the fused features of text and image can better express the desired visual effect. In this way, the accuracy and reliability of image editing can be improved, information loss in the initial image can be prevented, and the quality of the generated image can be improved.

[0037] Figure 2 A schematic diagram of an example of an image processing system 200 according to some embodiments of the present disclosure is shown. Figure 2 As shown, the image processing system 200 can be included or implemented in the electronic device 130. The following is in conjunction with... Figure 1 To describe the image processing system 200.

[0038] In some embodiments, the electronic device 130 may receive text input 110 for an initial image 120. The text input 110 describes the visual effects for the initial image 120.

[0039] For example, the initial image 120 can be an image provided by the user. The initial image 120 can also be an image provided by the electronic device 130 to the user. For example, in the case of a sharing application running on the electronic device 130, the user can select an image published, shared, or created in the application as the initial image 120. Figure 3 A schematic diagram of an example of an initial image 120 according to some embodiments of the present disclosure is shown. Figure 3 As shown, the initial image 120 includes visual elements such as objects (e.g., a puppy, a tennis ball) and image background.

[0040] continue Figure 2 Text input 110 provides the user with image editing instructions for electronic device 130. Text input 110 is used to describe the visual effects for initial image 120. For example, text input 110 may be for one or more objects in initial image 120, image background, or the size, color, etc. of initial image 120.

[0041] In some embodiments, a user can provide text input 110 to the electronic device 130 via text commands or voice. For example, text input 110 could be "Replace the black portion of the image background with white". In some embodiments, a user can provide text input 110 to the electronic device 130 via interactive icons provided by the electronic device 130. Interactive icons indicate different editing operations on the image, and may include, but are not limited to, an "Insert" icon configured to add elements to the image and a "Insert Element" corresponding to the insert icon, and a "Contrast Adjustment" icon configured to change the image contrast. For example, if a user triggers the "Insert" icon and the corresponding "Insert Element A" provided by the electronic device 130, the electronic device 130, in response to the user's operation, generates text input 110 stating "Insert element A into the original image".

[0042] like Figure 2 As shown, the image processing system 200 can generate a fusion feature 241 for the text input 110 and the initial image 120 based on the text input 110 and the initial image 120. For example, the fusion feature 241 can be obtained using a multimodal model 230. The initial image 120 and the text input 110 for the initial image 120 can be provided to the multimodal model 230. The image processing system 200 can use the multimodal model 230 to perform fusion processing on the initial image 120 and the text input 110 to generate the fusion feature 241.

[0043] The multimodal model 230 can be any model with text and image representation capabilities, and can be implemented using any suitable network structure. In some embodiments, the image processing system 200 can provide the text input 110 and the initial image 120 as inputs to the multimodal model 230 to obtain the output of a predetermined intermediate layer of the multimodal model 230. The image processing system 200 can determine the initial feature 241 based on the output of the predetermined intermediate layer. For example, the predetermined intermediate layer can be the penultimate layer of the multimodal model 230.

[0044] Figure 4 An example architecture of the multimodal model 230 is shown. Figure 4As shown, the multimodal model 230 acquires text input 110 and an initial image 120. The text input 110 is encoded to generate multiple text tokens 440. The initial image 120 is processed using a visual encoder 410 to obtain image features. The image features are adjusted using a linear layer 420 to obtain image tokens 430. Subsequently, the image tokens 430 and text tokens 440 are provided to a language model 450 to obtain the features output by the multimodal model 230. As mentioned above, in some embodiments, the fused features 241 can be determined based on the output features of intermediate layers (e.g., the penultimate layer) of the language model 450.

[0045] In some embodiments, the fusion feature 241 can be determined directly using the multimodal model 230. The resulting fusion feature 241 can then be incorporated into the subsequent diffusion model 270.

[0046] In some embodiments, to further ensure that the generated target image conforms to the text input, both the fusion features and the text features of the text input may need to be involved in the generation of the target image. In this case, the feature space of the generated fusion features needs to match the feature space of the encoded text input, for example, having the same distribution characteristics. To this end, in some embodiments, the image processing system 200 can determine initial features for fusing the text input 110 and the initial image 120 based on the text input 110 and the initial image 120. Subsequently, the image processing system 200 can determine the fusion features 241 by transforming the initial features into dimensions that match the text encoding.

[0047] like Figure 2 As shown, the multimodal model 230 obtains initial features by fusing multimodal features from the text input 110 and the initial image 120. The initial features are then input to the feature transformation model 240. Feature transformation is used to match the fused features 241 with the input feature space of the diffusion model 270.

[0048] The feature transformation model 240 can be implemented using any suitable network architecture. In some embodiments, an attention mechanism can be used to transform the initial features output by the multimodal model 230 to determine the fused features 241. That is, the feature transformation model 240 can be an attention-based model. For example, the feature transformation model 240 can determine key features and value features for the attention mechanism based on the initial features. Subsequently, the feature transformation model 240 can determine the fused features 241 using the attention mechanism based on the key features, value features, and pre-determined query features. For example, the query features can be determined during training.

[0049] The above describes an example implementation of the text and image fusion branch. In the image branch, the initial image 120 can be encoded or its features extracted to obtain the first image features 222 of the initial image 120. In some embodiments, such as Figure 2 As shown, noise can be introduced into the generation of the first image feature 222. For example, noise signal 210 and initial image 120 can be provided to image encoder 221. Image encoder 221 can generate the first image feature 222 based on initial image 120 and noise signal 210.

[0050] In some embodiments, the image processing system 200 can generate a target image 150 corresponding to the initial image 120 based on the first image features 222 and the fusion features 241 of the initial image 120. The target image 150 has visual elements related to visual effects. The visual elements in the target image 150 correspond to the text input 110.

[0051] As mentioned above, in some embodiments, both the fusion features and the text features of the text input can participate in the process of generating the target image. In this case, the image processing system 200 may include a branch for text processing. The text features obtained from the text branch can be further merged or combined with the fusion features 241. Figure 2 As shown, the text encoder 220 can generate a text code corresponding to the text input 110. The text code can be used to update the fusion feature 241 to obtain the updated fusion feature 251. For example, the fusion feature 241 can be updated by merging the text code and the fusion feature 241 to obtain the updated fusion feature 251. Furthermore, the diffusion model 270 can generate the target image 150 based on the updated fusion feature 251 and the first image feature 222 of the initial image 120.

[0052] In some embodiments, such as Figure 2 As shown, the electronic device 130 can perform a feature merging operation on the text encoding and fused feature 241 through the feature merging layer 250 to obtain an updated fused feature 251. For example, the feature merging layer 250 can perform an addition operation or a concatenation operation on the text encoding and fused feature 241. For instance, in the case of an addition operation, the values ​​of the text encoding and fused feature 241 can be directly added together according to their respective dimensions. Similarly, in the case of a concatenation operation, the text encoding and fused feature 241 can be concatenated together along a certain dimension.

[0053] The fusion feature 241 or the updated fusion feature 251 can be used by the diffusion model 270 in any suitable manner. In some embodiments, the fusion feature 241 or the updated fusion feature 251 can be injected into each layer of the diffusion model 270 via an attention mechanism.

[0054] In some embodiments, it is necessary to input as much image information as possible from the initial image 120 into the diffusion model 270 to prevent image information loss from affecting the quality of the generated target image 150. Therefore, the initial image 120 can be used to control the process of generating the target image 150. Figure 2 As shown, based on the initial image 120, a second image feature 261 can be generated using the control model 260. Subsequently, the diffusion model 270 can generate the target image 150 based on the first image feature 222, the second image feature 261, and the fusion feature 241 (or the updated fusion feature 251). The control model 260 can be constructed using any suitable mechanism or network structure. For example, the control model 260 can be implemented based on a control network.

[0055] In some embodiments, the diffusion model 270 can use the fusion feature 241 (or the updated fusion feature 251) and the second image feature 261 as control conditions to generate the target image 150. Figure 2 As shown, an initial image 120 is provided to an image encoder 221 to obtain a first image feature 222 representing the initial image 120. Subsequently, an electronic device 130 provides the first image feature 222 to a diffusion model 270. The diffusion model 270 uses a second image feature 261 and a fusion feature 241 (or an updated fusion feature 251) as control conditions to generate an encoded representation corresponding to the initial image 120. An image decoder 280 can perform a decoding operation on the encoded representation generated by the diffusion model 270 to generate a target image 150 corresponding to the text input 110.

[0056] The above is for reference only. Figure 2 An example implementation of the image processing system 200 is described. It should be understood that... Figure 2 The structures shown are merely exemplary and are not intended to limit the scope of this disclosure.

[0057] The following is a reference. Figure 2 Describe an example scenario. Electronic device 130 can generate target image 150 based on user-input text 110 and initial image 120. For example, for... Figure 3The initial image 120 shown (an image uploaded by the user and / or an image stored in the user-specified electronic device 130) is used. The user-inputted text 110 can be "replace the tennis ball in the image with a soccer ball". The electronic device 130 uses a multimodal model 230 to process the text input 110 and the initial image 120 to generate initial features for fusing the text input 110 and the initial image 120. Simultaneously, the electronic device 130 uses a text encoder 220 to perform an encoding operation on the text input 110 to obtain a text encoding. Subsequently, the electronic device 130 uses a feature merging layer 250 to concatenate or add the text encoding and the fused features 241 to obtain an updated fused feature 251. The electronic device 130 uses an image encoder 221 to perform an encoding operation on the initial image 120 and the random noise corresponding to the initial image 120 to obtain a first image feature 222. The first image feature 222, the updated fused feature 251, and the second image feature 261 generated by the control model 260 are provided to the diffusion model 270 to generate an image encoding.

[0058] For example, the second image feature 261 and the updated fusion feature 251 can be used as control conditions to generate an image code based on the first image feature 222 using a diffusion model 270. Subsequently, the target image 150 is generated from the image code using an image decoder 280. Figure 5 A schematic diagram of an example of a target image 150 according to some embodiments of the present disclosure is shown. Figure 5 As shown, based on the user's text input 110, the electronic device 130 replaces the tennis ball in the initial image 120 with a soccer ball to generate... Figure 5 The target image shown is 150.

[0059] The above describes an example embodiment of the image processing system 200 generating a target image. The following describes an example embodiment of training the image processing system 200. To train the image processing system 200, a corresponding training dataset can be constructed. For example, multiple initial images can be acquired, such as from any existing training image set. Then, the rendering effects of an image rendering tool can be used to add elements with a certain visual effect to the initial images (e.g., adding fireworks effects) or modify an element in the initial images to another visual effect (e.g., changing the color of a flower from red to yellow). Thus, an updated image corresponding to the initial image can be obtained, and a corresponding text description can be generated based on the rendering effect used. This yields training samples including the initial image, text description, and the updated image as a label or ground truth.

[0060] During training, the image processing system 200 can generate corresponding images based on the initial images and text descriptions in the training samples. The loss can be determined based on the difference between the generated images and the updated images in the training samples, thereby updating the parameters of at least a portion of the model in the image processing system 200.

[0061] As can be seen, in the embodiments of this disclosure, on the one hand, a target image with relevant visual elements corresponding to the initial image is generated based on the first image features of the initial image and the fusion features of the text input and the initial image. In this way, the fusion of text and image can more accurately represent the elements in the initial image related to the text input, thereby improving the accuracy and reliability of image editing. On the other hand, the second image features and fusion features generated by the control model are used as conditional controls to generate the target image to prevent information loss in the initial image and further improve the quality of the edited target image. Furthermore, the initial image and random noise signals are input into the image encoder to increase the diversity of input images and improve the quality of the target image.

[0062] Figure 6 A flowchart of an image processing procedure according to some embodiments of the present disclosure is shown. Procedure 600 can be implemented at an electronic device.

[0063] In box 610, text input for an initial image is received, which describes the visual effects for the initial image.

[0064] In box 620, fused features are generated based on the text input and the initial image.

[0065] In some embodiments, generating fusion features for text input and an initial image includes: determining initial features for fusing the text input and the initial image based on the text input and the initial image; and determining fusion features by converting the initial features into dimensions that match the text encoding.

[0066] In box 630, a target image corresponding to the initial image is generated based on the first image features and fusion features of the initial image. The target image has visual elements related to the visual effect.

[0067] In some embodiments, determining fusion features by converting initial fusion features into dimensions that match the text encoding includes: determining key features and value features for the attention mechanism based on the initial features; and determining fusion features using the attention mechanism based on the key features, value features, and pre-determined query features.

[0068] In some embodiments, the first image features of the initial image are determined by generating the first image features using an image encoder based on the initial image and the noise signal.

[0069] In some embodiments, generating a target image corresponding to an initial image includes: generating a second image feature of the initial image based on the initial image using a control model; and generating the target image based on the first image feature, the second image feature, and the fused feature.

[0070] In some embodiments, generating a target image based on a first image feature, a second image feature, and a fused feature includes: generating a target image based on the first image feature by using the second image feature and the fused feature as control conditions.

[0071] In some embodiments, process 600 further includes generating a text code corresponding to the text input using a text encoder based on the text input before generating the target image based on the first image features and the fusion features; and updating the fusion features using the text code.

[0072] Figure 7 A block diagram of an apparatus for image processing according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in an electronic device. Various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0073] As shown in the figure, the device 700 includes a receiving module 710 configured to receive text input for an initial image, the text input describing the visual effect of the initial image. The device 700 also includes a first generation module 720 configured to generate fused features for the text input and the initial image based on the text input and the initial image. The device 700 further includes a second generation module 730 configured to generate a target image corresponding to the initial image based on the fused features and first image features of the initial image, the target image having visual elements related to the visual effect.

[0074] In some embodiments, the second generation module 730 is further configured to generate a text code corresponding to the text input using a text encoder; obtain updated fusion features by performing a feature fusion operation on the text code and the fusion features; and generate a target image based on the updated fusion features and the initial image.

[0075] In some embodiments, the second generation module 730 is further configured to determine key features and value features for the attention mechanism based on initial features; and to determine fusion features using the attention mechanism based on the key features, value features, and pre-determined query features.

[0076] In some embodiments, the second generation module 730 is further configured to generate a first image feature based on the initial image and noise signal using an image encoder.

[0077] In some embodiments, the second generation module 730 is further configured to generate a second image feature of the initial image based on the initial image and using a control model; and to generate a target image based on the first image feature, the second image feature, and the fused feature.

[0078] In some embodiments, the second generation module 730 is further configured to generate a target image based on the first image features by using the second image features and the fusion features as control conditions.

[0079] In some embodiments, the apparatus 700 further includes an update module configured to, before generating a target image based on the first image features and the fusion features, generate a text code corresponding to the text input using a text encoder; and update the fusion features using the text code.

[0080] Figure 8 A block diagram is shown illustrating an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 8 The electronic device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to achieve Figure 1 Electronic devices 110.

[0081] like Figure 8 As shown, electronic device 800 is in the form of a general-purpose electronic device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 800.

[0082] Electronic device 800 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 800.

[0083] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0084] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0085] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0086] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0087] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0091] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An image processing method, comprising: Receive text input for an initial image, the text input describing the visual effect of the initial image; Based on the text input and the initial image, generate fusion features for the text input and the initial image; as well as Based on the first image features and the fusion features of the initial image, a target image corresponding to the initial image is generated, and the target image has visual elements related to the visual effect.

2. The method according to claim 1, further comprising: Before generating the target image based on the first image features and the fusion features. Based on the text input, a text encoder is used to generate a text code corresponding to the text input; and The fusion feature is updated using the text encoding.

3. The method of claim 1, wherein generating fusion features for the text input and the initial image comprises: Based on the text input and the initial image, determine initial features for fusing the text input and the initial image; as well as The fused features are determined by converting the initial features into dimensions that match the text encoding.

4. The method of claim 3, wherein determining the fused features by converting the initial features into dimensions that match the text encoding comprises: Based on the initial features, key features and value features for the attention mechanism are determined; as well as Based on the key features, the value features, and the pre-determined query features, the fusion features are determined using an attention mechanism.

5. The method of claim 1, wherein determining the initial features for fusing the text input and the initial image comprises: The text input and the initial image are provided as input to the multimodal model to obtain the output of the predetermined intermediate layer of the multimodal model; as well as The initial features are determined based on the output of the predetermined intermediate layer.

6. The method according to claim 1, wherein generating the target image corresponding to the initial image comprises: Based on the initial image, a second image feature of the initial image is generated using a control model; as well as A target image is generated based on the first image features, the second image features, and the fused features.

7. The method according to claim 6, wherein generating the target image based on the first image feature, the second image feature, and the fusion feature comprises: The target image is generated based on the first image feature by using the second image feature and the fused feature as control conditions.

8. The method of claim 1, wherein the first image feature of the initial image is determined by: Based on the initial image and noise signal, the first image feature is generated using an image encoder.

9. An apparatus for image processing, comprising: A receiving module is configured to receive text input for an initial image, the text input describing the visual effect of the initial image; The first generation module is configured to generate fusion features for the text input and the initial image based on the text input and the initial image; as well as The second generation module is configured to generate a target image corresponding to the initial image based on the first image features and the fusion features of the initial image, wherein the target image has visual elements related to the visual effect.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

Images