Image generation method and device and display equipment

Abstract

Embodiments of the invention provide an image generation method and apparatus, and a display device. The method comprises the steps of obtaining multi-modal information input by a user; processing each piece of modal information in the multi-modal information based on a coding network in the image generation model to obtain a feature vector corresponding to each piece of modal information; performing fusion processing on the feature vectors corresponding to the modal information based on a fusion network in the image generation model to obtain a fusion vector; and based on the multi-modal information, processing the fusion vector through a multi-stage network in the image generation model to obtain a target image corresponding to the multi-modal information. According to the image generation method, the corresponding image can be generated based on the multi-modal information.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to an image generating method, device and display device. Background Art

[0002] With the rapid development of image generation technology, artificial intelligence and machine learning technologies can realize the automatic generation of images. Related technologies can realize image generation based on single modal information, for example, generating images based on text information or generating corresponding images based on image information. However, with the increase of user personalized needs, this single modal information generation image can no longer meet user needs. Summary of the invention

[0003] The embodiments of the present application provide an image generation method, an apparatus, and a display device, which can generate corresponding images based on multimodal information to meet the personalized needs of users.

[0004] The first aspect of an embodiment of the present application provides an image generation method, which includes: first, obtaining multimodal information input by a user; wherein the multimodal information includes at least two items of audio information, text information, image information, and video information. Then, based on the encoding network in the image generation model, each modal information is processed separately to obtain a feature vector corresponding to each modal information. Secondly, based on the fusion network in the image generation model, the feature vectors corresponding to each modal information are fused to obtain a fusion vector. Finally, based on the multimodal information, the fusion vector is processed through the multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

[0005] The image generation method provided in the embodiment of the present application, after obtaining the multimodal information input by the user, inputs the multimodal information into the image generation model, and sequentially processes the encoding network, fusion network and multi-stage network in the image generation model to obtain a target image that meets the multimodal information. The image generation method provided in the embodiment of the present application can process multimodal information to generate an image that meets the requirements, meet the personalized needs of users, and improve the user experience.

[0006] In some embodiments, based on multimodal information, a fusion vector is processed through a multi-stage network in an image generation model to obtain a target image corresponding to the multimodal information, including: determining at least one auxiliary feature information corresponding to the multimodal information; processing the fusion vector and at least one auxiliary feature information based on at least one feature sub-network in the multi-stage network to obtain a target vector; decoding the target vector based on an image decoding sub-network in the multi-stage network to obtain a target image.

[0007] Based on the above scheme, auxiliary feature information is introduced when the multi-stage network processes the fusion vector. The auxiliary feature information can enable the multi-stage network to extract more realistic and vivid features, thereby improving the quality of the subsequent generated target image and enabling the image generation model to generate images that better meet user needs.

[0008] In some embodiments, a fusion vector and at least one auxiliary feature information are processed based on at least one feature subnetwork in a multi-stage network to obtain a target vector, including: processing the fusion vector and the first auxiliary feature information based on a first feature subnetwork in at least one feature subnetwork to determine a first output vector; wherein the at least one auxiliary feature information includes first auxiliary feature information, second auxiliary feature information, and third auxiliary feature information; processing the first output vector and the second auxiliary feature information based on an intermediate feature subnetwork in at least one feature subnetwork to determine a second output vector; and processing the second output vector and the third auxiliary feature information based on a second feature subnetwork in at least one feature subnetwork to determine a target vector.

[0009] Based on the above scheme, in a multi-stage network, the vector output by the previous feature sub-network and the auxiliary feature information corresponding to the current feature sub-network are input into the current feature sub-network. After being processed by the current feature sub-network, the feature vector output by the current feature sub-network is input into the next feature sub-network. By dividing the hierarchical relationship in the multi-stage network, each stage receives the output information and auxiliary information of the previous stage, realizing fine-grained control and realizing multi-stage controllable generation, improving the output quality of the image, and improving the execution efficiency and performance of the image generation model.

[0010] In some embodiments, the feature vectors corresponding to each modal information are fused based on a fusion network in an image generation model to obtain a fusion vector, including: determining a preset length; based on the preset length, the feature vectors corresponding to each modal information are fused through a fusion network to obtain a fusion vector of the preset length.

[0011] Based on the above scheme, the fusion network fuses the feature vectors corresponding to the multimodal information to generate a fusion vector of a preset fixed length. Generating a fusion vector of a preset length can improve the operating efficiency while ensuring the processing performance of the encoding network, and is more suitable for real-time processing and large-scale data analysis scenarios.

[0012] In some embodiments, based on a preset length, feature vectors corresponding to each modal information are fused through a fusion network to obtain a fusion vector of a preset length, including: performing feature selection processing on the feature vectors corresponding to each modal information through a fusion network to obtain a selected feature vector; mapping the selected feature vector to a preset feature space to obtain a mapped feature vector; based on a preset length, the mapped feature vector is fused to obtain the fusion vector of the preset length; wherein the fusion processing includes at least one fusion method selected from weighted fusion, feature-level fusion and decision-level fusion.

[0013] Based on the above scheme, the most useful features can be selected through feature selection during fusion processing, thereby reducing the dimension of the feature space and improving the generalization ability and computational efficiency of the image generation model. At the same time, through feature mapping and fusion processing, multiple feature vectors are fused into a new feature vector, realizing the fusion processing of multimodal information, and providing a basis for subsequent image generation based on multimodal information.

[0014] In some embodiments, each modal information is processed separately based on the encoding network in the image generation model to obtain a feature vector corresponding to each modal information, including at least one of the following: inputting audio information into the audio encoding subnetwork in the encoding network to generate an audio feature vector corresponding to the audio information; inputting text information into the text encoding subnetwork in the encoding network to generate a text feature vector corresponding to the text information; inputting image information into the image encoding subnetwork in the encoding network to generate an image feature vector corresponding to the image information; inputting video information into the video encoding subnetwork in the encoding network to generate a video feature vector corresponding to the video information.

[0015] Based on the above scheme, the encoding network includes multiple encoding sub-networks for processing different modal information, so that it can process various modal information input and output by users, providing a basis for the subsequent fusion network to fuse the feature vectors corresponding to the multi-modal information.

[0016] In some embodiments, audio information is input into an audio coding subnetwork in a coding network to generate an audio feature vector corresponding to the audio information, including: preprocessing the audio information to obtain preprocessed audio information; performing a feature extraction operation on the preprocessed modal information to obtain an initial feature vector; and encoding the initial feature vector to obtain an audio feature vector.

[0017] Based on the above scheme, the audio coding subnetwork in the coding network may encode the audio information including preprocessing, feature extraction and encoding processing, extracting meaningful feature representations from the audio information input by the user to improve the accuracy of subsequent image generation.

[0018] In some embodiments, the method further includes: obtaining multimodal training information; processing the multimodal training information based on the image generation model to be trained to generate a predicted image; wherein the image generation model to be trained includes a coding network to be trained, a fusion network to be trained and a multi-stage network to be trained; obtaining a sample image; using the predicted image as the initial training output information of the image generation model to be trained and the sample image as the supervision information, iterating the image generation model to be trained to obtain an image generation model.

[0019] Based on the above scheme, the embodiment of the present application also proposes a training process for the image generation model, in which the image generation model to be trained is trained by the acquired multimodal training information, and the image generation model to be trained is iteratively trained with sample images as supervisory information to obtain a trained image generation model. The trained image generation model can be used to convert the multimodal information into a corresponding target image, realize the processing of multimodal information to generate an image that meets the requirements, meet the personalized needs of users, and improve the user experience.

[0020] The second aspect of the embodiment of the present application provides an image generation device, including: an acquisition module and a determination module. The acquisition module is configured to obtain multimodal information input by a user; the multimodal information includes at least two items of audio information, text information, image information and video information. The determination module is configured to process each modal information separately based on the encoding network in the image generation model to obtain a feature vector corresponding to each modal information; to fuse the feature vectors corresponding to each modal information based on the fusion network in the image generation model to obtain a fusion vector; based on the multimodal information, the fusion vector is processed through the multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

[0021] The image generation device provided in the embodiment of the present application, after obtaining the multimodal information input by the user, inputs the multimodal information into the image generation model, and sequentially processes the encoding network, fusion network and multi-stage network in the image generation model to obtain a target image that meets the multimodal information. The image generation method provided in the embodiment of the present application can process multimodal information to generate an image that meets the requirements, meet the personalized needs of users, and improve the user experience.

[0022] The third aspect of the embodiment of the present application provides a display device, including a display and a controller coupled to the display. The display is configured to display an application interface of an image generation application; the application interface includes a first input area, a second input area, and an image display area. The controller is configured to: respond to receiving image information input by a user in the first input area, and text information input in the second input area; based on the encoding network in the image generation model, process each modal information in the multimodal information respectively to obtain a feature vector corresponding to each modal information; wherein the multimodal information includes image information and text information; based on the fusion network in the image generation model, the feature vectors corresponding to each modal information are fused to obtain a fusion vector; based on the multimodal information, the fusion vector is processed by the multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

[0023] The display device provided in the embodiment of the present application can receive image information input by a user in a first input area and text information input in a second input area, and generate a target image based on the image information and text information and a trained image generation model. The target image can meet the requirements of image information and text information. The display device provided in the embodiment of the present application can process multimodal information to generate images that meet the requirements, meet the personalized needs of users, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the hardware configuration of a display device provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the hardware configuration of a control device provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of an application interface provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of another application interface provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of an image display method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of an application interface provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of another image display method provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of another application interface provided in an embodiment of the present application;

[0034] Fig.10 A schematic diagram of another image display method provided in an embodiment of the present application;

[0035] Fig.11 A schematic diagram of another image display method provided in an embodiment of the present application;

[0036] Fig.12 A schematic diagram of another application interface provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of another image display method provided in an embodiment of the present application;

[0038] Fig.14 A schematic diagram of an image generation method provided in an embodiment of the present application;

[0039] Fig.15 A schematic diagram of an image generation model provided in an embodiment of the present application;

[0040] Fig.16 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0041] Fig.17 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0042] Fig.18 A schematic diagram of a multi-stage network structure provided in an embodiment of the present application;

[0043] Fig.19 A schematic diagram of a flow chart of an image display method provided in an embodiment of the present application;

[0044] Fig. 20 A flowchart of another image display method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0046] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0047] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0048] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0049] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0050] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0051] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0052] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, etc.

[0053] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0054] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0055] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0056] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0057] Figure 2 A hardware configuration block diagram of a display device 200 provided for some embodiments of the present application.

[0058] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0059] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0060] In some embodiments, the device interface 240 may include, but is not limited to, a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. The device interface 240 may also be a composite input / output interface formed by the above multiple interfaces.

[0061] The display device 200 can be connected to the external device 500 through the device interface 240. The external device 500 can be a set-top box, a game console, a PC, etc. When the display device 200 is connected to the external device 500 through the device interface 240, a corresponding signal source channel can be established, and an input signal can be received through the signal source channel. For example, the display device 200 can be connected to a game device through an HDMI interface to establish an HDMI channel, and receive audio and video signals sent by the game device based on the HDMI channel.

[0062] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0063] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0064] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0065] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0066] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0067] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0068] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0069] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0070] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0071] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0072] The operating system can be divided into different modules or levels according to the functions implemented. For example, in some embodiments, Figure 3 As shown, taking the Android system as an example, the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer) (referred to as "application layer"), the application framework layer (Application Framework layer) (referred to as "framework layer"), the system library layer and the kernel layer. In some embodiments, the operating system of the display device can also be other operating systems.

[0073] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0074] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0075] like Figure 3As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0076] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0077] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0078] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0080] In some embodiments, in order to generate a target image, the display device 200 may run an image generation application, which may be a system application built into the operating system of the display device 200. The image generation application may also be an independent application or a third-party application developed based on the operating system of the display device 200.

[0081] The display device 200 may receive control instructions input by the user during operation, wherein some control instructions are used to control the display device 200 to generate a target image, and such control instructions may be referred to as image generation instructions. Based on the different interaction modes supported by the display device 200, the user may input the image generation instructions in different ways.

[0082] In some embodiments, the display device 200 can obtain the image generation instruction by detecting the user's key input based on the matching control device 100. That is, when the display device 200 displays the application list interface, the user can control the focus cursor in the application list interface by using the direction keys on the control device 100. After the focus cursor moves to the image generation application icon, the user presses the confirmation key on the control device 100 to control the display device 200 to run the image generation application, that is, input the image generation instruction. The display device 200 can then run the image generation application according to the image generation instruction.

[0083] In some embodiments, when the display device 200 supports touch, voice, somatosensory, gesture and other interactive modes, the display device 200 can detect that the user can input an interactive signal through the corresponding interactive mode, and obtain an image generation instruction according to the interactive signal. For example, when the voice assistant application built into the display device 200 detects that the user input content is a voice instruction of "please generate a picture containing a bird for me, the bird has blue feathers on its back and white feathers on its belly", it determines that the user's intention is an image generation intention according to the keywords in the voice instruction. That is, it is determined that the current voice instruction is an image generation instruction. At this time, the display device 200 can run the image generation application in response to the image generation instruction.

[0084] In some embodiments, the image generation instruction can also be generated by the display device 200 by monitoring various operation events during the operation process and when a specific operation event is detected. For example, the display device 200 can run a search engine application such as "×× search", and the search interface can include a text input box and a picture input box. When the user enters the search text in the text input box and / or enters the search image in the picture input box, the display device 200 can automatically generate an image generation instruction and run the image generation application in response to the image generation instruction.

[0085] In response to the image generation instruction, the display device 200 may display an application interface of the image generation application by running the image generation application.

[0086] In some embodiments, the display device 200 starts the image generation application in response to the received start operation of the image generation application, and the display displays the application interface corresponding to the image generation application. The application interface of the image generation application may include two input areas, namely a first input area and a second input area. Among them, the first input area is used to input and display the image content in the input data, and the first input area may also be called a drawing area or an image drawing area. The second input area is used to input and display the text content in the input data, and the second input area may also be called a text input area.

[0087] Exemplarily, the first input area may be used to display an image, and the displayed image may be an image drawn by a user, or an image photographed by a user through an image collector on a display device, or an image uploaded through a terminal device.

[0088] For example, when the display device supports the touch function, the user can directly draw an image in the first input area, or can shoot an image through the image collector on the display device or upload an image through the terminal device. When the display device does not support the touch function, the user can shoot an image through the image collector on the display device or upload an image through the terminal device.

[0089] Exemplarily, the second input area is used to display the description text input by the user. The description text displayed in the second input area may be the description text input by the user, or may be the description text generated according to the description voice input by the user. For example, the description text may be the user's demand description.

[0090] In some embodiments, the application interface of the image generation application may further include an image display area for displaying the generated target image. The image display area may be displayed in parallel with the first input area to facilitate the user to compare the target image with the image content in the input data.

[0091] Exemplarily, the image display area is used to display the image generated after being processed by the image generation model, and the image display area may also be referred to as the generation area or the image generation area. For example, the image generation model may generate a target image based on the image displayed in the first input area and / or the description text displayed in the second input area, and display the target image in the image display area. The target image meets the requirements of the description image and the description text.

[0092] For example, the image display area can be used to display an image of the image displayed in the first input area after being processed by the image generation model, and can also be used to display an image of the description text displayed in the second input area after being processed by the image generation model, or it can also be used to display an image of the description image displayed in the first input area and the description text displayed in the second input area after being processed by the image generation model.

[0093] The first input area and the second input area can be expressed as different area shapes according to the application interface UI design specification of the image generation application, and distributed in different positions of the application interface. In some embodiments, the first input area, the second input area, and the image display area can be three independent areas in the application interface that do not have any intersection. For example, Figure 4 As shown, the first input area can be located in the upper left corner of the application interface. In order to facilitate the input and display of image content, the first input area can be a rectangular area. The second input area can be located at the bottom of the application interface. Similarly, in order to facilitate the input and display of text content, the second input area can be a long strip area. The image display area is located in the upper right corner of the application interface, and, in order to facilitate the display of the target image, the image display area can also be a rectangular area.

[0094] In some embodiments, there may also be an intersection area between the first input area, the second input area, and the image display area, and the display device 200 may determine the area to which the content belongs by the location of the focus mark. For example, when the display device 200 receives a signal that the user has moved the focus cursor to the first input control corresponding to the first input area, the full screen area of ​​the current application interface may be used to receive the user's input image operation, that is, the first input area at this time is the entire application interface with the focus mark in the first input control state. Similarly, when the display device 200 receives a signal that the user has moved the focus cursor to the second input control corresponding to the first input area, the first input area is the entire application interface with the focus mark in the second input control state.

[0095] In order to facilitate user input and image display, in some embodiments, the display device 200 can set a display hierarchical relationship between the first input area, the second input area, and the image display area. For example, when the display device 200 detects that the user performs image-related operations such as touch painting, image capture, and uploading pictures, the display level of the first input area can be set higher than the second input area and the image display area. When the display device 200 detects that the user performs text-related operations such as text input and voice interaction, the display level of the second input area can be set higher than the first input area and the image display area. Similarly, after the display device 200 detects that the image generation model generates a target image, the display level of the image display area can be set higher than the first input area and the second input area.

[0096] The display device 200 can dynamically adjust the display contents in the first input area, the second input area, and the image display area in different operation stages.

[0097] The first input area may display different contents at different interaction stages according to different interaction modes supported by the display device 200. In some embodiments, if the display device 200 supports touch interaction, for example, the display 260 of the display device 200 is connected to a touch component. The touch component and the display 260 may form a touch screen for detecting a touch signal input by a user based on the touch screen.

[0098] The first input area may be associated with a touch component, and the touch component detects the touch point input by the user through a change in capacitance or resistance. When it is detected that the user moves a finger or a stylus on the touch screen, the display device 200 continuously records the coordinates of the touch point. The touch screen driver and other touch screen interaction programs record these coordinate points to form a series of point data, that is, to form a drawing trajectory detection point set.

[0099] After obtaining the set of drawing trajectory detection points, the image drawing application will draw the trajectory according to these point data. That is, the image drawing application can draw lines on the first input area based on the line form corresponding to the drawing control selected by the user. For example, if it is detected that the user has selected a brush control, the image drawing application will display the lines in the first input area in the form of red, thick, solid lines according to the current line form of the brush control: red, thick, solid lines, and the line shape is consistent with the touch trajectory detection points. Among them, the image drawing application can be an independent application or a drawing function unit in an image generation application. The drawing of lines can be done by algorithms such as Bresenham.

[0100] After detecting and generating a drawing trajectory detection point set, the display device 200 can save the drawn trajectory as an image file, that is, the display device 200 converts the drawing trajectory detection points into a trajectory pixel point set to generate an image drawn by the user (also referred to as a first image). Similarly, the display device 200 can generate an image drawn by the user through an independent image drawing application, or through a drawing function unit in an image generation application.

[0101] In the process of generating a drawing image (such as the first image), the display device 200 can monitor the touch signal monitored by the touch component in real time, determine the touch node according to the touch signal, and generate the drawing image according to the touch trajectory on the appropriate touch node. Therefore, the first input area in the application interface is used to obtain the drawing image input by the user. That is, the display device 200 can extract the touch trajectory corresponding to the drawing instruction in response to receiving the drawing instruction input in the first input area, generate the drawing image according to the touch trajectory, and control the display to display the drawing image in the first input area.

[0102] In some embodiments, the display device 200 can generate a drawing image according to a preset detection cycle. That is, the display device 200 can obtain a preset detection cycle, and extract and obtain the touch trajectory of the user input based on the first input area from the first input area according to the preset detection cycle. Among them, the preset detection cycle is set according to the denoising round of the image generation model. For example, the denoising round of the image generation model is 30 rounds, and the total time is 3s, then the preset detection cycle is 1s, that is, a detection is performed every 10 rounds of denoising. The display device 200 can use the touch position when the input time reaches the preset touch response cycle as a touch node, so as to obtain the touch trajectory of the first input area every 1s, and generate a drawing image based on the touch trajectory.

[0103] In some embodiments, the display device 200 can generate a drawing image based on the touch events monitored by monitoring the touch events in the touch track. Among them, the touch events may include touch press events (ACTION_DOWN), touch move events (ACTION_MOVE), and touch lift events (ACTION_UP). After the display device 200 monitors the touch press event (ACTION_DOWN) through the touch component, it will continue to monitor the touch lift event (ACTION_UP) and use the touch lift event as an image generation node. That is, after monitoring the touch lift event, a drawing image is generated in response to the touch lift event. Then refresh the application interface according to the preset detection cycle to display the drawing image in the first input area.

[0104] In some embodiments, if the display device 200 does not support touch interaction, that is, the display device 200 does not include a touch component, the display device 200 can obtain a description image, wherein the description image is a file in a picture format input by the user. The description image can be acquired in real time by an image collector such as a camera, or can be uploaded by the user through the mobile terminal 300.

[0105] In order to collect the description image in real time, in some embodiments, the display device 200 may have a built-in or external image collector, that is, the display device 200 may include a device interface 240, and the device interface 240 is configured to connect to the image collector. The image collector has an image sensor such as a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) built in, which can take pictures or videos of the detection area to collect the description image.

[0106] For the description image collected in real time, the display device 200 can display a shooting interaction control for real-time image collection in the application interface. When the user inputs an image acquisition instruction based on the shooting interaction control, the display device 200 can run the image collector through the image acquisition application and obtain the description image through the image collector.

[0107] After the image collector starts running, the display device 200 can display the environment image captured by the image collector through the first input area. During the display of the environment image, the display device 200 can also receive a confirmation instruction input by the user. In response to the confirmation instruction, the environment image corresponding to the time when the confirmation instruction is input is obtained to obtain a description image.

[0108] For example, when the display device 200 does not support touch interaction, but has a built-in or external camera or other image collector, the user can draw a description image on a drawing medium such as paper, and then take a photo of the drawing medium through the image collector to obtain the description image.

[0109] In order to obtain the description image uploaded by the user, in some embodiments, the display device 200 further includes a communication device 220, and the communication device 220 is configured to establish a communication connection with the mobile terminal 300. The mobile terminal 300 has a camera device that can be used for scanning codes.

[0110] In some embodiments, the application interface of the image generation application may include a code scanning interactive control. When the display device 200 receives an image acquisition instruction input by the user based on the code scanning interactive control, an upload identification code for uploading an image may be displayed in the first input area, such as Figure 5As shown. The user operates the mobile terminal 300 to start the code scanning function. After scanning the QR code through the camera device, an image selection interface for uploading image files can be displayed on the mobile terminal 300. After the user selects the description image to be uploaded and enters a confirmation instruction, the mobile terminal 300 can send the selected description image to the display device 200 in response to the confirmation instruction. After receiving the description image, the display device 200 displays the description image in the first input area of ​​the application interface.

[0111] In some embodiments, the display device 200 can also obtain the description image uploaded by the user through the server 400. That is, the communication device 220 of the display device 200 is configured to establish a communication connection with the server 400. The server 400 also establishes a communication connection with the mobile terminal 300 to receive the description image uploaded by the user through the mobile terminal 300, and stores the uploaded description image in a specific Uniform Resource Locator (URL) address. When the display device 200 performs the step of receiving the description image uploaded by the mobile terminal 300 by scanning the upload identification code, the URL address issued by the server can be obtained first. That is, the storage address of the description image sent to the server 400 by the mobile terminal 300 after scanning the upload identification code is obtained. A picture receiving request is generated according to the URL address, and the picture receiving request is sent to the server, that is, the URL address issued by the server 400 is accessed, so that the server 400 sends the description image in response to the picture receiving request. The display device 200 then receives the description image issued by the server 400.

[0112] For example, the display device 200 controls the focus cursor to move in the image generation application interface in response to the key event of the user pressing the direction key of the control device 100. When the focus cursor moves to the code scanning interactive control, the display device 200 can receive a confirmation instruction input by the user by pressing the confirmation key of the control device 100. At this time, the display device 200 controls the display device 200 to display the upload identification code in response to the confirmation instruction, that is, the QR code for uploading the description image is displayed in the first input area. After the user starts scanning the QR code through the camera device of the mobile terminal 300, the selected description image (such as the seventh image) can be uploaded to the server 400. After receiving the description image uploaded by the user, the server 400 can send a URL address to the display device 200. After receiving the URL address, the display device 200 cancels the display of the QR code in the first input area and accesses the URL address to download the description image. And, after downloading the description image, the display device 200 displays the description image through the first input area.

[0113] It should be noted that the methods of acquiring images through the first input area provided in the above embodiments can be used alone or in combination. That is, for a display device 200 supporting touch interactive operations, the description image can also be acquired by acquiring images in real time through an image collector, or by uploading images through a mobile terminal 300.

[0114] In some embodiments, the display device 200 can be configured to obtain the drawing image and / or the description image through the first input area. Since both the drawing image and the description image can be displayed in the first input area, the display device 200 can obtain the drawing image or the description image by detecting whether the first input area displays an image.

[0115] In some embodiments, after generating the target image, the display device 200 can also perform image file operations such as saving and sharing on the target image in response to the user's interactive operation. To this end, the application interface of the image generation application can also include an image function area, and the image function area includes at least one interactive control. After displaying the target image, the display device 200 can receive an image interaction instruction input by the user based on the interactive control. And in response to the image interaction instruction, the control identification information of the target interactive control is obtained, and the target interactive control is the interactive control selected by the image interaction instruction. Then, based on the function mapping table, the function process corresponding to the control identification information is queried, and the function process is run.

[0116] like Figure 4 As shown, the image function area may include a save control. After displaying the target image, the display device 200 may obtain an image interaction instruction input by the user for the save control. The display device 200 may then respond to the image interaction instruction, obtain identification information "Save button" of the save control, and call an image file saving process, i.e., "ImageFile saving process", according to the identification information, to store the target image in a specific storage space based on the image file saving process.

[0117] In some embodiments, Figure 4 As shown, the image function area may also include a sharing control. The display device 200 may respond to the image interaction instruction input by the user based on the sharing control, and call the file sharing process "Image File sharing process" according to the identification information "sharebutton" of the sharing control. And send the target image to the target client according to the file sharing process.

[0118] The display device 200 can also send the generated target image to the server 400 through the file sharing process, so that the target image is stored through the server 400. The target client can download the target image from the server 400 by sending a request to obtain the target image to the server 400.

[0119] In some embodiments, the display device 200 may also support interactive operations on multiple target images. That is, the image function area may also include picture book interactive controls (such as Figure 4 After receiving the image interaction instruction input by the user based on the picture book interaction control, the display device 200 can obtain the target image saved in the preset generation cycle to generate a target image set. Then, the description text information input in the second input area during the target image generation process is obtained to generate a picture book file according to the target image set and the description text information.

[0120] In some embodiments, after generating the target image, the display device 200 may receive an image processing instruction input by the user. The image processing instruction may be input based on an image processing control in an image function area in the application interface. For example, the image function area may include image processing controls such as background beautification and image quality enhancement. The display device 200 may obtain the image processing instruction by detecting any image processing control selected by the user based on a touch interaction operation.

[0121] After obtaining the image processing instruction, the display device 200 can call the image processing algorithm model in response to the image processing instruction. Then perform corresponding image processing on the target image based on the image processing algorithm model. Among them, the image processing algorithm includes but is not limited to color and brightness adjustment, blurred background, edge detection and cutout, background replacement, depth estimation, image segmentation, style migration, etc. For example, the user inputs the image processing instruction by clicking the background beautification control, and the display device 200 can determine the foreground target and background area in the image based on the description text entered in the second input area. Then call the color and brightness adjustment algorithm to perform color, contrast, brightness and other parameter adjustments on the pixels in the background area of ​​the target image to make the background more harmonious or highlight the foreground theme.

[0122] Figure 6 A schematic diagram of an image display method provided in an embodiment of the present application. Figure 6 When the display device supports the touch function, the implementation process of the image display method provided in the embodiment of the present application is described. Figure 6 As shown, the method includes steps 610 to 640 as shown below.

[0123] Step 610 , in response to receiving a touch instruction inputted by a user in a first input area, controlling the display to display a first image drawn by the user in the first input area.

[0124] In some examples, the display device can monitor the touch command input by the user in the first input area in real time, and when the touch command is monitored, the image drawn by the user (i.e., the first image) can be displayed in the first input area according to the touch command. The touch command can be input by the user through the movement of a limb (such as a finger) or a touch device (such as a stylus) on the touch screen of the first input area. It should be noted that the following embodiments can be described by taking the user inputting the touch command through a touch device (such as a stylus) as an example.

[0125] For example, when a user moves a touch device in the first input area, the coordinates of the touch point will change. The display device can detect the change in the coordinates of the touch point to obtain the movement trajectory of the touch point, generate a map image based on the movement trajectory of the touch point, and display the first image drawn by the user in the first input area.

[0126] Figure 7 A schematic diagram of an application interface provided in an embodiment of the present application. Figure 7 As shown, the application interface 700 includes a first input area 710, a second input area 720 and an image display area 730. The first input area 710 on the left side of the application interface 700 displays a description image (such as a first image) drawn by a user.

[0127] Step 620, obtaining a second image generated based on the first image, and controlling the display to display the second image in the image display area.

[0128] For example, the second image displayed in the first input area can be generated by processing the first image through an image generation model. Based on various considerations such as device performance and product requirements, in actual applications, the image generation model can be deployed on a display device or on a server.

[0129] In some examples, the image generation model may be a trained model, and the image generation model may convert the input multimodal information such as image, text, audio and video into a corresponding reference image, and the reference image may meet the requirements of multimodal information. Fig.14 Detailed description will be given in the embodiment shown in FIG.

[0130] For example, after acquiring the second image, the display device may control the display to display the second image in the image display area. Figure 7 As shown, the image display area 730 on the right side of the application interface 700 displays a reference image (such as a second image) generated according to the description image (such as the first image) displayed in the first input area 710 .

[0131] In other words, the image generation model can realize the image-to-image function, converting the image drawn by the user into an image that is more realistic and meets the user's needs.

[0132] Figure 8 A schematic diagram of another image display method provided in an embodiment of the present application is shown below in conjunction with Figure 8 The implementation process of step 620 is described below when the image generation model is deployed on the display device and the server respectively. Figure 8 As shown, the above step 620 includes step 621 and step 622 as shown below.

[0133] Step 621: Process the first image based on the image generation model to obtain a second image, and control the display to display the second image in the image display area.

[0134] Exemplarily, when the image generation model is deployed on a display device, after acquiring the first image, the display device can process the first image based on the image generation model deployed thereon to obtain a second image, and display the second image in the image display area.

[0135] In some embodiments, step 621 may include: based on a preset detection cycle, using an image generation model to process the first image to obtain a second image, and controlling the display to display the second image in the image display area.

[0136] Exemplarily, the preset detection cycle can be set according to user needs, for example, the preset detection cycle can be a preset duration. The display device can input the image drawn by the user in the current detection cycle into the image generation model for processing every preset detection cycle to obtain a second image corresponding to the current detection cycle, and display the second image corresponding to the current detection cycle in the image display area.

[0137] In some examples, the first image displayed in the first input area may change or may not change in two adjacent detection cycles. In other words, the first images corresponding to two adjacent detection cycles may be the same or different.

[0138] For example, when the first input area displays the first image corresponding to the first detection cycle, the display device saves the first image corresponding to the first detection cycle. After entering the second detection cycle after the first detection cycle, if the user continues to draw an image in the first input area, the display device obtains the first image corresponding to the second detection cycle displayed in the first input area, and compares the first image corresponding to the second detection cycle with the first image corresponding to the first detection cycle to determine whether the first image displayed in the first input area has changed.

[0139] For example, the display device may compare the trajectory of the first image corresponding to the second detection cycle with the trajectory of the first image corresponding to the first detection cycle. When the trajectory of the first image corresponding to the second detection cycle completely overlaps with the trajectory of the first image corresponding to the first detection cycle, it indicates that the first image corresponding to the second detection cycle is the same as the first image corresponding to the first detection cycle, that is, the first image displayed in the first input area has not changed between the two detection cycles; when the trajectory of the first image corresponding to the second detection cycle does not completely overlap with the trajectory of the first image corresponding to the first detection cycle, it indicates that the first image corresponding to the second detection cycle is different from the first image corresponding to the first detection cycle, that is, the first image displayed in the first input area has not changed between the two detection cycles.

[0140] Based on the above scheme, the display method provided in the present application can process the first image to obtain the second image based on a preset detection cycle through an image generation model on the display device, thereby realizing the regular synchronization display of the first image drawn by the user in the first input area to the first input area, ensuring that the user can verify and confirm the drawn image, improving the portability of user interaction, and improving the user experience.

[0141] In some embodiments, if it is determined based on a preset detection cycle that the first image corresponding to the current detection cycle is different from the first image corresponding to the previous detection cycle, the image generation model is used to process the first image corresponding to the current detection cycle to obtain the second image corresponding to the current detection cycle; the display is controlled to display the second image corresponding to the current detection cycle in the image display area.

[0142] In some examples, if the display device determines that the first image corresponding to the current detection cycle is different from the first image corresponding to the previous detection cycle, that is, the first image displayed in the first input area has changed, the image generation model can be called to process the first image corresponding to the current detection cycle to obtain the second image corresponding to the current detection cycle, and display the second image in the image display area.

[0143] For example, since the first image corresponding to two adjacent detection cycles changes, the corresponding second image will also change. In order to ensure the consistency between the second image displayed in the image display area and the first image displayed in the first input area, when the first image displayed in the first input area changes, the second image displayed in the second input area needs to be updated accordingly.

[0144] Based on the above scheme, the display device compares the first images of two adjacent detection cycles, and when the first image of the current detection cycle changes from the first image of the previous detection cycle, the display device processes the first image of the current detection cycle through the image generation model, thereby avoiding repeated processing of the image generation model and improving image generation efficiency.

[0145] In some embodiments, if it is determined based on a preset detection cycle that the first image corresponding to the current detection cycle is the same as the first image corresponding to the previous detection cycle, the second image corresponding to the previous detection cycle continues to be displayed in the image display area.

[0146] In some examples, if the display device determines that the first image corresponding to the current detection cycle is the same as the first image corresponding to the previous detection cycle, that is, the first image displayed in the first input area has not changed, the second image corresponding to the previous detection cycle continues to be displayed in the image display area.

[0147] For example, since the first image corresponding to two adjacent detection cycles has not changed, the corresponding second image has not changed either. Therefore, there is no need to update the second image displayed in the second input area, nor is there a need for the image generation model to repeatedly process the first image corresponding to the current detection cycle, thereby improving the processing efficiency of the display device.

[0148] Based on the above scheme, the display device compares the first images of two adjacent detection cycles, and when the first images of two adjacent detection cycles have not changed, it maintains the display of the first image of the previous detection cycle, thereby avoiding repeated processing of the image generation model and improving image generation efficiency.

[0149] In some embodiments, the above step 621 may also include: monitoring a touch event corresponding to a touch instruction, processing the first image based on an image generation model according to the touch event to obtain a second image, and controlling the display to display the second image in the image display area.

[0150] In some examples, in addition to displaying the second image in the image display area based on the preset detection cycle as described above, the display device can also determine the second image displayed in the image display area by monitoring touch events corresponding to touch instructions input by the user in the first input area.

[0151] Exemplarily, the touch events corresponding to the touch instructions include touch down events, touch move events, and touch up events. For example, when a user triggers a touch event by inputting a touch instruction in the first input area through a touch device, the type of the touch event can be determined first.

[0152] For example, when the touch device changes from never touching the touch screen corresponding to the first input area to touching the touch screen corresponding to the first input area, the touch event corresponding to the input touch instruction is a touch down event. When the touch device continues to touch the touch screen corresponding to the first input area within a preset time, the touch event corresponding to the input touch instruction is a touch move event. When the touch device changes from touching the touch screen corresponding to the first input area to leaving the touch screen corresponding to the first input area, the touch event corresponding to the input touch instruction is a touch lift event.

[0153] In some examples, when a user draws an image in the first input area, a touch command input within a drawing cycle may trigger a touch-down event, a touch-move event, and a touch-up event in sequence. For example, when a touch command triggers a touch-down event, it indicates that the user begins drawing an image in the first input area, that is, begins drawing the image in the current drawing cycle; when a touch command triggers a touch-move event, it indicates that the user is drawing an image in the first input area; when a touch command triggers a touch-up event, it indicates that the user completes drawing the image in the first input area, that is, completes drawing the image in the current drawing cycle.

[0154] It should be noted that the touch command input by the user in a drawing cycle may also trigger a touch down event and a touch up event in sequence. In this case, the user may not adjust or update the first image of the first input area in the current drawing cycle.

[0155] Based on the above scheme, the display method provided by the present application can obtain the second image by processing the first image through the image generation model on the display device based on the touch event corresponding to the touch instruction, so that when the user completes a stage of drawing in the first input, the drawn image can be synchronously displayed to the first input area in time, ensuring that the user can verify and confirm the drawn image, improving the portability of user interaction, and improving the user experience.

[0156] In some embodiments, while monitoring a touch down event, the touch up event is continuously monitored; in response to monitoring the touch up event, the first image is processed based on an image generation model to obtain a second image, and the display is controlled to display the second image in the image display area.

[0157] In some examples, when the display device detects a touch-down event, it indicates that the current drawing cycle begins. The display device can continue to detect touch events. When a touch-up event is detected, it indicates that the current drawing cycle ends. The display device can respond to the touch-up event by processing the first image corresponding to the current drawing cycle displayed in the first input area to obtain a second image, and control the display to display the second image in the image display area.

[0158] That is to say, at the end of a drawing cycle, the display device can process the first image corresponding to the drawing cycle through the image generation model, thereby updating the second image in the image display area.

[0159] Based on the above scheme, when the touch down event is triggered, the touch up event is triggered, indicating that the user completes a drawing cycle and stops drawing. Therefore, the currently drawn image can be processed in time through the image generation model to ensure the timeliness of image display.

[0160] In some embodiments, the above step 621 may also include: in response to receiving a confirmation instruction input by the user, processing the first image based on the image generation model to obtain a second image, and controlling the display to display the second image in the image display area.

[0161] In some examples, in addition to displaying the second image in the image display area based on a preset detection cycle, or displaying the second image in the image display area by monitoring touch events corresponding to touch commands entered by the user in the first input area, the display device can also determine the second image displayed in the image display area through a confirmation command entered by the user.

[0162] Exemplarily, when the first image is displayed in the first input area, the user can input an instruction through the control device, or can also input a confirmation instruction through voice, and the confirmation instruction is used to indicate that the first image currently displayed in the first input area can be used to generate the corresponding second image. Therefore, after receiving the confirmation instruction, the display device responds to the confirmation instruction and processes the first image currently displayed in the first input area through the image generation model to obtain the second image, and displays the second image in the image display area.

[0163] In some examples, the application interface may include a confirmation control, and the user may input a confirmation instruction to the confirmation control through a control device (such as a remote control).

[0164] In some examples, the user can input a confirmation instruction at any time during the first image drawing process as required. Figure 7 When the user draws an image of a bird in the first input area 710, the user can input a confirmation command after completing the drawing of the entire bird image, or can input a confirmation command when completing the drawing of part of the bird image, which is not limited in this embodiment of the present application.

[0165] For example, the user can input a confirmation command when the drawing of a part of the bird image is completed, and the image generation model generates a corresponding reference image based on the part of the bird image, and displays it in the image display area 730. The user continues to draw the bird image in the first input area, and when the drawing of the remaining part of the image is completed, the user inputs a confirmation command again, and the image generation model generates a corresponding reference image based on the complete bird image currently displayed in the first input area 710, and displays it in the image display area 730.

[0166] Based on the above scheme, the display method provided in the present application can process the first image to obtain the second image based on the confirmation instruction input by the user through the image generation model on the display device, thereby realizing the synchronous display of the drawn image to the first input area according to user needs, thereby improving the user experience.

[0167] Step 622: Send the first image to the server to receive the second image generated by the server based on the first image, and control the display to display the second image in the image display area.

[0168] Exemplarily, when the image generation model is deployed on a server, after determining the first image, the display device may send the first image to the server, so that the server processes the first image to obtain the second image. For example, after receiving the first image sent by the display device, the server may process the first image through the image generation model deployed thereon to obtain the second image. The server sends the second image to the display device, and the display device controls the display to display the second image in the image display area.

[0169] In some embodiments, sending the first image to the server in the above step 622 includes: sending the first image to the server based on a preset detection cycle; or, monitoring a touch event corresponding to a touch instruction, and sending the first image to the server based on the touch event; or, sending the first image to the server in response to receiving a confirmation instruction input by the user.

[0170] In some examples, the process of the display device sending the first image to the server is similar to the process of the display device generating the second image based on the first image in the above step 621. For example, the display device may send the first image to the server based on a preset detection cycle, or send the first image to the server based on a touch instruction corresponding to a monitored processing instruction, or send the first image to the server based on a confirmation instruction input by the user.

[0171] In some embodiments, when sending a first image to a server based on a preset detection cycle, if the display device determines that the first image corresponding to the current detection cycle is different from the first image corresponding to the previous detection cycle, it indicates that the first image displayed in the first input area has been updated and the second image corresponding to the current detection cycle needs to be regenerated. In this case, the display device can send the first image corresponding to the current detection cycle to the server, and the server processes the first image corresponding to the current detection cycle based on the image generation model to obtain the second image corresponding to the current detection cycle.

[0172] Exemplarily, if the display device determines that the first image corresponding to the current detection cycle is the same as the first image corresponding to the previous detection cycle, it indicates that the first image displayed in the first input area has not been updated from the previous detection cycle to the current detection cycle, and therefore, the second image displayed in the image display area does not need to be updated, that is, the server does not need to regenerate the second image. In this case, the display device does not need to send the first image corresponding to the current detection cycle to the server, but controls the display to continue to display the second image determined in the previous detection cycle in the image display area.

[0173] In some embodiments, the display device may also continue to monitor the touch-up event when monitoring the touch-down event; and send the first image to the server in response to monitoring the touch-up event.

[0174] In some examples, when the display device detects a touch-up event, indicating that the current drawing cycle has ended, the first image corresponding to the current drawing cycle can be sent to the server, and the first image can be processed by an image generation model deployed on the server to generate a corresponding second image.

[0175] Exemplarily, when the display device receives a confirmation instruction input by the user, it can also send the first image currently displayed in the first input area to the server in response to the confirmation instruction, and process the first image through the image generation model deployed on the server to generate a corresponding second image.

[0176] Step 630: In response to receiving the first description text input by the user, control the display to display the first description text in the second input area.

[0177] Exemplarily, the first description text is a text input by the user or a text generated according to a description voice input by the user. The first description text is a text description of the desired image by the user. After determining the first description text, the display device controls the display to display the first description text in the second input area.

[0178] In some examples, the application interface may include a text input control and a voice input control, and the user may select the text input control through the control device to input descriptive text through the text input control; alternatively, the user may select the voice input control through the control device to input descriptive voice through the voice input control.

[0179] For example, the user can select a text input control through the remote control to call out a virtual input keyboard, and input corresponding text through the virtual input keyboard in the second input area 720 to obtain the first description text. Alternatively, the user can also select a voice input control to call out the voice input mode of the display device to input a description voice.

[0180] Fig. 9 A schematic diagram of another application interface provided in an embodiment of the present application. Fig. 9 As shown in (a) in FIG. 7 , the application interface 700 includes a text input control 910 and a voice input control 920 . The user can input a description text through the text input control 910 , and the description text displayed in the second input area 720 is “bird, blue feathers on the back; white feathers on the belly”.

[0181] In some embodiments, step 630 includes: in response to receiving a descriptive speech input by a user, processing the descriptive speech to obtain an audio spectrum; obtaining a first descriptive text obtained after processing the audio spectrum based on a speech recognition model, and controlling the display to display the first descriptive text in the second input area.

[0182] Exemplarily, when the first description text input by the user is a text generated according to the description voice input by the user, after obtaining the description voice input by the user, the display device first processes the description voice to obtain an audio spectrum. After obtaining the audio spectrum, the audio spectrum can be processed by a speech recognition model to obtain the first description text. The speech recognition model can be a pre-trained model, and the speech recognition model can be deployed on the display device or on a server.

[0183] In some examples, when the speech recognition model is deployed on a display device, the display device can process the audio spectrum corresponding to the acquired description speech through the speech recognition model deployed thereon to obtain the first description text. When the speech recognition model is deployed on a server, after obtaining the audio spectrum corresponding to the description speech, the display device first sends the audio spectrum to the server, and the server processes the audio spectrum through the speech recognition model deployed thereon to obtain the first description text, and sends the first description text to the display device. After obtaining the first description text, the display device controls the display to display the first description text in the second input area.

[0184] Based on the above scheme, the display method provided by the present application can convert the description voice into description text based on the description voice input by the user through the trained speech recognition model, and display the description text in the second input area, so as to facilitate the subsequent generation of the third image. By converting the description voice into description text, the user can avoid the need to enter text in the second input area, thereby improving input efficiency and user experience.

[0185] In some embodiments, controlling the display to display the first description text in the second input area in the above step 630 includes: obtaining the number of words of the third description text currently displayed in the second input area; if the number of words in the third description text is zero, adding the first description text to the second input area, and controlling the display to display the first description text in the second input area; if the number of words in the third description text is not zero, adding the first description text to the third description text to update the description text in the second input area, and controlling the display to display the updated description text in the second input area.

[0186] In some examples, after receiving the first description text input by the user, the display device may first determine whether the second input area currently displays the description text. For example, the display device may obtain the number of words of the third description text displayed in the second input area, and the number of words of the third description text may be zero or non-zero.

[0187] For example, when the number of characters in the third description text is zero, it indicates that the second input area is currently displayed as empty. In this case, the display device can display the first description text entered by the user in the second input area. When the number of characters in the third description text is not zero, it indicates that the second input area is currently displayed as not empty. In this case, the user input can be added to the first description text after adding it to the third description text to update the description text displayed in the second input area, and the display can be controlled to display the updated description text in the second input area. In other words, when the number of characters in the third description text is not zero, the second input area can display a new description text composed of the first description text and the third description text.

[0188] Continue to refer to Fig. 9 ,like Fig. 9 As shown in (b), the second input area 720 currently displays the third description text "bird, blue feathers on the back". In this case, if the first description text "white feathers on the belly" input by the user is received, the display device obtains the number of words in the third description text. Since the number of words in the third description text is not zero, the display device can add the first description text input by the user to the third description text, and display the updated description text "bird, blue feathers on the back; white feathers on the belly" in the second input area 720.

[0189] Based on the above scheme, when the display method provided by the present application receives the first description text input by the user, by obtaining the number of words of the third description text displayed in the second input area, and when it is determined that the number of words of the third description text is not zero, that is, there is a description text in the second input area, the newly input first description text is added to the original third description text, thereby realizing the supplement of the description text in the second input area and further improving the convenience of user interaction.

[0190] Step 640: Process the first description text and the first image to obtain a third image that conforms to the first description text, and control the display to display the third image in the image display area.

[0191] In some examples, after determining the first description text, the display device may process the first description text and the first image based on the first description text and the first image through an image generation model to obtain a third image that conforms to the first description text and the first image.

[0192] Fig.10A schematic diagram of another image display method provided in an embodiment of the present application is shown below in combination with Fig.10 The implementation process of step 640 when the image generation model is deployed on the display device and the server is described. Fig.10 As shown, the above step 640 includes the following steps 641 and 642. Among them, step 641 is the implementation process of step 640 when the image generation model is deployed on the display device, and step 642 is the implementation process of step 640 when the image generation model is deployed on the server.

[0193] Step 641, processing the first description text and the first image based on the image generation model to obtain a third image that conforms to the first description text, and controlling the display to display the third image in the image display area.

[0194] Exemplarily, when the image display area displays the second image, in response to the user inputting the first description text, the first description text and the first image are processed based on the image generation model deployed in the display device to obtain a third image. The third image meets the requirements of the first image displayed in the first input area and the requirements of the first description text displayed in the second input area.

[0195] In some examples, the third image may be different from the second image or the same as the second image. For example, when the number of words in the first description text is zero, the third image is the same as the second image, or when the content described in the first description text has been described in the first image, the third image may also be the same as the second image. This embodiment of the application is not limited to this.

[0196] In some examples, after generating the third image based on the image generation model, the display device may control the display to display the third image in the image display area. For example, when the third image is the same as the second image, the display device may control the display to maintain displaying the second image in the image display area; when the third image is different from the second image, the display device may control the display to exit displaying the second image in the image display area and display the third image.

[0197] In some embodiments, after step 641 above, the method further includes: in response to receiving a modification instruction for the first image input by the user in the first input area, determining a fourth image; if the fourth image is different from the first image, processing the first description text and the fourth image based on the image generation model to obtain a fifth image that conforms to the first description text, and controlling the display to display the fifth image in the image display area.

[0198] In some examples, the user may modify the first image displayed in the first input area, and after modifying the first image, the first input area may display the modified image (eg, the fourth image).

[0199] For example, when the first input area displays the first image, the user can move the touch screen corresponding to the first input area through a limb (such as a finger) or a touch device (such as a stylus) to input a modification instruction. When the display device detects the modification instruction, a modified fourth image can be generated according to the first image and the modification instruction. For example, the display device can generate the fourth image according to the movement trajectory corresponding to the first image and the modification instruction.

[0200] In some examples, after determining the fourth image, the display device may compare the fourth image with the first image previously displayed in the first input area to determine whether the fourth image is the same as the first image.

[0201] For example, when the first image is the same as the fourth image, it indicates that the user has not modified the first image, or the user's modification operation on the first image is unsuccessful. In this case, the display device controls the display to maintain displaying the currently displayed second image in the image display area.

[0202] For another example, when the first image and the fourth image are different, it indicates that the user has modified the first image. In this case, the display device needs to reprocess the fourth image and the first description text through the image generation model to generate a fifth image that conforms to the fourth image and the first description text. After generating the fifth image, the display device controls the display to exit the display of the second image in the image display area and display the fifth image.

[0203] Based on the above scheme, the present application can support users to modify the first image drawn in the first input area, and after the modification, reprocess the modified fourth image and the first description text displayed in the second input area through the image generation model to ensure that a fifth image is generated that conforms to the modified fourth image and the first description text.

[0204] In some embodiments, after step 641 above, the method further includes: in response to receiving a second descriptive text input by a user, controlling the display to display the second descriptive text in the second input area; processing the second descriptive text and the first image based on an image generation model to obtain a sixth image that conforms to the second descriptive text, and controlling the display to display the sixth image in the image display area.

[0205] In some examples, in addition to modifying the first image displayed in the first input area, the user can also modify the first description text displayed in the second input area as needed. For example, the user can re-enter the second description text in the second input area. The second description text is the text entered by the user or the text generated according to the description voice entered by the user, and the second description text is different from the first description text.

[0206] For example, the user may first delete the first description text displayed in the second input area, and after deleting the first description text, input the second description text and control the second input area to display the second description text.

[0207] After acquiring the second description text, the display device can process the second description text and the first image through the image generation model to obtain a sixth image that conforms to the first image and the second description text, and control the display to display the sixth image in the image display area.

[0208] Exemplarily, the user may also modify the first image displayed in the first input area and the first description text displayed in the second input area at the same time.

[0209] In some examples, the user modifies the first image displayed in the first input area to obtain a fourth image, and the timing starts when the fourth image is displayed in the first input area. Before the timing reaches the preset duration threshold, if the user re-enters the second description text in the second input area, the display device can process the fourth image and the second description text through the image generation model to obtain an image that conforms to the fourth image and the second description text, and display the image in the image display area. Among them, the fourth image is different from the first image, and the second description text is different from the first description text. The preset duration threshold can be set according to user needs, and the embodiments of the present application are not limited to this.

[0210] In other examples, after receiving the modification instruction of the first image input by the user in the first input area, the display device can display the modified fourth image in the first input area, process the fourth image and the first description text through the image generation model to obtain the modified image, and display the modified image in the display image. In this process, if the user re-enters the second description text in the second input area, the display device processes the fourth image and the second description text through the image generation model to obtain an image that conforms to the fourth image and the second description text, and displays the image in the image display area.

[0211] It should be noted that the display device can process the modifications of the first image in the first input area and the first description text in the second input area in sequence according to the order in which the user modifies them, or it can simultaneously process the modifications of the first image within a preset time threshold and the modifications within the first description text according to a preset time threshold. The embodiments of the present application are not limited to this.

[0212] Based on the above scheme, the present application can support users to modify the first description text displayed in the second input area, and after the modification, reprocess the first image displayed in the first input area and the modified second description text displayed in the second input area through the image generation model to ensure that a sixth image is generated that conforms to the first image and the modified second description text.

[0213] The display method provided by the embodiment of the present application is provided with an image generation model deployed on the display device, and the image generation model can process the first image input by the user in the first input area and / or the first description text input in the second input area, so as to generate a third image that meets the user's needs, and display the third image in the image display area. Therefore, the embodiment of the present application provides a function that can realize the generation of corresponding images based on multimodal information, and improves the accuracy of the image generated by the display device, thereby improving the user experience. At the same time, deploying the image generation model on the display device can further improve the security and generation efficiency of the third image generation process. In addition, since the first input area, the second input area and the image display area on the application interface can display different content at different stages, the convenience of user interaction is further improved.

[0214] Step 642, receiving a third image generated from the server based on the first description text and the first image, and controlling the display to display the third image in the image display area.

[0215] In some embodiments, when the image generation model is deployed on a server, after the above step 630, the method further includes: sending the first image and the first description text to the server.

[0216] Exemplarily, after receiving the first description text, the display device may send the first image and the first description text to the server. After receiving the first image and the first description text, the server processes the first image and the first description text through the image generation model deployed thereon to generate a third image, and displays the third image in the image display area.

[0217] In some examples, when the first description text is text input by the user, the display device can directly send the first description text to the server; when the first description text is text generated by the description voice input by the user, if the voice recognition model is deployed on the server, the display device can send the first image and the description voice to the server, and the voice recognition model on the server converts the description voice into the first description text. If the voice recognition model is not on the display device, the display device converts the description voice into the corresponding first description text through the voice recognition model, and then sends the first image and the first description text to the server.

[0218] For example, after receiving the first image and the first description text, the server processes the first image and the first description text through the image generation model deployed thereon, obtains a third image that conforms to the first image and the first description text, and sends the third image to the display device. After receiving the second image, the display device can display the third image in the image display area.

[0219] Exemplarily, if the image generation model is deployed on the server, when the user inputs a modification instruction for the first image in the first input area, the display device determines the modified fourth image, and when the fourth image is different from the first image currently displayed in the first input area, sends the first description text and the fourth image to the server, and the server processes the first description text and the fourth image through the image generation model deployed thereon, generates a fifth image that conforms to the first description text and the fourth image, and sends the fifth image to the display device, the display device receives the fifth image, and controls the display to display the fifth image in the image display area.

[0220] Correspondingly, if the image generation model is deployed on the server, when the display device receives the second description text input by the user, it can send the second description text and the first image to the server. The server processes the second description text and the first image through the image generation model deployed thereon, generates a sixth image that conforms to the second description text and the first image, and sends the sixth image to the display device. The display device receives the sixth image and controls the display to display the sixth image in the image display area.

[0221] It should be noted that when the image generation model is deployed on the server, the image displayed in the image display area is generated by the server and sent to the display device. The image display process in the image display area is similar to that in the above embodiment where the image generation model is deployed on the display device. To avoid repetition, it will not be repeated here.

[0222] The display method provided in the embodiment of the present application can send the first image and / or the first description text to the server after obtaining the first image drawn by the user in the first input area and / or the first description text input in the second input area, so as to process the first image and / or the first description text through the image generation model on the server, thereby generating a third image that meets the user's needs, and sending the generated third image to the display device, so that the display device displays the third image in the image display area. Therefore, the embodiment of the present application provides a function that can realize the generation of corresponding images based on multimodal information, and improves the accuracy of the image generated by the display device, thereby improving the user experience. At the same time, deploying the image generation model on the server can have higher processing power and more powerful computing resources, and is easy to manage and expand. In addition, since the first input area, the second input area and the image display area on the application interface can display different content at different stages, the convenience of user interaction is further improved.

[0223] Fig.11 A schematic diagram of another image display method provided in an embodiment of the present application is shown as follows: Fig.11 As shown, the image display method includes steps 1110 to 1140 as shown below.

[0224] Step 1110, acquiring the seventh image, extracting key feature points of the seventh image to obtain an eighth image, and controlling the display to display the eighth image in the first input area.

[0225] Exemplarily, the seventh image is an image captured by an image collector and / or an image uploaded by a mobile terminal. When the display device does not support touch operation or the user cannot draw the first image in the first input area, the display device can also use the image captured by the image collector or the image uploaded by the user through the mobile terminal as the seventh image and display the seventh image in the first input area. When the display device supports the touch function, if the user does not want to draw the first image in the first input area, the seventh image can also be captured by the image collector or uploaded by the mobile terminal.

[0226] In some examples, the application interface of the display includes an image acquisition control, which may include a shooting interaction control and a code scanning interaction control. The user may provide the seventh image to the display device through the shooting interaction control or the code scanning interaction control. Figure 7 The interface on the right side of the second input area 720 includes a shooting interaction control 740 and a code scanning interaction control 750.

[0227] In some embodiments, acquiring the seventh image in the above step 1110 includes: in response to receiving an image acquisition instruction input into the shooting interaction control of the application interface, receiving an image acquired by the image collector, and displaying the image acquired by the image collector in the first input area; in response to the shooting instruction input by the user, determining the captured image as the seventh image.

[0228] In some examples, the application interface of the display includes a shooting interaction control, and the user can input an image acquisition instruction to the shooting interaction control. For example, the user can input the image acquisition instruction to the shooting interaction control through a control device (such as a remote control), or the user can also input the image acquisition instruction to the shooting interaction control through voice, which is not limited in the embodiments of the present application.

[0229] For example, the image acquisition instruction is used to call the image collector in the display device. When the controller receives the image acquisition instruction, it starts the image collector on the display device so that the image collector can take pictures of the current environment. During the image capture process of the image collector, the first input area can display the image of the current environment captured by the image collector, so that the user can view the image captured by the image collector through the first input area and determine whether to confirm the capture. When the user enters the capture instruction, that is, enters the instruction to confirm the capture, the image collector acquires the captured image and sends the captured image to the controller. The controller determines the captured image as the seventh image and controls the display to display the seventh image in the first input area.

[0230] For example, Figure 7 As shown, when the user moves the focus to the shooting interaction control 740 through the remote control and presses the remote control to select the shooting interaction control, an image acquisition instruction can be input. The image acquisition instruction calls the image collector on the display device to start, enter the shooting mode, and display the shot image in the first input area.

[0231] In some embodiments, acquiring the seventh image in the above step 1110 also includes: in response to receiving an image acquisition instruction input into the scan code interactive control of the application interface, controlling the display to display the identification code in the first input area; and receiving the seventh image uploaded by the mobile terminal by scanning the identification code.

[0232] In some examples, the application interface of the display device may further include a code scanning interactive control. The user may input an image acquisition instruction to the code scanning interactive control to call out an identification code, which is used to upload the image after the terminal device scans the code. In response to the image acquisition instruction, the display device controls the display to display the identification code in the first input area, and the user scans the identification code through a mobile terminal (such as a mobile phone) to send the desired image (such as the seventh image) to the display device. The display device receives the seventh image and displays the seventh image in the first input area.

[0233] In some examples, after acquiring the seventh image, the first input area may exit the display of the identification code and display the seventh image. The identification code may be a QR code, a bar code, an RFID tag, or other forms, which is not limited in the embodiments of the present application. Alternatively, the display device also exits the display of the identification code in the first input area after detecting that the user has completed scanning the identification code, which is not limited in the embodiments of the present application.

[0234] For example, Figure 7 As shown, when the user moves the focus to the code scanning interactive control 750 through the remote control and presses the remote control to select the code scanning interactive control, an image acquisition instruction can be input. The image acquisition instruction can call an identification code and display the identification code in the first input area. The user scans the identification code through the mobile terminal to upload the seventh image.

[0235] In some embodiments, the above-mentioned receiving the seventh image uploaded by the mobile terminal by scanning the identification code includes: obtaining the URL address sent by the server; generating an image receiving request according to the URL address; sending the image receiving request to the server, so that the server sends the seventh image in response to the image receiving request; and receiving the seventh image sent by the server.

[0236] In some examples, the URL address is the storage address of the seventh image sent to the server by the terminal device after scanning the identification code.

[0237] For example, after scanning the identification code, the mobile terminal can send the image to be uploaded (such as the seventh image) to the server. The server stores the seventh image, generates a URL address according to the storage location of the seventh image, and sends the URL address to the display device. The display device responds to the image acquisition instruction, generates an image receiving request according to the URL address, and sends the image receiving request to the server. The server sends the seventh image to the display device according to the URL address in the image receiving request. The display device receives the seventh image and displays the seventh image in the first input area.

[0238] In some embodiments, the above step 1110 includes: acquiring a seventh image, and controlling the display to display the seventh image in the first input area; extracting key feature points of the seventh image to obtain an eighth image, and controlling the display to display the eighth image in the first input area.

[0239] In some examples, after acquiring the seventh image, the display device controls the display to display the seventh image in the first input area. The display device extracts key feature points of the seventh image displayed in the first input area, uses the feature point extraction map as the eighth image, and controls the display to display the eighth image in the first input area.

[0240] In some embodiments, the eighth image is obtained by extracting key feature points of the seventh image in the above-mentioned step 1110, including: determining the image type of the seventh image; if the image type of the seventh image is the first type, extracting the key feature points of the seventh image according to a skeleton feature extraction algorithm to obtain the eighth image; if the image type of the seventh image is the second type, extracting the key feature points of the seventh image according to an edge detection algorithm and a skeleton feature extraction algorithm to obtain the eighth image.

[0241] In some examples, the image type includes a first type and a second type, wherein the first type of image is a line drawing and the second type of image is a non-line drawing. Since the seventh image may be taken or uploaded by a user through a mobile terminal, the seventh image may not contain obvious line features. In order for the image generation model to generate a more accurate image based on the seventh image, the display device may perform image processing and feature extraction operations on the acquired seventh image.

[0242] For example, when the seventh image is a line drawing, the display device can extract key feature points of the seventh image according to a skeleton feature extraction algorithm to convert the seventh image into an eighth image. When the seventh image is a non-line drawing, the display device first converts the seventh image into a boundary image according to an edge detection algorithm, and then converts the boundary image into an eighth image according to a skeleton feature extraction algorithm. The eighth image can be a skeleton structure diagram.

[0243] In some examples, the display device performs grayscale conversion and binarization on the seventh image to generate a binary image, performs edge detection on the binary image using an edge detection algorithm to generate a boundary image, extracts morphological features from the boundary image to generate a skeleton structure map, and displays the generated skeleton structure map as the eighth image in the first input area.

[0244] In some examples, after determining the eighth image, the first input area may exit display of the seventh image and display the eighth image, or the first input area may display the seventh and eighth images at the same time, or the first input area may display the seventh or eighth image according to user selection.

[0245] In some embodiments, in response to the focus selecting the first option, the display is controlled to display the seventh image in the first input area; in response to the focus switching to the second option, the display is controlled to display the eighth image in the first input area.

[0246] In some examples, the application interface displayed on the display may include at least one option control.

[0247] For example, when the application interface includes an option control, such as a first option control, the first option control may correspond to both the first option and the second option. When the first input area displays the seventh image, it indicates that the first option control currently corresponds to the first option. In this case, when the user moves the focus to the first option control again through the remote control and selects the first option control through the remote control, the first option control corresponds to the second option. In this case, the display may exit the display of the seventh image in the first input area and display the eighth image.

[0248] For another example, when the application interface includes two option controls, such as a first option control and a second option control, the first option control may correspond to the first option, and the second option control may correspond to the second option. When the user moves the focus to the first option control through the remote control, in response to the focus selecting the first option corresponding to the first option control, the seventh image is displayed in the second input area; when the user moves the focus to the second option control through the remote control, in response to the focus selecting the second option corresponding to the second option control, the eighth image is displayed in the second input area.

[0249] Fig.12 A schematic diagram of another application interface provided in an embodiment of the present application. Fig.12 As shown in (a), the first input area 710 includes two option controls, a first option control 711 and a second option control 712. When the user selects the first option control 711 through the remote control, the display area of ​​the first input area 710 displays the seventh image. When the user selects the second option control 712 through the remote control, the display area of ​​the first input area 710 displays the eighth image.

[0250] Exemplarily, the first input area may also display the seventh image and the eighth image at the same time. For example, the first input area may be divided into two display areas, wherein one display area (such as the first display area) may display the seventh image, and the other display area (such as the second display area) may display the eighth image. Through the first input area, the user may view both the original image (such as the seventh image) that has been taken or uploaded and the skeleton image (such as the eighth image) generated based on the original image.

[0251] In some examples, the sizes of the first display area and the second display area may be the same or different. For example, when the sizes of the first display area and the second display area are different, the second display area may be larger than the first display area, such as the first display area may be a partial area of ​​the second display area. The embodiment of the present application does not limit the size and distribution of the first display area and the second display area.

[0252] For example, Fig.12 As shown in (b) of FIG. 7 , the display area of ​​the first input area 710 includes a display area for displaying the seventh image and a display area for displaying the eighth image. The display area for displaying the seventh image is a partial area of ​​the display area for displaying the eighth image.

[0253] Step 1120: Process the eighth image based on the image generation model to obtain a ninth image, and control the display to display the ninth image in the image display area.

[0254] In some examples, after determining the eighth image, the display device may process the eighth image through an image generation model to obtain a ninth image, and display the ninth image in the image display area.

[0255] It should be noted that step 1120 is similar to step 621 in the above embodiment, wherein the eighth image in step 1120 is equivalent to the first image in step 621, and the ninth image in step 1120 is equivalent to the second image in step 621. The image processing process of the image generation model and the image display process in the image display area have been described in detail in the above step 621. To avoid repetition, they will not be repeated here.

[0256] Step 1130: In response to receiving the first description text input by the user, control the display to display the first description text in the second input area.

[0257] It should be noted that step 1130 is similar to step 630 in the above embodiment and will not be described again here to avoid repetition.

[0258] Step 1140, processing the first description text and the eighth image based on the image generation model to obtain a tenth image that conforms to the first description text, and controlling the display to display the tenth image in the image display area.

[0259] In some examples, after acquiring the first descriptive text, the display device can process the first descriptive text and the eighth image based on the image generation model to obtain a tenth image that conforms to the first descriptive text and the eighth image, and display the tenth image in the image display area.

[0260] It should be noted that step 1140 is similar to step 640 in the above embodiment and will not be described again here to avoid repetition.

[0261] In some embodiments, after step 1140, the method further includes: in response to receiving a modification instruction for the eighth image input by the user in the first input area, determining the eleventh image; if the eleventh image is different from the eighth image, processing the first description text and the eleventh image based on the image generation model to obtain a twelfth image that conforms to the first description text, and controlling the display to display the twelfth image in the image display area.

[0262] In some examples, when the display device supports a touch function and the first input area displays the eighth image, the user can also modify the eighth image displayed in the first input area. For example, the user can move in the first input area through a limb (such as a finger) or a touch device (such as a stylus) to input a modification instruction. When the display device detects the modification instruction, a modified eleventh image can be generated based on the eighth image and the modification instruction. The display device determines whether the eleventh image is the same as the eighth image. When it is determined that the eleventh image is different from the eighth image, the eleventh image and the first description text displayed in the second input area are processed by the image generation model to obtain the twelfth image. Among them, the twelfth image meets the requirements of the eleventh image and the first description text.

[0263] It should be noted that the modification of the eighth image by the user is similar to the modification process of the first image in the above embodiment, and will not be described again here to avoid repetition.

[0264] In some embodiments, based on a preset detection cycle, an image generation model is used to process the first description text and the eleventh image to obtain the twelfth image, and the display is controlled to display the twelfth image in the image display area; or, a touch event corresponding to a touch instruction is monitored, and according to the touch event, the first description text and the eleventh image are processed based on the image generation model to obtain the twelfth image, and the display is controlled to display the twelfth image in the image display area; or, in response to receiving a confirmation instruction input by the user, the first description text and the eleventh image are processed based on the image generation model to obtain the twelfth image, and the display is controlled to display the twelfth image in the image display area.

[0265] It should be noted that the display device can also generate the twelfth image based on a preset detection cycle, a touch event corresponding to a touch instruction, or a confirmation instruction input by the user. The process of generating the twelfth image based on the first description text and the eleventh image in the embodiment of the present application is similar to the process of generating the second image based on the first image in the above embodiment. To avoid repetition, it will not be repeated here.

[0266] In some embodiments, after step 1140, the method further includes: in response to receiving a second descriptive text input by a user, controlling the display to display the second descriptive text in the second input area; processing the second descriptive text and the eighth image based on the image generation model to obtain a thirteenth image that conforms to the second descriptive text, and controlling the display to display the thirteenth image in the image display area.

[0267] In some examples, the display device may also update the first description text displayed in the second input area to obtain a second description text, and process the second description text and the eighth image displayed in the first input area through the image generation model to obtain a thirteenth image, and display the thirteenth image in the image display area. The thirteenth image meets the requirements of the second description text and the eighth image.

[0268] The display method provided by the embodiment of the present application has an image generation model deployed thereon, and the image generation model can process the seventh image input by the user in the first input area by shooting or uploading, and / or the first description text input in the second input area, so as to generate an image that meets the user's needs and display it in the image display area. Therefore, the embodiment of the present application provides a function that can realize the generation of corresponding images based on multimodal information, and improves the accuracy of the image generated by the display device, thereby improving the user experience. At the same time, deploying the image generation model on the display device can improve the security generation efficiency of the third image generation process. In addition, since the first input area on the application interface can display the images taken and uploaded by the user, the convenience of user interaction is further improved.

[0269] Fig.13 This is a schematic diagram of another image display method provided in an embodiment of the present application. It should be noted that: Fig.13 The image display method shown is similar to Fig.11 The difference between the image display methods shown is that Fig.12 In the image display method shown, the image generation model is deployed on the display device. Fig.13 In the image display method shown in FIG. 1 , the image generation model is deployed on the server. Fig.13 As shown, the method includes steps 1310 to 1340 as shown below.

[0270] Step 1310, acquiring the seventh image, extracting key feature points of the seventh image to obtain an eighth image, and controlling the display to display the eighth image in the first input area.

[0271] Among them, the seventh image is an image taken by an image collector and / or an image uploaded by a mobile terminal.

[0272] It should be noted that step 1310 is similar to step 1110 in the above embodiment and will not be described again here to avoid repetition.

[0273] Step 1320: Send the eighth image to the server to receive a ninth image generated from the server based on the eighth image, and control the display to display the ninth image in the image display area.

[0274] In some examples, step 1320 differs from step 1120 in the above embodiment in that after determining the eighth image, the display device needs to send the eighth image to the server, and obtain the ninth image by processing the eighth image through the image generation model on the server. The server sends the generated ninth image to the display device, and the display device displays the ninth image in the image display area.

[0275] Step 1330: In response to receiving the first description text input by the user, control the display to display the first description text in the second input area, and send the eighth image and the first description text to the server.

[0276] The first description text is text input by the user or text generated according to the description voice input by the user.

[0277] In some examples, step 1330 differs from step 1130 in the above embodiment in that after determining the first description text, the display device sends the first description text and the eighth image to the server for processing instead of being processed by the display device.

[0278] Step 1340, receiving a tenth image generated from the server based on the first description text and the eighth image, and controlling the display to display the tenth image in the image display area.

[0279] In some examples, step 1340 differs from step 1140 in the above embodiment in that, after receiving the first description text, the server processes the first description text and the eighth image through the image generation model deployed on the server to obtain the tenth image. The server sends the tenth image to the display device, and the display device controls the display to display the tenth image in the image display area.

[0280] The display method provided in the embodiment of the present application can send the first image and / or the first description text to the server after acquiring the seventh image input by the user in the first input area by shooting or uploading, and / or the first description text input in the second input area, so as to process the seventh image and / or the first description text through the image generation model on the server, thereby generating an image that meets the user's needs, and send the generated image to the display device so that the display device displays the image in the image display area. Therefore, the embodiment of the present application provides a function that can realize the generation of corresponding images based on multimodal information, and improves the accuracy of the image generated by the display device, thereby improving the user experience. At the same time, deploying the image generation model on the server can have higher processing power and more powerful computing resources, and is easy to manage and expand. In addition, since the first input area on the application interface can display images taken and uploaded by the user, the convenience of user interaction is further improved.

[0281] The above-mentioned embodiment provides a detailed description of the display device and the image display method. The image displayed in the image display area of ​​the display device is generated by an image generation model based on the image information displayed by the first input and the text information displayed in the second input area. The following describes the process of the image generation model in the above-mentioned embodiment generating an image based on the multimodal information input by the user.

[0282] In some examples, with the rapid development of image generation technology, artificial intelligence and machine learning technologies can realize the automatic generation of images, but related technologies are still focused on generating images based on single modal information. For example, generating images based on text information or generating corresponding images based on image information. With the increase in user personalized needs, this single modal information generation image can no longer meet user needs.

[0283] Based on this, an embodiment of the present application provides an image generation method, which processes the multimodal information input by the user through a trained image generation model to generate a target image that meets the requirements (i.e., the image displayed in the image display area in the above embodiment).

[0284] Combine the following Fig.14 The process of generating an image by the image generation model is described. It should be noted that since the image generation model can be deployed on a server or a display device, the image generation method provided in the following implementation can be implemented by a display device or a server, and the embodiments of the present application do not limit this.

[0285] Fig.14 A schematic diagram of an image generation method provided in an embodiment of the present application. Fig.14 As shown, the method includes steps 1410 to 1440 as shown below.

[0286] Step 1410, obtaining multimodal information input by the user.

[0287] Exemplarily, the modal information may be information of different forms (or types) input by the user, and the modal information may be audio information, text information, image information or video information. Among them, the multimodal information obtained in the embodiment of the present application may include at least two modal information of audio information, text information, image information and video information.

[0288] In some examples, the multimodal information may include any two modal information of audio information, text information, image information, and video information, or the multimodal information may include any three modal information of audio information, text information, image information, and video information, or the multimodal information may include audio information, text information, image information, and video information. The embodiments of the present application are not limited to this. For example, in the above embodiment, the multimodal information received by the display device includes image information input in the first input area and text information input in the second input area.

[0289] It should be noted that, in addition to audio information, text information, image information or video information, modal information may also include more modal information such as historical style status information, historical viewing content information, etc., which is not limited in the embodiments of the present application.

[0290] In some examples, a user may input multimodal information through a display device, wherein a display of the display device displays an application interface generated by an image, and the application interface may include multiple input areas, and a user may input different modal information through the multiple input areas.

[0291] For example, refer to Figure 7 The user can input first modal information in the first input area 710 of the application interface 700 and input second modal information in the second input area 720, wherein the first modal information can be image information (such as the first image in the above embodiment) and the second modal information can be text information (such as the first description text in the above embodiment).

[0292] In other examples, the application interface displayed by the display may include a third input area in addition to the first input area and the second input area, and the user may input third modal information through the third input area, wherein the third modal information may be audio information or video information. It should be noted that the application interface of the display may also include a greater number of input areas (such as a fourth input area), which is not limited in the embodiments of the present application. In other words, the user may input corresponding modal information in multiple input areas respectively, or may input corresponding modal information in any at least two of the multiple input areas.

[0293] In some examples, the user can input audio information to the display device through the microphone on the remote control, input text information to the display device through the remote control, and input image information or video information through the camera on the display device. Alternatively, the user can also input audio information through a sound event sensor, input modal information through an Internet of Things (IOT) device, etc. The embodiment of the present application does not specifically limit the input method of modal information.

[0294] Exemplarily, when the image generation model is deployed on a display device, the display device obtains multimodal information based on the modal information input by the user in each input area; when the image generation model is deployed on a server, the display device sends the modal information input by the user in each input area to the server, so that the server obtains multimodal information. The embodiment of the present application is exemplified by taking the deployment of the image generation model on a display device as an example.

[0295] Fig.15 A schematic diagram of an image generation model provided in an embodiment of the present application. Fig.15 As shown, the image generation model 1500 includes an encoding network 1510, a fusion network 1520, and a multi-stage network 1530. After obtaining the multimodal information input by the user, the multimodal information can be input into the encoding network 1510, and after being processed by the encoding network 1510, the fusion network 1520, and the multi-stage network 1530 in sequence, a target image that meets the requirements is generated.

[0296] The processing procedures of the encoding network 1510, the fusion network 1520 and the multi-stage network 1530 in the image generation model 1500 are described below.

[0297] Step 1420: Process each modal information separately based on the encoding network in the image generation model to obtain a feature vector corresponding to each modal information.

[0298] Exemplarily, the encoding network may include multiple encoding sub-networks, and each encoding sub-network may process its corresponding modality information respectively.

[0299] For example, Fig.15 As shown, when the modal information is audio information, text information, image information or video information, the encoding network 1510 may include an audio encoding subnetwork 1511, a text encoding subnetwork 1512, an image encoding subnetwork 1513 and a video encoding subnetwork 1514.

[0300] It should be noted that the coding network may also include a larger number of coding subnetworks, and the embodiment of the present application does not limit the number and type of coding subnetworks in the coding network. The embodiment of the present application takes the example that the coding network may include an audio coding subnetwork, a text coding subnetwork, an image coding subnetwork, and a video coding subnetwork as an example for exemplary description.

[0301] In some embodiments, step 1420 includes: inputting audio information into an audio coding subnetwork in the coding network to generate an audio feature vector corresponding to the audio information; inputting text information into a text coding subnetwork in the coding network to generate a text feature vector corresponding to the text information; inputting image information into an image coding subnetwork in the coding network to generate an image feature vector corresponding to the image information; and inputting video information into a video coding subnetwork in the coding network to generate a video feature vector corresponding to the video information.

[0302] In some examples, the audio encoding subnetwork may encode the audio information to obtain an audio feature vector corresponding to the audio information. The text encoding subnetwork may encode the text information to obtain a text feature vector corresponding to the text information. The image encoding subnetwork may encode the image information to obtain an image feature vector corresponding to the image information. The video encoding subnetwork may encode the video information to obtain a video feature vector corresponding to the video information.

[0303] For example, when the multimodal information input by the user includes image information and text information, the text information in the multimodal information can be input into the text encoding subnetwork for processing, and the image information can be input into the image encoding subnetwork for processing. In other words, the multiple encoding subnetworks in the encoding network can determine whether to perform encoding processing based on the input multimodal information, and the multiple encoding subnetworks do not necessarily need to perform encoding processing.

[0304] In some examples, the structures of the encoding sub-networks may be the same or different, wherein the encoding sub-networks may be pre-trained based on training data.

[0305] Exemplarily, the text encoding subnetwork can convert text information into a text feature vector of fixed length based on a transformer model.

[0306] In some examples, the text encoding subnetwork can convert text information into a fixed-length text feature vector through steps such as word segmentation, vocabulary mapping, word embedding, and encoding processing. For example, the text encoding subnetwork can convert the input text information into tokens of length 77.

[0307] For example, when text information is input into the text encoding subnetwork, the text encoding subnetwork first divides the text information into a series of tokens, which can be words, phrases, or other language units. Then, each token is mapped to a unique ID according to a predefined vocabulary, where the vocabulary is constructed based on a large amount of text data during the training phase, and each word or token can correspond to an index ID. Next, each token is converted into a corresponding word embedding vector, and the word embedding vector of each token is obtained by looking up the corresponding row of the embedding matrix. Since Transformer has no built-in sequential information, it is necessary to add a position encoding to each token to indicate its position in the sentence. The position encoding can be a predefined fixed vector or a trainable parameter, and these position encoding vectors will be added to the word embedding vector of each token. Finally, the text encoding subnetwork uses a model based on architectures such as Transformer to process these embedding vectors, capture the contextual relationship in the text information through mechanisms such as multi-head self-attention, and generate a series of text feature vectors.

[0308] Exemplarily, the image coding subnetwork uses a convolutional neural network (CNN) or a vision transformer to convert image information into a fixed-length image feature vector. For example, the image coding subnetwork can convert the input image information into tokens (i.e., image feature vectors) of length 196. Among them, CNN extracts image features through convolution operations and flattens the feature maps into one-dimensional vectors as tokens; VisionTransformer divides the image into small blocks and converts each small block into tokens containing features and context information through linear embedding and the Transformer coding subnetwork.

[0309] It should be noted that the video coding subnetwork and the image coding subnetwork have similar structures and similar processing processes. The difference is that the video information processing process needs to consider the temporal and spatial correlation between frames. To avoid repetition, the embodiment of the present application takes the image coding subnetwork as an example for explanation.

[0310] In some examples, the image encoding subnetwork may use CNN to extract image features from image information through processes such as convolution, feature extraction, and conversion of tokens, and convert them into a fixed-length image feature vector.

[0311] For example, when the image information is input into the image encoding subnetwork, CNN performs convolution operations on the input image based on the convolution kernel through the convolution layer to generate a feature map; different convolution kernels can extract different features, such as edges, textures, shapes, etc. As the convolution layer goes deeper, CNN can capture higher-level abstract features and build complex feature representations by stacking convolution layers, pooling layers (such as maximum pooling), and fully connected layers. Finally, the output of CNN (such as the feature map of the last convolution layer) is usually flattened into a one-dimensional vector, which can be regarded as the tokens of the image, and each token contains the feature information of a local area in the image.

[0312] In other examples, the image encoding subnetwork can also use Vision Transformer to extract image features from image information through image segmentation, linear embedding, position encoding, Vision Transformer encoding and other processes, and convert them into fixed-length image feature vectors.

[0313] For example, the Vision Transformer first divides the input image into a series of small patches, each of which is regarded as an independent token and contains pixel information of a local area in the image. Each patch is converted into a fixed-length vector (i.e., the embedding representation of the token) through a linear embedding layer. This process is usually implemented through a convolution layer, where the size of the convolution kernel is the same as the size of the patch and the number of output channels is equal to the length of the embedding vector. Secondly, a position encoding is added to each token, which can be fixed or a trainable parameter. After linear embedding and position encoding, all tokens are input into the Transformer encoding subnetwork for processing. The Transformer encoding subnetwork captures the contextual relationship between tokens through mechanisms such as Multi-Head Self-Attention. Finally, the output representation of each token contains the features and contextual information of the corresponding area in the image. The output of the Vision Transformer is a series of encoded tokens, each of which contains the feature representation of a local area in the image.

[0314] In some embodiments, the above-mentioned input of audio information into the audio coding subnetwork in the coding network to generate an audio feature vector corresponding to the audio information includes: preprocessing the audio information to obtain preprocessed audio information; performing feature extraction operations on the preprocessed modal information to obtain an initial feature vector; encoding the initial feature vector to obtain an audio feature vector.

[0315] Exemplarily, the audio coding subnetwork can use an autoregressive model (such as WaveNet or Transformer structure) to process the sound waveform. The audio coding subnetwork learns the intrinsic characteristics of the audio signal through the model and encodes it into a low-dimensional vector. For example, the audio coding subnetwork can convert the input audio information into tokens (i.e., audio feature vectors) of length 42.

[0316] In some examples, the preprocessing of the audio coding subnetwork may include sampling, quantization, noise reduction, and standardization. For example, the audio information (such as an audio signal) may be sampled first, that is, converted into a discrete time series, and then the noise and interference in the audio signal may be removed by a filter, and the audio signal may be standardized to have a uniform amplitude range, thereby obtaining the preprocessed audio information.

[0317] In some examples, the feature extraction process of the audio coding subnetwork is to extract meaningful features from the preprocessed audio information, and these features can be used in the subsequent tokenization process. For example, Fourier transform, Mel Frequency Cepstral Coefficients (MFCC), Linear Predictive Coding Coefficients (LPC), etc. can be used to extract useful features from the preprocessed audio information, and the extracted features are combined into feature vectors, which are called initial vector features. Processing vector features can reflect the temporal structure and frequency characteristics of audio information.

[0318] In some examples, after the initial feature vector is determined, the initial feature vector can be converted into tokens of fixed length. This process can also be called a token generation process, which can further convert the extracted features into a series of discrete and finite tokens.

[0319] For example, through the above step 1420, the multimodal information can output a fixed-length feature vector corresponding to each modal information through the encoding network, such as Fig.15 As shown, each fixed-length feature vector output by the encoding network 1510 can be input into the fusion network for further processing.

[0320] Step 1430: Based on the fusion network in the image generation model, the feature vectors corresponding to each modality information are fused to obtain a fusion vector.

[0321] In some examples, the fusion network can also be called a unified tokens encoding network. The fusion network can provide a common encoding space for different modal information (such as text information, image information, audio information, and video information), so that multimodal information can be processed and analyzed under a unified framework. For example, the fusion network can handle the different characteristics of various modal information, such as the sequential nature of text, the two-dimensional characteristics of images, and the time series nature of audio.

[0322] For example, the fusion network provided in the embodiment of the present application has multimodal compatibility, feature extraction capability, and scalability. First, the fusion network can process data from different modalities and map them to a common space. Secondly, the powerful feature extraction capability of the fusion network can extract meaningful feature representations from the original data. At the same time, the fusion network also has the characteristics of easy expansion to adapt to new data types and features. The fusion network provided in the embodiment of the present application can focus on operating efficiency while ensuring performance, and is suitable for scenarios of real-time processing and large-scale data analysis.

[0323] In some embodiments, the above step 1430 includes: determining a preset length; based on the preset length, fusing the feature vectors corresponding to each modal information through a fusion network to obtain a fusion vector of the preset length.

[0324] In some examples, the preset length can be set according to the requirements, and the embodiment of the present application does not limit the length of the fusion vector (i.e., the preset length). For example, the preset length can be 256, or the preset length can also be 1024. That is to say, after the feature vectors corresponding to each modal information are input into the fusion network, after being processed by the fusion network, a fusion vector with a length of 256 can be generated.

[0325] For example, when multimodal information includes image information, text information and audio information, after being processed by the encoding network, the encoding network can input a text feature vector of length 77, an image feature vector of length 196, and audio feature information of length 42 into the fusion network respectively. After the feature vectors of these three lengths are fused through the fusion network, a new vector of length 256 (i.e., a fused vector) is generated.

[0326] In some embodiments, feature selection processing is performed on the feature vectors corresponding to each modal information through a fusion network to obtain a selected feature vector; the selected feature vector is mapped to a preset feature space to obtain a mapped feature vector; based on a preset length, the mapped feature vector is fused to obtain a fused vector of a preset length.

[0327] Exemplarily, the fusion process includes at least one of weighted fusion, feature-level fusion and decision-level fusion.

[0328] In some examples, first, after obtaining the feature vectors corresponding to each modal information (hereinafter may also be referred to as modal feature vectors), feature selection processing can be performed on each modal feature vector to select the most useful features from each modal feature vector to obtain the selected feature vectors corresponding to each modality. Among them, feature selection processing can reduce the dimension of the feature space and improve the generalization ability and computational efficiency of the model. Secondly, after obtaining the selected feature vectors corresponding to each modality, feature mapping is performed on the selected feature vectors corresponding to each modality, and the selected feature vectors of different lengths are mapped to a common feature space to obtain each mapped feature vector. Finally, each mapped feature vector can be fused based on one or more combinations of feature fusion methods such as superposition, weighted average, feature-level fusion or decision-level fusion to fuse each mapped feature vector into a fusion vector.

[0329] In some examples, the fusion process of feature vectors is described by taking the specific form of feature vectors [B, N, C] as an example, where B represents Batchsize, which is the number of batches during training or inference, N is the feature dimension, and C is the number of Chanel channels.

[0330] For example, multiple feature vectors may be fused into a fusion vector in a weighted fusion manner. That is, the vector weights of the text feature vector, the audio feature vector, and the image feature vector may be calculated using the correlation coefficient method. Then, a weighted operation is performed on the text feature vector, the audio feature vector, and the image feature vector according to the vector weights to obtain a weighted text vector, a weighted audio vector, and a weighted image vector; and then the weighted text vector, the weighted audio vector, and the weighted image vector are added together to obtain a fusion vector.

[0331] For example, when the text feature vector is [B1, N1, C1], the speech feature vector is [B2, N2, C2], and the image feature vector is [B3, N3, C3], multiple feature vectors can be concatenated or weighted concatenated to form a fusion vector. That is:

[0332] [B0,N0,C0]=w1[B1,N1,C1]+w2[B2,N2,C2]+w3[B3,N3,C3];

[0333] Among them, [B0, N0, C0] is the fusion vector; w1, w2, w3 are learnable coefficients.

[0334] For another example, based on the cross-attention mechanism, multiple feature vectors can be fused to generate a fused vector, that is, the text feature vector, audio feature vector and image feature vector are respectively applied to define the query sequence, key sequence and value sequence, and then the similarity between the query sequence and the key sequence is calculated, and the similarity is normalized weighted by the Softmax function, and then weighted summation is performed according to the normalized weights to obtain a weighted value sequence; finally, the weighted value sequence is subjected to feature fusion with the query sequence and / or key sequence to obtain a fused vector.

[0335] For example, when the text feature vector is [B1, N1, C1], the speech feature vector is [B2, N2, C2], and the image feature vector is [B3, N3, C3], feature vector fusion can be performed based on the cross attention mechanism, that is:

[0336] [B0,N0,C0]=Trans_Attention([B1,N1,C1],[B2,N2,C2],[B3,N3,C3]);

[0337] Among them, [B0, N0, C0] is the fusion vector.

[0338] Step 1440, based on the multimodal information, the fusion vector is processed through a multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

[0339] Exemplarily, the target image obtained by processing the fusion vector by the multi-stage network meets the multimodal information requirements. The multi-stage network may include at least one feature sub-network. The number of the at least one feature sub-network may be one or more. When the number of the at least one feature sub-network is more than one, the multiple feature sub-networks are connected in sequence.

[0340] For example, Fig.15 As shown, the multi-stage network 1530 may include a first feature sub-network, a second feature sub-network, ..., and an Nth feature sub-network (i.e., the last feature sub-network), etc., wherein N is any integer greater than or equal to 3, and the output of the previous feature sub-network serves as the input of the next feature sub-network connected thereto.

[0341] It should be noted that the embodiment of the present application does not limit the number of characteristic sub-networks of the multi-stage. The embodiment of the present application takes the case where the number of characteristic sub-networks is greater than or equal to 3 as an example for illustrative description.

[0342] Fig.16 A schematic diagram of another image generation method provided in an embodiment of the present application is shown as follows: Fig.16 As shown, the above step 1440 includes steps 1441 to 1443 as shown below.

[0343] Step 1441, determining at least one auxiliary feature information corresponding to the multimodal information.

[0344] Exemplarily, the auxiliary feature information is information that assists the multi-stage network in extracting more accurate features from the fusion vector. The multi-stage network can extract the target vector from the fusion vector based on at least one auxiliary feature information to facilitate the subsequent generation of the target image. Among them, the auxiliary feature information can also be called conditional guidance information. The auxiliary feature information is related to the multimodal information. When the multimodal information input into the image generation model is different, the corresponding auxiliary feature information may also be different. For example, the auxiliary feature information includes feature information corresponding to the guidance auxiliary map, and the auxiliary guidance map may include images of style, content, structure and other types. For example, the features of the image can be extracted from the guidance auxiliary map through image encoding to obtain the auxiliary feature information.

[0345] In some examples, the guidance auxiliary map may include a human skeleton map, a depth map, and an edge structure map. Among them, the human skeleton map provides the basic skeleton structure of the character, which is used to guide the character posture and movement in the generated image, ensuring that the generated human posture is natural and reasonable. The depth map is used to express the depth information of the image, and can be used to create a realistic three-dimensional effect to make the image look more layered. Especially when generating complex scenes such as landscapes and urban landscapes, the depth map can help the model better understand the depth relationship of the scene, thereby generating a more realistic image. The edge map is used to emphasize the contours and boundaries in the image, which helps to maintain the clarity and details of the generated image. When generating fine patterns, text, or objects that need to maintain a specific shape, the edge map can ensure that the generated image has clear contours and accurate boundaries.

[0346] In other examples, the guidance auxiliary map may also include a color map, a texture map, a normal map, a height map, a reflection map, a lighting map, and a skeleton map of other animals (such as birds), etc. The embodiments of the present application do not limit the guidance auxiliary map.

[0347] Exemplarily, the auxiliary feature information can be determined based on the guidance auxiliary map. For example, feature extraction can be performed on the guidance auxiliary map to obtain the auxiliary feature information corresponding to each guidance auxiliary map. After determining the auxiliary feature information, multiple auxiliary feature information are pre-stored in the guidance feature table. Among them, the guidance feature table can be pre-stored in the display device or server, and the embodiment of the present application is not limited to this. Alternatively, the image generation model may also include a generation subnetwork to generate auxiliary feature information corresponding to each feature subnetwork, and input it into each corresponding feature subnetwork. The embodiment of the present application does not limit the generation and determination of auxiliary feature information.

[0348] In some examples, at least one auxiliary feature information corresponding to the multimodal information may be determined based on the multimodal information. For example, when the multimodal information involves a person, the corresponding auxiliary feature information includes feature information of a human skeleton map; when the multimodal information involves a landscape, the corresponding auxiliary feature information includes feature information corresponding to an edge map and a depth map.

[0349] For example, Fig. 9 As shown in (a), when the multimodal information includes image information displayed in the first input area (i.e., the image of a bird drawn by the user) and text information displayed in the second input area (such as the first descriptive text: bird, blue feathers on the back, white feathers on the belly), the corresponding auxiliary feature information can be determined based on the image information and text information, including at least feature information corresponding to the bird's skeleton structure diagram, feature information corresponding to the edge diagram, feature information corresponding to the depth of field diagram, and feature information corresponding to the color diagram.

[0350] In some examples, the number of at least one auxiliary feature information of the multimodal information may be 1 or more, for example, the number of auxiliary feature information corresponding to the multimodal information is M, where M is an integer greater than or equal to 1. The number M of auxiliary feature information corresponding to the multimodal information may be less than the number N of feature subnetworks in the multi-stage network (i.e., M>N), or may be greater than or equal to the number N of feature subnetworks.

[0351] For example, among multiple feature subnetworks, each feature subnetwork may correspond to auxiliary feature information, or, among multiple feature subnetworks, some feature subnetworks may correspond to auxiliary feature information, while another part of feature subnetworks may not have corresponding auxiliary feature information, wherein each feature subnetwork corresponds to different auxiliary feature information.

[0352] Exemplarily, when a feature sub-network corresponds to a plurality of auxiliary feature information, the plurality of auxiliary feature information may be weighted and then input into the feature sub-network for processing.

[0353] Step 1442: Process the fusion vector and at least one auxiliary feature information based on at least one feature sub-network in the multi-stage network to obtain a target vector.

[0354] Exemplarily, after determining at least one auxiliary feature information, the at least one auxiliary feature information and the fusion vector determined in step 1443 may be input into at least one feature sub-network in the multi-stage network to generate a target vector, wherein the target vector may reflect the characteristics of the multimodal information and the characteristics of the guided auxiliary graph.

[0355] In some examples, after determining the M auxiliary feature information corresponding to the multimodal information, the corresponding auxiliary feature information can be respectively allocated to the multi-stage network including the N feature network sub-networks.

[0356] In some examples, each feature subnetwork includes a processing process of multiple steps, and the multiple processing steps form a chain structure, thereby realizing the execution of processing tasks of different complexity corresponding to the feature subnetwork. Among them, the execution steps corresponding to each feature subnetwork can be the same or different. For example, when the complexity of the processing task corresponding to the feature subnetwork is higher, the corresponding steps will also be more.

[0357] like Fig.17 As shown, the above step 1442 includes the following steps 14421 to 14422. It should be noted that, Fig.17 An exemplary description is given by taking an example where at least one auxiliary feature information includes first auxiliary feature information, second auxiliary feature information and third auxiliary feature information.

[0358] Step 14421: Process the fusion vector and the first auxiliary feature information based on the first feature sub-network in at least one feature sub-network to determine a first output vector.

[0359] In some examples, the first feature subnetwork is the first feature subnetwork in the multi-stage network, i.e. Fig.15 The first feature sub-network in the multi-stage network 1530. After determining that the auxiliary feature information corresponding to the first feature sub-network is the first auxiliary feature information, the fusion vector output by the fusion network and the first auxiliary feature information can be input into the first feature sub-network.

[0360] For example, the fusion vector (i.e., the tokens vector) may be aligned and random noise may be added to obtain an input vector, and the input vector and the first auxiliary feature information may be input into the first feature subnetwork to obtain a first input vector. The random noise added to the aligned tokens vector may be Gaussian noise or other types of noise.

[0361] Step 14422: Process the first output vector and the second auxiliary feature information based on the intermediate feature sub-network in at least one feature sub-network to determine a second output vector.

[0362] In some examples, the number of intermediate feature subnetworks may be one or more. For example, the intermediate subnetwork may include the second feature subnetwork among multiple feature subnetworks, or may include the second feature subnetwork and the third feature subnetwork among multiple feature subnetworks, or may include more feature subnetworks. In other words, the intermediate feature subnetwork includes at least one feature subnetwork between the first feature subnetwork and the last feature subnetwork in the multi-stage network. This embodiment of the present application is not limited to this.

[0363] Exemplarily, after obtaining the first output vector output by the first feature sub-network, the first output vector and the auxiliary feature information corresponding to the intermediate feature sub-network can be input into the intermediate feature sub-network, and after being processed by the intermediate sub-network, the second output vector is output, wherein the second output vector is the feature vector output by the last intermediate feature sub-network in the intermediate feature sub-network.

[0364] For example, when the intermediate sub-network may include the second feature sub-network to the N-1th feature sub-network, the second output vector is a vector output by the N-1th feature sub-network.

[0365] In some examples, when the intermediate feature subnetwork includes the second feature subnetwork to the N-1th feature subnetwork among multiple feature subnetworks, each intermediate feature subnetwork may correspond to one auxiliary feature information (i.e., the second auxiliary feature information); or, some intermediate feature subnetworks correspond to auxiliary feature information, and another part of the intermediate feature vectors have no corresponding auxiliary feature information.

[0366] For example, when N is 5, the intermediate subnetwork includes the second feature subnetwork, the third feature subnetwork, and the fourth feature subnetwork. Among them, the second feature subnetwork corresponds to an auxiliary feature information (such as the second auxiliary feature information), and the third feature subnetwork and the fourth feature subnetwork have no corresponding auxiliary feature information. Therefore, the second feature subnetwork can process the first output vector and the second auxiliary feature information to output the first intermediate vector; the first intermediate vector is input to the third feature subnetwork, and after being processed by the third feature subnetwork, the second intermediate vector is generated; the second intermediate vector is input to the fourth feature subnetwork, and after being processed by the fourth feature subnetwork, the second output vector is generated.

[0367] Step 14423: Process the second output vector and the third auxiliary feature information based on the second feature subnetwork in at least one feature subnetwork to determine a target vector.

[0368] In some examples, the second feature subnetwork is the last feature subnetwork in the multi-stage network, that is, the Nth feature subnetwork. After determining the second output vector output by the intermediate feature subnetwork (that is, the N-1th feature subnetwork) and the third auxiliary information corresponding to the Nth feature subnetwork, the second output vector and the third auxiliary feature information are input into the Nth feature subnetwork to obtain the target vector.

[0369] Step 1443, based on the image decoding sub-network in the multi-stage network, the target vector is decoded to obtain the target image.

[0370] In some examples, the multi-stage network also includes an image decoding subnetwork (also referred to as an image decoder), which is used to convert the target vector into a target image, and the target image conforms to the multimodal information input by the user. For example, the image decoding process may include processing processes such as feature vector decoding, inverse quantization, inverse transformation, and pixel reconstruction. The image decoding subnetwork can recover pixel data close to the original image from the target vector.

[0371] Fig.18 A schematic diagram of a multi-stage network structure provided in an embodiment of the present application.

[0372] like Fig.18 As shown, the multi-stage network 1530 can obtain an input vector Z0 based on the fusion vector and random noise, and then input the input vector Z0 and the first auxiliary feature information into the first feature sub-network. After being processed by the first feature sub-network, a first output vector Z1 is obtained; and then the first output vector Z1 and the second auxiliary feature information are input into the second feature sub-network to obtain a second output vector Z2.

[0373] Similarly, the processing of other feature subnetworks can be continued until the Nth feature subnetwork, and the output vector and the third auxiliary feature information output by the previous feature subnetwork can be input into the Nth feature subnetwork, and after being processed by the Nth feature subnetwork, the target vector Zn is output. Finally, the target vector Zn is input into the image decoding subnetwork for decoding processing to obtain the target image.

[0374] It should be noted that each feature subnetwork in the multi-stage network is used to complete the execution of tasks in different aspects, such as style, content, structure, etc. The multi-stage network adopts an Adapter-based model, and realizes the individual and combined application of multiple tasks by adding elements such as edge contours and colors (i.e., auxiliary feature information). At the same time, a clear hierarchical relationship is defined in the multi-stage network, and each layer (such as each feature subnetwork) receives guidance information from the upper layer to achieve fine-grained control. In each feature subnetwork, through the design of a chain structure, this stage can be generated for tasks of different complexity. The chain structure is mainly for interactive conditional input and supports multi-stage controllable generation. Among them, the control condition output of the intermediate state is the implicit feature representation of the previous stage, and the dimension of the feature output of each stage remains consistent.

[0375] In some embodiments, the above-mentioned image generation method also includes: obtaining multimodal training information; processing the multimodal training information based on the image generation model to be trained to generate a predicted image; obtaining a sample image; using the predicted image as the initial training output information of the image generation model to be trained and the sample image as the supervision information, iterating the image generation model to be trained to obtain an image generation model.

[0376] Exemplarily, the image generation model to be trained includes a coding network to be trained, a fusion network to be trained, and a multi-stage network to be trained. After completing the iterative training, the image generation model obtained includes the coding network, the fusion network, and the multi-stage network. That is to say, during the image generation model training process, the entire model can be trained, or each network in the image generation model to be trained can be trained separately, which is not limited in the embodiments of the present application.

[0377] In some examples, when the image generation model to be trained is trained using multimodal training information, the loss value of the image generation model to be trained can be determined based on the sample image and the predicted image, and the model parameters of the image generation model to be trained can be adjusted according to the loss value until a preset number of iterations is reached or the image generation model converges, thereby obtaining a trained image generation model (i.e., the image generation model in the above embodiment). The trained image generation model can be used to convert the multimodal information into a corresponding target image.

[0378] The image generation method provided in the embodiment of the present application, after obtaining the multimodal information input by the user, inputs the multimodal information into the image generation model, and sequentially processes the encoding network, fusion network and multi-stage network in the image generation model to obtain a target image that meets the multimodal information. The image generation method provided in the embodiment of the present application can process multimodal information to generate an image that meets the requirements, meet the personalized needs of users, and improve the user experience.

[0379] Fig.19A schematic diagram of a flow chart of an image display method provided in an embodiment of the present application. Fig.19 , the image display method provided in the embodiment of the present application is described.

[0380] like Fig.19 As shown, the image display method provided in the embodiment of the present application can be applied to a display device, and the display device may include an application control module, a signal receiving module and an algorithm module. It should be noted that: Fig.19 The following is an illustrative explanation using the example of deploying an image generation model on a display device.

[0381] In some examples, the signal receiving module includes a voice receiving module, a text input module and a picture receiving module. When the display device supports a touch function, the signal receiving module may also include a touch module. Fig.19 An exemplary description is given by taking the example that the signal receiving module includes a touch control module.

[0382] In some examples, the algorithm module includes a speech recognition model, an image preprocessing model and an image generation model (such as the image generation model 1500 in the above embodiment); wherein the image generation model includes a text feature extractor (such as the text encoding subnetwork 1512 in the encoding network 1510 in the above embodiment), an image feature extractor (such as the image encoding subnetwork 1513 in the encoding network 1510 in the above embodiment), a feature fusion layer (such as the fusion network 1520 in the above embodiment), and a generation layer (such as the multi-stage network 1530 in the above embodiment).

[0383] like Fig.19 As shown, when a user opens an application (such as an image generation application) through a display device, the application control module can call each module in the signal receiving module to perform monitoring. For example, the application control module can call the voice receiving module for voice monitoring, call the touch module for touch monitoring, and call the text entry module for typing monitoring.

[0384] For example, Figure 7 As shown, when the user triggers the shooting interaction control 740 or the code scanning interaction control 750 in the image generation application, the image can be shot through the image collector or uploaded through the mobile terminal. The image receiving module can monitor the image and send the image to the algorithm module; the image preprocessing model in the algorithm module performs edge or skeleton extraction processing (such as key point feature extraction processing) on ​​the image and sends the processed image to the application control module. The application control module is in the drawing area of ​​the application interface (such as Figure 7 The first input area 710 in the drawing area displays the image, or the application control module updates the image currently displayed in the drawing area based on the processed image.

[0385] For example, Fig. 9 As shown in (a) of FIG. 1 , taking the drawing area as the first input area 710 as an example, when the user draws an image in the first input area 710 of the application interface, the touch receiving module can monitor the user's drawing operation and send the drawing information to the application control module, and the application control module displays the drawn image in the first input area 710, or based on the drawing information, updates the image currently displayed in the first input area 710. The application control module sends the image signal corresponding to the image displayed in the first input area 710 to the algorithm module, and the image feature extractor in the algorithm module extracts features from the image signal to obtain image features, and inputs the image features into the feature fusion layer.

[0386] like Fig.19 As shown, when the user inputs voice through far field or near field, the voice receiving module can monitor the voice signal and transmit the voice signal to the algorithm module. The audio feature extractor in the algorithm module extracts the features of the voice signal to obtain voice features, and inputs the voice features into the feature fusion layer.

[0387] For example, the voice receiving module can also send the monitored voice signal directly to the voice recognition model in the algorithm module, and the voice recognition model converts the voice signal into a text signal and sends the text signal to the application control module; or, when the user types text on the display device, the text entry module can monitor the typed text information and send the monitored text information to the application control module. After the application control module receives the text information, it displays the text information in the text input area of ​​the application interface (such as the second input area in the above embodiment), or the application control module updates the text currently displayed in the text input area based on the text information. The application control module sends the text signal corresponding to the text displayed in the text input area to the algorithm module, and the text feature extractor in the algorithm module extracts features from the text signal to obtain text features. And the text features are input into the feature fusion layer.

[0388] like Fig.19 As shown, after receiving the image features, voice features and text features, the feature fusion layer in the algorithm module can perform feature fusion processing on the image features, voice features and text features to obtain fusion features, and input the fusion features into the generation layer in the algorithm module, and after being processed by the generation layer, a generated image (such as the third image in the above embodiment) that conforms to each modal information (voice signal, image and text signal) is obtained. The generation layer feeds back the generated image to the application control module, and the application control module controls the generation area of ​​the application interface (such as the image display area in the above embodiment) to display the generated image.

[0389] like Fig.19As shown, when the user closes the application, the application control module can disconnect the monitoring actions of each module in the signal receiving module. For example, the application control module can disconnect the voice monitoring of the voice receiving module, disconnect the touch monitoring of the touch module, and disconnect the typing monitoring of the text entry module.

[0390] Fig. 20 A schematic diagram of another image display method provided in an embodiment of the present application. Fig. 20 As shown, the display device also includes an application module, which includes a saving module, a sharing module, a background beautification module, a picture book generation module, and a picture quality enhancement module.

[0391] like Fig. 20 As shown, after the generated image is displayed in the generation area of ​​the application interface, the generated image can be processed accordingly through the saving module, sharing module, background beautification module, picture book generation module and image quality enhancement module respectively.

[0392] For example, combined with Figure 4 As shown, in response to the user triggering the image quality enhancement control in the image generation application, the application control module calls the image quality enhancement interface to perform image quality enhancement processing on the generated image through the image quality enhancement module in the application module; the image quality enhancement module can feed back the processed high-definition image to the application control module, and the application control module controls the generation area to display the high-definition image. Figure 4 As shown, in response to the user triggering the picture book generation control in the image generation application, the application control module calls the picture book generation interface to process the generated image through the picture book generation module in the application module to generate corresponding picture book information. Figure 4 As shown, in response to the user triggering the background beautification control in the image generation application, the application control module calls the background beautification interface to beautify the generated image through the background beautification module in the application module; the background beautification module can feed back the processed beautified image to the application control module, and the application control module controls the generation area to display the beautified image. Figure 4 As shown, in response to the user triggering the sharing control in the image generation application, the application control module calls the sharing interface to share the generated image through the sharing module in the application module. Figure 4 As shown, in response to the user triggering the save control in the image generation application, the application control module calls the save interface to save the generated image through the save module in the application module.

[0393] like Fig. 20As shown, when the user closes the application, the sharing module in the application module sends a window switching instruction to the application control module, and the application control module switches the current shared application window to the background based on the window switching instruction. The picture book generation module sends a close application instruction to the application control module, and the application control module closes the current application.

[0394] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0395] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0396] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0397] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. An image generation method, characterized in that: The method comprises: Get multimodal information input by the user; Based on the encoding network in the image generation model, each modal information in the multimodal information is processed respectively to obtain a feature vector corresponding to each modal information; Based on the fusion network in the image generation model, the feature vectors corresponding to each modality information are fused to obtain a fusion vector; Based on the multimodal information, the fusion vector is processed through a multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

2. The method according to claim 1, characterized in that: The step of processing the fusion vector based on the multimodal information through a multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information includes: Determining at least one auxiliary feature information corresponding to the multimodal information; Processing the fusion vector and at least one auxiliary feature information based on at least one feature sub-network in the multi-stage network to obtain a target vector; The target vector is decoded based on the image decoding subnetwork in the multi-stage network to obtain the target image.

3. The method according to claim 2, characterized in that The step of processing the fused vector and at least one auxiliary feature information based on at least one feature sub-network in the multi-stage network to obtain a target vector includes: Processing the fusion vector and the first auxiliary feature information based on a first feature subnetwork in the at least one feature subnetwork to determine a first output vector; wherein the at least one auxiliary feature information includes the first auxiliary feature information, the second auxiliary feature information, and the third auxiliary feature information; Processing the first output vector and the second auxiliary feature information based on an intermediate feature subnetwork in the at least one feature subnetwork to determine a second output vector; The second output vector and the third auxiliary feature information are processed based on a second feature subnetwork in the at least one feature subnetwork to determine the target vector.

4. The method according to claim 1, characterized in that: The fusion network in the image generation model is used to fuse the feature vectors corresponding to each modality information to obtain a fusion vector, including: Determine the preset length; Based on the preset length, the feature vectors corresponding to the various modal information are fused through the fusion network to obtain the fused vector of the preset length.

5. The method according to claim 4, characterized in that The step of fusing the feature vectors corresponding to the respective modal information based on the preset length through the fusion network to obtain the fusion vector of the preset length includes: Performing feature selection processing on the feature vectors corresponding to the respective modal information through the fusion network to obtain a selected feature vector; Mapping the selected feature vector to a preset feature space to obtain a mapped feature vector; Based on the preset length, the mapping feature vector is fused to obtain the fused vector of the preset length; wherein the fusion process includes at least one fusion method of weighted fusion, feature-level fusion and decision-level fusion.

6. The method according to claim 1, characterized in that The multimodal information includes at least two items of audio information, text information, image information and video information; the encoding network in the image generation model processes each modal information respectively to obtain a feature vector corresponding to each modal information, including at least two items of the following: Inputting the audio information into the audio coding subnetwork in the coding network to generate an audio feature vector corresponding to the audio information; Inputting the text information into the text encoding subnetwork in the encoding network to generate a text feature vector corresponding to the text information; Inputting the image information into the image encoding subnetwork in the encoding network to generate an image feature vector corresponding to the image information; The video information is input into a video encoding subnetwork in the encoding network to generate a video feature vector corresponding to the video information.

7. The method according to claim 6, characterized in that The step of inputting the audio information into the audio coding sub-network in the coding network to generate an audio feature vector corresponding to the audio information includes: Preprocessing the audio information to obtain preprocessed audio information; Performing a feature extraction operation on the preprocessed modal information to obtain an initial feature vector; The initial feature vector is encoded to obtain the audio feature vector.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Obtain multimodal training information; Processing the multimodal training information based on a to-be-trained image generation model to generate a predicted image; wherein the to-be-trained image generation model includes a to-be-trained encoding network, a to-be-trained fusion network, and a to-be-trained multi-stage network; Get a sample image; The predicted image is used as the initial training output information of the image generation model to be trained, the sample image is used as the supervision information, and the image generation model to be trained is iterated to obtain the image generation model.

9. An image generation device based on multimodal information, characterized in that: include: The acquisition module is configured to: acquire multimodal information input by a user; Determine the module, which is configured as: Based on the encoding network in the image generation model, each modal information is processed respectively to obtain a feature vector corresponding to each modal information; Based on the fusion network in the image generation model, the feature vectors corresponding to the modal information are fused to obtain a fusion vector; Based on the multimodal information, the fusion vector is processed through a multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

10. A display device, characterized in that: include: A display configured to display an application interface of an image generation application; wherein the application interface includes a first input area, a second input area, and an image display area; A controller is coupled to the display and is configured to: In response to receiving image information input by a user in the first input area and text information input in the second input area; Based on the encoding network in the image generation model, each modal information in the multimodal information is processed respectively to obtain a feature vector corresponding to each modal information; wherein the multimodal information includes the image information and the text information; Based on the fusion network in the image generation model, the feature vectors corresponding to the modal information are fused to obtain a fusion vector; Based on the multimodal information, the fusion vector is processed through a multi-stage network in the image generation model to obtain a target image corresponding to the multimodal information.

Images