Display apparatus and control method of display apparatus

Abstract

The invention relates to a display device and a control method of the display device, and relates to the technical field of display devices. And at least one processor configured to execute instructions to cause the display device to: display a drawing interface; receiving a wireless signal of a remote controller, determining a moving track of the remote controller based on the wireless signal, and displaying the moving track on the drawing interface; an image generation instruction is received and responded to, an image generation interface is generated and displayed, the image generation interface comprises a second image generated based on the picture of the drawing interface and a first image thumbnail, the first image thumbnail comprises a thumbnail of the picture of the drawing interface, and when the second image is displayed on the image generation interface, the first image thumbnail of the picture of the drawing interface is displayed. The first image thumbnail does not completely shield the second image. According to the embodiment, the interaction performance of the display equipment can be further improved, and the rich drawing interaction requirements of the display equipment can be met.

Description

Technical Field

[0001] This application relates to the technical field of display devices, and in particular, to a display device and a control method for a display device. Background Art

[0002] With the development of display technology, more and more interactive functions can be achieved through display devices. For example, for the function items provided by the display device, relevant functions can be selected and applied through a remote control, and interaction can be achieved through voice interaction, etc. However, there is a further need to improve the interaction performance of display devices in the related art. Summary of the Invention

[0003] This application provides a display device and a control method for a display device to improve the interaction performance of the display device.

[0004] In a first aspect, some embodiments provide a display device, which includes:

[0005] A display;

[0006] And at least one processor configured to execute instructions to cause the display device to:

[0007] Display a painting interface;

[0008] Receive a wireless signal from a remote control, determine the movement trajectory of the remote control based on the wireless signal, and display the movement trajectory on the painting interface;

[0009] Receive and respond to an image generation instruction, and switch to display an image generation interface, where the image generation interface includes a second image generated based on the screen of the painting interface, and the screen of the painting interface includes the movement trajectory.

[0010] In a second aspect, some embodiments provide a control method for a display device, and the method includes:

[0011] Display a painting interface;

[0012] Receive a wireless signal from a remote control, determine the movement trajectory of the remote control based on the wireless signal, and display the movement trajectory on the painting interface;

[0013] Receive and respond to an image generation instruction, generate and display an image generation interface, where the image generation interface includes a second image generated based on the screen of the painting interface and a first image thumbnail, the first image thumbnail includes a thumbnail of the screen of the painting interface, and the first image thumbnail is displayed above the second image.

[0014] A display device and a control method thereof according to an embodiment of the present application. After displaying a painting interface, it receives a wireless signal of a remote controller, determines the movement trajectory of the remote controller through the wireless signal of the remote controller, and displays the movement trajectory on the painting interface, thereby realizing wireless painting on the painting interface of the display device by moving the remote controller, without being limited to the touch of a stylus and devices such as a wired-connected mouse. On the basis of obtaining the painting content determined by the movement trajectory of the remote controller, an image can be further generated on the basis of the picture on the painting interface according to an image generation instruction, and the generated second image is obtained and displayed on the image generation interface. Thus, on the basis of the movement trajectory drawn by the user through the remote controller, a richer second image can be further generated. At the same time, a thumbnail of the picture on the painting interface is also displayed on the image generation interface, so that the thumbnail of the picture on the painting interface before image generation and the generated second image can be viewed and compared, which can further improve the interaction performance of the display device and meet the needs of rich painting interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided by some embodiments of the present application;

[0017] Figure 2 A schematic diagram of the hardware configuration of a display device provided by some embodiments of the present application;

[0018] Figure 3 A schematic diagram of the hardware configuration of a control device provided by some embodiments of the present application;

[0019] Figure 4 A schematic diagram of the software configuration of a display device provided by some embodiments of the present application;

[0020] Figure 5 A schematic diagram of an application scenario provided by some embodiments of the present application;

[0021] Figure 6 、 Figure 7 A schematic diagram of the interface of a display device in some embodiments of the present application;

[0022] Figure 8 A schematic diagram of the principle of generating a second image based on a first image in some embodiments of the present application;

[0023] Figure 9 Schematic diagram of the principle for generating a second image based on control features in some embodiments of the present application;

[0024] Figure 10 Schematic diagram of the interaction process when a second image is generated by a display device in some embodiments of the present application;

[0025] Figure 11 Schematic diagram of the interaction process when a second image is generated by a server in some embodiments of the present application;

[0026] Figure 12 Schematic diagram of the process of the control method of a display device in some embodiments of the present application. Detailed implementation manners

[0027] Embodiments will be described in detail below, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application detailed in the claims.

[0028] It should be noted that the brief description of terms in the present application is only for facilitating the understanding of the following described embodiments, rather than intending to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood in their ordinary and common meanings.

[0029] The terms "first", "second", "third", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar or same-kind objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0030] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclusively include. For example, a product or device comprising a series of components does not necessarily have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0031] The term "module" refers to any known or later-developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code that can perform functions related to that element.

[0032] In the embodiments of the present application, the display device 200 generally refers to a device with the capabilities of displaying images and processing data. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc. In the following related embodiments of the present application, the display device 200 is taken as an example of a smart TV for illustration.

[0033] Figure 1 It is a schematic diagram of the operation scenario between the display device and the control device provided by some embodiments of the present application. As Figure 1 shown, the user can operate the display device 200 through touch operations, the mobile terminal 300, and the control device 100. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0034] The mobile terminal 300 can be used as a control device to perform human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device to establish a communication connection with the display device 200 for data interaction. In some embodiments, software applications can be installed on the mobile terminal 300 and the display device 200, and they can be connected and communicate through network communication protocols to achieve one-to-one control operations and data communication. It is also possible to transmit the audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve the synchronous display function.

[0035] As Figure 1 also shown, the display device 200 also communicates with the server 400 through various communication methods. The display device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0036] The display device 200 can provide a broadcast receiving TV function, and can also additionally provide a smart network TV function with computer support functions, including but not limited to, network TV, smart TV, Internet Protocol TV (IPTV), etc.

[0037] Figure 2 For some embodiments of the present application Figure 1 is the hardware configuration block diagram of the display device 200 in

[0038] In some embodiments, the display device 200 may include at least one of a tuner demodulator 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0039] In some embodiments, the detector 230 is used to collect signals from the external environment or for external interactions. For example, the detector 230 includes a light receiver ( Figure 2A sensor (not shown in the figure) for collecting the ambient light intensity; alternatively, the detector 230 includes an image collector, such as a camera, which can be used to collect the external environmental scene, the user's attributes or the user interaction gestures. Or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0040] In some embodiments, the display 260 includes a display function component for presenting a picture and a driving component for driving the image display. The display 260 is used to receive the image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, components of the menu control interface, and the user control UI interface, etc.

[0041] In some embodiments, the communication device 220 is a component for communicating with an external device or the server 400 according to various communication protocol types. The display device 200 can be provided with a plurality of communication devices 220 according to different supported communication methods. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0042] The communication device 220 can enable the display device 200 to communicate with the external device or the server 400 in a wireless or wired connection manner. Among them, the wired connection can connect the display device 200 with the external device through components such as data lines and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can directly establish a connection relationship with the external device, or can indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0043] In some embodiments, the controller 250 can include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and the first interface to the nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0044] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, that is, the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0045] In some embodiments, the user may input a user command through a Graphical User Interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the GUI.

[0046] In some embodiments, the audio output device 270 may be the built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0047] In some embodiments, the user input interface 280 can be used to receive instructions from the user input.

[0048] Figure 3 For some embodiments provided in this application Figure 1 is the hardware configuration block diagram of the control device. As Figure 3 shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0049] The control device 100 is configured to control the display device 200, receive input operation instructions from the user, and convert the operation instructions into instructions recognizable and responsive by the display device 200, serving as an interaction intermediary between the user and the display device 200.

[0050] In some embodiments, the control device 100 can be an intelligent device. For example, the control device 100 can install various applications for controlling the display device 200 according to user needs.

[0051] In some embodiments, as Figure 1 shown, after installing the application for controlling the display device 200, the mobile terminal 300 or other intelligent electronic devices can perform functions similar to those of the control device 100.

[0052] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and the data processing functions between the external and internal.

[0053] Under the control of the controller 110, the communication interface 130 realizes the communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other short-range communication modules.

[0054] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, the Bluetooth protocol, or the NFC protocol and send it to the display device 200. In the user input / output interface 140, the input interface includes at least one of other input interfaces such as a microphone 141, a touch panel 142, a sensor 143, and a button 144.

[0055] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0056] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0057] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0058] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0059] The operating system can be divided into different modules or layers according to the functions implemented, for example Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0060] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0061] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions of the applications in the application layer. Through the API interface, an application can access the resources in the system and obtain the services of the system during execution.

[0062] As Figure 4 shown, in the embodiments of the present application, the application framework layer includes a view system, managers, content providers, etc. Among them, the view system can design and implement the interfaces and interactions of applications. The view system includes lists, grids, text boxes, buttons, etc. The managers include at least one of the following modules: The activity manager is used to interact with all the activities running in the system; the location manager is used to provide access to the system location service for system services or applications; the package manager is used to retrieve various information related to the application packages currently installed on the device; the notification manager is used to control the display and clearing of notification messages; the window manager is used to manage the icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0063] In some embodiments, the activity manager is used to manage the life cycles of various applications and the general navigation back function, such as controlling the exit, opening, and backward movement of applications. The window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, taking screenshots, and controlling the changes of the display window. For example, shrinking the display window, jittering the display, distorting the display, etc.

[0064] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction libraries included in the system runtime layer, such as C / C++ instruction libraries, to implement the functions that the framework layer is to achieve.

[0065] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, as Figure 4As shown in the figure, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0066] It should be noted that the above examples are only simple divisions of the functions of the operating system, and do not limit the specific form of the operating system of the display device 200 in the embodiments of the present application. According to factors such as the functions of the display device and the type of the operating system, the number of levels and the specific level types included in the operating system may be in other forms.

[0067] Regarding the interactive performance of the display device in the related art, if you want to implement the painting function, usually an input device connected to the display device needs to be equipped to achieve painting, such as a mouse, a stylus, etc. connected to the display device, which limits that the user needs to be in a space very close to the display device or needs to be in direct contact with the display device at a short distance when performing painting interaction, seriously affecting the user experience. Also taking the painting function as an example, if the user does not have good painting skills, it will be difficult to meet the painting interaction requirements when drawing on the display device through interaction.

[0068] After research, it is found that a display device usually can be equipped with a remote controller that can communicate wirelessly with the display device. By interacting with the display device through the remote controller, relevant options of the display device can be selected to send control instructions, so that the display device can execute corresponding control functions according to the control signals of the remote controller. Therefore, the remote controller can be located by the wireless signal of the remote controller to obtain the position information of the remote controller, and then the movement trajectory of the remote controller can be obtained by combining the position information of the remote controller, so as to realize remote wireless painting for the display device. On the basis of obtaining the movement trajectory of the display device, a graph with rich colors can be further generated for the graph obtained for the movement trajectory to meet the requirements of painting interaction and further improve the interactive performance of the display device. The specific application scenario can be as Figure 5 As shown in the figure, the user moves the remote controller 501, and the display device 500 obtains the movement trajectory of the remote controller by receiving the wireless signal of the remote controller and displays the movement trajectory. The schematic diagram of the displayed movement trajectory can be displayed on the drawing interface 502. On the basis of the movement trajectory, for the graph including the movement trajectory, a graph with rich colors can be further generated. The schematic diagram when the graph is displayed full screen is shown in interface 503.

[0069] Accordingly, some embodiments of the present application provide a display device, and the display device includes:

[0070] A display;

[0071] and at least one processor configured to execute instructions to cause a display device to:

[0072] display a painting interface;

[0073] receive a wireless signal of a remote controller, determine a movement trajectory of the remote controller based on the wireless signal, and display the movement trajectory on the painting interface;

[0074] receive and respond to an image generation instruction, generate and display an image generation interface, the image generation interface including a second image generated based on a screen of the painting interface and a first image thumbnail, the first image thumbnail including a thumbnail of the screen of the painting interface, and while the second image is being displayed on the image generation interface, the first image thumbnail does not completely obscure the second image.

[0075] Wherein, the first image thumbnail not completely obscuring the second image may be that the first image thumbnail does not obscure the second image at all, that is, there is no overlapping part between the display areas of the first image thumbnail and the second image, or the first image thumbnail partially obscures the second image, but does not completely obscure the second image, that is, there is an overlapping part between the display areas of the first image thumbnail and the second image, but some content of the second image is still visible.

[0076] There is an overlapping part between the display areas of the first image thumbnail and the second image, which may be that the display area of the first image thumbnail is completely included in the display area of the second image, or only a part of the display area of the first image thumbnail overlaps with the display area of the second image. Among them, the proportion of the overlapping part between the display areas of the first image thumbnail and the second image in the display area of the second image is not limited. Usually, it is sufficient that the display of the main part of the image of the second image is not affected.

[0077] The painting interface is an interface where handwriting can be written and erased using an input device within a specific area. The specific area for writing and erasing handwriting on the painting interface may be referred to as a painting area or a drawing board area, etc. in the embodiments of the present application. In the following embodiments, the painting area is taken as an example for description.

[0078] The manner of entering the painting interface is not limited. After entering the application interface of the target application, the painting interface can be entered by receiving and responding to a painting instruction. The target application refers to an application that can receive the position signal of the remote controller, determine the movement trajectory of the remote controller, and further generate a second image with rich image content based on the graph including the movement trajectory. The name of the target application is not limited, such as Figure 6 "AI Magic Doodle" shown in (1) and 6(2), but not limited thereto.

[0079] In some embodiments, application prompt information of the target application may also be displayed on the application interface of the target application. For example, Figure 6 "Turn on the remote control pointing mode, wave the pen to sketch freely, and start an unprecedented creative graffiti experience" shown in (1). It can be understood that in other embodiments, the application prompt information of the target application may also be set otherwise, and the embodiments of the present application do not make specific limitations thereto.

[0080] In some embodiments, multiple historical movement trajectories of the remote control, as well as thumbnails of multiple second images (also referred to as second image thumbnails in the embodiments of the present application) generated based on the multiple historical movement trajectories, may also be displayed on the application interface of the target application. Or thumbnails of multiple second images generated based on a pre-set line graph and the line graph are displayed, such as Figure 6 thumbnails of Image 1, Image 2, Image 3... shown in (1) and 6(2), etc., to intuitively indicate the functions of the target application.

[0081] In some embodiments, while multiple second image thumbnails are displayed on the application interface of the target application, one or more thumbnails of the first image (also referred to as first image thumbnails in the embodiments of the present application) may also be displayed. Among them, the first image includes a picture of one or more historical movement trajectories of the remote control, such as Figure 6 a first image thumbnail 621 is also shown in (2) to display the thumbnail of the drawn first image that has been stored.

[0082] The manner of entering the application interface of the target application is not limited. It may be entering the application interface of the target application by selecting the target application among the applications on the home page of the display device. In some embodiments, it may also be entering the application interface of the target application by receiving a voice command or a preset gesture command. In other embodiments, the target application may also be a sub-application of the first application of the display device, that is, the first application includes the target application and one or more second applications. After entering the first application, the target application is selected from multiple applications of the first application and the application interface of the target application is entered. In other embodiments, the application interface of the target application may also be entered by other means.

[0083] The painting area on the painting interface is usually located in the middle area of the painting interface, such as Figure 6 the painting area 601 in (3) and 6(4). However, it is not limited thereto. For example, in some examples, the full-screen area of the display may be used as the painting area to achieve a full-screen painting effect.

[0084] The method of receiving the wireless signal of the remote control and determining the movement trajectory of the remote control based on the wireless signal is not limited, as long as the position of the remote control can be determined based on the received wireless signal of the remote control, and the movement trajectory of the remote control can be determined based on multiple positions of the remote control.

[0085] In some embodiments of the present application, the positioning of the remote control can be implemented in combination with the UWB (Ultra Wide Band) positioning method. Among them, a UWB transmitter is provided in the remote control. The remote control provided with the UWB transmitter can also be called an absolute pointing remote control. The display device is provided with a UWB receiver. At this time:

[0086] The processor is further configured to execute instructions to cause the display device to:

[0087] Obtain the UWB wireless signal received by the ultra-wideband UWB receiver, and determine the position information of the remote control based on the UWB wireless signal.

[0088] During the process of the user operating the remote control, the UWB transmitter of the remote control will emit a UWB wireless signal, and the UWB receiver of the display device will receive the UWB wireless signal. Thus, the position information of the remote control can be determined according to the UWB wireless signal. Furthermore, the movement trajectory of the remote control can be determined according to multiple UWB wireless signals, and the movement trajectory can be drawn in the drawing area of the display interface.

[0089] The method of determining the position information of the remote control based on the UWB wireless signal is not limited. For example, an antenna array is provided in the UWB transmitter, and an antenna array is also provided in the UWB receiver. The remote control sends a UWB wireless signal through the antenna array of the UWB transmitter, and the UWB wireless signal can carry the transmission time. The display device receives the UWB wireless signal through the antenna array of the UWB receiver, combines the time difference between the transmission time and the reception time of the UWB wireless signal to determine the propagation time of the UWB wireless signal, and then calculates and determines the distance between the antenna of the remote control and the antenna of the display device according to the propagation time and the propagation speed (speed of light) of the UWB signal. Since the pulse width of the UWB signal is very short, therefore, the propagation time can very accurately reflect the propagation distance of the signal. Combining the UWB wireless signals received by multiple antennas of the antenna array of the UWB receiver, multiple propagation distances can be calculated and determined, and then the position information of the remote control can be determined in combination with multiple propagation distances. The position information in the display screen of the display device can be determined in combination with this position information.

[0090] It can be understood that in other embodiments, other methods may also be adopted to determine the position information of the remote controller based on the UWB wireless signal. This application does not specifically limit the method for determining the position information of the remote controller. After determining the movement trajectory of the remote controller in combination with the position information, the movement trajectory can be displayed on the painting interface, such as Figure 6 the movement trajectory drawn in the painting area 601 in (3).

[0091] The image generation interface is an interface for displaying the generated second image. Among them, there can be only one second image displayed in the image generation interface, or there can be more than two, such as Figure 6 in the image generation interface shown in (7), multiple second images are displayed in the display area 672, such as thumbnails of Image 1, Image 2, Image 3, etc., and Image 1 is displayed in the display area 671.

[0092] Among them, the second image on the image generation interface is generated based on the screen of the painting interface. Among them, when generating the second image based on the screen of the painting interface, it can be generated by the display device or by the server.

[0093] Taking the generation of the second image by the display device as an example, the display device can receive and respond to the image generation instruction, and generate the first image based on the screen of the painting interface. Among them, the screen of the painting interface includes the movement trajectory; perform content understanding on the first image to obtain the text features of the first image; perform image feature recognition on the first image to obtain the image features of the first image, and perform image generation on the first image based on the text features and image features to obtain the generated second image.

[0094] Taking the generation of the second image by the server as an example, the display device can receive and respond to the image generation instruction, and generate the first image based on the screen of the painting interface; send an image generation request to the server, and the image generation request includes the first image. After receiving the image generation request, the server generates the second image based on the first image in the image generation request, and returns an image generation response to the display device, and the image generation response includes the second image. The display device receives the image generation response returned by the server and displays the second image included in the image generation response. Among them, when the server generates the second image based on the image generation request, it can perform content understanding on the first image to obtain the text features of the first image; perform image feature recognition on the first image to obtain the image features of the first image, and perform image generation on the first image based on the text features and image features to obtain the generated second image.

[0095] According to the display device of the embodiment of the present application, after displaying the painting interface, it receives the wireless signal of the remote control, determines the moving trajectory of the remote control through the wireless signal of the remote control, and displays the moving trajectory on the painting interface, thereby realizing wireless painting on the painting interface of the display device by moving the remote control, without being limited by the touch of a stylus and devices such as a wired mouse. On the basis of the painting content determined by the moving trajectory of the remote control, it is also possible to further generate an image on the basis of the image generation instruction on the basis of the screen of the painting interface to obtain the generated second image, so that on the basis of the moving trajectory drawn by the user through the remote control, a richer second image can be further generated, which can further improve the interaction performance of the display device and meet the needs of rich painting interactions.

[0096] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0097] Obtain target image style information,

[0098] Display an image generation interface, and the second image displayed on the image generation interface also matches the target image style information.

[0099] The image style refers to the information of the visual expression method and artistic style of the image. Some possible image styles include sketch style, watercolor style, oil painting style, comic style, hand-painted style, abstract style, science fiction style, retro style, cyberpunk style, movie sense, black and white art style, natural scenery style, cartoon style, etc., but not limited to this. The target image style information refers to the information of the image style of the second image expected to be generated, which specifies the information of the image style of the second image.

[0100] The method of obtaining the target image style information is not limited. In some embodiments, the information of the image style preset in the display device can be determined as the target image style information. There can be at least one piece of image style information preset in the display device. If there is only one piece of image style information preset in the display device, the information of this image style can be directly used as the target image style information. If there are multiple pieces of image style information preset in the display device, one or more of them can be randomly selected as the target image style information.

[0101] In other embodiments, the display device can receive and respond to an image style determination instruction to obtain target image style information. In some embodiments, a style selection area can be set on the painting interface, and multiple style selection items are set in the style selection area. Each style selection item corresponds to an image style information. The corresponding target image style information can be obtained by receiving a selection instruction for one or more style selection items.

[0102] In some other embodiments, the display device may analyze the content displayed on the painting interface or a first image containing the content displayed on the painting interface to generate recommended image style information. Accordingly, in some embodiments, the processor is further configured to:

[0103] Understand the content of the first image to obtain semantic label information of the first image;

[0104] Match and obtain target image style information associated with the semantic label information from a preset plurality of image style information.

[0105] The type of the semantic label information is not limited. For example, the semantic label information identified in some examples may be, for example, a warm cottage, a future mecha, etc. The method of matching and obtaining the target image style information associated with the semantic label information is not limited. In some examples, style matching may be performed in combination with a style inference model (such as Prompt-to-Style). Through the style inference model, one or more image style information with the strongest correlation with the semantic label information can be output.

[0106] Among them, the style inference model can be trained. After the style inference model is trained by a server or other device, the trained style inference model can be deployed to the display device or the server to generate recommended image style information for the first image.

[0107] The method of training to obtain the style inference model is not limited. In some embodiments, the method of training to obtain the style inference model may be as described below.

[0108] First, obtain a sample data set. Each sample data in the sample data set includes a line drawing and the image style corresponding to the line drawing. Among them, the number of lines in the line drawing is not limited. The line drawing may only have lines, or it may include lines and at the same time have corresponding filling colors in the enclosed or semi-enclosed areas of the lines. The label is the image style information of the line drawing. After obtaining the sample data set, divide the sample data set into a training set, a validation set, and a test set.

[0109] After the sample data set is divided, the model training process can be carried out. During the model training process, the style inference model is trained with the training set, and during or after the training process, the validation set is used to evaluate the performance of the style inference model, and the test set is used to finally test the trained style inference model to ensure the performance of the model in actual applications. It can be understood that during the training of the style inference model, the style inference model understands the content of the sample data to obtain the corresponding image style information, and compares the image style information with the image style information labeled for the sample data to calculate the training loss, and adjusts the style inference model in combination with the calculated training loss to achieve the update of the style inference model. After the model training is completed, the style inference model updated for the last time is used as the trained style inference model.

[0110] Thus, when obtaining the target image style information, based on the semantic label information obtained by understanding the content of the first image, the target image style information associated with the semantic label information is matched, so that the obtained target image style information is matched with the content of the first image, which can improve the adaptability of the obtained target image style information to the content of the first image and help to further improve the interaction performance of the display device.

[0111] Among them, the obtained image style information can be only one, or can include two or more. In the case where the image style information includes two or more, the second image includes two or more, and each second image is matched with at least one target image style information. For example, one or more of the second images are images corresponding to the first image style, and one or more of the other second images are images corresponding to the second image style, and one or more of the second images are images with two or more image styles at the same time, etc., to obtain an image that can have two or more different image styles at the same time, or the effect of style comparison of different image styles.

[0112] It can be understood that in the case where the server generates the second image, the image generation request sent by the display device to the server may also include the target image style information. In some other embodiments, in the case where the server generates the second image, after the display device sends an image generation request to the server, it may also be that the server analyzes the first image to generate recommended image style information.

[0113] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0114] Based on the resolution of the display device, perform image magnification processing on the second image, and display the second image after the image magnification processing.

[0115] The resolution of a display device refers to the number of physical pixels that the display device can display, usually expressed in the format of width × height, such as 1920×1080, 3840*2160, etc., but not limited to this.

[0116] When performing image magnification processing on the second image, it can be combined with the resolution of the generated second image and the resolution of the display device. Among them, the resolution of the generated second image refers to the number of pixels of the generated second image, which can be determined by magnifying the size or pixel value information array of the generated second image.

[0117] The method of performing image magnification processing on the second image based on the resolution of the display device is not limited, as long as the resolution of the second image after image magnification can be consistent with the resolution of the display device. Among them, being consistent with the resolution of the display device can mean that the resolution of the second image after image magnification is the same as the resolution of the display device, or the difference between the resolution of the second image after image magnification and the resolution of the display device is within a preset difference range, as long as the second image after image magnification can be displayed within the effective display area of the display device and has high clarity.

[0118] In some embodiments, image magnification processing can be performed by means of image interpolation of the generated second image to obtain the magnified second image. In other embodiments, an image magnification model can be used to perform image magnification processing on the generated second image.

[0119] Based on this embodiment, after obtaining the second image, image magnification processing can also be performed on the second image based on the resolution of the display device and then displayed. Thus, when generating the second image, a second image with a relatively small resolution can be generated to improve the efficiency of generating the second image, and further combined with the resolution of the display device to perform image magnification processing on the second image and then display it, so as to meet the display requirements of the resolution of the display device and simultaneously meet the requirements of image generation efficiency and the resolution of the display device.

[0120] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0121] Receive and respond to a painting instruction, display a painting interface, and display guiding prompt information in a first preset area of the painting interface. The guiding prompt information includes prompt information for painting on the display device via a remote controller.

[0122] The specific position of the first preset area is not limited. For example Figure 6 The first preset area 602 at the lower left corner of the display interface shown in (4). The specific information content included in the guiding prompt information is not limited. For example Figure 6"Turn on the pointing mode of the remote control, select the brush color, hold down the OK button to draw, or drag the color to fill." Among them, when displaying the guiding prompt information, it can also be combined with an image mark for prompting. For example, Figure 6 Pattern 1 in (4), the image mark can be a pattern of a related cartoon or other image, so as to achieve the effect of being prompted by an object such as a person marked by the image mark. It can be understood that in other embodiments, the image mark and the guiding prompt information can also be set in other ways.

[0123] Based on this embodiment, when displaying the painting interface in response to the received painting instruction, guiding prompt information is also displayed in the first preset area of the painting interface to prompt how to paint on the display device through the remote control, so as to guide the user to paint on the painting interface through the remote control, thereby further improving the interaction performance of the display device and enhancing the user experience.

[0124] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0125] Display a brush interface in the second preset area of the painting interface. The brush interface includes multiple brush selection items, and the widths of the lines drawn by the brushes corresponding to different brush selection items are different.

[0126] The specific position of the second preset area is not limited, as long as it is convenient for the user to select different brushes. For example, Figure 6 The second preset area 603 shown in (3). As Figure 6 As shown in (3), there are three brushes 6031, 6032, and 6033 with different thicknesses preset, and the widths of the lines drawn by the brushes with different thicknesses are different. It can be understood that in other embodiments, the number of brushes can also be set in other ways.

[0127] Based on this embodiment, a brush interface can be displayed in the second preset area of the painting interface displayed on the display, so that multiple different brush selection items can be provided, thereby allowing the user to select different brush selection items to draw lines with different widths through different brushes, which helps to draw lines with different widths and further improves the interaction performance of the display device.

[0128] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0129] Display a color interface in the third preset area of the painting interface. The color interface includes multiple color selection items, and the colors corresponding to different color selection items are different.

[0130] The specific position of the third preset area is not limited, as long as it is convenient for the user to select different colors of the brush. For example, Figure 6The third preset area 604 shown in (3). As Figure 6 As shown in (3), the third preset area 604 is provided with a plurality of different color selection items, such as Figure 6 The color selection item 6041 shown in (4). After selecting different color selection items, the color of the drawn line will be different. It can be understood that in other embodiments, the number of color selection items in the third preset area 604 can also be set otherwise.

[0131] Based on this embodiment, a color interface can be displayed in the third preset area of the painting interface displayed on the display, so that a plurality of different color selection items can be provided, so that the user can select different color selection items to draw lines of different colors with different color brushes, which helps to realize the drawing of lines of different colors and further improves the interaction performance of the display device.

[0132] Among them, the colors corresponding to the multiple color selection items in the third preset area can be multiple different colors pre-set by the display device.

[0133] In some embodiments, the colors corresponding to the multiple color selection items in the third preset area can also be colors recommended by combining the content displayed on the painting interface and display the selection items of these colors. At this time, the processor is further configured to execute instructions to enable the display device to:

[0134] Perform semantic segmentation on the picture of the painting interface to obtain a plurality of first object contents;

[0135] Based on the plurality of first object contents and the picture, determine the recommended color information and display the recommended color information in the third preset area. For example, display the recommended color information in the color interface.

[0136] The method of performing semantic segmentation is not limited. In some embodiments, it can be performed through a semantic segmentation model. The semantic segmentation model can adopt any model that can perform semantic analysis on images. In the embodiments of the present application, it can include the SAM (Segment Anything Model) semantic segmentation model, but is not limited thereto. Among them, in some specific examples, it can be to generate a first image of the picture including the painting interface and then perform semantic segmentation on the first image.

[0137] Thus, the display device can also perform semantic segmentation on the picture of the painting interface and determine the recommended color information based on the plurality of first object contents obtained by the segmentation, so that the recommended color information is matched with the content displayed on the painting interface, and colors that are more matched with the content displayed on the painting interface can be provided for the user to select and use, further improving the interaction performance of the display device.

[0138] Among them, when it is possible to generate recommended colors for the screen in combination with the painting interface, the multiple color selection items in the third preset area can all be selection items for the colors generated in combination with the screen of the painting interface, or some of the color selection items can be the color selection items preset by the display device in advance. For example Figure 6 each color selection item in the first sub-region 6042 shown in (3), and some of the color selection items are selection items for the colors generated in combination with the screen of the painting interface. For example Figure 6 each color selection item in the second sub-region 6043 shown in (3), but not limited to this.

[0139] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0140] Receive and respond to a color selection instruction, and set the color of the brush for drawing the movement trajectory to the color corresponding to the color selection instruction.

[0141] There is no limit to the way of receiving the color selection instruction. In some examples, the color selection instruction can be received by operating on the color selection item. In some examples, the color selection instruction can also be received by a voice instruction. For example, by voice inputting "select the first color", the color corresponding to the first color selection item in the arrangement order of the third preset area 604 can be determined as the color corresponding to the color selection instruction.

[0142] In some embodiments, referring to Figure 6 as shown in (3), the third preset area 604 can also be provided with a preset control 6044. By operating on the preset control 6044, an interface with more color selection items can be displayed, such as Figure 6 the color selection interface 6045 shown in (5), which displays 18 color selection items, so that more color selection items can be selected accordingly.

[0143] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0144] Receive and respond to an operation instruction for generating a new color, determine a new color according to the colors corresponding to two or more selected color selection items, and display the new color.

[0145] There is no limit to the way of receiving the operation instruction for the new color. Several of the ways are exemplified below.

[0146] In some embodiments, it can be to receive the operation instruction for generating a new color through a control provided on the painting interface corresponding to generating a new color, the collected voice information, or the received gesture information, etc.

[0147] In some other embodiments, it may be that after moving the focus of the remote control to a certain color selection item, by pressing the first preset button of the remote control, within a preset duration, after receiving the operation instruction of pressing the first preset button after the focus moves to another color selection item, it is considered that the preset operation instruction is received, that is, if the interval duration of the operation instruction for the first preset button received is within the preset duration, it is considered that the preset operation instruction is received, and the new color is determined according to the colors corresponding to two or more color selection items.

[0148] The method for determining the new color according to the colors corresponding to two or more selected color selection items is not limited. In some embodiments, the pixel values of the colors of each color selection item can be weighted and summed according to the number of clicks on each color selection item to obtain the color corresponding to the weighted sum pixel value. In some other embodiments, it may also be that according to the residence duration of the focus of the remote control on each color selection item, the pixel values of the colors of each color selection item are weighted and summed, and different residence durations correspond to different weighting coefficients, to obtain the color corresponding to the weighted sum pixel value. In other embodiments, it may also be to use other methods to determine the new color according to the colors corresponding to two or more color selection items. In some other embodiments, color overlay calculation can be performed based on the LAB color space, which can make the mixing result more conform to the visual law of the human eye and help to further improve the interaction performance of the display device.

[0149] In some embodiments, when receiving the operation instruction for two or more color selection items, it can be considered that the trigger instruction for generating a new color is received. Based on the received trigger instruction for generating a new color, when clicking on the color selection item, a particle diffusion animation can also be triggered, and the particle movement speed is associated and linked with the color concentration parameter, so as to visually and intuitively know the color selection and color fusion effect. In some embodiments, when there is a color palette control on the color interface, when clicking on different color selection items, a plurality of arranged rings or sectors can be generated at the edge of the color palette control. The number of rings or sectors is the same as the number of clicked color selection items, and the colors of each ring or sector are different, so that it can be intuitively known how many color selection items have been selected. In some examples, the number of layers of the plurality of rings or sectors can be configured to achieve the effect of a gradient color halo, so as to further achieve the effect of intuitive display.

[0150] In some embodiments, when there is a color palette control, different display effects can also be set for the color palette control, such as a metallic texture effect, different metallic reflection intensities, etc.

[0151] In some other embodiments, during the process of generating a new color, after the user has clicked on one or more color selection items, a recommended color complementary to the selected color can also be determined, and the color selection item of the complementary recommended color can be highlighted, and a prompt message can be displayed to help the user select the complementary color to generate a new color, thereby enabling artistic creation accordingly.

[0152] Based on this embodiment, the display device can also determine a new color based on operation instructions for two or more color selection items, and accordingly can personalize the configuration and setting of the new color, further improving the interaction performance of the display device.

[0153] In some embodiments, it is further configured to execute instructions to cause the display device to:

[0154] Set the color of the cursor displayed on the painting interface to the color corresponding to the color selection instruction.

[0155] Combined with Figure 6 As shown in (3), the color of the cursor on the painting interface is the same as the color of the third color selection item in the third display area 604.

[0156] Based on this embodiment, according to the received color selection instruction, the color of the cursor displayed on the drawing interface of the display can also be set to the color corresponding to the color selection instruction, so that the user of the display device can intuitively know the color of the brush during the current drawing based on the color of the cursor, further improving the interaction performance of the display device.

[0157] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0158] Receive and respond to a color filling instruction, and fill the positioning area where the target moving position of the remote control is located with the currently selected color, and the positioning area is determined according to the target moving position and the moving trajectory.

[0159] The color filling instruction is an instruction for filling a partial area of the screen of the content displayed on the painting interface with a color. By receiving the color filling instruction and filling the positioning area where the target moving position of the remote control is located with the currently selected color, on the basis of drawing lines such as the moving trajectory through the remote control, the positioning area can be further filled with the selected color, further improving the interaction performance of the display device.

[0160] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0161] Receiving and in response to a color filling instruction, displaying the cursor shown on the painting interface as a second state, where the second state is associated with image filling.

[0162] The second state is a state that can indicate the availability for image filling. For example, in some examples, the second state can be the state of an oil bucket, as shown by the cursor 606 in Figure 6 (6).

[0163] Based on this embodiment, after receiving the color filling instruction, the cursor shown on the painting interface is also displayed as a second state associated with image filling, so that the second state of the cursor can intuitively indicate that an image filling operation is to be performed.

[0164] The determination method of the positioning area where the target moving position is located is not limited. In some embodiments, it may include a closed area formed by the moving trajectory on the painting interface, such as Figure 6 the area 607 in (6). In other embodiments, it may also include a semi-closed area formed by the moving trajectory and the boundary of the painting interface, but is not limited thereto.

[0165] The method of color filling the positioning area is not limited. In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0166] Generate a layer of the currently selected color in the positioning area.

[0167] Based on this embodiment, the color filling of the positioning area can be achieved by generating a layer of the currently selected color in the positioning area, so that the filled part is located on different layers. The filled area and other areas can be processed through different layers, and it is helpful for further processing of the filled area, etc., further improving the interaction performance of the display device.

[0168] Receiving and in response to a coloring instruction for the painting interface, performing intelligent coloring processing on the content displayed on the painting interface to obtain a colored picture.

[0169] The coloring instruction is an instruction used to indicate automatically adding colors to the content displayed on the painting interface.

[0170] The method of performing intelligent coloring processing on the content displayed on the painting interface is not limited. In some embodiments, it may be that after obtaining a first image based on the content displayed on the painting interface, the first image is input into a coloring model, and the coloring model automatically performs intelligent coloring processing on the first image to obtain a colored picture.

[0171] Among them, the coloring model can be a coloring model constructed based on a generative adversarial network (GAN). The coloring model can be a model obtained through training, and can use the line drawing as the sample object. With the manual line drawing of the line drawing as the label, the generator model is trained to enable it to automatically learn the mapping relationship from lines to colors. After obtaining the trained coloring model, the first image is directly colored and output for the whole image through the coloring model, so that fast coloring processing with consistent effects and without manual intervention can be generated.

[0172] In some embodiments, the processor is further configured to:

[0173] In response to the coloring instruction, perform semantic segmentation on the content displayed on the painting interface to obtain multiple first object contents;

[0174] Based on the multiple first object contents and the screen, determine the color information corresponding to each first object content, and use the color information corresponding to each first object content to perform coloring processing on the corresponding first object content to obtain the colored screen.

[0175] Based on this embodiment, it is also possible to receive a coloring instruction. After performing semantic segmentation on the content displayed on the painting interface to obtain multiple first object contents, determine the color information corresponding to each first object content and automatically perform coloring processing. Thus, after drawing lines and other contents on the painting interface, intelligent coloring processing can also be performed, further improving the interaction performance of the display device.

[0176] The manner of using the color information corresponding to each first object content to perform coloring processing on the corresponding first object content is not limited. In some embodiments, the processor is further configured to execute instructions to enable the display device to:

[0177] Generate a layer mask for each first object content;

[0178] Based on the multiple first object contents and the screen, determine the color information corresponding to each first object content, and use the color information corresponding to each first object content to perform coloring processing on the corresponding layer mask to obtain the colored screen.

[0179] Among them, the manner of determining the color information corresponding to each first object content is not limited. In some embodiments, the model for semantic segmentation can be a multi-modal semantic segmentation model. While obtaining multiple first object contents, it can also combine the regions to which each first object content belongs and the relative position relationships, etc., to generate scene description information, such as "a forest cottage under the sun". Furthermore, based on the scene description information, the recommended colors for each first object content are determined. For example, when the first object content is the sky, the recommended color is gradient blue, and when the first object content is vegetation, the recommended colors are shades of green, etc.

[0180] A layer mask, also known as a masked layer, is a layer that covers the layers of the interface displayed on the display screen. Moreover, this layer can be transparent, semi-transparent, or completely opaque. Thus, certain parts of an image can be modified by setting a visible or transparent layer on the image. By painting or filling specific areas on the mask, specific parts of the image can be selectively hidden, shown, or modified, thereby achieving the effect of adjusting the display effect of the image.

[0181] Accordingly, during the process of performing intelligent coloring on the first object content, it is carried out by generating a layer mask and then coloring the corresponding positions of the layer mask, so that targeted coloring can be performed on the area corresponding to the first object content, further improving the interaction performance of the display device.

[0182] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0183] Use the color information corresponding to each first object content to perform color coordination processing on the boundaries of adjacent layer masks.

[0184] Based on this embodiment, after coloring the layer masks corresponding to each first object content, color coordination processing is also performed on the boundaries of adjacent layer masks, so that the color coordination of the boundaries of adjacent layer masks can be improved, and further contribute to maintaining the overall color harmony of the picture.

[0185] Among them, during the process of drawing on the painting interface, that is, during the processes of generating the movement trajectory of the remote control and displaying it, performing color filling, and intelligent coloring, etc., backward and forward operations can also be performed on these operations. The ways to control the backward and forward operations are not limited. In some examples, a backward operation control and a forward operation control can be displayed on the painting interface, such as Figure 6 the backward operation control 641 and the forward operation control 642 shown in (6). By receiving a backward operation instruction through the backward operation control 641, the most recent painting operation can be withdrawn, for example, clearing the just-drawn movement trajectory. By receiving a restoration operation instruction through the forward operation control 642, the withdrawn painting operation can be restored, for example, restoring the just-cleared and just-drawn movement trajectory. Among them, when all the content corresponding to the painting operations has been displayed on the painting interface and there is no recoverable operation, the forward operation control 642 can be set to an inoperable state, for example, set to gray, as shown in Figure 6 (3), 6(5), 6(6).

[0186] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0187] Receive and respond to a clear instruction to clear the content of the screen on the painting interface. That is, all the drawn content on the painting interface can be cleared.

[0188] After receiving the clear instruction, a clear confirmation interface can also be displayed. The clear confirmation interface in some examples is as Figure 7 (1) shown, for the user to confirm the drawn content to be cleared on the painting interface, and then clear the content of the screen on the painting interface after receiving the confirmation instruction, so as to reduce the possibility of clearing the screen content due to misoperation. Among them, when the clear confirmation interface is displayed, if a cancel instruction is received, the painting interface can be returned.

[0189] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0190] Receive and respond to a storage instruction for the content displayed on the painting interface, generate a first image containing the screen content of the painting interface, and store the generated first image. That is, the first image directly contains the screen content of the painting interface, such as containing the movement trajectory drawn on the painting interface, and the content after color filling operation, etc.

[0191] The generated and saved first image can be displayed when it is necessary to view the historical original image, for the user to view the historical first image, make further edits or deletions, etc. In some examples, when entering the application interface of the target application, a thumbnail of the first image can be displayed on the application interface of the target application, as Figure 6 (2) shown, the first image thumbnail 621.

[0192] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0193] Receive and respond to a deletion instruction for the first image, and delete the stored first image.

[0194] Taking Figure 6 (2) shown as an example, it can be that when the focus of the remote control is on the first image thumbnail 621, the user issues a deletion instruction by pressing a preset button on the remote control to indicate deleting the first image corresponding to the first image thumbnail 621.

[0195] In some embodiments, after receiving the deletion instruction, a deletion confirmation interface can also be displayed. The deletion confirmation interface in some examples is as Figure 7 (4) shown, for the user to confirm the deletion of the first image, and then delete the first image after receiving the confirmation instruction, so as to reduce the possibility of deleting the first image due to misoperation. Among them, when the deletion confirmation interface is displayed, if a cancel instruction is received, the interface before receiving the deletion instruction can be returned, as Figure 6The interface shown in (2).

[0196] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0197] Receive and respond to a viewing instruction for a first displayed image, display a painting interface, and display the screen content of the first image on the painting interface.

[0198] Since the first image is an image containing the screen content of the historical movement trajectory, by receiving the viewing instruction for the first image, the painting interface containing the screen content of the first image can be directly displayed, which helps to further operate on the basis of the stored first image.

[0199] Among them, when displaying the screen content of the first image on the painting interface, in the case where the screen content of the first image contains different movement trajectories, for the lines of different movement trajectories, different line objects can be used for display, so as to facilitate further operations on different lines, such as deletion, position movement, color change, etc., but not limited to this.

[0200] It can be understood that taking Figure 6 the application interface shown in (2) as an example, a deletion instruction for the first image thumbnail 621 can be issued through one of the preset buttons on the remote control or a combination of a group of preset buttons, and a viewing instruction for the first image thumbnail 621 can be issued through another preset button on the remote control or a combination of another group of preset buttons, as long as the deletion instruction and the viewing instruction can be distinguished.

[0201] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0202] Receive and respond to an image generation instruction, switch to display a picture prompt interface, the picture prompt interface is used to prompt that an image is being intelligently generated, and after the second image generation is completed, switch to display an image generation interface.

[0203] The picture prompt interface is an interface used to prompt that an image is being intelligently generated, and the specific content displayed on the picture prompt interface is not limited. A specific example of the picture prompt interface is as Figure 7 shown in (3).

[0204] Based on this embodiment, after receiving the image generation instruction, it can first switch to display the picture prompt interface to prompt that the image is being intelligently generated, and after the second image generation is completed, switch to display the image generation interface to display the generated second image.

[0205] Among them, the processor is further configured to execute instructions to cause the display device to:

[0206] During the process of displaying the illustration prompt interface, if a cancellation generation instruction is received, switch to display the painting interface.

[0207] In some embodiments, after receiving the cancellation generation instruction, a cancellation confirmation interface may also be displayed. The cancellation confirmation interface in some examples is as Figure 7 shown in (2), for the user to confirm the cancellation of image generation, and then switch to display the painting interface after receiving the confirmation instruction, so as to reduce the possibility of canceling the image generation process due to misoperation. Among them, in the case of displaying the cancellation confirmation interface, if a cancellation instruction is received, return to display the illustration prompt interface to continue the image generation process.

[0208] Based on this embodiment, during the process of intelligently generating the second image based on the image generation instruction, the intelligent image generation process can also be canceled by receiving a cancellation instruction and switch to display the painting interface, thereby further improving the interaction performance of the display device.

[0209] The image generation interface is an interface for displaying the generated second image. The specific manner of the image generation interface is not limited. In some embodiments, the image generation interface includes a first display area and a second display area. The first display area displays thumbnails of multiple second images generated based on the screen of the painting interface, and the second display area displays the second image corresponding to the thumbnail where the focus of the remote control is located.

[0210] Among them, the first display area is an area for displaying thumbnails of multiple second images, and the second display area is an area for displaying one specific second image. The specific positions of the first display area and the second display area on the display screen are not limited. The setting positions in a specific example can be as Figure 6 shown in (7). The first display area 671 is set in the area below the display, and the second display area 672 is set in the area above the display. Thumbnails of multiple different second images generated are displayed in the first display area 671. For example, thumbnails of Image 1, Image 2, Image 3, and Image 4 are displayed in the first display area 671. The second display area 671 displays the image corresponding to the thumbnail of Image 1 where the focus is located.

[0211] Based on this embodiment, when displaying the generated image, the image generation interface simultaneously includes a first display area and a second display area, and a plurality of second image thumbnails are simultaneously displayed in the first display area. Thus, a plurality of second images can be generated simultaneously based on the content of the screen of the painting interface and their thumbnails are displayed, and the second image corresponding to the second image thumbnail where the focus of the remote control is located is displayed in the second display area, that is, the magnified image of the second image thumbnail can be displayed in the second display area. Thus, while conveniently viewing a plurality of different second image thumbnails generated, the corresponding magnified image can be viewed, further improving the interaction performance of the display device.

[0212] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0213] Display a first image thumbnail at at least one of a preset position in the second display area, a preset position in the first display area, and a preset position of the thumbnail of the second image.

[0214] The first image is an image containing the content of the screen of the painting interface, which includes the movement trajectory of the painting interface. The first image thumbnail is a thumbnail of the first image. The specific position of the preset position is not limited, and it can be any position in the image generation interface as long as it does not obscure or completely obscure the second image and the second image thumbnail. In some embodiments, the preset position includes the boundary position of the first display area, and can also be the boundary position of the second image, such as Figure 6 the lower left corner of the image 1 displayed in the first display area 672 as shown in (7).

[0215] At the boundary of the first image thumbnail, original image prompt information can also be displayed, such as Figure 6 the words "original image" as shown in (7), to prompt that the image corresponding to the first image thumbnail is the original image used to generate the second image. In other embodiments, the specific text content of the original image prompt information can also be set otherwise.

[0216] In some embodiments, at the preset position of one or more second image thumbnails, a first image thumbnail can also be displayed to indicate that the second image corresponding to the second image thumbnail is generated based on the image corresponding to the first image thumbnail. As shown in Figure 6 (7) Figure 1 , Figure 2 , Figure 3 the lower left corners of the second image thumbnails of all show the corresponding first image thumbnails.

[0217] Based on this embodiment, when displaying the image generation interface, a thumbnail of the screen content of the painting interface is also displayed at a preset position of the image generation interface, so as to compare and view the screen content of the painting interface before image generation and the second image obtained after image generation, which helps to further improve the interaction performance of the display device.

[0218] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0219] Receive and respond to a full-screen display instruction, and perform full-screen display of the second image displayed in the second display area. In one example, for Figure 6 the full-screen display interface after the full-screen display instruction is received for the interface of (7) is as Figure 6 (8) shown.

[0220] Wherein, when performing full-screen display of the second image, a first image thumbnail may also be displayed at a preset position of the full-screen display interface. The first image thumbnail includes a thumbnail of the screen of the painting interface, as Figure 6 (8) shown, so that when performing full-screen display, it is also possible to compare and view the screen content of the painting interface before image generation and the second image obtained after image generation, which helps to further improve the interaction performance of the display device.

[0221] Based on this embodiment, it is also possible to perform full-screen display of the second image displayed in the second display area. By performing full-screen display of the second image, it is possible to view the image details in the second image more closely.

[0222] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0223] When performing full-screen display of the second image displayed in the second display area, receive and respond to an image return instruction, and switch to display the image generation interface. For example, when displaying Figure 6 the full-screen display interface shown in (8), if an image return instruction is received, then return to display the image generation interface as shown in Figure 6 (7).

[0224] Based on this embodiment, during the process of full-screen display of the second image, it is also possible to switch to display the image generation interface according to the image return instruction, so that it is possible to switch between full-screen display of the second image and the image generation interface that simultaneously displays multiple second image thumbnails and the second image, which helps to further improve the interaction performance of the display device.

[0225] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0226] Receive and respond to the home page save instruction, and set the second image corresponding to the home page save instruction as the image of the application home page of the target application. That is, when the page of the application home page of the target application is displayed next time, the second image will be displayed on the application home page. In the case where there are a large number of images on the application home page, the second image can be found and displayed by dragging or moving the focus of the remote control.

[0227] Among them, in some specific examples, when the home page save instruction is received when the image generation interface is displayed, it can be the second image corresponding to the second image thumbnail where the focus of the remote control is located, or the second image displayed in the first display area, which is set as the image of the application home page of the target application. When the home page save instruction is received when the full-screen display interface of the second image is displayed, the second image displayed on the full-screen display interface is set as the image of the application home page of the target application.

[0228] In some specific examples, when setting the second image as the image of the application home page of the target application, the second image and its corresponding first image thumbnail can be set as the image of the application home page of the target application, and when the second image and the first image thumbnail are displayed on the application home page of the target application, they can be displayed according to a preset positional relationship. For example, the first image thumbnail is located at the lower left corner of the thumbnail of the second image, as shown in Figure 6 (1), 6(2).

[0229] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0230] Receive and respond to the trajectory adjustment instruction, adjust the movement trajectory, and display the adjusted movement trajectory on the painting interface, where the trajectory adjustment instruction is an instruction sent by the remote control, an instruction obtained by collecting voice information, or an instruction obtained by gesture recognition.

[0231] The trajectory adjustment instruction is an instruction used to indicate the adjustment of the movement trajectory displayed on the painting interface. The type of adjustment of the movement trajectory is not limited, including but not limited to moving the position, zooming in, zooming out, etc. Different operation instructions for the movement trajectory can be issued through the remote control, voice, or gesture recognition. The specific way of issuing the operation instructions is not limited as long as different instructions can be distinguished.

[0232] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0233] Receive the wireless signals of two or more remote controls, respectively determine the position information of the two or more remote controls based on the wireless signals, and obtain the movement trajectories of the two or more remote controls according to the multiple position information during the movement of the remote controls;

[0234] Display the movement trajectories of two or more remote controls on the painting interface.

[0235] Based on this embodiment, two or more remote controls can simultaneously send wireless signals to the display device, so as to simultaneously draw movement trajectories on the display interface through two remote controls, enabling two or more users to separately use different remote controls to draw simultaneously. On the basis of combining the trajectories of the remote controls used by two or more different users, the effect of further generating a second image with rich colors is achieved, further improving the interaction performance of the display device.

[0236] Among them, taking the wireless signal sent by the remote control as a UWB wireless signal as an example, the display device can be provided with two or more UWB receivers to respectively receive the UWB wireless signals of different remote controls, or only one UWB receiver can be set to receive the UWB wireless signals of different remote controls. Since different remote controls are in different spatial positions, based on the received UWB wireless signals, two different position information can be located, and by positioning and tracking the position information, different remote controls can be identified and the movement stacks of different remote controls can be obtained. In some other embodiments, it can also be that the wireless signal sent by the remote control includes the remote control identifier of the remote control, so that different remote controls can be distinguished through the remote control identifier, but it is not limited to this, as long as the movement trajectories of different remote controls can be distinguished.

[0237] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0238] Record the remote control identifier corresponding to the movement trajectory of the remote control;

[0239] Receive and respond to the trajectory adjustment instruction sent by the remote control. If the remote control identifier of the movement trajectory corresponding to the trajectory adjustment instruction is the same as the remote control identifier corresponding to the remote control, adjust the movement trajectory corresponding to the trajectory adjustment instruction, and display the adjusted movement trajectory on the painting interface.

[0240] Based on this embodiment, when the trajectory adjustment instruction is received, if the trajectory adjustment instruction is an instruction sent by the remote control, only the movement trajectory corresponding to the remote control can be adjusted, and the movement trajectory corresponding to another remote control cannot be adjusted, so as to achieve personalized custom adjustment while jointly drawing the trajectory.

[0241] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0242] Receive and respond to a trajectory adjustment instruction, adjust the movement trajectory, and display the adjusted movement trajectory on the painting interface, where the trajectory adjustment instruction is an instruction obtained through the collected voice information or an instruction obtained through gesture recognition.

[0243] Based on this embodiment, when receiving a trajectory adjustment instruction, if the trajectory adjustment instruction is an instruction obtained through the collected voice information or an instruction obtained through gesture recognition, the movement trajectories on the painting interface can be adjusted without distinguishing which remote control the movement trajectory corresponds to, thereby further improving the collaborative interaction performance when two or more users use the remote control to draw movement trajectories at the same time.

[0244] It can be understood that in some possible implementation manners, when the display device receives a trajectory adjustment instruction sent by the remote control, the movement trajectories on the painting interface can be adjusted without distinguishing which remote control the movement trajectory corresponds to, so as to further improve the collaborative interaction performance.

[0245] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0246] Receive and respond to a sharing instruction, and display an image sharing page, where the image sharing page includes a graphic recognition code generated based on a second image. In some specific examples, the graphic recognition code can be a QR code. Thus, the second image can be shared with a third party, such as sent to an instant messaging user, by using a mobile scanning device, such as a mobile terminal, to scan the graphic recognition code.

[0247] Among them, in some specific examples, when receiving a sharing instruction when the image generation interface is displayed, the graphic recognition code can be generated based on the second image corresponding to the second image thumbnail where the focus of the remote control is located, or the second image displayed in the first display area. When receiving a sharing instruction when the full-screen display interface of the second image is displayed, the graphic recognition code is generated based on the second image displayed in the full-screen display interface.

[0248] The specific page form of the image sharing page is not limited. The image sharing interface in some specific examples is as Figure 7 (5) shown. Among them, when the generation of the graphic recognition code fails, a prompt interface indicating the failure of the graphic recognition code generation can also be displayed, such as Figure 7 (6) shown. In other embodiments, the image sharing page and the prompt page for the failure of the image recognition code generation can also be designed otherwise.

[0249] In some examples, the content displayed on the painting interface can also be shared with a mobile terminal so that the mobile terminal can synchronously edit the content displayed on the painting interface. Accordingly, the processor is further configured to execute instructions to cause the display device to:

[0250] Receive and respond to a sharing instruction, and share the content displayed on the painting interface with the mobile terminal;

[0251] Receive and respond to the edited image transmitted by the mobile terminal, and update the content displayed on the painting interface.

[0252] There is no limit to the way of sharing the content displayed on the painting interface with the mobile terminal. For example, after establishing a Bluetooth connection between the display device and the mobile terminal, the content displayed on the painting interface can be shared with the mobile terminal, or when the display device and the mobile terminal are connected to the same local area network, such as accessing the same WiFi hotspot, the content displayed on the painting interface can be shared with the mobile terminal through the wireless local area network. It can also be through an image sharing interface that displays a graphic recognition code generated based on the content displayed on the painting interface, and the mobile device scans the graphic recognition code to share the content displayed on the painting interface with the mobile terminal.

[0253] In some examples, the processor is further configured to execute instructions to cause the display device to:

[0254] Determine at least one moving speed of the remote control according to the distances of multiple position information during the movement of the remote control, and associate and store the moving speed with the moving trajectory;

[0255] Generate an image of the second image based on the text features, image features, and moving speed for the first image.

[0256] Among them, one position information of the remote control corresponds to a set of coordinate data, the position coordinates on the display screen of the display device. Through two moving position information, the distance between the two position information can be calculated, and the sampling times between the two information and the sampling duration of each sampling can be combined, so that the moving speed of the remote control for these two position information can be determined. Among them, the sampling duration refers to the positioning duration of obtaining the position information of the remote control during the movement of the remote control, and it is also the duration between two adjacent position information obtained, which can be related to the receiving frequency of the display device receiving the signal of the remote control.

[0257] Among them, for any moving trajectory, one or more moving speeds can be calculated. When calculating one moving speed, the moving speed can be calculated by combining the position information of the starting point and the ending point of the moving trajectory. When calculating multiple moving speeds, the moving speed can be calculated by combining the starting point, the ending point, and one or two or more intermediate points of the moving trajectory.

[0258] Based on this embodiment, the first image can also be used to generate an image in combination with the moving speed of the moving trajectory, so that the generated second image can also reflect content related to the moving speed. For example, the states of some objects in the second image are related to the speed. If the speed is associated with the wind speed, when the generated objects include leaves, grass, water ripples, etc., the leaves, grass, water ripples, etc. will show their states at different wind speeds, but this is not limited thereto.

[0259] It can be understood that when sending an image generation request to the server and the server generates the second image, after the display device calculates the moving speed, it can send the moving trajectory and the associated moving speed to the server. The server combines the text features, image features, and moving speed to generate the first image to obtain the generated second image. It can also be that the display device sends the time stamp of each position information it records to the server at the same time. After the server calculates the moving speed, it combines the text features, image features, and moving speed to generate the first image to obtain the generated second image.

[0260] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0261] During the process of displaying the image generation page, receive and respond to the return to drawing board instruction, and switch to display the painting interface.

[0262] In some embodiments, after receiving the return to drawing board instruction, a return to drawing board confirmation interface can also be displayed. The return to drawing board confirmation interface in some examples is as Figure 7 (7) shown, for the user to confirm the return to the drawing board, and after receiving the confirmation instruction, switch to display the painting interface to reduce the possibility of returning to the painting interface due to misoperation. Among them, if a cancellation instruction is received during the process of the return to drawing board confirmation interface, it can return to continue displaying the image generation interface.

[0263] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0264] During the process of displaying the image generation page, receive and respond to the return to regenerate instruction, regenerate the second image according to the screen of the painting interface again, and refresh the image generation interface. Among them, when the display device generates the second image, the display device regenerates the second image according to the screen of the painting interface again. When the server generates the second image, the display device sends an image generation request to the server again to obtain the second image regenerated by the server.

[0265] In some embodiments, after receiving the regeneration instruction, a regeneration confirmation interface may also be displayed for the user to confirm the return to the drawing board, and the second image may be regenerated after receiving the confirmation instruction. When there is a limit on the number of times the display device generates images, the regeneration confirmation interface may also include information about the remaining number of image generations, such as Figure 7 (8) shows, but is not limited to this.

[0266] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0267] Receive a modification instruction for the displayed second image;

[0268] In response to the modification instruction, perform semantic segmentation on the displayed second image to obtain a plurality of second object contents;

[0269] Receive and respond to an editing operation on the target object content among the plurality of second object contents, redraw the second image based on the operation on the target object content to obtain the redrawn second image, and display the redrawn second image.

[0270] The modification instruction is an instruction used to indicate that the displayed second image needs to be modified, indicating that the user needs to perform custom adjustments on the displayed second image. Among them, the displayed second image can be, for example, Figure 6 (7) The second image displayed on the image generation interface shown, or it can be, for example, Figure 6 (8) The second image displayed on the full-screen display interface shown.

[0271] The method of performing semantic segmentation on the displayed second image is not limited. The semantic segmentation model can be used to perform semantic segmentation on the displayed second image to obtain a plurality of object contents after semantic segmentation, which are referred to as second object contents in the embodiments of the present application. The semantic segmentation model can be a SAM model, a vision-language model (Grounding-SAM), etc., but is not limited to this.

[0272] When obtaining a plurality of second object contents, a labeled layer mask may be output, and the label of the layer mask may be the corresponding second object content.

[0273] After performing semantic segmentation to obtain a plurality of second object contents, editing can be performed on the second object contents as objects, such as deleting, moving, re-filling colors, etc., but is not limited to this.

[0274] The target object content is the second object content for the editing operation. The target object content can be selected from multiple second object contents in various possible ways. For example, after moving the focus of the remote control to a certain second object content, by pressing a preset button on the remote control, the target object content can be determined. Another example is that after splitting to obtain multiple second object contents, each second object content is numbered, and by receiving a voice editing instruction containing the number, the target object content corresponding to the voice editing instruction can be determined. It can be understood that in other embodiments, the target object content can also be selected and determined in other ways.

[0275] After obtaining the editing operation on the target object content, the target object content can be processed based on the editing operation. Taking the editing operation including a deletion operation as an example, the target object content can be deleted. Taking the editing operation including a movement operation as an example, the position of the target object content in the second image can be moved. After processing the target object content, the second image can be redrawn based on the editing operation on the target object content to obtain the redrawn second image.

[0276] It can be understood that for the displayed second image, multiple editing operations can be sequentially performed on the same target object content. Each editing operation can be the same or different, or one or more editing operations can be respectively performed on different target object contents. In the related embodiments of the present application, it can be that each time an editing operation is completed, the second image is redrawn based on the editing operation to obtain the redrawn second image, or multiple editing operations are sequentially performed, and after an editing operation is completed, if no new editing operation is received within a preset time period, the second image is redrawn based on the received editing operation to obtain the redrawn second image, but it is not limited thereto.

[0277] Based on this embodiment, during the process of displaying the second image, a modification instruction for the second image can also be received, and the content displayed on the painting interface is semantically segmented based on the modification instruction to obtain multiple second object contents. Thus, the target object content can be selected from the multiple second object contents for editing, and the second image is redrawn based on the editing operation, so as to obtain the redrawn second image for display. Thus, on the basis of the second image generated by image generation, the second image can be further personalized edited, further improving the interaction performance of the display device.

[0278] The method of redrawing the second image based on the operation on the target object content to obtain the redrawn second image is not limited. Some methods are exemplified below.

[0279] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0280] Receive and respond to an editing operation on the target object content among multiple second object contents, and generate a new layer based on the operation on the target object content;

[0281] Based on the second image and the new layer, perform image redrawing to obtain the redrawn second image.

[0282] Among them, the generated new layer can be a mask layer. When generating a new layer based on the operation on the target object content, only the pixel positions where the target object content is located can be set to opaque, while other positions are set to transparent, so as to achieve the effect of not blocking the response image content of the second image.

[0283] After generating the new layer, when performing image redrawing based on the second image, the new layer, and the second image, the layer where the second image is located and the new layer can be superimposed, and the new layer can be set above the layer where the second image is located, so as to obtain the redrawn second image. Among them, after superimposing the layer where the second image is located and the new layer, color coordination processing can be further performed on the boundary area of the layer superposition to improve the color coordination of the boundary area of the layer.

[0284] Based on this embodiment, when redrawing the second image based on the editing operation, a new layer is generated based on the operation on the target object content, and image redrawing is performed in combination with the second image and the obtained new layer.

[0285] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0286] Generate corresponding image layers for multiple second object contents respectively, and record the layer stacking order of each image layer;

[0287] Receive and respond to an editing operation on the target object content among multiple second object contents, and adjust the layer stacking order of each image layer;

[0288] Based on the adjusted layer stacking order of the image layers, obtain the redrawn second image.

[0289] Among them, the layer stacking order of the image layers refers to the up-and-down arrangement relationship of different layers. The layers are superimposed in the order from bottom to top, and the upper layer content will cover the overlapping part of the lower layer. By adjusting the layer stacking order, the front and back relationship of elements can be determined. For example, the main body is placed on the upper layer and the background is on the lower layer, enhancing the sense of space of the picture. And adjusting a single layer alone will not affect other contents, which is convenient for modifying details and improves the flexibility of adjusting the second image.

[0290] Among them, each image layer can store corresponding layer parameters, including but not limited to color, transparency, depth, physical material, etc.

[0291] There is no limitation on the method for obtaining the layer stacking order of the image layer. In a related embodiment, the second image can be estimated through a monocular depth estimation model (Monocular Depth Estimation, MDE), and a three-dimensional relationship graph can be constructed in combination with the estimated depth information. The layer stacking order of the layers corresponding to each second object content can be determined in combination with the three-dimensional relationship graph.

[0292] When the layer stacking order of each image layer is adjusted in response to an editing operation, the transformation matrix of the layer order of each image layer can be determined in combination with the editing operation, and the occlusion conflict area after the editing operation is detected. The upper and lower order of each image layer can be adjusted in combination with the occlusion conflict area, and the specific adjustment method is not limited.

[0293] Among them, after the layer stacking order of the image layer, the boundary area of the layer stacking can be further subjected to color coordination processing to improve the color coordination of the boundary area of the layer.

[0294] Based on this embodiment, when the second image is redrawn based on the editing operation, corresponding image layers are generated for the multiple second object contents respectively, and the layer stacking order of each image layer is recorded. After receiving the editing operation, the image is redrawn by adjusting the layer stacking order of each image layer. Therefore, when redrawing the image, it can be adjusted through the stacking order of different layers, which can well take into account the front and back occlusion of different second object contents.

[0295] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0296] If a hole area exists in the screen after the layer stacking order of the image layer is adjusted, image completion is performed on the hole area to obtain a redrawn second image.

[0297] The hole area refers to the hole where the second object content was originally located after the layer stacking order of the image layer is adjusted and after a second object content is moved, for example, the second object content is deleted or moved, resulting in a hole, for example, the mask value will be set to 0, thus generating a hole area. Therefore, it is necessary to perform image completion on the hole area to improve the integrity of the second image.

[0298] Based on this embodiment, when redrawing an image, after adjusting the stacking order of different layers, if there are empty areas, the image can be completed in the empty areas, thereby improving the integrity of the picture after adjusting the stacking order of the image layers, and helping to further improve the interactive performance of the display device.

[0299] The manner of image completion for the hollow area is not limited. In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0300] Determine the hollow area of the screen after adjusting the layer stacking order of the image layer;

[0301] Based on the pixel information around the hollow area, perform low-resolution pixel completion on the hollow area;

[0302] Generate a texture for the hollow area after low-resolution pixel completion;

[0303] Based on the information of the hollow area after texture generation and the screen after adjusting the layer stacking order of the image layer, perform physical lighting analysis, and based on the result of the physical lighting analysis, adjust the shadow area and / or highlight area of the hollow area after texture determination generation, and perform pixel adjustment on the hollow area according to the shadow area and / or highlight area to obtain the redrawn second image.

[0304] Based on this embodiment, when performing image completion on the hollow area after adjusting the stacking order of different layers, first perform low-resolution pixel completion on the hollow area with the pixel information around the hollow area, then generate a texture for the hollow area after low-resolution pixel completion, and further combine the screen after adjusting the layer stacking order of the image layer. After performing physical lighting analysis, perform pixel adjustment on the shadow area and / or highlight area of the hollow area, so that pixel adjustment is also combined with the result of physical lighting analysis for the hollow area, which can improve the matching degree between the lighting condition of the completed hollow area and the environment in the second image.

[0305] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0306] Perform image preprocessing on the content displayed on the painting interface to generate a first image. The image preprocessing includes at least one of filling in discontinuous lines and noise filtering.

[0307] Based on this embodiment, before generating the first image, image preprocessing such as filling in discontinuous lines and noise filtering is also performed on the content displayed on the painting interface, which can effectively filter out noise and improve the smoothness and accuracy of the lines in the obtained first image.

[0308] Among them, in the above embodiments, the second image in the image generation interface displayed in response to the image generation instruction may be generated based on the content displayed on the painting interface. In some specific examples, a first image may be generated based on the content displayed on the painting interface, and the displayed content includes a movement trajectory; the content of the first image is understood to obtain the text features of the first image; the image features of the first image are recognized to obtain the image features of the first image, and image generation is performed on the first image based on the text features and the image features to obtain the generated second image.

[0309] There is no limit to the method of understanding the content of the first image to obtain the text features of the first image. In some embodiments, the content of the first image may be understood by a content understanding model to obtain the text features of the first image.

[0310] There is no limit to the method of recognizing the image features of the first image to obtain the image features of the first image. In some embodiments, the image features of the first image may be recognized by a control network model to obtain the image features of the first image.

[0311] When performing image generation on the first image based on the text features and the image features, the first image may be subjected to image generation by an extension model based on the text features and the image features to obtain the generated second image.

[0312] Accordingly, in some embodiments, the processor is further configured to execute instructions to cause the display device:

[0313] Understand the content of the first image through a content understanding model to obtain the text features of the first image, recognize the image features of the first image through a control network model to obtain the image features of the first image, and perform image generation on the first image based on the text features and the image features through an extension model to obtain the generated second image.

[0314] Accordingly, as shown in reference Figure 8 When obtaining the second image based on the first image, the content of the first image may be understood through a content understanding model to obtain the text features of the first image. After preprocessing the first image, the image features of the first image are recognized through a control network model to obtain the image features of the first image. The control features can be obtained by combining the text features, the image features, and the obtained image style information, and the extension model is controlled by the control features to generate the second image.

[0315] Among them, when the extension model generates the second image under the control of the control features, in some embodiments, as shown in reference Figure 9 It can be combined with a pure noise image for generation. A pure noise image refers to an image in which the pixel values are randomly distributed and there is no specific pattern or structure.

[0316] Based on this embodiment, different models can be used to perform content understanding and image feature recognition on the first image, and image generation can be performed on the first image based on text features and image features. Thus, processing can be carried out in combination with the trained models, which can improve the efficiency and accuracy of image generation. By deploying the corresponding models to the server for processing, the requirement for the processing capacity of the display device can also be reduced. Combining the processing capacity of the server for processing can further improve the processing efficiency.

[0317] The content understanding model is an information that can understand the content in the image to obtain the content description information in the image. The information content of the obtained content description information is not limited. For example, the objects corresponding to the line drawing, the relationship between the lines, when the lines correspond to multiple objects, the relationship between the multiple objects, etc., but not limited thereto. The type of the object corresponding to the line drawing is not limited, such as animals, plants, human figures, mountains, etc., but not limited thereto.

[0318] Among them, the content understanding model can be obtained through training. After the content understanding model is trained through the server or other devices, the trained content understanding model can be deployed to the display device or the server to perform content understanding on the first image and obtain the text features of the first image. The type of the specific model of the content understanding model is not limited. For example, the LMM (Large Multimodal Model) model. For example, the LMM model can be a model based on the LLAVA (Large Language and Vision Assistant) structure, which can use the pre-trained language model embedding to greatly reduce the cost and difficulty of multi-modal model training. The specific model structure of the LMM model is not limited. For example, in some embodiments, it can be the interval8B model.

[0319] The manner of training to obtain the content understanding model is not limited. In the embodiments of the present application, the content understanding model can be obtained by fine-tuning on the basis of the existing large language model. Among them, when performing fine-tuning, LoRA (Low-Rank Adaptation) can be used for fine-tuning. The LoRA fine-tuning method can reduce the number of training parameters through low-rank matrix decomposition, so as to achieve efficient parameter adjustment for large-scale pre-trained models.

[0320] In some embodiments, the manner of training to obtain the content understanding model can be as described below.

[0321] First, obtain a sample data set. Each piece of sample data in the sample data set includes a line graph and the label corresponding to the line graph. Among them, the number of lines in the line graph is not limited. The line graph can only have lines, or it can include lines and at the same time have corresponding filling colors in the closed or semi-closed areas of the lines. The label is the text description information of the line graph. After obtaining the sample data set, divide the sample data set into a training set, a validation set, and a test set. For example, 70% of the sample data constitutes the training set, 15% of the sample data constitutes the validation set, and the remaining 15% of the sample data constitutes the test set to evaluate the generalization ability of the model.

[0322] After dividing the sample data set, the model training process can be carried out. During the model training process, use the training set to train the content understanding model, and during or after the training process, use the validation set to evaluate the performance of the content understanding model, and use the test set to conduct a final test on the trained content understanding model to ensure the performance of the model in actual applications. It can be understood that during the process of training the content understanding model, the content understanding model conducts content understanding on the sample data to obtain corresponding description information, and compares this description information with the text description information annotated in the sample data to calculate the training loss, and adjusts the content understanding model in combination with the calculated training loss to achieve the update of the content understanding model. After the model training is completed, use the content understanding model updated for the last time as the content understanding model obtained through training.

[0323] The control network model is a model that can identify features related to the characteristics of an image in the image to obtain image features, and then can control the image generated by the expansion model based on the obtained image features. Thus, based on the features of the first image with only brief content such as lines, etc., it can control the expansion model to generate an image with rich information. The type of image recognized by the control network model from the first image is not limited. For example, it can include one or more of the image style category, image subject information, position information of the image subject, scale information of the image subject, relationship information between image subjects, color information of the image subject, edge map, pose key points, depth map, segmentation map, normal map, and the first image itself, etc., but is not limited to this.

[0324] Among them, the control network model can be obtained through training. After obtaining the control network model through training on a server or other device, the trained control network model can be deployed to a display device or a server to identify the image features of the first image. The specific type of the control network model is not limited. In the embodiments of the present application, it can be a ControlNet network. ControlNet is a neural network that controls a pre-trained image diffusion model (such as StableDiffusion). It allows the input of an adjustment image, and then uses this adjustment image to manipulate image generation. The type of the adjustment image is not limited, such as scribbles, edge maps, pose key points, depth maps, segmentation maps, normal maps, etc., but not limited thereto.

[0325] The method of training to obtain the control network model is not limited. It can be understood that when training the extended model, it can be based on an existing control network model, and the control network model can be fine-tuned through the training process to obtain the control network model of the embodiments of the present application. In the embodiments of the present application, the control network model can be obtained by fine-tuning on the basis of an existing control network model. Among them, when fine-tuning, LoRA (Low-Rank Adaptation) can be used for fine-tuning. The LoRA fine-tuning method can reduce the number of training parameters through low-rank matrix decomposition, so as to achieve efficient parameter adjustment for large-scale pre-trained models.

[0326] The extended model is a model that can further expand and generate richer images based on the input control features on the basis of the input image. The control features can include the text features and image features as described above. The type of the extended model is not limited. In the embodiments of the present application, the extended model can be a model based on the SDXL model.

[0327] The extended model can be obtained through training. It can be understood that when training the extended model, it can be based on an existing extended model, and the extended model can be fine-tuned through the training process to obtain the extended model of the embodiments of the present application. In the embodiments of the present application, the extended model can be obtained by fine-tuning on the basis of an existing extended model. Among them, when fine-tuning, LoRA (Low-Rank Adaptation) can be used for fine-tuning. The LoRA fine-tuning method can reduce the number of training parameters through low-rank matrix decomposition, so as to achieve efficient parameter adjustment for large-scale pre-trained models.

[0328] Among them, in some embodiments of the present application, the model training processes for the content understanding model, the control network model, and the extension model can be carried out separately, or an image processing model including two or more of the content understanding model, the control network model, and the extension model can be constructed, and by training the image processing model, the content understanding model, the control network model, and the extension model can be trained simultaneously, so as to improve the stability of the comprehensive result of the obtained second image.

[0329] Taking the construction of an image processing model including the control network model and the extension model as an example, an image processing model including the control network model and the extension model can be obtained by training the image processing model including the control network model and the extension model. During the training process, a sample data set can be constructed, and each piece of sample data in the sample data set includes a line drawing and the corresponding effect drawing of the line drawing. Among them, the number of lines in the line drawing is not limited. The line drawing can only have lines, or it can include lines and at the same time have corresponding filling colors in the enclosed or semi-enclosed areas of the lines. The effect drawing is the label corresponding to the line drawing. After obtaining the sample data set, the sample data set is divided into a training set, a validation set, and a test set.

[0330] After dividing the sample data set, the model training process can be carried out. During the model training process, the image processing model is trained using the training set, and during or after the training process, the validation set is used to evaluate the performance of the image processing model, and the test set is used to finally test the trained image processing model to ensure the performance of the model in actual applications. It can be understood that during the training process of the image processing model, the control network model in the image processing model identifies the image features of the sample data, and the extension model in the image processing model generates a colorful image based on the control of the graphic features, and compares the image with the effect drawing corresponding to the sample data to calculate the training loss, and adjusts the control network model and / or the extension model in the image processing model in combination with the calculated training loss to realize the update of the image processing model. After the model training is completed, the last updated image processing model is used as the trained image processing model.

[0331] It can be understood that in the case of constructing an image processing model that simultaneously includes a content understanding model, a control network model, and an extension model, the specific training process can be similar to the above method and will not be elaborated here.

[0332] In some embodiments, the processor is further configured to execute instructions to cause the display device to:

[0333] Obtain the first reliability of the text feature and the second reliability of the image feature;

[0334] Based on text features, image features, a first reliability, and a second reliability, the extended model generates an image for a first image to obtain a generated second image.

[0335] Among them, the first reliability refers to the reliability of the text features obtained by the content understanding model for content understanding of the first image, and the first reliability can be reflected by the maximum probability of the output result of the content understanding model for processing the first image. The second reliability refers to the reliability of the image features obtained by the control network model for image feature recognition of the first image, and the second reliability can be reflected by the maximum probability of the output result of the control network model for image feature recognition of the first image. By inputting the first reliability and the second reliability into the extended model at the same time, the extended model can generate the second image by combining the reliability of the text features and the reliability of the image features at the same time, which can further improve the correlation between the generated second image and the first image, and can further improve the interaction performance of the display device.

[0336] Based on the above-described embodiments, the following will be described by way of examples of specific interaction processes. It can be understood that in the description of the following examples, only some processing modules and some processing processes are shown. In the actual technical processing process, other processing modules and processing processes may also be involved.

[0337] Taking the display device generating the second image as an example, refer to Figure 10 As shown, after the display device is powered on, the user controls the display device through a remote controller or voice, opens the application interface of the target application of the display device, and opens the painting interface based on the received painting instruction.

[0338] After opening the painting interface, the user opens the pointing mode of the remote controller by pressing a preset button on the remote controller or the like and moves the remote controller. The UWB transmitter of the remote controller emits a UWB wireless signal.

[0339] The UWB receiver of the display device receives the UWB wireless signal, converts it into a digital signal, and provides it to the trajectory generation module. The trajectory generation module determines the position information of the remote controller based on the received digital signal, combines the position information of multiple times, determines the movement trajectory of the remote controller, and displays the movement trajectory on the painting interface through the display module.

[0340] After the drawing is completed or during the drawing process, the image processing module generates a second image based on the image generation instruction sent by the remote control or voice, etc., received by the display device, and based on the content such as the movement track displayed on the drawing interface, and after performing image magnification processing on the generated second image, it is displayed through the display module. It can be understood that when displaying through the display module, it can be displayed based on the image display interface as described above, or it can be full-screen display, but not limited to this. The image processing module and the track generation module can be different modules or integrated in the same module.

[0341] Taking the generation of the second image by the server as an example, refer to Figure 11 As shown, after the display device is powered on, the user controls the display device through the remote control or voice, opens the application interface of the target application of the display device, and based on the received drawing instruction, opens the drawing interface.

[0342] After opening the drawing interface, the user opens the pointing mode of the remote control by pressing a preset button on the remote control or other means and moves the remote control. The UWB transmitter of the remote control emits UWB wireless signals.

[0343] The UWB receiver of the display device receives the UWB wireless signal, converts it into a digital signal, and provides it to the track generation module. The track generation module determines the position information of the remote control based on the received digital signal, combines the position information of multiple times, determines the movement track of the remote control, and displays the movement track on the drawing interface through the display module.

[0344] After the drawing is completed or during the drawing process, the display device receives the image generation instruction sent by the remote control or voice, etc., sends an image generation request to the server through the display device communication module, and the image generation request includes the first image, where the first image includes the content such as the movement track displayed on the drawing interface. After receiving the image generation request, the server generates the second image based on the image generation request and returns the second image to the display device. The display device communication module receives the second image, and the image processing module performs image magnification processing on the second image and then displays it through the display module. It can be understood that when displaying through the display module, it can be displayed based on the image display interface as described above, or it can be full-screen display, but not limited to this. The image processing module and the track generation module can be different modules or integrated in the same module.

[0345] Among them, in the display devices of the above-described embodiments, the manner of receiving instructions is not limited. The instructions here include, but are not limited to, the painting instructions, image generation instructions, image style determination instructions, color selection instructions, color filling instructions, coloring instructions, clearing instructions, storage instructions, confirmation instructions, cancellation instructions, deletion instructions, viewing instructions, backtracking operation instructions, full-screen display instructions, image return instructions, home page saving instructions, trajectory adjustment instructions, sharing instructions, return to drawing board instructions, regeneration instructions, and modification instructions as described above.

[0346] For example, controls corresponding to the instructions may be displayed on a relevant interface. After moving the focus of the remote control to the corresponding control, a preset button such as the OK button is pressed to issue a painting instruction. For another example, corresponding instructions can be issued by pressing preset buttons on the remote control. Among them, different instructions correspond to different single preset buttons or groups of preset buttons, or even if they correspond to the same or the same group of preset buttons, but are triggered under different display interfaces, so that different control instructions can be received.

[0347] For another example, the display device can obtain different control instructions included in the voice information by collecting voice information. The painting instruction can be obtained through the collected voice information or received gesture information. For another example, the display device can collect an image frame through an image sensor and determine whether a corresponding control instruction is received by identifying whether there is gesture information corresponding to the corresponding control instruction in the image frame. It can be understood that in other embodiments, control instructions can also be received through other means as long as different control instructions can be distinguished.

[0348] Reference Figure 12 As shown, some embodiments of the present application provide a control method for a display device. The method is executed by the display device and includes the following steps:

[0349] Step S121: Display a painting interface;

[0350] Step S122: Receive the wireless signal of the remote control, determine the movement trajectory of the remote control based on the wireless signal, and display the movement trajectory on the painting interface;

[0351] Step S123: Receive and respond to an image generation instruction, generate and display an image generation interface. The image generation interface includes a second image generated based on the screen of the painting interface and a first image thumbnail. While the second image is displayed on the image generation interface, the first image thumbnail does not completely block the second image.

[0352] In some embodiments, the method further includes:

[0353] Obtain target image style information;

[0354] The image generation interface is displayed, and the second image displayed on the image generation interface also matches the target image style information.

[0355] In some embodiments, the method further includes: displaying a color interface in a third preset area of the painting interface, the color interface including a plurality of color selection items, and the colors corresponding to different color selection items being different.

[0356] In some embodiments, the method further includes: receiving and responding to a color filling instruction, and filling the positioning area where the target moving position of the remote control is located with the currently selected color, the positioning area being determined according to the target moving position and the moving trajectory.

[0357] In some embodiments, the method further includes: performing semantic segmentation on the screen displayed on the painting interface to obtain a plurality of first object contents; determining recommended color information based on the plurality of first object contents and the screen, and displaying the recommended color information in the third preset area.

[0358] In some embodiments, the method further includes: receiving and responding to an operation instruction for two or more color selection items, determining a new color according to the colors corresponding to the two or more color selection items, and displaying the new color.

[0359] In some embodiments, the method further includes: receiving and responding to a color filling instruction, and displaying the cursor displayed on the painting interface in a second state, the second state being associated with image filling.

[0360] In some embodiments, the method further includes: receiving and responding to a coloring instruction for the painting interface, performing semantic segmentation on the screen of the content displayed on the painting interface to obtain a plurality of first object contents; determining color information corresponding to each first object content based on the plurality of first object contents and the screen, and performing coloring processing on the corresponding first object content with the color information corresponding to each first object content to obtain a colored screen.

[0361] In some embodiments, the image generation interface includes a first display area and a second display area. The first display area displays thumbnails of a plurality of second images generated based on the screen of the painting interface, and the second display area displays the second image corresponding to the thumbnail where the focus of the remote control is located.

[0362] In some embodiments, the method further includes: displaying a first image thumbnail at a preset position of the image generation interface, the first image thumbnail including a thumbnail of the screen of the painting interface.

[0363] In some embodiments, the method further includes: receiving a modification instruction for a second displayed image; in response to the modification instruction, performing semantic segmentation on the second displayed image to obtain a plurality of second object contents; receiving and responding to an editing operation on a target object content among the plurality of second object contents, redrawing the second image based on the editing operation on the target object content to obtain a redrawn second image, and displaying the redrawn second image.

[0364] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential either, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0365] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the control method of the display device in any of the above-described embodiments are implemented.

[0366] In some embodiments, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the control method of the display device in any of the above-described embodiments are implemented.

[0367] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0368] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0369] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A display device, characterized in that, The display device includes: A display; And at least one processor configured to execute instructions to cause the display device to: Display a painting interface; Receive a wireless signal of a remote controller, determine a movement trajectory of the remote controller based on the wireless signal, and display the movement trajectory on the painting interface; Receive and respond to an image generation instruction, generate and display an image generation interface, the image generation interface includes a second image generated based on the screen of the painting interface and a first image thumbnail, the first image thumbnail includes a thumbnail of the screen of the painting interface, while the second image is being displayed on the image generation interface, the first image thumbnail does not completely obscure the second image.

2. The display device according to claim 1, wherein The processor is further configured to execute instructions to cause the display device to: Obtain target image style information; Display the image generation interface, and the second image displayed on the image generation interface also matches the target image style information.

3. The display device according to claim 1, wherein The processor is further configured to execute instructions to cause the display device to: Perform semantic segmentation on the screen of the painting interface to obtain a plurality of first object contents; Determine recommended color information based on the plurality of first object contents and the screen, and display the recommended color information in a third preset area.

4. The display device according to claim 1, characterized in that, The processor is further configured to execute instructions to cause the display device to: Receive and respond to an operation instruction to generate a new color, determine a new color according to the colors corresponding to two or more selected color options, and display the new color.

5. The display device according to claim 1, wherein The processor is further configured to execute instructions to cause the display device to: Receive and respond to a color filling instruction, and display the cursor displayed on the painting interface in a second state, and the second state is associated with image filling.

6. The display device according to any one of claims 1 to 5, characterized in that, The processor is further configured to execute instructions to cause the display device to: Receive and respond to a coloring instruction for the painting interface, perform semantic segmentation on the screen of the content displayed on the painting interface to obtain a plurality of first object contents; Determine color information corresponding to each of the first object contents based on the plurality of first object contents and the screen, and perform a coloring process on the corresponding first object content using the color information corresponding to each of the first object contents to obtain a colored screen.

7. The display device according to any one of claims 1 to 6, characterized in that The processor is further configured to execute instructions to cause the display device to: The image generation interface includes a first display area and a second display area, the first display area displays thumbnails of a plurality of second images generated based on the screen of the painting interface, and the second display area displays the second image corresponding to the thumbnail where the focus of the remote controller is located.

8. The display device according to claim 7, wherein The processor is further configured to execute instructions to cause the display device to: Display the first image thumbnail at at least one of a preset position in the second display area, a preset position in the first display area, and a preset position of the thumbnail of the second image.

9. The display device according to claim 7, wherein The processor is further configured to execute instructions to cause the display device to: Receive a modification instruction for the displayed second image; In response to the modification instruction, perform semantic segmentation on the displayed second image to obtain a plurality of second object contents; Receive and respond to an editing operation on a target object content among the plurality of second object contents, redraw the second image based on the editing operation on the target object content to obtain a redrawn second image, and display the redrawn second image.

10. A control method for a display device, characterized in that, The method includes: Display a painting interface; Receive a wireless signal of a remote controller, determine a movement trajectory of the remote controller based on the wireless signal, and display the movement trajectory on the painting interface; Receive and respond to an image generation instruction, generate and display an image generation interface, where the image generation interface includes a second image generated based on the screen of the painting interface and a first image thumbnail, the first image thumbnail includes a thumbnail of the screen of the painting interface, and while the second image is displayed on the image generation interface, the first image thumbnail does not completely obscure the second image.

Images