Display device and display method

Abstract

The invention discloses a display device and a display method.The method comprises the steps that in response to an instruction for opening a picture book application, a function page of the picture book application is displayed on a displayer, and a prompt for generating a picture book through voice interaction is displayed on the function page; in response to the voice interaction instruction, displaying a picture book generation waiting page on a display, the picture book generation waiting page displaying the progress of picture book generation; after the picture book is generated, a picture book playing page is displayed on a displayer, the picture book playing page comprises an image display area and a text display area, and the generated picture book comprises at least one section of story content in sequence and an image corresponding to each section of story content; and sequentially displaying at least one section of story content in the text display area according to the sequence, performing voice broadcast on the target story content being displayed in the text display area, and displaying an image corresponding to the target story content in the image display area. By adopting the method, the picture book meeting user requirements can be automatically generated according to the voice interaction instruction, and the picture book is displayed and played.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display method. Background Art

[0002] With the rapid development of display devices, the functions that display devices can provide to users are becoming more and more abundant. At present, display devices include smart TVs, smart set-top boxes, smart boxes, and products with smart display screens. Taking smart TVs as an example, smart TVs enable more and more scenarios. They are not only used as devices for watching TV programs at home, but also for playing games, playing electronic photo albums, and displaying information.

[0003] The user can view preset images or videos through the display device. For example, the user can view preset picture books through the display device, but cannot view non-preset picture books. Therefore, traditional display devices have the problem of being unable to display picture books that meet the user's personalized needs. Summary of the invention

[0004] The present application provides a display device and a display method, which can automatically generate a picture book that meets user needs according to voice interaction instructions, and display and play it.

[0005] In a first aspect, a display device is provided, comprising:

[0006] a display configured to display content from a broadcast system or network and / or a user interface;

[0007] and at least one processor connected to the display and configured to execute instructions to cause the display device to:

[0008] In response to an instruction to open the picture book application, a function page of the picture book application is displayed on the display, and the function page displays a prompt for generating a picture book through voice interaction;

[0009] In response to the voice interaction instruction, a picture book generation waiting page is displayed on the display, and the picture book generation waiting page displays the progress of the picture book generation;

[0010] After the picture book is generated, a picture book playback page is displayed on the display, the picture book playback page includes an image display area and a text display area, and the generated picture book includes at least one story content with a sequence and an image corresponding to each story content;

[0011] At least one story content is displayed in the text display area in sequence, the target story content being displayed in the text display area is voiced, and an image corresponding to the target story content is displayed in the image display area.

[0012] In the above embodiment, a display device is provided, in which a picture book application is preinstalled. In response to an instruction to open the picture book application, a function page of the picture book application is displayed, in which a prompt for generating a picture book through voice interaction is displayed, so as to instruct the user to input a voice interaction instruction for generating a picture book; in response to the voice interaction instruction, a picture book generation waiting page is displayed, and the progress of the picture book generation is displayed on the generation waiting page, so that the user can timely understand the progress of the picture book generation; after the picture book is generated, a picture book playback page is displayed, so that at least one section of story content is displayed in sequence in the text display area of ​​the picture book playback page, the target story content being displayed is voice broadcasted, and at the same time, the image corresponding to the target story content is displayed in the image display area of ​​the picture book playback page. In this way, a picture book can be automatically generated using the input voice interaction instruction, and at least one section of the story content in the picture book and the image corresponding to each section of the story content can be displayed and played in sequence on the display, and the display device can display a picture book that meets the personalized needs of the user, thereby improving the user experience.

[0013] In one embodiment, the function page further displays at least one picture book generated historically, and the processor is further configured to:

[0014] In response to a triggering operation on a first target picture book in at least one picture book, displaying a picture book playing page on the display, and playing the first target picture book on the picture book playing page;

[0015] In response to a deletion operation on a second target picture book in the at least one picture book, the second target picture book is deleted from the at least one picture book.

[0016] In the above embodiment, the processor can display at least one picture book generated historically in the function page, and the user can trigger any picture book in the at least one picture book to play the picture book on the picture book playback page. In this way, the user can repeatedly watch the picture books generated historically, avoiding the consumption of resources and time caused by the repeated generation of picture books with the same requirements, which is conducive to improving the user's usage experience; in addition, the user can also delete any picture book in at least one picture book to delete the picture book that the user is not satisfied with, which is conducive to improving the user's usage experience.

[0017] In one embodiment, the processor is further configured to:

[0018] After the target story text is generated based on the voice interaction instruction, the target story text is displayed on the generation waiting page of the picture book.

[0019] In the above embodiment, after the processor generates the target story text based on the voice interaction instruction, it displays the target story text on the picture book generation waiting page, so that the user can understand the story content of the generated picture book in time before the image generation of the picture book is completed, thereby improving the user experience.

[0020] In one embodiment, the processor is further configured to:

[0021] Display a static avatar on the picture book generation waiting page;

[0022] The process of voice broadcasting the target story content being displayed in the text display area, and controlling the virtual image to make corresponding actions according to the voice broadcast.

[0023] In the above embodiment, the processor can display a static virtual image in the picture book generation waiting page and a dynamic virtual image in the picture book playback page. The virtual image makes corresponding actions according to the voice broadcast, which is conducive to improving the richness and vividness of the displayed content and enhancing the user experience.

[0024] In one embodiment, the processor is further configured to:

[0025] Receive interactive voice input by the user, and generate voice interactive instructions based on the interactive voice;

[0026] In response to the voice interaction instruction, convert the interactive speech into a target text;

[0027] Based on the target text and the preset first prompt word, a first input text is assembled, and the first prompt word is used to guide the language macro model to generate character information and story theme that meet the first target requirement; the first input text is input into the language macro model to obtain the target character information and the target story theme;

[0028] Based on the target role information, the target story theme and the preset second prompt word, assembling to obtain a second input text, the second prompt word is used to guide the language macro model to generate a story title, a story introduction and a story content that meet the second target requirements; inputting the second input text into the language macro model to obtain a target story text, the target story text including a target story title, a target story introduction and a target story content;

[0029] Based on the target role information, the target story theme, the target story text and the preset third prompt word, assembling to obtain a third input text, the third prompt word is used to guide the language macro model to generate a screen description text that meets the third target requirement; inputting the third input text into the language macro model to obtain a target description text;

[0030] The target picture book style is determined based on the keywords in the target text, the style prompt words corresponding to the target picture book style are obtained in the preset style library, the target description text and the style prompt words are spliced ​​to obtain the spliced ​​text; the spliced ​​text is input into the text-generated picture model to generate a picture book.

[0031] In the above embodiment, the target text in the interactive voice input by the user is assembled with the first prompt word into the first input text, thereby guiding the language model to generate the target character information and the target story theme, and then the target character information, the target story theme and the second prompt word are assembled into the second input text, thereby guiding the language model to generate the target story text, and then, the target character information, the target story text and the third prompt word are assembled into the third input text, thereby guiding the language model to generate the target description text, and the target description text is a plurality of continuous story contents, and then based on the target description text, the text graph capability of the text graph model is used to generate the images corresponding to the plurality of continuous story contents, and obtain the picture book, thereby realizing the automatic generation of the picture book that meets the user's needs by using the input interactive voice, and the user can indicate the picture book style through the interactive voice, thereby realizing the automatic generation of the picture book that meets the user's style requirements, simplifying the user operation and improving the user experience.

[0032] In one embodiment, the processor is further configured to:

[0033] Determine the picture book style to be adjusted, where the picture book style to be adjusted is a picture book style that has been manually evaluated and has an unsatisfactory generated effect;

[0034] Obtain multiple sample images of the picture book style to be adjusted, and obtain the content description text corresponding to each sample image. Each sample image and the corresponding content description text constitute a training sample, and multiple training samples constitute a training set.

[0035] For each sample image in the training set, the content description text corresponding to the current sample image is input into the Wensheng graph network in the Wensheng graph model; a target sample image is randomly selected from other sample images in the training set except the current sample image; the target sample image is input into the adapter layer in the Wensheng graph model, and the adapter layer extracts features from the target sample image to obtain a feature representation of the target sample image; the feature representation is output to the Wensheng graph network;

[0036] The text-based graph network processes the received content description text and feature representation and outputs a predicted image;

[0037] The model loss is calculated based on the current sample image and the predicted image, the parameters of the adapter layer are adjusted based on the model loss, and training continues until the model loss meets the preset conditions.

[0038] In the above embodiment, for the picture book style to be adjusted that has unsatisfactory generation effect in manual evaluation, the method of fine-tuning parameters of the adapter layer of the Wensheng picture model can be used to improve the generation effect of the Wensheng picture model on the picture book style to be adjusted. At the same time, the parameter fine-tuning method can significantly reduce the computational complexity and video memory usage during model training, and has the advantages of retaining knowledge, reducing resource overhead, and improving generalization ability and flexibility.

[0039] In a second aspect, a display method is provided, comprising:

[0040] In response to an instruction to open the picture book application, a function page of the picture book application is displayed on the display, and the function page displays a prompt for generating a picture book through voice interaction;

[0041] In response to the voice interaction instruction, a picture book generation waiting page is displayed on the display, and the picture book generation waiting page displays the progress of the picture book generation;

[0042] After the picture book is generated, a picture book playback page is displayed on the display, the picture book playback page includes an image display area and a text display area, and the generated picture book includes at least one story content with a sequence and an image corresponding to each story content;

[0043] At least one story content is displayed in the text display area in sequence, the target story content being displayed in the text display area is voiced, and an image corresponding to the target story content is displayed in the image display area.

[0044] In the above embodiment, a display method is provided, wherein a picture book application is preinstalled in a display device, and in response to an instruction to open the picture book application, a function page of the picture book application is displayed, wherein a prompt for generating a picture book through voice interaction is displayed in the function page, so as to instruct the user to input a voice interaction instruction for generating a picture book; in response to the voice interaction instruction, a picture book generation waiting page is displayed, and the progress of the picture book generation is displayed on the generation waiting page, so that the user can promptly understand the progress of the picture book generation; after the picture book is generated, a picture book playback page is displayed, so that at least one section of story content is displayed in sequence in the text display area of ​​the picture book playback page, the target story content being displayed is voice broadcasted, and at the same time, an image corresponding to the target story content is displayed in the image display area of ​​the picture book playback page, so that a picture book can be automatically generated using the input voice interaction instruction, and at least one section of the story content in the picture book and the image corresponding to each section of the story content can be displayed and played in sequence on the display, and the display device can display a picture book that meets the personalized needs of the user, thereby improving the user experience.

[0045] In one embodiment, the function page further displays at least one picture book generated historically, and the display method further includes:

[0046] In response to a triggering operation on a first target picture book in at least one picture book, displaying a picture book playing page on the display, and playing the first target picture book on the picture book playing page;

[0047] In response to a deletion operation on a second target picture book in the at least one picture book, the second target picture book is deleted from the at least one picture book.

[0048] In the above embodiment, the processor can display at least one picture book generated historically in the function page, and the user can trigger any picture book in the at least one picture book to play the picture book on the picture book playback page. In this way, the user can repeatedly watch the picture books generated historically, avoiding the consumption of resources and time caused by the repeated generation of picture books with the same requirements, which is conducive to improving the user's usage experience; in addition, the user can also delete any picture book in at least one picture book to delete the picture book that the user is not satisfied with, which is conducive to improving the user's usage experience.

[0049] In one embodiment, the display method further includes:

[0050] After the target story text is generated based on the voice interaction instruction, the target story text is displayed on the generation waiting page of the picture book.

[0051] In the above embodiment, after the processor generates the target story text based on the voice interaction instruction, it displays the target story text on the picture book generation waiting page, so that the user can understand the story content of the generated picture book in time before the image generation of the picture book is completed, thereby improving the user experience.

[0052] In one embodiment, the display method further includes:

[0053] Display a static avatar on the picture book generation waiting page;

[0054] The process of voice broadcasting the target story content being displayed in the text display area, and controlling the virtual image to make corresponding actions according to the voice broadcast.

[0055] In the above embodiment, the processor can display a static virtual image in the picture book generation waiting page and a dynamic virtual image in the picture book playback page. The virtual image makes corresponding actions according to the voice broadcast, which is conducive to improving the richness and vividness of the displayed content and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 An operation scenario between a display device and a control device according to some embodiments is shown;

[0058] Figure 2 shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0059] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0060] Figure 4 shows a software configuration diagram in a display device 200 according to some embodiments;

[0061] Figure 5 A flowchart of a display method according to some embodiments is exemplarily shown;

[0062] Figure 6 A schematic diagram exemplarily illustrates a picture book application according to some embodiments;

[0063] Figure 7 A schematic diagram exemplarily shows a function page according to some embodiments;

[0064] Figure 8 A schematic diagram exemplarily shows a picture book generation waiting page according to some embodiments;

[0065] Fig. 9 A schematic diagram of a picture book playback page according to some embodiments is exemplarily shown;

[0066] Fig.10 A schematic diagram exemplarily shows another picture book playback page according to some embodiments;

[0067] Fig.11 A schematic diagram showing the composition of a Wensheng graph model according to some embodiments is exemplified;

[0068] Fig.12 A timing diagram exemplarily illustrates a display method according to some embodiments;

[0069] Fig.13 The figure exemplarily shows an internal structure diagram of a computer device according to some embodiments. DETAILED DESCRIPTION

[0070] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0071] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0072] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0073] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0074] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is a specific implementation of the display device of the present application.

[0075] Figure 1 FIG. 1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0076] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, etc.

[0077] In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0078] In some embodiments, the display device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0079] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0080] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0081] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0082] like Figure 3 The display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0083] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0084] The display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from the controller, and display video content, image content, and menu control interface components and user control UI interface.

[0085] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0086] The communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0087] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0088] The detector 230 is used to collect signals from the external environment or the external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0089] The external device interface 240 may include, but is not limited to, any one or more of the following interfaces: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0090] The tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0091] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0092] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0093] In some embodiments, the controller includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0094] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

[0095] "User interface" is the medium interface for interaction and information exchange between applications or operating systems and users. It realizes the conversion between the internal form of information and the form acceptable to users. The commonly used form of user interface is the Graphical User Interface (GUI), which refers to the user interface related to computer operation displayed in a graphical way. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0096] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0097] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0098] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0099] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0100] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0101] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0102] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0103] Traditional picture books are books that are mainly paintings with a small amount of text. With the rapid development of display devices, the functions that display devices can provide to users are becoming more and more abundant. Users can watch preset images or videos through display devices. For example, they can watch the contents of picture books through display devices. Picture books can be pre-set in the display device, and each picture book contains multiple images or videos. The display device can display multiple images in the picture book or display the videos in the picture book in sequence, so that users can watch the pre-set picture books through the display device. However, if the user needs to watch a picture book with a specific theme, but the display device does not pre-set the picture book with the theme, the user cannot watch it. Therefore, traditional display devices have the problem of being unable to display picture books that meet the personalized needs of users.

[0104] In order to solve the problem that traditional display devices cannot display picture books that meet the personalized needs of users, an embodiment of the present application provides a display method. In this method, a picture book application is pre-installed in the display device. In response to an instruction to open the picture book application, a function page of the picture book application is displayed, and a prompt for generating a picture book through voice interaction is displayed in the function page to instruct the user to input a voice interaction instruction for generating a picture book; in response to the voice interaction instruction, a picture book generation waiting page is displayed, and the progress of picture book generation is displayed on the generation waiting page, so that the user can timely understand the progress of picture book generation; after the picture book is generated, a picture book playback page is displayed, so that at least one story content is displayed in sequence in the text display area of ​​the picture book playback page, the target story content being displayed is voice broadcasted, and the image corresponding to the target story content is displayed in the image display area of ​​the picture book playback page. In this way, the picture book can be automatically generated by using the input voice interaction instruction, and at least one story content in the picture book and the image corresponding to each story content can be displayed and played in sequence on the display, and the display device can display the picture book that meets the personalized needs of the user, thereby improving the user experience.

[0105] like Figure 5 As shown, Figure 5 A flowchart of a display method according to some embodiments is exemplarily shown. The method comprises:

[0106] S100. In response to an instruction to open a picture book application, a function page of the picture book application is displayed on a display, and the function page displays a prompt for generating a picture book through voice interaction.

[0107] Among them, the display device is pre-installed with multiple applications, and the picture book application is an application pre-installed in the display device for generating and displaying picture books. Figure 6 The schematic diagram of the picture book application according to some embodiments is exemplarily shown. Among them, the application named AI Speed ​​Picture Book in the figure is the icon of the picture book application. After the user triggers the icon of the picture book application, the processor of the display device generates an instruction to open the picture book application in response to the triggering operation of the icon of the picture book application.

[0108] The function page refers to a page used to display the functions of the picture book application. The function page displays a prompt for generating a picture book through voice interaction, which is used to prompt the user to automatically generate a picture book by inputting a voice interaction instruction. Figure 7 A schematic diagram of a function page according to some embodiments is exemplified. The prompt in the function page may be: only one sentence is needed to draw a colorful children's picture book for you, for example, "Tell me a story about the adventures of a kitten".

[0109] In some embodiments, after displaying the function page of the picture book application, the voice recording function is turned on to receive the voice interaction instruction input by the user. For example, the voice interaction instruction may be: the child does not want to take medicine, please tell him a story to teach him to take medicine. In some embodiments, the text corresponding to the voice interaction instruction may be displayed in the function page.

[0110] In some embodiments, the function page also displays at least one voice interaction example text. The voice interaction example text is used to provide users with examples of voice interaction. Figure 7 , a sample text for voice interaction could be: Try saying: Tell me a story about a kitten’s adventures.

[0111] In some embodiments, the function page also includes a picture book generation start control, refer to Figure 7 The name of the button is AI picture book generation. In response to the triggering operation of the picture book generation start button, the voice recording function is started to receive the voice interaction command input by the user.

[0112] S200. In response to the voice interaction instruction, a picture book generation waiting page is displayed on the display, where the picture book generation waiting page displays the progress of picture book generation.

[0113] Among them, the voice interaction instruction refers to the instruction obtained after processing the interactive voice input by the user. The picture book generation waiting page refers to the page displayed on the display during the process of the processor generating the picture book. In some embodiments, the picture book generation waiting page displays a picture book generation waiting prompt to prompt the user to wait for the picture book generation to be completed. For example, the picture book generation waiting prompt can be: the picture book is being drawn quickly.

[0114] The picture book generation is performed by the picture book generation module of the processor, and the picture book generation module sends the progress information of the picture book generation to the processor. The processor displays the progress of the picture book generation on the picture book generation waiting page based on the progress information. For example, the progress information can be the percentage of the duration of the current picture book generation to the preset duration. Figure 8 The schematic diagram of the picture book generation waiting page according to some embodiments is exemplarily shown. The picture book generation waiting page can display the progress of the picture book generation in the form of a progress bar.

[0115] In some embodiments, the processor has a timeout exception handling function. If the waiting time for picture book generation exceeds a preset time, that is, the picture book generation is not completed after exceeding the preset time, a failure prompt is returned to indicate that the current picture book generation has failed.

[0116] S300: After the picture book is generated, a picture book playback page is displayed on the display, the picture book playback page includes an image display area and a text display area, and the generated picture book includes at least one story content with a sequence and an image corresponding to each story content.

[0117] The processor determines to generate a picture book when the received progress information indicates that the picture book generation is complete. The generated picture book often contains at least one story content with a sequence and an image corresponding to each story content. The sequence of at least one story content refers to the contextual logical sequence of at least one story content. At least one story content is combined in sequence to form a complete picture book story. The image corresponding to each story content shows the main information of the story content.

[0118] The picture book playback page refers to the page for playing the generated picture book. Since the generated picture book includes story content and images, the picture book playback page can include an image display area and a text display area, so as to display the story content and images in different areas. Fig. 9 The schematic diagram of a picture book playback page according to some embodiments is exemplarily shown, wherein the picture book playback page includes an image display area and a text display area, the text display area displays the first story content of the picture book, and the image display area displays the image corresponding to the first story content.

[0119] S400, displaying at least one story content in sequence in the text display area, performing voice broadcasting on the target story content being displayed in the text display area, and displaying an image corresponding to the target story content in the image display area.

[0120] Among them, reference Fig. 9 , the text display area of ​​the picture book playback page is used to display at least one section of the story content, and one section of the story content is displayed each time. The target story content refers to the story content currently being displayed in the text display area. When the target story content is displayed in the text display area, the target story content can be voice broadcast. Exemplarily, a preset timbre can be used to voice broadcast the target story content through a voice broadcast module. The user's timbre can also be determined by performing timbre recognition on the voice interaction instructions, and the user's timbre is used to voice broadcast the target story content through a voice broadcast module. The timbre used for voice broadcasting can be determined according to the user's choice.

[0121] In addition, when the target story content is displayed in the text display area, the image corresponding to the target story content can also be displayed in the image display area. In this way, the user can watch the picture book through the story content, images displayed in the display device, and the story content played by voice, which improves the user experience.

[0122] For example, in the order of at least one section of story content, the i-th section of story content is displayed in the text display area, the i-th section of story content is voice broadcast, and the image corresponding to the i-th section of story content is displayed in the image display area. After the voice broadcast of the i-th section of story content is completed, the i+1-th section of story content, image display and voice broadcast are continued until the last section of story content is completed, where i is a positive integer.

[0123] like Fig.10 A schematic diagram of another picture book playback page according to some embodiments is exemplified. After the last paragraph (the 9th paragraph) of the story content is played, the picture book playback completion text is displayed in the text display area, the voice playback is stopped, and the picture book playback completion screen is displayed in the image display area.

[0124] In the above embodiment, a display method is provided, wherein a picture book application is preinstalled in a display device, and in response to an instruction to open the picture book application, a function page of the picture book application is displayed, wherein a prompt for generating a picture book through voice interaction is displayed in the function page, so as to instruct the user to input a voice interaction instruction for generating a picture book; in response to the voice interaction instruction, a picture book generation waiting page is displayed, and the progress of the picture book generation is displayed on the generation waiting page, so that the user can promptly understand the progress of the picture book generation; after the picture book is generated, a picture book playback page is displayed, so that at least one section of story content is displayed in sequence in the text display area of ​​the picture book playback page, the target story content being displayed is voice broadcasted, and at the same time, an image corresponding to the target story content is displayed in the image display area of ​​the picture book playback page, so that a picture book can be automatically generated using the input voice interaction instruction, and at least one section of the story content in the picture book and the image corresponding to each section of the story content can be displayed and played in sequence on the display, and the display device can display a picture book that meets the personalized needs of the user, thereby improving the user experience.

[0125] In some exemplary embodiments, the function page also displays at least one picture book generated historically, and the display method also includes: in response to a trigger operation on a first target picture book in at least one picture book, displaying a picture book playback page on the display, and playing the first target picture book on the picture book playback page; in response to a deletion operation on a second target picture book in at least one picture book, deleting the second target picture book from at least one picture book.

[0126] Each time a picture book is generated, the processor may save the generated picture book to a picture book library, and the picture book library stores the picture books generated in history. The function page may also display at least one picture book generated in history, refer to Figure 7 , at least one picture book generated historically is displayed in the function page in the form of a triggerable control.

[0127] The first target picture book refers to a picture book triggered by a user from at least one of the displayed picture books. In response to the triggering operation on the first target picture book, the processor displays a picture book playing page on the display and plays the first target picture book on the picture book playing page.

[0128] The second target picture book refers to the picture book selected by the user from at least one displayed picture book. In some embodiments, when the focus frame of the remote control moves to the second target picture book, the processor responds to a trigger operation on the menu key of the remote control to delete the second target picture book, and the function page no longer displays the second target picture book. In other embodiments, when the focus frame of the remote control moves to the second target picture book, a play option and a delete option are displayed, and in response to a trigger operation of the play option, a picture book play page is displayed on the display, and the second target picture book is played on the picture book play page; in response to a trigger operation of the delete option, the second target picture book is deleted from at least one picture book, and the function page no longer displays the second target picture book.

[0129] In some embodiments, the function page can display M picture books generated historically each time. When the focus frame of the remote control moves to the jth picture book among the M picture books, the focus frame continues to move to the Mth picture book. If the focus frame continues to move, the M+1th picture book is displayed in the function page, and the 1st picture book is moved out of the function page. By analogy, other picture books generated historically can be displayed. Wherein, j and M are both positive integers, and j is less than or equal to M.

[0130] In an embodiment of the present application, the processor can display at least one picture book generated historically in the function page, and the user can trigger any picture book in the at least one picture book to play the picture book on the picture book playback page. In this way, the user can repeatedly watch the picture books generated historically, avoiding the consumption of resources and time caused by the repeated generation of picture books with the same requirements, which is conducive to improving the user experience; in addition, the user can also delete any picture book in at least one picture book to delete the picture book that the user is not satisfied with, which is conducive to improving the user experience.

[0131] In some exemplary embodiments, the display method further includes: after generating the target story text based on the voice interaction instruction, displaying the target story text on the generation waiting page of the picture book.

[0132] The target story text refers to all story contents in the generated picture book. In one embodiment, the generation process of the picture book is to first generate the target story text based on the voice interaction instruction, and then generate the image corresponding to each story content based on at least one story content in the target story text, that is, the generation of the target story text is earlier than the generation of the image.

[0133] After the target story text is generated, the target story text can be displayed on the picture book generation waiting page until all the images corresponding to the story contents are generated, or until the progress information indicates that the picture book generation is completed.

[0134] For example, refer to Figure 8 When the progress information is 51%, since the target story text has been generated, the target story text can be displayed on the picture book generation waiting page, while the image generation is not completed and the picture book generation will continue.

[0135] In some embodiments, the picture book generation waiting page also includes a prompt to return to the home page, so as to prompt the user to stop the picture book generation and return to the home page by triggering the return key of the remote control. The user can return to the home page by triggering the return key of the remote control during the picture book generation process to stop the picture book generation, so as to improve the user experience.

[0136] In an embodiment of the present application, after the processor generates the target story text based on the voice interaction instruction, the target story text is displayed on the picture book generation waiting page, so that the user can understand the story content of the generated picture book in time before the image generation of the picture book is completed, thereby improving the user's usage experience.

[0137] In some exemplary embodiments, the display method also includes: displaying a static virtual image on the picture book generation waiting page; and voice broadcasting the target story content being displayed in the text display area, and controlling the virtual image to perform corresponding actions according to the voice broadcast.

[0138] The virtual image refers to a digital image displayed on a display, which can be displayed in two-dimensional or three-dimensional form. The type of the virtual image can be a virtual person, a virtual animal, etc.

[0139] refer to Figure 8 ,In the picture book generation waiting page, the virtual image can be displayed statically.

[0140] refer to Fig. 9 ,In the picture book playback page, the virtual image can be displayed dynamically. ,For example, the processor voice broadcasts the target story content being displayed in the text display area, and ,the virtual image can make corresponding actions as the voice broadcasts.

[0141] In an embodiment of the present application, the processor can display a static virtual image in the picture book generation waiting page and a dynamic virtual image in the picture book playback page. The virtual image makes corresponding actions according to the voice broadcast, which is conducive to improving the richness and vividness of the displayed content and enhancing the user experience.

[0142] In some exemplary embodiments, the display method further includes: receiving interactive speech input by a user, generating a speech interaction instruction based on the interactive speech; in response to the speech interaction instruction, converting the interactive speech into a target text; assembling a first input text based on the target text and a preset first prompt word, the first prompt word being used to guide the language model to generate character information and a story theme that meet the first target requirement; inputting the first input text into the language model to obtain target character information and a target story theme; assembling a second input text based on the target character information, the target story theme and a preset second prompt word, the second prompt word being used to guide the language model to generate a story title, a story introduction and a story content that meet the second target requirement; Input the second input text into the language big model to obtain the target story text, which includes the target story title, target story introduction and target story content; assemble the third input text based on the target role information, target story theme, target story text and the pre-set third prompt words, and the third prompt words are used to guide the language big model to generate a picture description text that meets the third target requirement; input the third input text into the language big model to obtain the target description text; determine the target picture book style based on the keywords in the target text, obtain the style prompt words corresponding to the target picture book style in the preset style library, splice the target description text and the style prompt words to obtain the spliced ​​text; input the spliced ​​text into the Wenshengtu model to generate a picture book.

[0143] The processor receives the interactive voice input by the user through an internally configured module for acquiring voice commands or through an externally set voice control device, and then converts the interactive command into a voice interactive command, and performs voice recognition on the interactive voice in response to the voice interactive command to obtain the target text corresponding to the interactive voice. For example, the target text may be: A Little Adventure of Four Seasons, The Friendship of the Snowman and the Little Black Cat, etc.

[0144] The first target requirement is used to indicate the requirements that the language model needs to follow when outputting the target character information and the target story theme. The first prompt word refers to the prompt word pre-set in the processor, which is used to guide the language model to generate the character information and story theme that meet the first target requirement. The processor can assemble the target text and the first prompt word into a first input text, and input the first input text into the language model to generate the target character information and the target story theme.

[0145] The first prompt word may include an extraction requirement text and a requirement description text. The extraction requirement text indicates the object to be extracted and the content to be extracted from the object to be extracted. The requirement description text indicates the format of the output result and an example of the format.

[0146] For example, the extraction requirement text of the first prompt word may be: Please extract the roles and themes contained in qurey according to the qurey I input. Among them, the object to be extracted is qurey, and the content to be extracted from the object to be extracted is the roles and themes.

[0147] The requirement description text of the first prompt word can be: Do not output my input content, directly output the result, including role information role_name, story theme story_info; the output format is output as a JSON object, including two keys, role_name and story_info; an example of the format: {"role_name":[],"story_info":""}.

[0148] The processor takes the target text as the object to be extracted, and assembles it together with the first prompt word to obtain the first input text.

[0149] The first target requirement indicates the requirements that the language model needs to follow when outputting target character information and target story themes, including the format of the output results, key information to complete the character image, separation of character name and character body, storage format of multiple character images, character information extraction requirements, and story theme extraction requirements.

[0150] For example, the first target requirement may be:

[0151] The output content strictly follows the following requirements:

[0152] 1. Directly output string data in standard JSON format and block other content;

[0153] 2. Provide a suitable description of the extracted character image in the format of [character name] + description. You can use your imagination, but it must be reasonable. For example, for people, you can define skin color, hairstyle, clothing, decoration, etc., and for animals, you can define color, decoration, etc.;

[0154] 3. Each [role name] contains only one role. If a word contains two or more roles, they should be defined separately. For example, if you want to define grandpa and grandma separately, you should define grandpa and grandma separately.

[0155] 4. Use Python list format to store the value in role_name. The number of roles is greater than 0 and less than 3.

[0156] 5. If the role cannot be found in qurey, give a suitable role based on the context information;

[0157] 6. If there is a character with the same name in the qurrey, give the character a proper name;

[0158] 6. Keep the story theme as brief as possible, but retain the key points in the qurrey.

[0159] Based on the target text and the first prompt word, an example of generating target character information and target story theme through the language model is as follows:

[0160] Example 1: The target text is a story about a puppy wandering alone in the forest. The output of the language model can be:

[0161] {"role_name":["[Puppy] A yellow Shiba Inu with a red collar around its neck"],

[0162] "story_info":"The story of a puppy who started his forest adventure out of curiosity and eventually gained courage"}.

[0163] Example 2: The target text is: Xiaohong and Xiaoming go to school; the output of the language model can be:

[0164] {"role_name":["[Xiaohong] A little Chinese girl, with short black hair, a white T-shirt, a black school skirt, yellow sneakers, and a red scarf","[Xiaoming] A little Chinese boy, with a black crew cut, a blue T-shirt, blue jeans, white sneakers, and a red scarf"],"story_info":"The story of Xiaohong and Xiaoming helping each other in school and eventually gaining friendship"}.

[0165] Example 3: The target text is: Two children go to school; the output of the language model can be:

[0166] {"role_name":["[Xiaohong] A little Chinese girl, with short black hair, a white T-shirt, a black school skirt, yellow sneakers, and a red scarf","[Xiaoming] A little Chinese boy, with a black crew cut, a blue T-shirt, blue jeans, white sneakers, and a red scarf"],"story_info":"The story of Xiaohong and Xiaoming helping each other and eventually gaining friendship"}.

[0167] After the target character information and the target story theme are generated based on the target text and the preset first prompt word, the target story text can be further generated using the language macro model.

[0168] The second target requirement is used to indicate the requirements that the language model needs to follow when outputting the target story text. The second prompt word is a prompt word pre-set in the processor, which is used to guide the language model to generate a story title, story introduction and story content that meet the second target requirement. The processor can assemble the target character information, the target story theme and the second prompt word into a second input text, and input the second input text into the language model to generate a target story text including a target story title, a target story introduction and a target story content.

[0169] The second prompt word may include story generation requirement text and requirement description text. The story generation requirement text indicates the generation basis of the story text, the type of the story text and the text paragraph requirement text. The requirement description text indicates the format of the output result and the example of the format.

[0170] For example, the story generation requirement text of the second prompt word can be: Please create a children's story based on the story role Role_name and story summary Story_summary I entered, and the paragraphs must be clear. Among them, the generation basis of the story text is the story role Role_name and story summary Story_summary, the type of the story text is a children's story, and the text paragraph requirement text is clear paragraphs.

[0171] The requirement description text for the second prompt word can be: Do not output my input content, directly output the result, including the title title, story summary and story content; the output format is output as a JSON object, including three keys, title, summary and content; an example of the format:

[0172] {"title":"","summary":"","content":""}.

[0173] The processor uses the target role information as the story role Role_name in the second prompt word, uses the target story theme as the story summary Story_summary in the second prompt word, and assembles them together with the second prompt word to obtain the second input text.

[0174] The second target requirement indicates the requirements that the language model needs to follow when outputting the target story text, including the format of the output results, the format requirements of multiple story paragraphs, the story segmentation requirements, the story genre type, the character naming requirements, the sensitive word requirements, the character processing requirements, etc.

[0175] For example, the second target requirement may be:

[0176] The output content strictly follows the following requirements:

[0177] 1. Directly output string data in standard JSON format and block other content;

[0178] 2. The content in the output JSON data needs to be wrapped in double quotes. If quotes are required inside the content, only single quotes can be used;

[0179] 3. The story needs to have multiple paragraphs. Each paragraph MUST start with PAGE: and cannot contain special symbols, such as (“\"\\| / ) etc.

[0180] 4. Directly output the story outline without providing page numbers, page segments and other useless information;

[0181] 5. The story should have 6-10 paragraphs, at least 6 and at most 10. Do not divide the story into sections.

[0182] 6. The story is narrative-based and does not require character dialogue;

[0183] 7. Don’t give random names to characters;

[0184] 8. Sensitive words are strictly prohibited in the content;

[0185] 9. Double quotes are not allowed in text content;

[0186] 10. Each story cannot exceed 2 sentences;

[0187] 11. Only characters in Role_name can appear in the story. Other characters or interactive characters are not allowed.

[0188] Based on the target character information, the target story theme and the second prompt word, an example of generating the target story text through the language model is as follows:

[0189] Example 1: target role information Role_name: ["[Paul] A little Shiba Inu with a red collar and brown eyes."], target story theme Story_summary: ["The story of Paul exploring the city and finally finding happiness"].

[0190] Output target story text:

[0191] {"title":"Paul's Amazing City Adventure",

[0192] "summary":"Paul is a brave little Shiba Inu who wears a red collar and explores every corner of the city with his curious brown eyes",

[0193] "content":"PAGE:On a warm spring afternoon, Paul set out on a journey to explore the city. His red collar sparkled in the sun.PAGE:Paul sniffed the various aromas in the market.PAGE:Then, Paul found a small park and rolled happily on the grass.PAGE:As the sky gradually darkened, Paul found a tall tower and decided to climb it to overlook the entire city.PAGE:After descending from the tower, Paul encountered a group of pigeons, which surrounded him and made Paul feel very warm.PAGE:As night fell, Paul found his way home in the moonlight.PAGE:Paul rested at home for a night and prepared for new adventures.PAGE:The next day, Paul found a small river and played by the river.PAGE:Paul found some beautiful stones by the river and took them back home.PAGE:Paul's adventure made him very happy and he looked forward to more exploration."}.

[0194] After the target story text is generated based on the target role information, the target story theme and the preset second prompt words, the target description text can be further generated using the language model.

[0195] The third target requirement is used to indicate the requirements that the language model needs to follow when outputting the target description text. The third prompt word is a prompt word pre-set in the processor, which is used to guide the language model to generate a screen description text that meets the third target requirement. The processor can assemble the target character information, the target story theme, the target story text and the third prompt word into a third input text, and input the third input text into the language model to generate a target description text. Since the target story text includes multiple story texts, the target description text can include multiple description texts, each description text is an information description of the corresponding image.

[0196] The third prompt word may include a description text generation basis and a description text generation requirement. The description text generation basis indicates a generation basis of the description text.

[0197] For example, the third prompt word can be: Generate the screen description text corresponding to each PAGE according to the provided Title, Summary, Content, Role_list and Role_info, and the PAGE and the description text are strictly one-to-one corresponding! Among them, the description text generation basis is Title, Summary, Content, Role_list and Role_info, and the description text generation requirement is to generate the screen description text corresponding to each PAGE, and the PAGE and the description text are strictly one-to-one corresponding.

[0198] The processor uses the target story title, target story introduction, and target story content as Title, Summary, and Content in the third prompt word, respectively. The target role information includes the name and basic information of at least one role. The name of at least one role is used as Role_list in the third prompt word, and the basic information of each role is used as Role_info. Together with the third prompt word, they are assembled to obtain the third input text.

[0199] The third target requirement indicates the requirements that the target description text output by the language large model needs to follow, including the format of the output results, the description requirements at the beginning of the text, the action description requirements, the correspondence between the elements in the description text and the characters, the vocabulary and grammar requirements, the correspondence between the picture book pages and the description text, the correspondence between the picture book pages and the picture scenes, the requirements for the number of picture book pages, the output requirements for the character names, etc.

[0200] For example, the third target requirement may be:

[0201] The output content strictly follows the following requirements:

[0202] 1. There is a Role_name in the picture, which is the role description. The description text format is A: [Role_name] action / posture / position / background. The beginning of the sentence is represented by [Role_name];

[0203] 2. There is no Role_name in the picture, it is a description of the scenery or background, the description text paradigm B: [NC] Scenery description, the beginning of the sentence is indicated by [NC]. [scenery description] For example, "a large grassland, full of vitality";

[0204] 3. For the actions of Role_name, use simple action words, such as "walk", "sit", "stand", "touch" and other simple body action words to describe. Increase the diversity of close-up and long-range views;

[0205] 4. If there is no obvious action for Role_name, expand the posture according to the scene. For example, sleeping can be expanded to "lying flat" or "closing eyes";

[0206] 5. Each PAGE description text should contain the positional relationship between each element and Role_name, such as "There is a green tree next to Paul" and "There is a sun above Jack's head";

[0207] 6. Each PAGE description text should not contain element psychology or emotional actions. Delete relevant psychological descriptions such as "very happy", "feeling very fulfilled inside", "dreaming", etc., and only keep the description of body movements and element positions;

[0208] 7. Make sure you use simple words, including simple grammar, simple word usage, no modifiers, no adjectives, so that the image generation model can understand it;

[0209] 8. Output uses simple Chinese grammar combinations, and each description text is no longer than 25 words;

[0210] 9. Do not output irrelevant information or analytical information, and try to keep one character in each sentence;

[0211] 10. [NC] only describes the landscape;

[0212] 11. One PAGE only outputs one text description;

[0213] 12. First output how many pages you want to generate, and make sure the two numbers are consistent. Note! The current number of screens is at least 6;

[0214] 13. Determine the number of PAGES according to the Content, and do not output other content exceeding this number;

[0215] 14. Do not output notes! Do not output content that does not match the PAGE in Content;

[0216] 15. [Role_name] is the name of the role that appears. When the name is output, brackets are added on both sides, such as [name]. The same name can only appear once in each paragraph.

[0217] An example of generating a target description text through a language model based on target character information, target story text, and a preset third prompt word is as follows:

[0218] Example 1, Title: Jack's Beach Camping Adventure.

[0219] Summary:Jack decided to go to the seaside alone for a beach camping adventure during the summer vacation.

[0220] Content:PAGE:Jack put on his small backpack and set out for the beach. He planned to spend an unforgettable night on the beach. PAGE:When they arrived at the beach, the sun was slowly setting and Jack found a perfect camping spot. PAGE:He skillfully set up the tent and prepared for the arrival of the night. PAGE:When night fell and the stars twinkled in the sky, Jack lit a bonfire. PAGE:He roasted marshmallows and enjoyed the tranquility brought by the sound of the waves. PAGE:Suddenly a gust of sea breeze blew over and Jack felt a little cool, but he was not afraid. PAGE:He took out a flashlight and strolled on the beach to explore the mysteries of the night. PAGE:Under the moonlight, Jack found a group of small crabs busy on the beach. PAGE:The next morning, when the first ray of sunlight shone on the tent, Jack was awakened by the warm light. PAGE:He packed up the tent and set out on the way home with a heart full of memories.

[0221] Role_list:[Jack].

[0222] Role_info:[Jack is a black kitten].

[0223] The output target description text may be: Since there are 10 picture book pages PAGES, the screen description texts of the 10 picture book pages are output, and 10 scenes are output.

[0224] 1. [Jack] is walking on the road in front of the beach with a small backpack on his back, with blue sky and white clouds in the background.

[0225] 2.[NC] In the evening, the red sun hangs in the sky, the sea water is reflected red by the sun, and the background is the sunset on the beach.

[0226] 3. [Jack] is standing next to a tent on the beach, with a red sunset over the sea and a sunset on the beach in the background.

[0227] 4. [Jack] Sitting by a bonfire on the beach with stars in the sky and the beach at night in the background.

[0228] 5. [Jack] sitting by a campfire, holding a marshmallow in his hand, with the sea in the distance, the moon hanging in the sky, and the night at the beach in the background.

[0229] 6.[NC] Waves on the sea, a tent on the beach, the moon hanging in the sky, the background is the beach at night.

[0230] 7. [Jack] Walking on the beach, the moon in the sky casts silver moonlight.

[0231] 8. [Jack] was standing on the beach with a group of little crabs next to him and the moon hanging in the sky.

[0232] 9. [Jack] is lying in a tent by the beach, with the sun in the sky and the morning in the background.

[0233] 10.[Jack] walked home in the morning sunshine, carrying his duffel bag.

[0234] The Wensheng graph model refers to a large model based on deep learning, which generates an image corresponding to the text description by parsing and understanding the input text description. In some embodiments, the target description text is input into the Wensheng graph model to generate at least one image corresponding to the target description text. Each description text in the target description text corresponds to an image, and multiple description texts and the images corresponding to each description text constitute a picture book.

[0235] In other embodiments, the processor extracts keywords from the target text to obtain keywords in the target text, and determines the target picture book style based on the keywords related to the style in the keywords. The target picture book style includes but is not limited to comic style, cartoon style, illustration style, etc. If the keywords do not contain keywords related to the style, the default style is used as the target picture book style. For example, the default style is a cartoon style.

[0236] The preset style library is used to store the correspondence between picture book styles and style prompt words. By querying the preset style library, the style prompt words corresponding to the target picture book style can be determined. The processor splices the target description text with the style prompt words to obtain a spliced ​​text, and inputs the spliced ​​text into the text-generated graph model to generate a picture book with the target picture book style.

[0237] In an embodiment of the present application, the target text in the interactive voice input by the user is assembled with the first prompt word into the first input text, thereby guiding the language model to generate the target character information and the target story theme, and then the target character information, the target story theme and the second prompt word are assembled into the second input text, thereby guiding the language model to generate the target story text, and then the target character information, the target story text and the third prompt word are assembled into the third input text, thereby guiding the language model to generate the target description text, and the target description text is a plurality of continuous story contents, and then based on the target description text, the text graph capability of the text graph model is used to generate images corresponding to the plurality of continuous story contents, and a picture book is obtained, thereby realizing the automatic generation of a picture book that meets the user's needs by using the input interactive voice, and the user can indicate the picture book style through the interactive voice, thereby realizing the automatic generation of a picture book that meets the user's style requirements, simplifying the user operation and improving the user experience.

[0238] In some exemplary embodiments, the display method also includes: determining a picture book style to be adjusted, the picture book style to be adjusted is a picture book style that has been manually evaluated and has an unsatisfactory generation effect; obtaining multiple sample images under the picture book style to be adjusted, obtaining a content description text corresponding to each sample image, each sample image and the corresponding content description text constitute a training sample, and multiple training samples constitute a training set; for each sample image in the training set, inputting the content description text corresponding to the current sample image into the Vincent graph network in the Vincent graph model; randomly selecting a target sample image from other sample images in the training set except the current sample image; inputting the target sample image into an adapter layer in the Vincent graph model, the adapter layer extracts features from the target sample image to obtain a feature representation of the target sample image; outputting the feature representation to the Vincent graph network; the Vincent graph network processes the received content description text and feature representation, and outputs a predicted image; calculating a model loss based on the current sample image and the predicted image, adjusting parameters of the adapter layer based on the model loss, and continuing training until the model loss meets a preset condition.

[0239] Among them, since the Wensheng picture model may output images that are inconsistent with reality or have a large difference from the actual style type, the images output by the Wensheng picture model can be manually evaluated in whole or in part to screen out images with unsatisfactory generation effects. The picture book style corresponding to the image with unsatisfactory generation effect is determined as the picture book style to be adjusted. For the picture book style to be adjusted, the Wensheng picture model can be adjusted by adjusting the parameters to improve the generation effect of the Wensheng picture model on the picture book style to be adjusted.

[0240] Sample images refer to images generated by the non-textual image model and having the style of the picture book to be adjusted. Content description text refers to the description of the content of the sample image. Each sample image and its corresponding content description text constitute a training sample, and the training set consists of multiple training samples.

[0241] like Fig.11 The composition diagram of the Wensheng graph model according to some embodiments is exemplarily shown. The Wensheng graph model includes a Wensheng graph network and an adapter layer. When the parameters of the Wensheng graph model are adjusted using a training set, the parameters of the Wensheng graph network can be frozen, and only the parameters of the adapter layer can be fine-tuned, which can significantly reduce the computational complexity and video memory usage during model training, while having the advantages of retaining knowledge, reducing resource overhead, and improving generalization ability and flexibility.

[0242] Specifically, for each sample image in the training set, the content description text corresponding to the current sample image is input into the Wensheng graph network in the Wensheng graph model, and the target sample image in other sample images is input into the adapter layer. The adapter layer is used to extract features from the target feature image to obtain the feature representation of the target sample image. The feature representation is input into the Wensheng graph model, so that the content description text and the feature representation are processed by the Wensheng graph model to obtain a predicted image. The processor calculates the model loss based on the current sample image, the predicted image and the preset loss function, and adjusts the parameters of the adapter layer based on the model loss, and continues training until the model loss meets the preset conditions to obtain the Wensheng graph model with adjusted parameters. Exemplarily, LoRA (Low-Rank Adaptation) can be used to fine-tune the parameters of the Wensheng graph model.

[0243] In an embodiment of the present application, for the picture book style to be adjusted whose generation effect is not ideal in manual evaluation, the method of fine-tuning parameters of the adapter layer of the Wensheng picture model can be used to improve the generation effect of the Wensheng picture model on the picture book style to be adjusted. At the same time, the parameter fine-tuning method can significantly reduce the computational complexity and video memory usage during model training, and has the advantages of retaining knowledge, reducing resource overhead, and improving generalization ability and flexibility.

[0244] To explain the display device, display method and effect in this solution in detail, a most detailed embodiment is described below:

[0245] A display device is provided in an embodiment of the present application, including: a display, configured to display content from a broadcast system or network and / or a user interface; and at least one processor, which is connected to the display and configured to execute instructions to enable the display device to: in response to an instruction to open a picture book application, display a function page of the picture book application on the display, the function page displays a prompt for generating a picture book through voice interaction; in response to a voice interaction instruction, display a picture book generation waiting page on the display, the picture book generation waiting page displays the progress of picture book generation; after the picture book is generated, display a picture book playback page on the display, the picture book playback page includes an image display area and a text display area, the generated picture book includes at least one story content with a sequence and an image corresponding to each story content; display at least one story content in the text display area in sequence, voice broadcast the target story content being displayed in the text display area, and display the image corresponding to the target story content in the image display area. In some embodiments, the function page also displays at least one picture book generated historically, and the processor responds to a trigger operation on a first target picture book in at least one picture book, displays a picture book playback page on the display, and plays the first target picture book on the picture book playback page; responds to a deletion operation on a second target picture book in at least one picture book, and deletes the second target picture book from at least one picture book.

[0246] In some embodiments, after the processor generates the target story text based on the voice interaction instruction, it generates a waiting page for displaying the target story text in the picture book.

[0247] In some embodiments, the processor displays a static virtual image on the picture book generation waiting page; during the process of voice broadcasting the target story content being displayed in the text display area, the processor controls the virtual image to perform corresponding actions according to the voice broadcast.

[0248] In some embodiments, the processor receives interactive speech input by a user, generates a speech interaction instruction based on the interactive speech; in response to the speech interaction instruction, converts the interactive speech into a target text; assembles a first input text based on the target text and a preset first prompt word, the first prompt word is used to guide the language model to generate character information and a story theme that meet the first target requirement; inputs the first input text into the language model to obtain target character information and a target story theme; assembles a second input text based on the target character information, the target story theme and a preset second prompt word, the second prompt word is used to guide the language model to generate a story title, a story introduction and a story content that meet the second target requirement; The target story text is input into the language model to obtain the target story text, which includes the target story title, the target story introduction and the target story content; based on the target role information, the target story theme, the target story text and the pre-set third prompt words, a third input text is assembled to obtain the third prompt word, which is used to guide the language model to generate a picture description text that meets the third target requirement; the third input text is input into the language model to obtain the target description text; based on the keywords in the target text, the target picture book style is determined, the style prompt words corresponding to the target picture book style are obtained in the preset style library, and the target description text and the style prompt words are spliced ​​to obtain the spliced ​​text; the spliced ​​text is input into the Wenshengtu model to generate a picture book.

[0249] In some embodiments, the processor determines a picture book style to be adjusted, which is a picture book style that has been manually evaluated and has an unsatisfactory generated effect; obtains multiple sample images of the picture book style to be adjusted, obtains content description text corresponding to each sample image, each sample image and the corresponding content description text constitute a training sample, and multiple training samples constitute a training set; for each sample image in the training set, inputs the content description text corresponding to the current sample image into the Vincent graph network in the Vincent graph model; randomly selects a target sample image from other sample images in the training set except the current sample image; inputs the target sample image into the adapter layer in the Vincent graph model, the adapter layer extracts features from the target sample image, and obtains a feature representation of the target sample image; outputs the feature representation to the Vincent graph network; the Vincent graph network processes the received content description text and feature representation, and outputs a predicted image; calculates the model loss based on the current sample image and the predicted image, adjusts the parameters of the adapter layer based on the model loss, and continues training until the model loss meets a preset condition.

[0250] like Fig.12 The timing diagram of the display method according to some embodiments is exemplified. The display system includes an application control module, an information perception module, a control platform, a language large model key information extraction / completion module, a multimodal generation large model, etc.

[0251] The application control module is used to call voice monitoring in response to an instruction to open the picture book application after the user opens the picture book application.

[0252] The information perception module is used to receive the interactive voice input by the user, perform voice recognition on the interactive voice, and obtain the target text corresponding to the interactive voice.

[0253] The control platform includes a picture book central control and a database. The picture book central control is used for the overall scheduling of the display method, and the database is used to store key information returned to the picture book central control.

[0254] The key information extraction / completion module of the language model includes role extraction, theme extraction, style extraction and broadcast timbre recommendation. Among them, role extraction, theme extraction, style extraction and broadcast timbre recommendation can be separate functional modules, and the role information, story theme, style information and timbre information can be obtained by performing corresponding information extraction on the target text. The role information may include role name and role attribute information. Alternatively, the target text can be extracted and completed through the voice model to output role information, story theme and style information. It is also possible to perform timbre extraction on the interactive voice to obtain the user's timbre information.

[0255] The multimodal generation model includes modules such as story generation, text-image rewriting, text-image, speech synthesis, and music generation. Among them, the character name and character attribute information are used as the target character information, the story theme is used as the target story theme, and the recommended style is determined based on the style information; the multimodal generation model is used to generate the target story text based on the target character information and the target story theme, and to rewrite the target story text with text-image to generate the target description text, and then combine the target description text with the recommended style, and generate the image corresponding to the target description text through the multimodal generation model to obtain the picture book. In addition, the multimodal generation model is also used to perform speech synthesis based on the recommended timbre and generate background music based on the recommended style.

[0256] The picture book central control returns the picture book's images, story content, synthesized voice and background audio to the picture book application, and plays the picture book after audio and video alignment. For example, the story content in the picture book can be played using synthesized voice, background audio can be played, and the corresponding images and story content can be displayed at the same time.

[0257] In an embodiment of the present application, a display device and a display method are provided, wherein a picture book application is pre-installed in the display device, and in response to an instruction to open the picture book application, a function page of the picture book application is displayed, wherein a prompt for generating a picture book through voice interaction is displayed in the function page, so as to instruct a user to input a voice interaction instruction for generating a picture book; in response to the voice interaction instruction, a picture book generation waiting page is displayed, and the progress of picture book generation is displayed on the generation waiting page, so that the user can promptly understand the progress of picture book generation; after the picture book is generated, a picture book playback page is displayed, so that at least one story content is displayed in sequence in a text display area of ​​the picture book playback page, a target story content being displayed is voice broadcasted, and an image corresponding to the target story content is displayed in an image display area of ​​the picture book playback page, so that a picture book can be automatically generated using the input voice interaction instruction, and at least one story content in the picture book and the image corresponding to each story content can be displayed and played in sequence on the display, so that the display device can display a picture book that meets the personalized needs of the user, thereby improving the user experience.

[0258] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0259] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a display method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0260] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0261] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0262] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0263] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0264] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0265] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0266] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0267] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display device, characterized in that: include: a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor connected to the display and configured to execute instructions to cause the display device to: In response to an instruction to open a picture book application, a function page of the picture book application is displayed on the display, wherein the function page displays a prompt for generating a picture book through voice interaction; In response to the voice interaction instruction, displaying a picture book generation waiting page on the display, wherein the picture book generation waiting page displays the progress of picture book generation; After the picture book is generated, a picture book playback page is displayed on the display, the picture book playback page includes an image display area and a text display area, and the generated picture book includes at least one story content in sequence and an image corresponding to each story content; The at least one story content is displayed in the text display area in sequence according to the sequence, a voice broadcast is performed on the target story content being displayed in the text display area, and an image corresponding to the target story content is displayed in the image display area.

2. The display device according to claim 1, characterized in that The function page further displays at least one picture book generated historically, and the processor is further configured to: In response to a triggering operation on a first target picture book among the at least one picture book, displaying a picture book playing page on the display, and playing the first target picture book on the picture book playing page; In response to a deletion operation on a second target picture book in the at least one picture book, the second target picture book is deleted from the at least one picture book.

3. The display device according to claim 1, characterized in that The processor is further configured to: After the target story text is generated based on the voice interaction instruction, the target story text is displayed on the generation waiting page of the picture book.

4. The display device according to claim 1, characterized in that The processor is further configured to: Displaying a static virtual image on the picture book generation waiting page; The process of voice broadcasting the target story content being displayed in the text display area, and controlling the virtual image to perform corresponding actions according to the voice broadcast.

5. The display device according to claim 1, characterized in that The processor is further configured to: Receiving interactive voice input by a user, and generating the voice interactive instruction based on the interactive voice; In response to the voice interaction instruction, converting the interactive voice into a target text; Assembling a first input text based on the target text and a preset first prompt word, wherein the first prompt word is used to guide the language macro model to generate character information and a story theme that meet the first target requirement; inputting the first input text into the language macro model to obtain target character information and a target story theme; Assembling a second input text based on the target role information, the target story theme and a preset second prompt word, wherein the second prompt word is used to guide the language macro model to generate a story title, a story introduction and a story content that meet the second target requirement; inputting the second input text into the language macro model to obtain a target story text, wherein the target story text includes a target story title, a target story introduction and a target story content; Assembling a third input text based on the target character information, the target story theme, the target story text and a preset third prompt word, wherein the third prompt word is used to guide the language macro model to generate a screen description text that meets the third target requirement; inputting the third input text into the language macro model to obtain a target description text; The target picture book style is determined based on the keywords in the target text, the style prompt words corresponding to the target picture book style are obtained in a preset style library, the target description text and the style prompt words are spliced ​​to obtain a spliced ​​text; the spliced ​​text is input into a text-generated picture model to generate a picture book.

6. The display device according to claim 5, characterized in that The processor is further configured to: Determining a picture book style to be adjusted, where the picture book style to be adjusted is a picture book style that has been manually evaluated and has an unsatisfactory generation effect; Acquire multiple sample images of the picture book style to be adjusted, and acquire content description text corresponding to each sample image, wherein each sample image and the corresponding content description text constitute a training sample, and multiple training samples constitute a training set; For each sample image in the training set, inputting the content description text corresponding to the current sample image into the text graph network in the text graph model; Randomly select a target sample image from other sample images in the training set except the current sample image; input the target sample image to an adapter layer in a Vincent graph model, wherein the adapter layer extracts features from the target sample image to obtain a feature representation of the target sample image; and output the feature representation to the Vincent graph network; The text-based graph network processes the received content description text and feature representation and outputs a predicted image; A model loss is calculated based on the current sample image and the predicted image, parameters of the adapter layer are adjusted based on the model loss, and training is continued until the model loss meets a preset condition.

7. A display method, characterized in that: include: In response to an instruction to open a picture book application, a function page of the picture book application is displayed on a display, wherein the function page displays a prompt for generating a picture book through voice interaction; In response to the voice interaction instruction, displaying a picture book generation waiting page on the display, wherein the picture book generation waiting page displays the progress of picture book generation; After the picture book is generated, a picture book playback page is displayed on the display, the picture book playback page includes an image display area and a text display area, and the generated picture book includes at least one story content in sequence and an image corresponding to each story content; The at least one story content is displayed in the text display area in sequence according to the sequence, a voice broadcast is performed on the target story content being displayed in the text display area, and an image corresponding to the target story content is displayed in the image display area.

8. The method according to claim 7, characterized in that The function page further displays at least one picture book generated historically, and the method further includes: In response to a triggering operation on a first target picture book among the at least one picture book, displaying a picture book playing page on the display, and playing the first target picture book on the picture book playing page; In response to a deletion operation on a second target picture book in the at least one picture book, the second target picture book is deleted from the at least one picture book.

9. The method according to claim 7, characterized in that: The method further comprises: After the target story text is generated based on the voice interaction instruction, the target story text is displayed on the generation waiting page of the picture book.

10. The method according to claim 7, characterized in that The method further comprises: Displaying a static virtual image on the picture book generation waiting page; The process of voice broadcasting the target story content being displayed in the text display area, and controlling the virtual image to perform corresponding actions according to the voice broadcast.

Images