Display device and picture book media asset generation method

Abstract

Embodiments of the invention provide a display device and a picture book media asset generation method, the generation method responds to a picture book generation instruction, detects a target description text associated with the picture book generation instruction, and extracts an initial keyword of the target description text, first feature information used for describing role features and second feature information used for describing theme features are obtained according to the initial keyword; and generating a first image frame according to the first feature information and the second feature information. And extracting supplementary feature information of the first image, performing iteration on the first feature information according to the supplementary feature information, generating a second image frame according to the iterated feature information, and synthesizing the picture book media asset according to the first image frame and the second image frame. According to the method, feature extraction is performed on the generated image frames, and character features of the picture book media assets are iterated through the extracted feature information, so that role effects presented by the image frames in the picture book media assets can be kept consistent, and the generation quality of the picture book media assets is improved.

Description

Technical Field

[0001] This application relates to the field of display device technology, and more particularly to a display device and a method for generating picture book media assets. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, equipped with open operating systems and chips, possessing open application platforms, enabling two-way human-computer interaction, and integrating multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] Display devices can achieve corresponding functions by running different AI models through different applications. For example, a display device can be configured with a picture book application, which can call and run the Diffusion model. This model uses a neural network architecture to understand and analyze the text content input by the user, converting the text content into image frames to form the final picture book media asset. In other words, the Diffusion model can progressively denoise the text content, generating image frames that match the text content at the pixel level to produce the final picture book media asset.

[0004] Because picture book assets describe a single story, the content presented between image frames is related and continuous. For example, when a picture book asset describes the story of the crow drinking water, the crow presented in each image frame should be consistent. However, due to the randomness of AI models, even if each image frame of a picture book asset is generated from the same text content, the content presented in each image frame may still not be consistent. For instance, if the user-input text content only describes the protagonist of the story as a crow, the crow's beak shape, plumage, or leg shape presented in different image frames generated by the display device may be inconsistent, affecting the user's viewing experience. Summary of the Invention

[0005] This application provides a display device and a method for generating picture book media assets to solve the problem of inconsistent content among image frames in picture book media assets generated by the display device.

[0006] In a first aspect, some embodiments of this application provide a display device, including a display and a controller. The display is configured to display an application interface for a picture book application, and the controller is configured to execute the following program steps:

[0007] In response to a picture book generation command input by the user based on the application interface, target description text is detected, wherein the target description text is the description text associated with the picture book generation command;

[0008] Extract initial keywords from the target description text;

[0009] First feature information and second feature information are obtained based on the initial keywords. The first feature information is used to describe the character features, and the second feature information is used to describe the theme features.

[0010] Generate prompt text based on the first feature information and the second feature information;

[0011] Generate a first image frame according to the first feature information and the prompt text;

[0012] Extract supplementary feature information from the first image frame, wherein the supplementary feature information is the character feature extracted based on the first image frame;

[0013] A second image frame is generated based on iterative feature information, wherein the iterative feature information is generated according to the first feature information, the prompt text, and the supplementary feature information;

[0014] The picture book media is synthesized based on the first image frame and the second image frame.

[0015] Secondly, some embodiments of this application also provide a method for generating picture book media assets, including the following steps:

[0016] In response to a user-inputted picture book generation command, the target description text is detected. The picture book generation command is an interactive command input based on the application interface of the picture book application, and the target description text is the description text associated with the picture book generation command.

[0017] Extract initial keywords from the target description text;

[0018] First feature information and second feature information are obtained based on the initial keywords. The first feature information is used to describe the character features, and the second feature information is used to describe the theme features.

[0019] Generate prompt text based on the first feature information and the second feature information;

[0020] Generate a first image frame according to the first feature information and the prompt text;

[0021] Extract supplementary feature information from the first image frame, wherein the supplementary feature information is the character feature extracted based on the first image frame;

[0022] A second image frame is generated based on iterative feature information, wherein the iterative feature information is generated according to the first feature information, the prompt text, and the supplementary feature information;

[0023] The picture book media is synthesized based on the first image frame and the second image frame.

[0024] As can be seen from the above technical solutions, the display device and picture book media asset generation method provided in some embodiments of this application can respond to a picture book generation instruction, detect the target description text associated with the picture book generation instruction, extract the initial keywords of the target description text, and then obtain first feature information and second feature information based on the initial keywords. The first feature information is used to describe character features, and the second feature information is used to describe theme features. Then, prompt text is generated based on the first and second feature information, and a first image frame is generated based on the first feature information and the prompt text. Supplementary feature information of the first image is extracted, and the first feature information is iterated based on the supplementary feature information. A second image frame is then generated according to the iterated feature information, so as to synthesize the picture book media asset based on the first and second image frames. The method extracts features from the generated image frames and iterates on the character features using the extracted feature information, which can ensure that the character effects of each image frame in the picture book media asset remain consistent, thereby improving the generation quality of the picture book media asset. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0027] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0028] Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0029] Figure 4 A schematic diagram illustrating the forward and reverse diffusion processes of the diffusion model provided in some embodiments of this application;

[0030] Figure 5 Examples of comparison diagrams showing the effects of differences in image frames in picture book media assets provided in some embodiments of this application;

[0031] Figure 6 A flowchart illustrating a method for generating picture book media assets, provided for some embodiments of this application;

[0032] Figure 7These are schematic diagrams illustrating the effect of the homepage interface of the picture book application provided in some embodiments of this application;

[0033] Figure 8 A schematic diagram illustrating the display effect of target description text provided in some embodiments of this application;

[0034] Figure 9 A flowchart illustrating the extraction of the first keyword and the second keyword provided for some embodiments of this application;

[0035] Figure 10 This is a schematic diagram of the process for obtaining first feature information provided in some embodiments of this application;

[0036] Figure 11 A schematic diagram illustrating the generation process of each image frame in a picture book media asset provided in some embodiments of this application;

[0037] Figure 12 This is a schematic diagram of the process for extracting supplementary feature information provided in some embodiments of this application;

[0038] Figure 13 This is a flowchart illustrating the process of synthesizing media asset data provided in some embodiments of this application;

[0039] Figure 14 A schematic diagram illustrating the effect of the picture book generation process prompt interface provided in some embodiments of this application;

[0040] Figure 15 These are schematic diagrams illustrating the effect of the generated result interface provided in some embodiments of this application;

[0041] Figure 16 These are comparison images of the effects of image frames in picture book media provided in some embodiments of this application. Detailed Implementation

[0042] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0043] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0044] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0045] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0046] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0047] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0048] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0049] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0050] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0051] like Figure 1The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0052] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0053] Figure 2 This is a hardware configuration block diagram of a display device 200 provided in some embodiments of this application.

[0054] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0055] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0056] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0057] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0058] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0059] In some embodiments, the controller 250 may include at least one central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device 200 and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0060] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0061] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0062] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0063] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0064] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device 200 to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0065] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for the display device 200.

[0066] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0067] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0068] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0069] like Figure 3As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0070] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0071] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0072] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0073] It should be noted that the above examples are only simple divisions of the functions of the operating system, and do not limit the specific form of the operating system of the display device 200 in the embodiments of the present application. Depending on factors such as the functions of the display device and the type of the operating system, the number of levels and the specific level types included in the operating system may be in other forms.

[0074] To implement different functions, in some embodiments, different types of applications can be configured in the application layer of the display device 200, and users can run different applications in the display device 200 to achieve different functions. For example, as Figure 3 shown in the application layer, Application 1 can be a media asset playback application for obtaining media asset data and performing decoding and playback on the media asset data; Application 2 can be a whiteboard application for implementing graphic drawing in response to the touch operations input by the user; Application 3 can be a picture book application for generating and playing picture book media assets.

[0075] For the generation of picture book media assets, in some embodiments, the picture book application of the display device 200 can generate picture book media assets through a pre-set picture book generation technology, and the picture book media assets can be video data, image data, etc. For example, the picture book generation technology pre-set in the picture book application can be AI models such as GAN (Generative Adversarial Networks) and Diffusion Models to achieve the automated creation of picture book media assets.

[0076] For the convenience of description, in some embodiments of the present application, the picture book generation technology pre-set in the picture book media assets is taken as an example of the technology based on Diffusion Models for description, but it is not a limitation. The model pre-set in the picture book media assets can also be other networks or AI models for image generation.

[0077] In some embodiments, the picture book application can receive control instructions input by the user based on different interaction methods, such as voice input, touch input, or instructions input by the user through the control device 100 supporting the display device 200. After the display device 200 starts the picture book application, it can control the display 260 to display the application interface of the picture book application. The application interface can include a home page interface, a picture book generation prompt interface, a picture book playback interface, etc. The application interface of the picture book application can include multiple Views (views) and controls to assist the user in operating and using the picture book application.

[0078] In some embodiments, the display device 200 can obtain the picture book story text through a picture book application. This picture book story text can be a preset text or a story text automatically generated by the display device 200 based on a user-input description text using an AI language model. The picture book story text is then input into a Diffusion model trained in the picture book application, such as Stable Diffusion, to generate corresponding images by calling the Diffusion model through the picture book story text, thereby synthesizing the picture book media asset corresponding to the picture book story text.

[0079] To adapt to different storylines, in some embodiments, the picture book application of the display device 200 marks the picture book story text as multiple consecutive sub-texts and calls the Diffusion model through these consecutive sub-texts to generate image frames corresponding to those story text segments. Picture book media assets are then generated from these multiple image frames. The Diffusion model works by progressively adding noise to the image data (forward diffusion process) and progressively removing noise from it to recover the original image data (backward diffusion process).

[0080] For example, such as Figure 4 As shown, the forward diffusion process is Figure 4 The process proceeds from right to left, starting with the original image data x0, and at step t, x... t-1 Based on adding noise, we get x t x t Only with x t-1 Related, until after T steps, x T It completely transforms into Gaussian noise. The reverse process (reverse diffusion process) is represented as... Figure 4 The process proceeds from left to right. First, a Gaussian noise x is given. T The original image data x0 is recovered by progressively denoising. After the model is trained, various images can be generated by providing Gaussian noise.

[0081] However, in the backdiffusion process of the Diffusion model, randomness is introduced at each step to simulate a realistic noise removal process and ensure the diversity of image generation. Due to the randomness of the model, the image frames generated by the Diffusion model for different paragraphs of the picture book story text will differ. This will result in variations in the character appearances presented in different image frames within the picture book media generated by the display device 200. For example, the facial features, skin tone, body shape, or gender of the characters presented in consecutive image frames may be inconsistent, leading to discontinuity in the picture book media and affecting the user's viewing and usage experience. For example, as... Figure 5 As shown, Figure 5The panda is the main character in the Chinese picture book, and the panda's appearance differs greatly between the two image frames.

[0082] To this end, some embodiments of this application provide a display device 200, which can extract character features used to describe characters and then add the character features to an image generation model (such as a diffusion model) to improve the consistency of characters in generated images. After each image frame is generated by the image generation model, the display device 200 performs feature extraction on the image to determine whether there are new character features in the generated image frame. When new character features exist, the newly extracted character features are used to iterate the character features of the image generation model, so that the character effects presented in each image frame generated by the image generation model can be consistent, thereby improving the coherence of picture book media assets and enhancing the user experience.

[0083] For ease of distinction and explanation, in some embodiments of this application, data information describing role characteristics is represented as first feature information, and data information describing theme characteristics is represented as second feature information.

[0084] like Figure 6 As shown, in some embodiments, the display device 200 may include a display 260 and a controller 250. The display 260 is configured to display the application interface of the picture book application, such as a homepage interface, a picture book generation prompt interface, a picture book playback interface, etc.; the controller 250 is configured to execute the following program steps:

[0085] S601: In response to a picture book generation instruction input by the user based on the application interface, detect the target description text, wherein the target description text is the description text associated with the picture book generation instruction.

[0086] When the display device 200 starts the picture book application, the controller 250 creates or reuses the process of the picture book application based on the operating system, so that the application interface of the picture book application can be displayed in the foreground of the display device 200, that is, the display 260 is controlled to display the application interface of the picture book application, so that the user can perform related operations based on the application interface of the picture book application.

[0087] In some embodiments, after launching the picture book application, the user can select a pre-set story template in the application to generate picture book media assets; alternatively, the user can also input descriptive text of the picture book story into the application interface to generate picture book media assets. For example, after launching the picture book application, the display device 200 displays... Figure 7 On the homepage interface shown, users can select template text on the homepage interface or input descriptive text for user-generated picture book media assets via voice or text input. Then, by performing a selection operation on the "Generate Now" option control 701, the user can input picture book generation instructions to the display device 200.

[0088] In some embodiments, the picture book generation instruction includes associated descriptive text, such as template text selected by the user or input text content. For ease of description, embodiments of this application represent the descriptive text associated with the picture book generation instruction as target descriptive text. The target descriptive text can be multiple segments of story text, a short story text, or a single sentence summary of the story; this application does not impose any limitations on this.

[0089] Upon receiving a picture book generation instruction, the display device 200 responds by detecting the target description text associated with the picture book generation instruction, so that the display device 200 can generate picture book media materials that match the target description text.

[0090] In some embodiments, when a user inputs target description text to the display device 200 via voice input, the display device 200 can respond to the voice command input via voice input by performing voice recognition, such as ARS (Automatic Speech Recognition) technology, to convert the voice command into text content to form the target description text.

[0091] In some embodiments, after the display device 200 converts a voice command into target description text, it can display the target description text on the input control of the homepage interface, allowing the user to judge the accuracy of the voice conversion through the text content displayed in the input control. For example, if a user speaks the synopsis of the picture book media asset to be generated, such as "The protagonist is a little rabbit who lives in a beautiful forest," the display device 200 can convert it into the corresponding target description text through ARS and display the converted target description text on an input control such as... Figure 8 The input control 801 shown.

[0092] In some embodiments, the display device 200 can also record the historical generation records of the picture book application, generate similar story templates based on the historical generation records, and display the story templates on the homepage of the picture book application. In this way, users can select a story template on the homepage, and the corresponding picture book media assets will be automatically generated.

[0093] In some embodiments, when the display device 200 detects the target descriptive text, it can also control the display 260 to display a style selection interface based on the picture book application. The style selection interface includes option controls for different visual styles, such as animation, sketch, 3D cartoon, watercolor, and flat illustration. Different visual styles can correspond to different image generation models. Users can select the display effect of image frames through the various option controls in the style selection interface to generate picture book media assets of different styles using the corresponding image generation models.

[0094] S602: Extract the initial keywords from the target description text.

[0095] After detecting the target description text associated with the picture book generation instruction, the display device 200 can extract initial keywords from the target description text. These initial keywords may include keywords representing character names, keywords representing locations, and keywords representing events or themes.

[0096] like Figure 9 As shown, in some embodiments, when the display device 200 extracts initial keywords from the target description text, it can construct a request template based on the target description text (S901). The request template includes the target description text and instructions for extracting initial keywords. The initial keywords include a first keyword and a second keyword. The first keyword is used to characterize the role of the target description text, and the second keyword is used to characterize the theme of the target description text.

[0097] For example, if the target description text associated with the picture book generation instruction is "Little Rabbit Picking Mushrooms", then the first keyword is "Little Rabbit" and the second keyword is "picking mushrooms"; if the target description text is "Xiao Wang's Workplace Life", then the first keyword is "Xiao Wang" and the second keyword is "workplace life"; if the target description text is "The Running Race Between Little Cow and Little Horse", then the first keyword is "Little Cow" and "Little Horse" and the second keyword is "running race".

[0098] After the request template is constructed, it is input into the Large Language Model (LLM model) (S902), which is used to perform context learning, instruction regulation, and task reasoning. Then, the first response data returned by the LLM model based on the request template is obtained (S903), and the first keyword and second keyword contained in the first response data are parsed (S904).

[0099] For example, the display device 200 can construct a request template that conforms to the LLM interface requirements based on the target description text. The request template includes instructions for extracting story characters and themes, as well as the target description text as context. The LLM model is then invoked, sending the constructed request template to the LLM model and waiting for its response. Upon receiving the response from the LLM model (such as a JSON-formatted string), the response is parsed to obtain the data, namely the story characters and themes.

[0100] In some embodiments, the story characters in the picture book media may include protagonists and supporting characters. There may be one or more protagonists and one or more supporting characters. The display device 200 may extract and process different story characters separately, so that the presentation effect of a single character can be consistent.

[0101] In some embodiments, if the target description text does not include a story character, the large language model of the display device 200 will return a specific character or an empty first keyword to indicate that the target description text associated with the current picture book generation instruction does not include a story character. For example, if the target description text is "The Wonderful Adventures of the Candy House", the LLM model of the display device 200 will return "NONE" and "The Adventures of the Candy House", where "NONE" is used to indicate that the story protagonist, i.e., the first keyword, is empty.

[0102] To distinguish between the first keyword and the second keyword, in some embodiments, the large language model of the display device 200 can return the first keyword and the second keyword in a preset format. For example, {"role":"X", "input":"X"}, where role is the story character and input is the story theme.

[0103] S603: Obtain the first feature information and the second feature information based on the initial keywords.

[0104] After extracting the initial keywords, the display device 200 can obtain first feature information and second feature information through the initial keywords. The first feature information can be used to describe character characteristics, and the second feature information can be used to describe theme characteristics. That is, the display device 200 can generate first feature information based on the first keyword and second feature information based on the second keyword. The first and second feature information are feature information expanded from the initial keywords. The first feature information may include the character's gender, facial features, skin color, clothing, personality, etc.

[0105] For example, when the first keyword extracted by the display device 200 is "rabbit", the first feature information generated by the display device 200 may include "rabbit", "gray fur", "female", "three-lobed mouth", "red eyes", "three whiskers", "white shorts", "naughty", "cute", "two rabbit teeth", etc.

[0106] like Figure 10 As shown, in scenarios where the first response data contains a first keyword, in some embodiments, when the display device 200 obtains the first feature information based on the initial keyword, it can query preset feature information based on the first keyword (S1001). The preset feature information refers to the character feature information preset in the picture book application of the display device 200. For example, the display device 200 may have preset feature information corresponding to "Sun Wukong," including "Sun Wukong, monkey, male, bright eyes, pointed ears, wide mouth, round head, moderate height, strong, golden battle robe, red belt, golden headband, brave, mischievous, righteous," etc.

[0107] After querying the preset feature information based on the first keyword, if the preset feature information includes the target information corresponding to the first keyword, it means that the display device 200 has stored the feature information (i.e., target information) corresponding to the current story character, and can extract the target information to obtain the first feature information (S1002); if the preset feature information does not include the target information corresponding to the first keyword, it means that the display device 200 has not stored the feature information corresponding to the current story character, and the display device 200 can generate the feature information of the first keyword through the large language model to obtain the first feature information (S1003).

[0108] In some embodiments, after the display device 200 generates the first feature information through the large language model, it can associate and store the first feature information with the first keyword, and store the first feature information generated by the large language model as the preset feature information corresponding to the first keyword.

[0109] In scenarios where the first response data does not contain the first keyword, in some embodiments, when the display device 200 obtains the first feature information based on the initial keyword, it can randomly access preset feature information (S1004) and extract the accessed preset feature information to obtain the first feature information (S1005).

[0110] In other words, when the target description text associated with the picture book generation instruction does not include keywords of the story character, that is, when the target description text does not include the first keyword, the display device 200 can randomly select a character from the preset feature information as the protagonist of the picture book media asset and generate the picture book media asset corresponding to that protagonist.

[0111] In scenarios where the first response data does not contain the first keyword, in some embodiments, the display device 200 can also display a character selection control in the application interface of the picture book application. Users can use the character selection control to select a story character corresponding to preset feature information or input user-defined character information.

[0112] In some embodiments, the display device 200 may also input the second keyword into a large language model to generate second feature information corresponding to the second keyword. It is understood that the principle of the large language model generating feature information is the same, and can be referred to the above embodiments; therefore, it will not be repeated here.

[0113] In some embodiments, the display device 200 may also directly label the extracted first keyword as first feature information, and / or directly label the extracted second keyword as second feature information. That is, the extracted keywords are directly used as feature information without any other processing.

[0114] S604: Generate a prompt text based on the first feature information and the second feature information.

[0115] After acquiring the first feature information and the second feature information, the display device 200 can generate prompt text based on the first feature information and the second feature information. The prompt text is the picture book story text, used to describe the plot of the picture book media. The prompt text can be generated in the form of segments, so that the display device 200 can generate image frames for different segments of the plot.

[0116] In some embodiments, when the display device 200 generates prompt text based on the first feature information and the second feature information, it can construct a prompt word template based on the first feature information and the second feature information. The prompt word template includes the first feature information, the second feature information, and instructions for generating the prompt text. The prompt word template is then input into a large language model, and second response data returned by the large language model based on the prompt word template is obtained. Then, the initial prompt text in the second response data is parsed, and segmentation processing is performed on the initial prompt text to generate the final prompt text. The initial prompt text is continuous text, i.e., text that has not been segmented.

[0117] For example, after acquiring character features (first feature information) and theme features (second feature information), the display device 200 generates a prompt word template: a prompt word template for the picture book story is generated based on the character features and theme features. The prompt word template may include introducing characters, setting scenes, and proposing tasks. The generated prompt word template is then sent as input to the LLM model, which generates continuous text describing the storyline (initial prompt text) based on the prompt word template and returns it to the display device 200 for processing. The display device 200 then segments the generated continuous text into paragraphs to ensure that each paragraph is relatively independent, forming the final prompt text.

[0118] In some embodiments, after the display device 200 performs segmentation processing on the initial prompt text, it also performs detection on the initial prompt text according to preset rules, filtering out text content that does not meet the requirements, in order to generate the final prompt text. For example, filtering rules can be set according to preset age of picture book media asset viewers, and text content that does not meet the requirements can be deleted according to the filtering rules.

[0119] S605: Generate a first image frame according to the first feature information and the prompt text.

[0120] After the display device 200 generates the prompt text, it can generate the first image frame using the first feature information and the prompt text. That is, it generates the corresponding image frame based on the character characteristics and the text describing the storyline. Each prompt text segment can generate one image frame, meaning each image frame can serve as a separate scene in the picture book media asset.

[0121] like Figure 11 As shown, in some embodiments, the display device 200 can generate target keywords based on the first feature information and the prompt text (S1101), and then input the target keywords into the diffusion model so that the diffusion model generates a first image frame by denoising the target keywords (S1102). The diffusion model is a probability distribution transformation-based model, i.e., a diffusion model. Then, the first image frame returned by the diffusion model based on the target keywords is obtained (S1103). In other words, the display device 200 can combine character features and generated prompt text used to describe the storyline into keywords, and generate corresponding image frames for each prompt text using the diffusion model.

[0122] In scenarios where the diffusion model supports image input, in some embodiments, the display device 200 can also generate a reference image based on feature information used to describe character characteristics, and input the reference image and prompt text into the diffusion model, so that the diffusion model generates a first image frame according to the reference image and prompt text.

[0123] In some embodiments, when the display device 200 generates a reference image based on feature information describing character features, it can also generate multiple images of different dimensions based on the feature information describing character features, and input the generated multiple images as reference images into the diffusion model. For example, the feature information describing character features includes features of the front, back, and sides of the character. The display device 200 can classify the above features and generate multiple images of the front, back, and sides of the character to improve the consistency of the character presentation.

[0124] S606: Extract supplementary feature information from the first image frame, wherein the iterative feature information is generated based on the first feature information, the prompt text, and the supplementary feature information.

[0125] After the display device 200 generates a first image frame, it performs image understanding and feature extraction on the first image frame to detect the character features of the story characters in the first image frame. For ease of distinction, some embodiments of this application represent the character features extracted from the first image frame as supplementary feature information, that is, the supplementary feature information is the character features extracted based on the first image frame.

[0126] like Figure 12 As shown, in some embodiments, when the display device 200 extracts supplementary feature information of the first image frame, it can perform preprocessing on the first image frame to generate a preprocessed image. Then, it extracts key features from the preprocessed image based on an image recognition algorithm. It queries the first feature information and determines supplementary feature information based on the first feature information and the key features, and then extracts the supplementary feature information. The supplementary feature information consists of key features not included in the first feature information. That is, the supplementary feature information consists of role features that differ from the first feature information among all role features extracted from the first image frame; in other words, the supplementary feature information is a newly added role feature obtained by the display device 200 through feature recognition of the first image frame.

[0127] S607: Generate a second image frame according to the iterative feature information, wherein the iterative feature information is generated based on the first feature information, the prompt text, and the supplementary feature information.

[0128] After extracting the supplementary feature information, the display device 200 can generate iterative feature information based on the first feature information, the prompt text, and the supplementary feature information, and then generate a second image frame according to the iterative feature information. That is, the display device 200 can iterate the first feature information using the supplementary feature information.

[0129] like Figure 11As shown, in some embodiments, the display device 200 can append supplementary feature information to the first feature information to generate iterative feature information (S1104). For example, for character features, the first feature information input to the diffusion model when generating the first image frame is "rabbit," "white fur," and "short tail"; the image generated by the diffusion model adds the character feature of "white shorts" in addition to the above features, then the character feature of "white shorts" is supplementary feature information. The display device 200 then appends this supplementary feature information to the first feature information to form iterative feature information: "rabbit," "white fur," "short tail," and "white shorts."

[0130] After generating iterative feature information, the display device 200 generates iterative keywords based on the iterative feature information and prompt text (S1105), and inputs the iterative keywords into the diffusion model so that the diffusion model generates a second image frame by denoising the iterative keywords (S1106). Then, the second image frame fed back by the diffusion model based on the iterative keywords is obtained (S1107).

[0131] In other words, when the display device 200 uses the prompt text in each segment to call the diffusion model to generate image frames, each time a new image frame is generated, the display device 200 performs image understanding and feature extraction on the newly generated image frame to extract the character features in the newly generated image. Then, the newly extracted character features are used to iterate the character features input to the model so that the character presentation effect of the preceding and following image frames can be consistent, thereby improving the coherence of the picture book media assets.

[0132] For certain special storylines, such as scenarios where character traits change in response to prompts in the text, the image frames generated after the picture book media assets also need to be changed accordingly. For example, if the character trait input by the large language model is "white shorts," and the prompt text indicates that the protagonist has changed into a yellow dress, the image frame generated by the prompt text needs to be changed to show "yellow dress."

[0133] Therefore, in some embodiments, during the generation of a first image frame or a second image frame, the display device 200 may also detect whether there are mutually exclusive features between the first feature information and the prompt text, or iterate whether there are mutually exclusive features between the feature information and the prompt text. If there are mutually exclusive features between the first feature information and the prompt text, the first feature information is iterated according to the prompt text, and the first image frame is generated according to the iterated feature information and the prompt text. Similarly, if there are mutually exclusive features between the first feature information and the prompt text, the display device 200 iterates the iterated feature information according to the prompt text, and generates a second image frame according to the iterated feature information and the prompt text. In this way, the picture book media assets generated by the display device 200 can correctly change along with the prompt text.

[0134] S608: Synthesize picture book media based on the first image frame and the second image frame.

[0135] After generating the first image frame and the second image frame through the above steps S601-S607, the display device 200 can synthesize the final picture book media assets based on the first image frame and the second image frame. The picture book media assets synthesized by the display device 200 can be video data or image data.

[0136] In some embodiments, the display device 200 further detects whether the prompt text generates a corresponding image frame. If all prompt texts generate image frames, the display device 200 executes step S608; otherwise, it continues to generate image frames using the prompt texts and feature information until all prompt texts have generated corresponding image frames.

[0137] like Figure 13 As shown, to improve the generation effect of picture book media assets, in some embodiments, the display device 200 can generate audio playback data based on the prompt text (S1301) and generate a subtitle file based on the prompt text (S1302). Steps S1301 and S1302 can be processed in parallel or sequentially. In sequential processing, the display device 200 can execute S1301 first or S1302 first; this application does not limit this. Then, according to the prompt text, the audio playback data, subtitle file, first image frame, and second image frame are synthesized to generate the picture book media assets (S1303).

[0138] In some embodiments, when generating audio playback data, the display device 200 can also annotate the prompt text with pitch according to its content. The pitch annotations are used to present different emotional sound effects. The display device 200 can match the corresponding pitch annotations to the content of each segment of the prompt text and generate audio playback data according to the pitch annotations, so that the generated sound data can contain rich emotions, thereby improving the quality of the picture book media assets. For example, if the prompt text generated by the display device 200 contains scenes of a rabbit picking mushrooms and a rabbit encountering a big bad wolf, the display device 200 can annotate the scene describing the rabbit picking mushrooms with a pitch annotation to present a lighthearted mood; and the display device 200 can annotate the scene describing the rabbit encountering the big bad wolf with a pitch annotation to present a tense mood, and so on.

[0139] In some embodiments, when compositing picture book media assets, the display device 200 can also match background music suitable for the picture book media assets according to the prompt text and synthesize the background music into the picture book media assets to enhance the richness of the picture book media assets. For example, background music that matches the theme features or the semantics of the prompt text.

[0140] Furthermore, since the generation of picture book media assets may take some time, in some embodiments, the display device 200 can also respond to the picture book generation command, calculate the predicted generation time of the picture book media assets, and display a generation process prompt interface through the picture book application control display 260. The picture book generation process prompt interface is used to display a message indicating that the picture book media assets are being generated, as well as the predicted generation time. For example, as... Figure 14 The illustrated picture book generation process prompt interface shows how users can... Figure 14 The application interface shown indicates that the current picture book application is generating picture book media assets, and obtains the estimated generation time information of the picture book media assets.

[0141] In some embodiments, after the display device 200 completes the synthesis of the picture book media assets, it can control the display 260 to display the generated result interface through a picture book application. The generated result interface may include a preview image of the picture book media assets, playback controls for the picture book media assets, etc. For example, as... Figure 15 As shown, the generated result interface of the picture book media asset may include a playback control 1501 and a preview image 1502. The preview image 1502 is a view that does not support manipulation and may be an image frame in the picture book media asset. The playback control 1501 is a view that supports selection operations. Users can play the picture book media asset by performing selection operations on the playback control 1501.

[0142] In some embodiments, the display device 200 can also annotate the prompt text for scenes in picture book media assets, such as labeling the same scenes with the same tags or annotation information. Then, it can update and iterate the text using the scene's feature information to ensure consistent presentation of the same scene.

[0143] For example, the home of the story character in the prompt text generated by the display device 200 may include the interior of the bedroom and the interior of the living room. The display device 200 can generate the same scene for the bedroom interior and the same scene for the living room interior using the above-mentioned annotations and feature information.

[0144] In scenarios where there are multiple characters in a picture book media asset, similarly in some embodiments, when the picture book media asset includes multiple characters, the display device 200 can perform the above processing on each character separately so that each character can maintain a consistent presentation effect, thereby improving the coherence of the picture book media asset.

[0145] In some embodiments, the display device 200 can save the generated picture book media assets in local storage or upload them to a cloud server for storage, allowing users to play them repeatedly. When storing picture book media assets, the display device 200 can also associate the assets with specific story titles, allowing users to query and retrieve the assets by story title.

[0146] In some embodiments, the picture book application also includes a sharing control for picture book media assets. After the user performs a selection operation on the sharing control, the display device 200 can generate a sharing message including the URL of the picture book media asset associated with the sharing control. The user can share the picture book media asset with other devices, such as other mobile terminals configured with the picture book application, through the sharing message generated by the display device 200.

[0147] Based on the above embodiments, the display device 200 can expand the keywords describing a character to generate corresponding character features. Then, by performing image understanding and feature extraction on each image frame, and iterating the original character features using the extracted features, such as... Figure 16 As shown, each image frame in the picture book media asset generated by the display device 200 can present a consistent display effect.

[0148] Based on the aforementioned display device 200, some embodiments of this application also provide a method for generating picture book media assets, which can be applied to the display device 200 provided in the above embodiments, and can also be applied to other terminal devices. It should be noted that the terminal device to which the generation method is applied can be hardware or specific software, and this application does not limit this. Figure 6 As shown, the generation method may include the following steps:

[0149] S601: In response to a picture book generation instruction input by the user, detect the target description text, wherein the picture book generation instruction is an interactive instruction input based on the application interface of the picture book application, and the target description text is the description text associated with the picture book generation instruction;

[0150] S602: Extract the initial keywords from the target description text;

[0151] S603: Obtain first feature information and second feature information based on the initial keywords. The first feature information is used to describe the role features of the target descriptive text, and the second feature information is used to describe the topic features of the target descriptive text.

[0152] S604: Generate a prompt text based on the first feature information and the second feature information;

[0153] S605: Generate a first image frame according to the first feature information and the prompt text;

[0154] S606: Extract supplementary feature information of the first image frame, wherein the supplementary feature information is the character feature extracted based on the first image frame;

[0155] S607: Generate a second image frame according to iterative feature information, wherein the iterative feature information is generated based on the first feature information, the prompt text, and the supplementary feature information;

[0156] S608: Synthesize picture book media based on the first image frame and the second image frame.

[0157] As can be seen from the above technical solutions, the display device and picture book media asset generation method provided in some embodiments of this application can respond to a picture book generation instruction, detect the target description text associated with the picture book generation instruction, extract the initial keywords of the target description text, and then obtain first feature information and second feature information based on the initial keywords. The first feature information is used to describe character features, and the second feature information is used to describe theme features. Then, prompt text is generated based on the first and second feature information, and a first image frame is generated based on the first feature information and the prompt text. Supplementary feature information of the first image is extracted, and the first feature information is iterated based on the supplementary feature information. A second image frame is then generated according to the iterated feature information, so as to synthesize the picture book media asset based on the first and second image frames. The method extracts features from the generated image frames and iterates on the character features using the extracted feature information, which can ensure that the character effects of each image frame in the picture book media asset remain consistent, thereby improving the generation quality of the picture book media asset.

[0158] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0159] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0161] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the application interface of the picture book application. The controller is configured as follows: In response to a picture book generation command input by the user based on the application interface, target description text is detected, wherein the target description text is the description text associated with the picture book generation command; Extract initial keywords from the target description text; Based on the initial keywords, first feature information and second feature information are obtained. The first feature information is used to describe the role characteristics of the target descriptive text, and the second feature information is used to describe the topic characteristics of the target descriptive text. Generate prompt text based on the first feature information and the second feature information; Generate a first image frame according to the first feature information and the prompt text; Extract supplementary feature information from the first image frame, wherein the supplementary feature information is the character feature extracted based on the first image frame; A second image frame is generated based on iterative feature information, wherein the iterative feature information is generated according to the first feature information, the prompt text, and the supplementary feature information; The picture book media is synthesized based on the first image frame and the second image frame.

2. The display device according to claim 1, characterized in that, The controller is configured to extract initial keywords from the target description text. A request template is constructed based on the target description text. The request template includes the target description text and instructions for extracting the initial keywords. The initial keywords include a first keyword and a second keyword. The first keyword is used to characterize the role of the picture book media asset, and the second keyword is used to characterize the theme of the picture book media asset. The request template is input into a large language model, which is used to perform context learning, instruction regulation, and task reasoning. Obtain the first response data returned by the large language model based on the request template; The first keyword and the second keyword contained in the first response data are parsed to generate the first feature information based on the first keyword and the second feature information based on the second keyword.

3. The display device according to claim 2, characterized in that, The controller is configured to obtain first feature information based on the initial keywords, specifically as follows: When the first response data contains the first keyword, query the preset feature information based on the first keyword; If the preset feature information includes target information corresponding to the first keyword, extract the target information to obtain the first feature information; If the preset feature information does not include the target information corresponding to the first keyword, the feature information of the first keyword is generated through the large language model to obtain the first feature information.

4. The display device according to claim 2, characterized in that, The controller is configured to obtain first feature information based on the initial keywords, specifically as follows: When the first response data does not contain the first keyword, preset feature information is randomly accessed; Extract the accessed preset feature information to obtain the first feature information.

5. The display device according to claim 1, characterized in that, The controller generates prompt text based on the first feature information and the second feature information, specifically configured as follows: A prompt word template is constructed based on the first feature information and the second feature information. The prompt word template includes the first feature information, the second feature information, and instructions for generating prompt text. The prompt word template is input into a large language model, which is used to perform context learning, instruction fine-tuning, and reasoning tasks. Obtain the second response data returned by the large language model based on the prompt word template; Parse the initial prompt text of the second response data, wherein the initial prompt text is continuous text; The initial prompt text is segmented to generate the new prompt text.

6. The display device according to claim 1, characterized in that, The controller is configured to generate a first image frame according to the first feature information and the prompt text, specifically as follows: Generate target keywords based on the first feature information and the prompt text; The target keywords are input into the diffusion model, so that the diffusion model generates the first image frame by denoising the target keywords; The diffusion model is based on probability distribution transformation; Obtain the first image frame returned by the diffusion model based on the target keyword.

7. The display device according to claim 1, characterized in that, The controller is configured to extract supplementary feature information from the first image frame, specifically as follows: Preprocessing is performed on the first image frame to generate a preprocessed image; Key features of the preprocessed image are extracted based on image recognition algorithms; Query the first feature information; Supplementary feature information is determined based on the first feature information and the key feature, wherein the supplementary feature information is the key feature not included in the first feature information; Extract the supplementary feature information.

8. The display device according to claim 1, characterized in that, The controller is configured to generate a second image frame based on iterative feature information, specifically as follows: The supplementary feature information is appended to the first feature information to generate the iterative feature information; Generate iterative keywords based on the iterative feature information and the prompt text; The iterative keywords are input into the diffusion model, so that the diffusion model generates the second image frame by denoising the iterative keywords; the diffusion model is a model based on probability distribution transformation. Obtain the second image frame returned by the diffusion model based on the iterative keywords.

9. The display device according to claim 1, characterized in that, The controller is configured to synthesize picture book media assets based on the first image frame and the second image frame, specifically as follows: Audio playback data is generated based on the prompt text, and a subtitle file is generated based on the prompt text; According to the prompt text, the audio broadcast data, the subtitle file, the first image frame, and the second image frame are synthesized to generate the picture book media asset.

10. A method for generating picture book media assets, characterized in that, include: In response to a user-inputted picture book generation command, the target description text is detected. The picture book generation command is an interactive command input based on the application interface of the picture book application, and the target description text is the description text associated with the picture book generation command. Extract initial keywords from the target description text; First feature information and second feature information are obtained based on the initial keywords. The first feature information is used to describe the character features, and the second feature information is used to describe the theme features. Generate prompt text based on the first feature information and the second feature information; Generate a first image frame according to the first feature information and the prompt text; Extract supplementary feature information from the first image frame, wherein the supplementary feature information is the character feature extracted based on the first image frame; A second image frame is generated based on iterative feature information, wherein the iterative feature information is generated according to the first feature information, the prompt text, and the supplementary feature information; The picture book media is synthesized based on the first image frame and the second image frame.

Images