Multimodal content generation and playing method and device of teaching picture book and electronic equipment

Abstract

The present application relates to the technical field of artificial intelligence content generation, and discloses a multi-modal content generation and playing method and device for teaching picture books, an electronic device and a computer readable medium. The method obtains educational theme information and theme materials from an educational theme library according to an access request, and generates a scene script sequence. According to the role description of each scene script, a role reference image is generated, and a scene image is generated with the role reference image as an initial image. The audio corresponding to the script text is obtained, and the script text, the role description, the scene description, the role reference image, the scene image and the audio are written into a scene record according to the scene identifier. The scene playing sequence is determined according to the scene script sequence, the scene image is displayed, and the corresponding audio is played. According to the scene description, the method also matches the pre-stored fixed video materials, and selects the playing resources according to the loading state, so that the scene corresponding relationship of the multi-modal content can be determined, and the change of the visual attributes of the roles in different scenes can be reduced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence-generated content and computer multimedia processing technology. Specifically, it relates to a method for generating and playing multimodal content for educational picture books, a device for generating and playing multimodal content, an electronic device, and a computer-readable medium. Background Technology

[0002] Digital educational picture books can organize script text, scene images, and audio readings according to teaching themes, and can be displayed and played in scene order on mobile phones, tablets, personal computers, or smart display devices. Existing educational picture books usually use pre-edited scripts and images, or generate content based on user-input themes by calling a generative model.

[0003] When generating picture book content based on a teaching theme, the user-input theme text is usually short. Directly generating a script based on this theme text can easily lead to problems such as the generated content lacking pedagogical knowledge support or being insufficiently related to the target teaching theme. When generating images for different scenes, the characters in each scene may also change in appearance, clothing, or color characteristics, resulting in a lack of visual continuity for the same character in the teaching picture book.

[0004] Furthermore, script text, character descriptions, scene descriptions, scene images, and audio are generated through different processing steps, and each type of content needs to be associated with a specific scene. Organizing playback resources solely based on file generation time or return order can easily lead to scene mismatches between script text, images, and audio. For pre-stored video footage, it is also necessary to match it with the corresponding scene and determine whether to display images or video footage based on the resource loading status.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a method, device, electronic device, and computer-readable medium for generating and playing multimodal content in educational picture books, thereby solving, to at least a certain extent, the technical problems such as insufficient utilization of teaching theme information, discontinuous visual expression of characters in different scenes, and unclear scene correspondence between multimodal content.

[0007] According to one aspect of the present invention, a method for generating and playing multimodal content for educational picture books is provided. The method includes: obtaining an access request for a target teaching topic; determining topic retrieval information based on topic input text in the access request; obtaining educational topic information and topic materials corresponding to the target teaching topic from an educational topic database based on the topic retrieval information, and establishing a content generation instance corresponding to the current access; performing script generation processing based on the educational topic information and topic materials to obtain a sequence of scene scripts arranged in playback order, each scene script including a scene identifier, script text, character description, and scene description; determining a target character description based on the character descriptions in each scene script; performing character image generation processing based on the target character description to obtain a character image that has a mapping relationship with the target character description. Color reference image; for each scene script, the character reference image is used as the initial image, and image-to-image processing is performed according to the scene description in the scene script to obtain a scene image that has a mapping relationship with the character reference image and the scene description in the scene script; audio corresponding to the script text in each scene script is obtained; for each scene script, according to the scene identifier in the scene script, the corresponding script text, character description, scene description, character reference image, scene image and audio are written into the scene record identified by the scene identifier; the scene playback sequence is determined according to the scene script sequence, and the scene images in each scene record and the audio in the same scene record are displayed according to the scene playback sequence.

[0008] According to one aspect of the present invention, a multimodal content generation and playback device for educational picture books is provided. The device includes: an access acquisition module configured to acquire an access request for a target teaching topic; a topic retrieval module configured to determine topic retrieval information based on topic input text in the access request, acquire educational topic information and topic materials corresponding to the target teaching topic from an educational topic database based on the topic retrieval information, and establish a content generation instance corresponding to the current access; a script generation module configured to perform script generation processing based on the educational topic information and topic materials to obtain a sequence of scene scripts arranged in playback order; and a character image generation module configured to determine a target character description based on the character descriptions in each scene script, and generate character images based on the target character descriptions. The system generates character reference images; a scene image generation module is configured to use the character reference image as the initial image for each scene script and perform image-to-image processing based on the scene description in the scene script to obtain the scene image; a multimodal resource binding module is configured to acquire the audio corresponding to the script text in each scene script and write the script text, character description, scene description, character reference image, scene image, and audio for each scene into the scene record identified by the scene identifier; and a playback module is configured to determine the scene playback sequence based on the scene script sequence and display the scene images in each scene record and play the audio in the same scene record according to the scene playback sequence.

[0009] In some embodiments of the present invention, based on the above technical solutions, the topic input text can be obtained from the topic text or teaching content text input by the user, or it can be obtained through speech recognition of the topic speech input by the user. Topic retrieval information can be determined based on the topic input text and the subject, language, and style selected by the user.

[0010] In some embodiments of the present invention, based on the above technical solutions, the educational topic database includes a topic knowledge base and a topic material database. Based on topic retrieval information, topic knowledge texts corresponding to the target teaching topic are determined in the topic knowledge base, and candidate topic materials are determined in the topic material database based on the topic retrieval information. Then, the topic material currently being accessed and used is determined from the candidate topic materials.

[0011] In some embodiments of the present invention, based on the above technical solutions, when an access request for the same target teaching topic is obtained again, a new content generation instance corresponding to the re-access is established, a random seed corresponding to the re-access is generated, the topic material to be used for the re-access is determined from the candidate topic materials according to the random seed, and script generation processing, character image generation processing, and scene image image generation processing are performed according to the random seed and the topic material to be used for the re-access.

[0012] In some embodiments of the present invention, based on the above technical solutions, character prompt information representing character identity and appearance is obtained from the target character description. Scene image prompt information corresponding to each scene identifier is determined according to the scene description and character prompt information in each scene script. The character reference image is used as the initial image for each scene, and the initial image is transformed according to the scene image prompt information to obtain the scene image for each scene. The scene image prompt information for different scenes includes at least partially identical character prompt information.

[0013] In some embodiments of the present invention, based on the above technical solutions, the audio corresponding to the script text can be returned along with the scene script, or it can be obtained by performing audio generation processing on the script text. The scene identifier, scene order, script text, character description, scene description, character reference image address, scene image address, and audio address are written into the scene record identified by the scene identifier, and one or more scene records are associated with the corresponding content generation instance.

[0014] In some embodiments of the present invention, based on the above technical solutions, for a target scene in a scene playback sequence, a scene identifier of the target scene is obtained, video retrieval information is determined according to the scene description in the scene record identified by the scene identifier, and fixed video material with a mapping relationship to the target scene is obtained from the video material library according to the matching relationship between the video retrieval information and the video material tag, and the video material identifier or video address of the fixed video material is written into the scene record identified by the scene identifier.

[0015] In some embodiments of the present invention, based on the above technical solutions, a video material loading task is established for fixed video materials, and the number of concurrent video material loading tasks or the status query interval is determined according to the current network status. When the fixed video materials have not been loaded, the scene image in the scene record identified by the scene identifier of the target scene is displayed and the audio in the same scene record is played; when the fixed video materials are loaded, the fixed video materials are displayed when entering the target scene or reaching the scene switching node of the target scene.

[0016] According to one aspect of the present invention, an electronic device is provided. The electronic device includes a processor and a memory, the memory storing executable instructions of the processor, and the processor being configured to execute the multimodal content generation and playback method described above by executing the executable instructions.

[0017] According to one aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the multimodal content generation and playback method described above.

[0018] In the technical solution provided by the embodiments of the present invention, topic retrieval information is determined based on the topic input text, and educational topic information and topic materials are obtained from the topic knowledge base and topic material base respectively, so that the input of the script generation process includes knowledge information and material information corresponding to the target teaching topic, thereby maintaining the data association between the teaching topic and the generated content during the generation process.

[0019] Based on the character descriptions in each scene script, the target character description is determined. First, a character reference image is generated based on the target character description. Then, this character reference image is used as the initial image for each scene during image-to-image processing. As a result, each scene image shares the character reference image and at least some of the same character cue information, reducing the visual attribute variations caused by generating characters separately for each scene.

[0020] By writing script text, character descriptions, scene descriptions, character reference images, scene images, and audio into the corresponding scene records through scene identifiers, and determining the scene playback sequence based on the scene script sequence, the multimodal content obtained from different processing processes can establish a clear data correspondence according to the scene.

[0021] Fixed video clips are pre-stored in a video clip library. Video retrieval information is determined based on the scene description of the target scene and matched with video clip tags, eliminating the need to generate video on demand for the current access request. Scene images and audio are displayed while the fixed video clips are not fully loaded; the fixed video clips are displayed at scene entry points or scene transition points after loading, allowing the playback process to adapt to different resource loading states.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] In the attached diagram: Figure 1 An exemplary system architecture diagram illustrating the application of the technical solution of the present invention is shown.

[0025] Figure 2 The flowchart illustrating the steps of the method for generating and playing multimodal content of educational picture books in some embodiments of the present invention is shown.

[0026] Figure 3 The diagram illustrates the data relationship between content generation instances, scene script sequences, and scene records in some embodiments of the present invention.

[0027] Figure 4 The flowchart illustrating the generation process of character reference images and scene images in some embodiments of the present invention is shown schematically.

[0028] Figure 5 The flowchart illustrating the matching, loading, and playback resource selection of fixed video materials in some embodiments of the present invention is shown schematically.

[0029] Figure 6 The diagram illustrates the interface flow of a terminal device in an embodiment of the present invention. Detailed Implementation

[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0031] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Specific details are provided in the following description to give a thorough understanding of the embodiments of the invention. Those skilled in the art can implement the technical solutions of the invention without one or more of the specific details, and may also employ other methods, components, apparatuses, or steps. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring the technical solutions of the invention.

[0032] The block diagrams shown in the attached figures are merely functional entities and do not necessarily correspond to physically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network, processor, or microcontroller devices.

[0033] The term "target teaching theme" as used in this article refers to the teaching theme object targeted in the current educational picture book generation process. The term "theme input text" as used in this article refers to the theme text or teaching content text directly entered by the user, or the text obtained through speech recognition from the user's input of the theme's voice. The target teaching theme and theme input text respectively represent the processing object and the text input, and should not be understood as the same data object.

[0034] The "educational topic library" referred to in this article includes a topic knowledge base and a topic material library. The topic knowledge base stores topic knowledge text, while the topic material library stores topic materials that correspond to different teaching topics. The video material library stores fixed video materials and is a different data resource from the topic material library. Topic materials are used for script generation or image generation, while fixed video materials are used for scene playback.

[0035] The term "content generation instance" as used in this document refers to a data organization object corresponding to a single access. A single access can correspond to one content generation instance, which may include an instance identifier, current access association information, and one or more scenario records. The current access association information may include at least one of the following: session identifier, request sequence number, access time, or user terminal identifier.

[0036] The term "character description" in this article refers to the raw descriptive data used to describe the corresponding characters in each scene script. "Target character description" is descriptive data determined based on the character descriptions in each scene script and used to generate a common character reference diagram. "Character hints" are obtained from the target character description and are used to characterize the character's identity and appearance. Character descriptions, target character descriptions, and character hints are different data sets with a sequential processing relationship.

[0037] The term "scene description" in this article refers to descriptive data in the scene script used to characterize the scene environment, character actions, or visual content. "Scene image prompts" are determined based on the scene description and character prompts, and are used to generate scene images. "Video retrieval information" is determined based on the scene description in the scene record identified by the target scene's scene identifier, and is used to match fixed video footage in the video footage library. Scene image prompts and video retrieval information are used in different processing branches.

[0038] The "fixed video content" referred to in this article refers to video content pre-stored in the video content library before the current access request is received, and is not the video content generated in response to the current access request. "Video content tags" are used to characterize the content category and applicable scenario category of the fixed video content. "Video content identifiers" are used to identify fixed video content, and "video addresses" are used to locate or access fixed video content.

[0039] Figure 1 An exemplary system architecture diagram illustrating the application of the technical solution of the present invention is shown schematically. For example... Figure 1 As shown, a multimodal content generation and playback system may include a user terminal, a server, an educational topic library, a video material library, and an artificial intelligence processing engine. The user terminal and the server can communicate via wired or wireless networks.

[0040] The user terminal can be a smartphone, tablet, personal computer, smart display device, or other electronic device capable of receiving user input and outputting images and audio. The user terminal can acquire the user's input of topic text, teaching content text, or topic audio, and obtain the user's selected subject, language, and art style to generate an access request.

[0041] In one client-side implementation, the user terminal is an iPad running iOS, and the client-side application is implemented using Objective-C and the UIKit framework. The application lifecycle is managed by AppDelegate. SceneDelegate creates a UIWindow, sets KiddoStoryBookViewController (used to generate the homepage) as the root view controller, and embeds it into UINavigationController. After generating the homepage and completing content request and response parsing, StoryBookDisplayViewController, carrying the picture book data model, is pushed into the navigation controller to enter the landscape picture book reading interface.

[0042] The KiddoStoryBookViewController displays the topic input box, voice input button, sample cards, and subject, language, and art style filter controls. When the user clicks the voice input button, the client requests voice recognition and microphone permissions, uses AVAudioEngine to capture audio buffer data, and obtains the voice-recognized text through SFSpeechRecognizer with the language region set to zh-CN. The client then fills the topic input box with the voice-recognized text and uses it as the topic input text.

[0043] In this client-side implementation, StoryBookDisplayViewController serves as the reader container, internally creating a UIPageViewController to manage multiple pages. StoryBookPageViewController is used to display the script text and scene images for a single page. The reader also includes an audio session and a bottom control bar, providing controls for previous page, next page, play, pause, and progress adjustment. When the reader enters display mode, it can switch to landscape mode and read the image, audio, or fixed video material corresponding to the current page.

[0044] Domain data on the client side can be carried by StoryBookModel and StoryBookPageModel. StoryBookModel can include storyName, cover, introduce, firstPage, and pages, where storyName stores the book title, cover and firstPage store cover-related data, introduce stores the book introduction, and pages stores an array of StoryBookPageModels arranged in playback order. StoryBookPageModel can include page, indexTag, content, imageUrl, audioUrl, and videoUrl, which are used to store the page number, page index, page text, scene image address, audio address, and fixed video address, respectively.

[0045] The server-side can include an access acquisition module, a topic search module, a script generation module, a character image generation module, a scene image generation module, a multimodal resource binding module, and a playback module. These modules can be deployed on the same server, or distributed across multiple servers or cloud computing nodes.

[0046] In one implementation, the client-side uses DataRequestManager to encapsulate the network processing for sending access requests to the server, and uses TTSRequestManager to encapsulate optional audio generation requests. Remote image, audio, and fixed video materials are then processed by the resource management logic in the reader. The access acquisition module, topic retrieval module, script generation module, character image generation module, scene image generation module, and multimodal resource binding module can also be encapsulated by a comprehensive content generation service, providing a unified business interface to the client-side.

[0047] The AI ​​processing engine may include at least one of a script generation model, an image generation model, and an audio generation model. The script generation model is used to generate scripts based on educational topic information and thematic materials; the image generation model is used to generate character images and process scene images; and the audio generation model is used to generate audio based on the script text. These models may be provided by the same model service or by different model services.

[0048] In some optional implementations, the script generation model can employ a neural network model with text generation capabilities, the image generation model can employ a diffusion model or other generation models with text-to-image and image conversion capabilities, and the audio generation model can employ a text-to-speech model. The specific network structure, number of parameters, and deployment method of the model can be configured according to server resources and application conditions, and this invention does not impose any special limitations on them.

[0049] Figure 2 The flowchart illustrating the steps of a multimodal content generation and playback method for educational picture books in some embodiments of the present invention is shown schematically. This method can be executed by the server, or jointly by the user terminal and the server. Figure 2 As shown, the method mainly includes the following steps.

[0050] Step S210: Obtain an access request for the target teaching topic.

[0051] In one implementation, the user terminal acquires the user-inputted topic text or teaching content text and identifies it as the topic input text. In another implementation, the user terminal acquires the user-inputted topic speech, performs speech recognition on the topic speech to obtain speech-recognized text, and identifies the speech-recognized text as the topic input text. The user terminal can also acquire the user-selected subject, language, and art style, and write the topic input text, subject, language, and art style into the access request.

[0052] In the iPad implementation described above, generating the homepage can be achieved by calling `DataRequestManager.sendPostRequest`, where the theme input text, art style, language, and subject are written into the `input`, `style`, `language`, and `subject` fields, respectively. The client-side can serialize these fields into `paramJson` and include the `apiCode` (used to identify the business interface) and a `stream` field (with a value of `false`) in the request body. Then, it uses AFNetworking or NSURLSession to call the `common / invoke` interface of the comprehensive content generation service via a POST request. In one specific implementation, this comprehensive content generation service can be deployed on Alibaba APaaS. Even if the user hasn't entered a theme text, the client-side can also write a pre-configured sample theme into the `input` field to provide a sample preview entry.

[0053] After receiving an access request, the content generation service can sequentially execute the steps of topic retrieval, script generation, character image generation, scene image generation, and audio acquisition within the service, and return a structured result. Upon receiving the response, the client-side first verifies the `code` and `success` fields in the response; when `code` is 200 and `success` is true, it reads the result string from the `data.result` field and performs JSON parsing on the result string again to obtain a `structData.output` object.

[0054] In an exemplary data contract, `structData.output` includes a `title` field and a `content` array. The `title` field stores the book title, and each element in the `content` array corresponds to a scene record and includes at least `text`, `image`, and `voice` fields. The client writes the `title` field to the `storyName` of the `StoryBookModel`, and writes the `text`, `image`, and `voice` fields from the `content` array to the `content`, `imageUrl`, and `audioUrl` of the `StoryBookPageModel`, respectively. The client can also set the `page` and `indexTag` according to the arrangement in the `content` array, and write the parsed multiple `StoryBookPageModel`s to the `pages` array of the `StoryBookModel`.

[0055] Step S220: Determine the topic retrieval information based on the topic input text in the access request, retrieve the educational topic information and topic materials corresponding to the target teaching topic from the educational topic database based on the topic retrieval information, and create a content generation instance corresponding to the current access.

[0056] The topic retrieval module can perform word segmentation, keyword extraction, or semantic representation processing on the topic input text, and combine the processing results with subject, language, and art style to form topic retrieval information. In one embodiment, the topic retrieval information may include topic keywords, subject identifiers, language identifiers, and art style identifiers. In another embodiment, the topic retrieval information may also include text feature representations obtained from the topic input text.

[0057] The topic retrieval module performs keyword retrieval, vector retrieval, or a hybrid retrieval combining keywords and vectors in the topic knowledge base based on the topic retrieval information to identify one or more topic knowledge texts that correspond to the target teaching topic. Multiple topic knowledge texts can be sorted according to search relevance, and a preset number of the top-ranked topic knowledge texts are identified as educational topic information.

[0058] The topic retrieval module also identifies candidate topic materials in the topic material library that correspond to the target teaching topic based on the topic retrieval information. Topic materials may include at least one of the following: plot elements for script generation, teaching examples, character elements, environmental elements, or image reference information. Topic materials may possess attribute information such as topic category, subject category, language category, or art style category.

[0059] In one implementation, the currently accessed topic material can be determined from the candidate topic materials based on the degree of matching between the candidate topic materials and the topic retrieval information. In another implementation, a random selection can be made from candidate topic materials that meet the matching conditions of topic category and subject category. Topic knowledge text is identified as educational topic information, and the selected topic material is identified as the topic material.

[0060] The topic retrieval module can organize educational topic information and topic materials into a retrieval-enhanced generation context, and then provide this context to the script generation module. The retrieval-enhanced generation context can include the target teaching topic, topic knowledge text, selected topic materials, subject, language, and art style. Therefore, subsequent script generation can utilize content obtained from the educational topic database, rather than relying solely on short text input by the user.

[0061] The topic search module also creates a content generation instance for the current access. The content generation instance stores an instance identifier and information associated with the current access. The instance identifier can be generated by the server, and the information associated with the current access can consist of at least one of the following: session identifier, request serial number, or access time.

[0062] Step S230: Based on the educational theme information and theme materials, script generation processing is performed to obtain a sequence of scene scripts arranged in the playback order.

[0063] The script generation module submits educational topic information, topic materials, and the subject, language, and art style from the access request to the script generation model. The script generation model returns one or more scene scripts according to a pre-defined output structure. Each scene script includes at least a scene identifier, script text, character description, and scene description.

[0064] In one implementation, the scene script may further include a scene order. The scene order can be directly output by the script generation model, or it can be determined by the script generation module based on the position of the scene scripts in the scene script sequence. Scene identifiers can be returned by the script generation model, or they can be generated by the script generation module according to the scene order.

[0065] Character descriptions describe the identity or appearance of characters in a scene, while scene descriptions describe the environment, character actions, or visual content of the scene identified by the scene identifier in the scene script. Therefore, character and scene descriptions can constitute visual description data directly output by the script generation process, without requiring the playback module to reconstruct it from the script text.

[0066] In one exemplary output structure, a scene script may include a scene identifier field, a scene order field, a script text field, a character description field, and a scene description field. The script text field stores the teaching content, the character description field stores the character's name, appearance, clothing, or color information, and the scene description field stores the scene environment and the character's actions within the scene. This output structure is merely an example; the field names can be adjusted according to the data interface.

[0067] When the integrated content generation service returns simplified page data to the client, the script generation module and the multimodal resource binding module can maintain scene identifiers, scene order, character descriptions, scene descriptions, character reference image addresses, and scene image addresses in the content generation instance and scene record on the server side, and only output the title and content arrays required for playback to the client side. The StoryBookPageModel on the client side and the scene record on the server side establish a correspondence through page, indexTag, or scene identifier, so that the simplified fields on the client side and the complete scene data on the server side can point to the same scene.

[0068] In other implementations, the client-side can also utilize StoryBookParser to parse text content organized according to preset tags, extracting story titles, page numbers, image addresses, and audio addresses. Structured JSON parsing and tagged text parsing can serve as two input adaptation methods; the parsed data is converted into StoryBookModel and StoryBookPageModel for display on the same reader.

[0069] Step S240: Determine the target character description based on the character description in each scene script, and perform character image generation processing based on the target character description to obtain a character reference image.

[0070] The character image generation module can iterate through the character descriptions in the scene script sequence to obtain the names of characters that appear repeatedly in each scene, along with their corresponding appearance, clothing, color, and body type information. For multiple descriptions of the same attribute in different scenes, the module can select the description that appears most frequently, the description that appears earlier in the scene sequence, or a description that is pre-specified as a basic attribute to form the target character description.

[0071] In another implementation, the script generation model can output basic character descriptions separately along with the scene script sequence. The character image generation module then identifies these basic character descriptions as the target character descriptions. The target character descriptions can describe a main character or multiple target characters that need to appear together in the character reference image.

[0072] The character image generation module obtains character hints from the target character description. Character hints may include at least one of the following: character name, physical features, clothing features, color features, and body shape features. The character image generation module submits the character hints and the art style from the access request to the image generation model for character image generation processing, resulting in a character reference image that maps to the target character description.

[0073] In one implementation, the character reference image can be an image containing only the target character with a simple background, so that subsequent image processing preserves the character's visual attributes. In another implementation, the character reference image can contain multiple target characters, maintaining a distinguishable positional relationship between each target character in the image. The character reference image is stored in the content generation instance, and the character reference image address is written to each scene record that needs to use the character reference image.

[0074] Step S250: For each scene script, the character reference image is used as the initial image, and image-to-image processing is performed according to the scene description in the scene script to obtain a scene image that has a mapping relationship with the character reference image and the scene description in the scene script.

[0075] The scene image generation module determines scene image prompts corresponding to each scene identifier based on the scene description and character prompts in each scene script. The scene image prompts may include at least partially identical character prompts, and include at least one of scene environment information and character action information associated with the corresponding scene.

[0076] The scene image generation module uses the character reference image as the initial image and the scene image prompt information as text conditions. It calls the image-to-image processing interface of the image generation model to perform image conversion, thereby obtaining a scene image that corresponds to the corresponding scene identifier. The image-to-image processing can control the degree of correlation between the initial image and the output image through image conversion intensity or prompt information weight. The above parameters can be configured according to the art style and scene type.

[0077] For multiple scenes in a scene script sequence, the scene image generation module reuses the same character reference image and reuses at least partially identical character prompts in the scene image prompts of different scenes. Different scene environment information and character action information can be added to different scenes to give the generated scene images different scene content while maintaining the identity and appearance attributes of the target character.

[0078] Figure 4 The flowchart illustrating the generation process of character reference images and scene images in some embodiments of the present invention is shown schematically. For example... Figure 4 As shown, the character descriptions in each scene script are used to determine the target character description, which is then used to generate the character reference image. For the first, second, and third scenes, the character reference image serves as the initial image. The descriptions of the first, second, and third scenes are combined with the character prompts to form corresponding scene image prompts, thereby generating the first, second, and third scene images.

[0079] Step S260: Obtain the audio corresponding to the script text in each scene script, and write the multimodal data into the scene record.

[0080] In one implementation, the script generation service can return the audio or audio address corresponding to the script text along with the scene script. In another implementation, the multimodal resource binding module submits the script text from each scene script to the audio generation model for audio generation processing, obtaining the audio corresponding to the respective script text.

[0081] In one specific implementation, the audio generation model can use a text-to-speech service, with the model identifier set to qwen-tts and the timbre parameter set to Serena. The client or server submits the script text as input to the text-to-speech service, obtains the audio URL, and writes it to the audio address or the audioUrl of the StoryBookPageModel. This specific model and timbre are only used to illustrate one executable audio generation method and do not restrict the use of other models or timbres for audio generation.

[0082] For each scene script, the multimodal resource binding module writes the scene order, script text, character description, scene description, character reference image address, scene image address and audio address of the scene image generated based on the scene script into the scene record identified by the scene identifier, according to the scene identifier in the scene script.

[0083] Figure 3 The diagram schematically illustrates the data relationship between content generation instances, scene script sequences, and scene records in some embodiments of the present invention. For example... Figure 3 As shown, a content generation instance corresponds to a single access and is associated with one or more scene records. A scene record is identified by a scene identifier and stores script text, description data, image addresses, and audio addresses related to the same scene. The scene identifier is used to determine the scene to which the data belongs, and the scene order is used to determine the scene's position in the scene script sequence and scene playback sequence.

[0084] Step S270: Determine the scene playback sequence based on the scene script sequence, and display the scene images in each scene record and play the audio in the same scene record according to the scene playback sequence.

[0085] The playback module can sort scene records according to scene order, forming a paginated or continuous scene playback sequence. In paginated playback mode, users can enter the next scene by turning pages; in continuous playback mode, the playback module can switch scenes based on audio playback end events, preset display times, or user actions.

[0086] When entering a scene, the playback module reads the corresponding scene record based on the scene identifier, displays the scene image pointed to by the scene image address, and plays the audio pointed to by the audio address. Thus, the playback module reads both the scene image and audio from the same scene record, rather than temporarily combining them based on the return order of different resources.

[0087] In some optional implementations, the scene recorder can also save the display format of the script text, audio playback duration, subtitle timing information, or page interaction information. The playback module can display the script text while showing scene images, and control the text display progress based on the subtitle timing information.

[0088] In the iPad implementation described above, StoryBookDisplayViewController creates multiple StoryBookPageViewControllers based on the firstPage and pages of StoryBookModel. The cover page can display storyName, cover, and introduce, while content pages can display script text and scene images based on the content and imageUrl of StoryBookPageModel. Images can be loaded using SDWebImage or other asynchronous image loading components based on the imageUrl, audio can be played using AVAudioPlayer or AVPlayer based on the audioUrl, and fixed video clips can be loaded based on the videoUrl and played by a video player.

[0089] The reader can configure the landscape orientation, audio session, and current page playback status during the viewWillAppear stage. When the user clicks the previous or next page control, the reader retrieves the adjacent StoryBookPageModel based on the page or indexTag and performs the page switch; when the user clicks the play or pause control, the reader controls the audio or fixed video material on the current page; the progress bar updates according to the current media playback time and total duration, and accepts user drag operations to adjust the playback progress.

[0090] In a local sample mode, when a user clicks on a sample card on the homepage, no network request is sent to the integrated content generation service. Instead, MockData directly constructs the StoryBookModel and its pages array. The imageUrl, audioUrl, and videoUrl on the page point to pre-stored files in the demo1 to demo4 directories or the client-side resource directory, respectively. This sample mode can display the cover, paginated text and images, audio readings, and fixed video materials when the network service is unavailable, and verify the client-side page model, scene order, and media playback logic.

[0091] Figure 6 The diagram schematically illustrates the interface flow of a terminal device in an embodiment of the present invention. For example... Figure 6 As shown, in one type of client-side display, after the application starts, it first displays application identification information, such as the "INFINITY AI STORYTELLING" logo displayed in actual applications, followed by the character creation interface. The character creation interface displays character creation prompts, such as "Create a protagonist unique to you," and provides input controls for the target character name and controls for capturing images with the front-facing camera. The user inputs the target character name and captures an image of the user through the front-facing camera. The client uses the target character name and the appearance description information obtained from the user image as reference information for character generation.

[0092] User images can be analyzed by the client or server to obtain appearance descriptions such as clothing features, hairstyle features, facial contours, or color features. The character generation reference information does not directly replace the character description in the scene script, but rather serves as supplementary input to the script generation process, enabling the script generation model to output a character description containing the target character's name and appearance features. The character image generation module then determines the target character description based on the character descriptions in each scene script and generates a character reference image that maps to the target character description. The main character object in the character reference image is called the target character. For example, the target character could be a 3D cartoon character generated from the user image; in practical applications, the target character's name is "Allen." In subsequent descriptions, the target character will be used to refer to this character object.

[0093] After the character reference image is generated, the client enters the mode selection interface. The mode selection interface includes a random selection mode and a user-selected mode. The random selection mode corresponds to the processing method of determining the theme material from candidate theme materials based on a random seed; the user-selected mode corresponds to the user continuing to select a subject, target art style, or auxiliary character material, and then determining the theme search information and theme material processing method based on the selected content.

[0094] In self-selection mode, the client enters the subject selection interface. This interface displays a self-directed exploratory learning menu and offers subject options such as Chinese, Mathematics, and English. When the user selects Mathematics, the client sets the `subject` field to the corresponding subject identifier. The client can also display entry points for themes such as Art, Chinese Classics, and Science & Technology; the selected theme entry point is used to define the target teaching theme or input text for the theme.

[0095] The client then enters the art style and companion selection interface. This interface can display multiple preset art style cards and can group these cards according to different user groups. The preset art style selected by the user is designated as the target art style. To illustrate the visual constraints that different target art styles impose on character reference images and scene images, the target art style used in the first teaching content display process is referred to as the first target art style, for example, the "Magic Castle" art style in actual applications; the target art style used in the second teaching content display process is referred to as the second target art style, for example, the "Heroic Romance" art style in actual applications; and the target art style used in the third teaching content display process is referred to as the third target art style, for example, the "Cyber ​​Abyss" art style in actual applications. In the following description, the first target art style, second target art style, and third target art style will be used to refer to the above art styles respectively. The interface grouping information is only used to organize the client-side display; the style field corresponding to the target art style is written in the generation request, and the age field is not written into the topic search information or submitted to the content generation service.

[0096] The art style and companion selection interface can also display auxiliary character candidates that correspond to the target art style. When a user selects an auxiliary character candidate, the client uses the candidate's name, category, ability attributes, and appearance elements as theme material or reference information for character generation. The auxiliary character corresponding to the first target art style is called the first auxiliary character, for example, a winged pony, named "Babby" in actual application; the auxiliary character corresponding to the second target art style is called the second auxiliary character, for example, a female character with traditional clothing, named "Narcissus" in actual application; the auxiliary character corresponding to the third target art style is called the third auxiliary character, for example, a female character with futuristic technology, named "Eva" in actual application. In the following description, the first, second, and third auxiliary characters will be used to refer to the above auxiliary characters respectively. Based on this, the script generation model generates character descriptions for the target character and corresponding auxiliary characters in the scene script, and the character image generation module and scene image generation module generate character reference images containing the target character and corresponding auxiliary characters and scene images for each scene, respectively.

[0097] Once the target role, subject, target art style, and auxiliary role are set, the client can play fixed video footage indicating the transition to the teaching content presentation flow. This fixed video footage can be, for example, a wormhole animation displayed in a real application, pre-stored in a video footage library or the client's resource directory. Video footage tags can include the flow start category, scene transition category, and corresponding art style category. In the following description, "fixed video footage for scene transition" will be used to refer to this animation. The playback module calls the fixed video footage for scene transition based on mode switching points or scene switching nodes; it does not need to generate a transition video for the current access request.

[0098] In the first teaching content presentation process, the target teaching theme is the pattern of number arrangement, the subject field is mathematics, the art style field is the first target art style, and the auxiliary character is the first auxiliary character. The theme knowledge base provides theme knowledge texts of arithmetic sequences or even number arrangements, and the theme material library provides candidate theme materials such as scene environment materials, number clue materials, interactive device materials, and prompting character materials. According to the scene order in the scene script sequence, the scene where the target character and the first auxiliary character obtain number clues is called the first scene. The scene environment material of the first scene is, for example, a fountain area, and the number clue material is, for example, a glowing blueprint. The scene where missing numbers are filled according to the pattern of 2, 4, 6, and 8 is called the second scene. The scene where the prompting character receives prompt information and operates the interactive device is called the third scene. The prompting character is, for example, a magician character in actual application. The above specific objects are only used to illustrate the role of scene environment materials, number clue materials, prompting character materials, and interactive device materials in the scene script. In the following description, the first scene, the second scene, and the third scene refer to the above scenes.

[0099] The scene image generation module uses a character reference image containing the target character and the first auxiliary character as the initial image, and generates multiple scene images based on the scene environment, numerical clues, numerical sequences, prompting characters, and interactive devices of each of the first to third scenes. The multimodal resource binding module writes the mathematical explanatory text, scene images, and audio readings of the first to third scenes into the corresponding scene records, and the playback module displays the questions, prompts, and solutions in the scene order.

[0100] In the second teaching content presentation process, the target teaching theme is couplet knowledge, the subject field is Chinese, the art style field is the second target art style, and the auxiliary role is the second auxiliary role. The theme knowledge base provides the correspondence between the number of characters, parts of speech, or semantics of the upper and lower lines of the couplet, while the theme material base provides theme materials such as second scene environment materials, prompting character materials, and target object materials. The second scene environment material is, for example, a canyon town in a real application; the prompting character material is, for example, an old man; and the target object material is, for example, a treasure object. The script generation model generates a scene script in which the target character and the second auxiliary role complete the couplet problem-solving in the second scene environment. The script text can include the antithesis between the first and second phrases and the plot where the prompting character provides clues. In the following description, the second scene environment material, the prompting character material, and the target object material refer to the above specific materials.

[0101] In the third teaching content demonstration process, the target teaching theme is the principle of electromagnets, the subject field is physics, the art style field is the third target art style, and the auxiliary character is the third auxiliary character. The theme knowledge base provides theme knowledge text on how electric current generates a magnetic field through a coil and how ferromagnetic materials are attracted. The theme material library provides theme materials such as third scene environment materials, target materials, conductive wires, and coils. The third scene environment material is, for example, a spaceship in a real application, and the target material is, for example, a swarm of insect objects. The script generation model generates a scene script in which the target character and the third auxiliary character wind conductive wires into a coil and pass electricity to form an electromagnet, using magnetic force to act on the target. In the following description, the third scene environment material and the target material refer to the above specific materials.

[0102] The first, second, and third teaching content display flows described above correspond to different target teaching themes, theme knowledge texts, theme materials, and target art styles, but they can use the same client-side data structure and playback flow. Each page in the teaching content display flow is carried by a StoryBookPageModel, and the playback order is maintained through page or indexTag; content displays the teaching script text, imageUrl points to the scene image, audioUrl points to the reading audio, and videoUrl points to optional scene transition fixed video material or task completion fixed video material.

[0103] When the final scene of a teaching content demonstration completes, the user enters the task completion page. This page displays the target character image and a task completion message, such as "MISSION ACCOMPLISHED" in a real-world application. The target character image can be a reference image or a thumbnail generated from it. The fixed video clip for task completion can be a clip from the video library. After the user confirms task completion, the user clears the current page playback and returns to the character creation interface, redisplaying the application identifier and description information. This description could be, for example, the phrase "Making every child easily teachable" in a real-world application, allowing the user to begin creating a new content instance. In the following description, "task completion message" and "application description" will refer to the specific text mentioned above.

[0104] The following describes the usage of random seeds and the topic resource library in conjunction with the implementation of repeated access. When an access request for the same target teaching topic is obtained again, the topic retrieval module creates a new content generation instance corresponding to the repeated access and generates a random seed corresponding to the repeated access.

[0105] The random seed can be generated by a random number generator, or determined based on at least one of a new instance identifier, request serial number, or access time. The random seed is stored in the new content generation instance to determine the theme material to be used for re-access from candidate theme materials, and can also serve as the generation control input for script generation models and image generation models.

[0106] In one implementation, the number of candidate topic materials is N, where N is a positive integer. An integer S is obtained based on a random seed, and the index of the candidate topic material is determined by taking S modulo N. The candidate topic material pointed to by this index is determined as the topic material to be used again. In another implementation, the order of the candidate topic materials can be adjusted according to the random seed, and a preset number of materials can be selected from the adjusted candidate topic materials.

[0107] The topic retrieval module provides random seeds and topic materials for revisiting the site to the script generation module, character image generation module, and scene image generation module. The script generation module performs script generation processing based on the random seeds and topic materials for revisiting the site; the character image generation module and scene image generation module perform corresponding character image generation processing and raw image processing based on the random seeds.

[0108] The new content generation instances store the scene script sequence, character reference images, scene images, audio, and scene recordings generated during the subsequent visits. Each previous visit and subsequent visit has a corresponding content generation instance, and each content generation instance is associated with the corresponding visit through an instance identifier and current visit association information.

[0109] Figure 5 The flowchart illustrating the matching, loading, and playback resource selection of fixed video materials in some embodiments of the present invention is shown schematically. Figure 5 As shown, fixed video clips are pre-stored in the video clip library. Each fixed video clip has a video clip tag, which can include at least one of the following: content category, applicable scene category, subject category, style category, or action category.

[0110] For a target scene in the scene playback sequence, the playback module or the topic retrieval module obtains the scene identifier of the target scene and reads the scene description from the scene record identified by that scene identifier. Video retrieval information is determined based on the scene description. The video retrieval information may include at least one of the following: scene category, environment category, character action category, or teaching topic category.

[0111] The category items in the video retrieval information can be matched with the video material tags of each fixed video material. In one implementation, the fixed video material with the most identical category items is selected. In another implementation, the matching results for scene category, environment category, and action category are calculated separately, and multiple matching results are combined according to preset weights. The fixed video material whose combined result meets the preset selection criteria is then selected.

[0112] The selected fixed video clips are mapped to the target scene. The video clip identifier or video address of the fixed video clip is written into the fixed video clip information in the scene record identified by the scene identifier of the target scene. The video clip identifier is used to identify the video clip object, and the video address is used to locate or access the video clip object. Both have different data functions than the video clip tag.

[0113] In some implementations, the fixed video material can be a video file stored locally on the user's terminal. In this case, the playback module can directly read the local video based on the video material identifier. In other implementations, the fixed video material can be stored on a server, a content delivery network, or an object storage system. In this case, the playback module establishes a video material loading task based on the video address.

[0114] In the iPad implementation described above, the reader can maintain a queue of fixed video clip loading tasks, where each task corresponds to a StoryBookPageModel or scene identifier. An example initial concurrency parameter, `maxConcurrentTasks`, can be set to 2. When the current network status changes, the playback module can increase or decrease the concurrency based on this initial value. For fixed video clips that require querying the remote resource preparation status, an example status query interval can be set to approximately 5 seconds, and this interval can be extended or shortened based on network latency or service response status.

[0115] When the video address of the fixed video material is available, the playback module writes the video address into the videoUrl of the corresponding StoryBookPageModel. The StoryBookPageViewController reads the videoUrl when displaying the page; if the videoUrl is empty or the fixed video material has not finished loading, it displays the scene image pointed to by the imageUrl and plays the audio pointed to by the audioUrl. If the videoUrl is not empty and the fixed video material has finished loading, it plays the fixed video material when entering the target scene or at a scene transition node.

[0116] The video footage loading task is used to download, cache, or prefetch fixed video footage, not to generate video footage. The playback module obtains the current network status and determines the concurrency of the video footage loading task or the status query interval based on the current network status. The current network status can be characterized by at least one of the following: available bandwidth, network latency, packet loss status, or connection type.

[0117] In one implementation, when the available bandwidth is greater than a first bandwidth threshold, the concurrency of video material loading tasks is increased or the status query interval is shortened; when the available bandwidth is less than a second bandwidth threshold, the concurrency of video material loading tasks is decreased or the status query interval is extended. The first bandwidth threshold may be greater than or equal to the second bandwidth threshold. The concurrency and status query interval can be further adjusted according to the processing power of the user terminal and the size of the video file.

[0118] When the fixed video footage has not finished loading, the playback module displays the scene image from the scene record identified by the scene identifier of the target scene, and plays the audio from the same scene record. When the fixed video footage has finished loading, the playback module displays the fixed video footage when entering the target scene or reaching the scene transition node of the target scene.

[0119] Scene transition nodes can be the point when the user performs a page-turning operation, the point when the audio playback of the previous scene ends, or the point when the preset scene transition conditions are reached in continuous playback mode. When the scene images and audio of the current scene have already begun to be displayed and played, the fixed video material does not need to immediately interrupt the current scene after loading; instead, it can be used the next time the user enters the target scene or reaches a scene transition node.

[0120] When a fixed video clip fails to load, the video address is invalid, or loading is not completed within a preset time, the playback module can continue to display scene images of the target scene and play audio from the same scene recording. Therefore, the loading status of the fixed video clip does not affect the use of scene images and audio already written into the scene recording.

[0121] The following describes an embodiment of the apparatus of the present invention, which can be used to execute the multimodal content generation and playback method described above. For details not disclosed in the apparatus embodiment, please refer to the method embodiment described above.

[0122] like Figure 1 As shown, the multimodal content generation and playback device may include an access acquisition module, a topic retrieval module, a script generation module, a character image generation module, a scene image generation module, a multimodal resource binding module, and a playback module.

[0123] The access acquisition module is configured to acquire access requests for the target teaching topic. The topic retrieval module is configured to determine topic retrieval information based on the topic input text in the access request, retrieve educational topic information and topic materials corresponding to the target teaching topic from the educational topic database based on the topic retrieval information, and create a content generation instance corresponding to the current access.

[0124] The script generation module is configured to generate scripts based on educational theme information and theme materials, resulting in a sequence of scene scripts arranged in playback order. The character image generation module is configured to determine the target character description based on the character descriptions in each scene script, and then generate character reference images based on the target character descriptions.

[0125] The scene image generation module is configured to use a character reference image as the initial image for each scene script, and perform image-to-image processing based on the scene description in the scene script to obtain the corresponding scene image. The multimodal resource binding module is configured to acquire the audio corresponding to the script text in each scene script, and write the script text, character description, scene description, character reference image address, scene image address, and audio address of the same scene into the scene record identified by the scene identifier.

[0126] The playback module is configured to determine the scene playback sequence based on the scene script sequence, and then display the scene images from each scene record and play the audio from the same scene record according to the scene playback sequence. The playback module can also perform steps such as matching fixed video materials, creating video material loading tasks, and selecting playback resources based on resource loading status.

[0127] The above modules can be implemented using software, hardware, or a combination of both. When implemented in software, each module can be implemented by the processor executing the corresponding program instructions in memory; when implemented in hardware, each module can be implemented using application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices.

[0128] The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, which is configured to perform the aforementioned multimodal content generation and playback method by executing the executable instructions. The electronic device may also include at least one of a communication interface, a display, a speaker, and an input device.

[0129] The processor may be a central processing unit, a graphics processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling program execution. The memory may be a read-only memory, random access memory, solid-state memory, magnetic memory, or other computer-readable storage device capable of storing programs and data.

[0130] A computer program may be stored on a computer-readable medium, which, when executed by a processor, implements the aforementioned multimodal content generation and playback method. The computer-readable medium may be a portable computer disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0131] It should be noted that although the steps of the method in this invention are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, or one step may be broken down into multiple steps.

[0132] Other embodiments of the invention will be understood by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art. This specification and embodiments are to be considered exemplary only, and the actual scope of protection of the invention is defined by the claims.

Claims

1. A method for generating and playing multimodal content of educational picture books, characterized in that, include: Obtain an access request for the target teaching topic; Based on the topic input text in the access request, determine the topic retrieval information, obtain educational topic information and topic materials corresponding to the target teaching topic from the educational topic database based on the topic retrieval information, and create a content generation instance corresponding to the current access; Based on the educational theme information and the theme materials, a script generation process is performed to obtain a sequence of scene scripts arranged in the playback order. Each scene script in the scene script sequence includes a scene identifier, script text, character description, and scene description. The target character description is determined based on the character description in each scene script. The character image generation process is performed based on the target character description to obtain a character reference image that has a mapping relationship with the target character description. For each scene script, the character reference image is used as the initial image, and image-to-image processing is performed according to the scene description in the scene script to obtain a scene image that has a mapping relationship with the character reference image and the scene description in the scene script. Obtain the audio corresponding to the script text in each scene script; For each scene script, according to the scene identifier in the scene script, the script text, character description and scene description in the scene script, as well as the character reference image, the scene image generated according to the scene script and the audio corresponding to the script text in the scene script, are written into the scene record identified by the scene identifier. The scene playback sequence is determined based on the scene script sequence, and the scene images in each scene record and the audio in the same scene record are displayed according to the scene playback sequence.

2. The multimodal content generation and playback method according to claim 1, characterized in that, The steps of obtaining an access request for the target teaching topic and obtaining educational topic information and materials corresponding to the target teaching topic from the educational topic database based on the topic retrieval information include: Obtain the topic text or teaching content text input by the user, and determine the topic text or teaching content text as the topic input text; or, obtain the topic speech input by the user, perform speech recognition on the topic speech to obtain speech recognition text, and determine the speech recognition text as the topic input text. Obtain the subject, language, and art style selected by the user, and determine the topic search information based on the input text and the subject, language, and art style. The educational topic database includes a topic knowledge base and a topic material database. Based on the topic retrieval information, topic knowledge texts that correspond to the target teaching topic are determined in the topic knowledge base. Based on the topic retrieval information, candidate topic materials that correspond to the target teaching topic are determined in the topic material library, and the topic material currently accessed and used is determined from the candidate topic materials; The topic knowledge text and the currently accessed topic materials are identified as the educational topic information and the topic materials, respectively.

3. The multimodal content generation and playback method according to claim 2, characterized in that, When a request for access to the same target teaching topic is received again, the method further includes: Create a new content generation instance corresponding to the next visit; Generate a random seed corresponding to the revisit; The theme material to be used for the next visit is determined from the candidate theme materials based on the random seed; Based on the random seed and the theme material used for the revisit, script generation processing, character image generation processing, and scene image image generation processing are performed to obtain a scene script sequence, character reference image, and scene image corresponding to the new content generation instance.

4. The multimodal content generation and playback method according to claim 1, characterized in that, For each scene script, the character reference image is used as the initial image, and image-to-image processing is performed based on the scene description in the scene script, including: Obtain character hint information from the target character description to characterize the character's identity and appearance. The character hint information includes at least one of the following: character name, appearance features, clothing features, color features, and body shape features. Based on the scene descriptions in each scene script and the character prompt information, scene image prompt information corresponding to the scene identifiers of each scene script is determined. The scene image prompt information also includes at least one of scene environment information and character action information associated with the scene identified by the corresponding scene identifier. The character reference image is used as the initial image for each scene, and the initial image is transformed according to the scene image prompt information to obtain the scene image for each scene; The scene image prompts in different scenarios include at least some of the same character prompts.

5. The multimodal content generation and playback method according to claim 1, characterized in that, The process of acquiring the audio corresponding to the script text in each scene script, and writing the audio into the scene record according to the scene identifier in each scene script, includes: For each scene script, obtain the audio returned along with the scene script; or, perform audio generation processing on the script text in the scene script to obtain the audio corresponding to the script text in the scene script. Write the scene identifier in the scene script, the scene order of the scene script in the scene script sequence, the script text, character description and scene description in the scene script, the character reference image address of the character reference image, the scene image address of the scene image generated according to the scene script and the audio address of the audio corresponding to the script text in the scene script into the scene record identified by the scene identifier. Establish a correspondence between one or more of the scene records and the content generation instance, including instance identifier and current access association information.

6. The multimodal content generation and playback method according to claim 1, characterized in that, The method further includes: For the target scene in the scene playback sequence, obtain the scene identifier of the target scene, determine the video retrieval information based on the scene description in the scene record identified by the scene identifier, and obtain fixed video materials with a mapping relationship with the target scene from the video material library based on the matching relationship between the video retrieval information and the video material tags. The video material tags are tag information used to characterize the content category and applicable scene category of the fixed video materials. Write the video material identifier or video address of the fixed video material into the scene record identified by the scene identifier; When displaying the target scene, the fixed video footage is displayed; The fixed video material refers to video material that is pre-stored in the video material library before the access request is received.

7. The multimodal content generation and playback method according to claim 6, characterized in that, The step of displaying the fixed video material when displaying the target scene includes: A video material loading task is created for the aforementioned fixed video material; Obtain the current network status, and determine the concurrent number of video material loading tasks or the status query interval based on the current network status; When the fixed video material has not been fully loaded, display the scene image in the scene record identified by the scene identifier of the target scene, and play the audio in the same scene record; When the fixed video material is loaded, it is displayed when entering the target scene or reaching the scene switching node of the target scene.

8. A multimodal content generation and playback device for educational picture books, characterized in that, include: The access acquisition module is configured to acquire access requests for the target teaching topic; The topic retrieval module is configured to determine topic retrieval information based on the topic input text in the access request, retrieve educational topic information and topic materials corresponding to the target teaching topic from the educational topic database based on the topic retrieval information, and establish a content generation instance corresponding to the current access. The script generation module is configured to perform script generation processing based on the educational theme information and the theme materials to obtain a sequence of scene scripts arranged in the playback order. Each scene script in the scene script sequence includes a scene identifier, script text, character description, and scene description. The character image generation module is configured to determine the target character description based on the character description in each scene script, perform character image generation processing based on the target character description, and obtain a character reference image that has a mapping relationship with the target character description. The scene image generation module is configured to take the character reference image as the initial image for each scene script, and perform image-to-image processing according to the scene description in the scene script to obtain a scene image that has a mapping relationship with the character reference image and the scene description in the scene script. The multimodal resource binding module is configured to acquire audio corresponding to the script text in each scene script, and for each scene script, according to the scene identifier in the scene script, write the script text, character description, scene description, character reference image, scene image generated according to the scene script, and audio corresponding to the script text in the scene script into the scene record identified by the scene identifier. The playback module is configured to determine a scene playback sequence based on the scene script sequence, and to display scene images from each scene record and play audio from the same scene record according to the scene playback sequence.

9. An electronic device, characterized in that, include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the multimodal content generation and playback method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal content generation and playback method according to any one of claims 1 to 7.

Images