Picture book generation method and device, equipment and storage medium
Through big model technology, story outlines, storyboard understanding and multimedia generation are generated, and the creation of picture books is automatically solved, which relies on manual and time-consuming problems in traditional picture books production, and efficient and personalized picture books are achieved, lowering the threshold and improving the User experience.
Patent Information
- Application Number
- CN202411001402.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-24
AI Technical Summary
In the prior art, painting production relies on manual labor and requires mastering specific technical knowledge and skills, which makes it difficult for users in non-related industries to complete independently, and the traditional manual production process consumes a lot of time and energy.
Using a large-model-based picture book generation method, we automatically create target picture books, including text, images and audio content, through generating story outlines, storyboard understanding and multimedia generation, to achieve automated processes and personalized customization.
It improves the efficiency and quality of picture book production, lowers the threshold, realizes automated processes, can generate content that meets user needs, and improves user experience.
Smart Images

Figure CN119027528A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of data processing and image processing, and in particular to the technical fields of artificial intelligence, big data, and image generation, and can be applied to application fields such as intelligent document editing, intelligent assistants, and virtual assistants. Background Art
[0002] At present, the production of picture books mainly relies on manual labor and involves certain technical requirements, such as the need to master specific technical knowledge and skills, which makes it difficult for ordinary users in non-related industries (such as fine arts) to complete the production of picture books independently. Even if there is a customized demand for picture books, ordinary users cannot achieve it by themselves. In addition, the traditional manual production process, including drawing pictures, writing copy, graphic typesetting, and audio and video production, all requires a lot of time and energy. Summary of the invention
[0003] The present invention provides a method, device, equipment and storage medium for generating a picture book.
[0004] According to one aspect of the present disclosure, a method for generating a picture book is provided, comprising:
[0005] Based on the determined theme content, generate a story outline around the theme content; wherein the story outline includes N paragraphs around the theme content; N is an integer greater than or equal to 2;
[0006] Performing storyboard understanding on each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph; wherein the plurality of prompt information for a paragraph can describe the scene screen to which the paragraph relates;
[0007] Based on the multiple prompt information of each paragraph in the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
[0008] According to another aspect of the present disclosure, a device for generating a picture book is provided, comprising:
[0009] A text processing unit, configured to generate a story outline around the determined theme content based on the determined theme content; wherein the story outline includes N paragraphs around the theme content; wherein N is an integer greater than or equal to 2;
[0010] A multimedia generation unit is used to perform storyboard understanding on each of the N paragraphs included in the story outline to obtain multiple prompt information for each paragraph; wherein the multiple prompt information for a paragraph can describe the scene image to which the paragraph corresponds; based on the multiple prompt information for each of the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0012] at least one processor; and
[0013] a memory communicatively connected to the at least one processor; wherein,
[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any embodiment of the present disclosure.
[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.
[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method according to any one of the embodiments of the present disclosure is implemented.
[0017] In this way, the disclosed scheme can understand the generated storyline and obtain multiple prompt information for describing the scene pictures corresponding to each paragraph, and then generate a picture book (such as a target picture book) that matches the theme content based on the multiple prompt information of each paragraph, thereby effectively improving the production efficiency and quality of the picture book. Compared with the existing manual production methods, the disclosed scheme effectively avoids the tedious process of traditional picture book production, lowers the threshold for picture book production, and realizes the automated process of picture book generation. Moreover, it can generate picture book content that meets user needs, meet the user's personalized needs, and thus improve the user experience.
[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0020] Figure 1 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 1 ;
[0021] Figure 2 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 2 ;
[0022] Figure 3This is a schematic diagram of the effect of displaying a target picture book according to an embodiment of the present application;
[0023] Figure 4(a) to Figure 4(d) It is a schematic diagram of a processing flow of realizing a book-turning effect in a target picture book according to an embodiment of the present application;
[0024] Figure 5 is a schematic diagram of a book turning effect of a target picture book according to an embodiment of the present application;
[0025] Figure 6 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 3 ;
[0026] FIG. 7( a ) and FIG. 7( b ) are schematic diagrams of an application process of a method for generating a picture book according to an embodiment of the present application in a specific scenario;
[0027] Figure 8 is a schematic diagram of a picture book generation system according to an embodiment of the present application;
[0028] Fig. 9 is a structural schematic diagram of a picture book generating device according to an embodiment of the present application;
[0029] Fig.10 It is a block diagram of an electronic device used to implement the sketchbook generating method of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0031] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.
[0032] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.
[0033] The following is a description of the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure, and they all belong to the protection scope of the embodiments of the present disclosure.
[0034] At present, the production of picture books mainly relies on manual labor and involves certain technical requirements, such as the need to master specific technical knowledge and skills, which makes it difficult for ordinary users in non-related industries (such as fine arts) to complete the production of picture books independently. Even if there is a customized demand for picture books, ordinary users cannot achieve it by themselves. In addition, the traditional manual production process, including drawing pictures, writing copy, graphic typesetting, and audio and video production, all requires a lot of time and energy.
[0035] Based on this, the disclosed solution proposes a large model-based picture book generation method to automatically and efficiently generate a target picture book.
[0036] Specifically, Figure 1 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.
[0037] Further, the method includes at least part of the following contents. Figure 1 As shown, including:
[0038] Step S101: Based on the determined theme content, generate a story outline surrounding the theme content.
[0039] Here, the story outline can be understood as a summary of the story, such as, in one example, including the main plot, characters, conflicts, and ending.
[0040] Furthermore, in order to facilitate automatic generation of a picture book, the story outline generated around the theme content may include N paragraphs around the theme content, where N is an integer greater than or equal to 2. For example, each paragraph contains a main plot, so that it is convenient to generate a picture book in units of paragraphs later.
[0041] For example, in one example, the subject content may specifically be text content input by the target object, such as one keyword or multiple keywords, etc.; the present disclosure does not impose any specific limitation on the output subject content.
[0042] Step S102: performing storyboard analysis on each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph.
[0043] Here, the multiple prompts of the paragraphs obtained after the storyboard is understood can describe the scene that the paragraph is targeting. This lays the foundation for the subsequent accurate generation of images. At the same time, it can also effectively avoid the angles of the subsequent generated images being the same, effectively improving the visualization effect of the generated target picture book.
[0044] Here, it should be noted that "storyboard understanding" can also be called "storyboard understanding". Its purpose is to ensure that the generated image (also called storyboard image) accurately matches the scene picture of the paragraph, thereby ensuring the consistency of the image and text, and at the same time, improving the visual effect and thus improving the user experience.
[0045] Step S103: Based on the plurality of prompt information of each of the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
[0046] In one example, the image of each paragraph may also include subtitles (eg, including the text content of the paragraph), so that a target picture book combining images and texts can be obtained, thereby effectively improving the readability of the target picture book.
[0047] In this way, the disclosed scheme can understand the generated storyline and obtain multiple prompt information for describing the scene pictures corresponding to each paragraph, and then generate a picture book (such as a target picture book) that matches the theme content based on the multiple prompt information of each paragraph, thereby effectively improving the production efficiency and quality of the picture book. Compared with the existing manual production methods, the disclosed scheme effectively avoids the tedious process of traditional picture book production, lowers the threshold for picture book production, and realizes the automated process of picture book generation. Moreover, it can generate picture book content that meets user needs, meet the user's personalized needs, and thus improve the user experience.
[0048] In a specific example, the story outline can be obtained in the following manner, including: obtaining multiple theme prompt words for a determined theme content, inputting the multiple theme prompt words of the theme content into a target content generation model to generate a story outline around the theme content, so as to quickly infer the story outline around the theme content, laying the foundation for the subsequent accurate generation of picture book content.
[0049] Here, the target content generation model can be obtained by training the preset content generation model based on prompt words and a small number of learning samples (few-shot) using the supervised fine-tuning (SFT) data pre-training technology. This provides strong support for efficiently and accurately generating a story outline around the input theme content, and at the same time, it also provides strong support for the subsequent generated picture book content to have better completeness and readability.
[0050] Furthermore, the content generation model described above can be specifically a large model, and further, can be specifically a large language model. Alternatively, it can also be other text generation models, as long as the above method can be implemented, and the present disclosure does not limit this.
[0051] It should be noted that the story outline described above can be obtained by inference using a content generation model, or can also be obtained by using other text generation methods, and the present disclosure does not impose any specific restrictions on this.
[0052] Furthermore, in a specific example, the disclosed solution also provides editing functions, such as editing functions for the subject content, or editing functions for the generated images, etc.; for example, after obtaining a story outline surrounding the subject content (for example, after step S101 described above), the generated story outline can also be edited, for example, after displaying the generated story outline, the story outline after adjusting the story outline can also be obtained, so that the target object can personalize the generated content according to its own needs, thereby further improving the user experience and making the subsequently generated picture book more meet the user needs.
[0053] In a specific example of the disclosed solution, after obtaining the target picture book for the subject content (for example, step S103 described above), the target multimedia picture book can also be obtained in the following manner, specifically including:
[0054] Step S104: Generate audio data for each paragraph based on the text content of each paragraph.
[0055] For example, in one example, the text content of each paragraph can be converted into audio data using the Text-to-Speech (TTS) technology. It should be noted that the disclosed solution does not limit the synthesis technology from text to audio.
[0056] Step S105: Assemble the audio data of each paragraph with the images in the target picture book to obtain a target multimedia picture book for the subject content.
[0057] In one example, the disclosed solution also provides an editing function for each page in a multimedia picture book, and the editable content of each page may specifically include at least one of the following: text, audio, image details, etc.; in this way, the interactivity with the user is enhanced to further ensure that a picture book that meets the user's needs can be generated, thereby improving the user's experience.
[0058] In this way, the disclosed solution provides an instant audio automation generation solution, which can effectively combine the audio data of each paragraph with the target picture book, thereby further improving the user's reading experience.
[0059] Figure 2 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.
[0060] Further, the method includes at least part of the following contents. Figure 2 As shown, including:
[0061] Step S201: Based on the determined theme content, generate a story outline surrounding the theme content.
[0062] Here, the story outline includes N paragraphs surrounding the theme content; N is an integer greater than or equal to 2.
[0063] Here, please refer to the above instructions for how to obtain the story outline, which will not be repeated here.
[0064] Step S202: performing storyboard analysis on each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph.
[0065] Here, the multiple prompt information of the paragraph can describe the scene screen that the paragraph is related to.
[0066] Step S203: obtaining an image for each paragraph based on a plurality of prompt information of each of the N paragraphs.
[0067] Step S204: According to the story development sequence of the N paragraphs included in the story outline, the images of each paragraph are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state.
[0068] Here, the target picture book can dynamically display a book-turning effect around a preset symmetry axis during playback, and each image of the target picture book can be displayed in full size on the display interface.
[0069] Here, "full size" can be understood as the maximum size that can be displayed on the display interface. In other words, each image in the target picture book can be displayed at the maximum size of the display interface.
[0070] For example, in one example, Figure 3 As shown, when the target picture book is played on the display interface, each image can be displayed in full size on the display interface, thus improving the reading experience.
[0071] Furthermore, the target picture book can also dynamically display the book turning effect, for example, around a preset symmetry axis (for example, taking the display interface as a rectangle, at this time, the preset symmetry axis can be the vertical midline of the rectangular display interface, for example, recorded as AA'), thereby improving the user's reading experience.
[0072] In this way, the disclosed solution can utilize the story development sequence of each paragraph to overlay each image, so as to facilitate playing the target picture book with a flipping book effect, thereby enhancing the sense of reality and further improving the user's reading experience.
[0073] In a specific example of the disclosed solution, the following method can be used to perform superposition processing on the images of each paragraph to obtain the target picture book with each image in a superimposed state, thereby providing support for achieving a book-turning effect and full-size display of each page. Specifically, the above-described superimposition processing of the images of each paragraph according to the story development sequence of the N paragraphs included in the story outline to obtain the target picture book with each paragraph image in a superimposed state (for example, the above-described step S204) specifically includes:
[0074] Step S204 - 1 : according to the story development sequence of the N paragraphs contained in the story outline, determine the i-th image and the i+1-th image having a front-to-back display sequence.
[0075] Step S204-2: According to the following superposition processing steps, the i-th image (which can be understood as the current page) and the i+1-th image (which can be understood as the next page of the current page) are superimposed to obtain the target picture album in which the images of each paragraph are in a superimposed state.
[0076] Here, in one example, the i-th image can be understood as the currently displayed page, that is, the current page; the i+1-th image can be understood as the next page of the current page. In other words, after "turning the book" from the current page, the i+1-th page needs to be displayed in full size.
[0077] Furthermore, the above-mentioned superposition processing method (such as step S204-2) specifically includes:
[0078] Step S204-2-1: Overlay a blank display area on the layer above the i+1th image (eg, the next page of the current page). For example, overlay the blank display area on the i+1th image.
[0079] Here, the area size of the blank display area is the same as the area size of the i-th image. In other words, the blank display area can also be displayed in full size on the display interface.
[0080] Step S204-2-2: placing the sub-image located on one side of the preset symmetry axis (also referred to as the first side of the first i-th image) in the ith image on one side of the blank display area (also referred to as the first side of the blank display area).
[0081] That is to say, after placing the blank display area on the upper layer of the (i+1)th image, in order to achieve the book-turning effect, it is also necessary to place part of the image in the (i)th image, for example, the sub-image located on one side of the preset symmetry axis, on one side of the blank display area.
[0082] Step S204-2-3: Flip the sub-image in the i+1th image located on one side of the preset symmetry axis (also referred to as the first side of the i+1th image), and after flipping, attach it to the back of the sub-image in the i-th image located on the other side of the preset symmetry axis (also referred to as the second side of the i-th image).
[0083] Step S204-2-4: placing the pasted image on the other side of the blank display area (also referred to as the second side of the blank display area) to obtain an overlaid image.
[0084] That is to say, after placing part of the image in the ith image on one side of the blank display area, for the other side of the blank display area, the sub-image on the other side of the preset symmetry axis in the ith image needs to be placed on the other side of the blank display area. At this time, in order to achieve the book turning effect, before placing the sub-image on the other side of the preset symmetry axis in the ith image on the other side of the blank display area, the back of the sub-image on the other side of the preset symmetry axis in the ith image fits the part to be displayed in the i+1th image after the page is turned, thus laying the foundation for achieving the page turning effect.
[0085] For example, if the book flipping effect is to flip from the right side of the display interface to the left side of the display interface, at this time, as shown in FIG4(a), a blank display area, such as an empty document object model (DOM), is superimposed on the i+1th image; secondly, as shown in FIG4(b), the image at the preset symmetry axis BB in the i-th image is copied. ′The sub-image on the left side of the image (i.e., sub-image 1 of the i-th image) is copied, and the copied sub-image is added to the left area of the empty DOM (i.e., the preset symmetry axis AA) by using the absolute positioning layout technology. ′ ) to obtain the DOM of sub-image 1 containing the i-th image (which can be recorded as DOM-1); again, as shown in FIG4(c), copy the i-th image located on the preset symmetry axis BB ′ The sub-image on the right side (i.e., sub-image 2 of the i-th image) and the sub-image on the preset symmetry axis AA in the i+1-th image ′ The left sub-image of the image (i.e., the sub-image 1 of the i+1th image), wherein the sub-image 1 of the i+1th image is flipped, for example, by calling the rotateY function to rotate the sub-image 1 of the i+1th image around the preset symmetry axis AA ′ Rotate 180 degrees; use the absolute positioning layout technology to paste the flipped sub-image to the back of sub-image 2 of the i-th image. At this time, the pasted image is a double-sided image, the front side of the double-sided image is sub-image 2 of the i-th image, and the back side is sub-image 1 of the i+1-th image; finally, as shown in Figure 4(d), the pasted image can be placed in the right area of DOM-1 (that is, the preset symmetry axis AA). ′ Further, a transition animation technique can be set for the pasted image placed in the right area of DOM-1. This provides strong support for achieving the flip effect. Thus, the following can be obtained by using the overlay processing method: Figure 5 The flipping book effect from the i-th image to the i+1-th image is shown.
[0086] It should be noted that in this example, the rotateY function can realize an element around the vertical axis (that is, the y axis, such as the preset symmetry axis AA ′ ) to achieve the book-turning effect of the disclosed solution; in addition, in actual applications, parameters such as transform-style and perspective can also be called to enhance the book-turning effect of the picture book; here, the transform-style parameter is used to set whether the elements in the picture book are in 3D space, such as setting the transform-style parameter to preserve-3d, so that the elements in the picture book are in 3D space; the perspective parameter is used to set the distance between the observer and the plane, such as the plane with z=0 (the z-axis is the coordinate axis perpendicular to the y-axis), so that the picture book elements with three-dimensional position transformation produce a perspective effect. In this way, by calling the above parameters, the 3D effect can be enhanced while achieving the book-turning effect, thereby further enhancing the user experience.
[0087] In addition, it should be noted that in actual applications, in order to achieve a vertical flip effect, the value of the rotateY function usually needs to be set to 180 degrees, but for a specific system environment (such as the IOS system), the value of the rotateY function needs to be set to -180 degrees; further, when using the rotateY function for vertical flipping, the value of the translateX function can also be set to -0.5px to avoid possible gap problems when displaying the picture book. In addition, in order to avoid being unable to cover the elements on the back of the picture book page, the rotateY function and the backfaceVisibility parameter can also be used in conjunction, such as setting the value of the rotateY function to -180 degrees and the backfaceVisibility parameter to hidden.
[0088] It should be noted that the above is only an exemplary description. In actual applications, superposition processing can be performed based on the specific needs of the book turning effect, and the present disclosure does not limit this.
[0089] In this way, the disclosed solution utilizes the overlay processing method to perform overlay processing on the obtained images to obtain a specific book-turning effect, thereby improving the quality of picture book production and providing users with a personalized picture book reading experience. Moreover, the interactive function can enhance the interactivity between users and picture books, increase users' reading interest and participation, and thus improve users' usage experience.
[0090] Furthermore, since the disclosed solution realizes the book-turning effect by passing in a complete DOM, and each page is a complete DOM that fills the entire screen in full size, compared to the method of passing in half a page of DOM alone, the disclosed solution can make the text, buttons and other elements on each page of the picture book well adapted to the book-turning animation, and the above process can realize the book-turning effect of the picture book without relying on jQuery plug-ins. Therefore, the disclosed solution is more universal.
[0091] Figure 6 This is a schematic flow chart of a method for generating a picture book according to an embodiment of the present application. Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figures 1 to 5 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.
[0092] Further, the method includes at least part of the following contents. Figure 6 As shown, including:
[0093] Step S601: Based on the determined theme content, generate a story outline surrounding the theme content.
[0094] Here, the story outline includes N paragraphs surrounding the theme content; N is an integer greater than or equal to 2.
[0095] Step S602: Determine the current segment that needs to be understood.
[0096] That is to say, after obtaining N paragraphs surrounding the subject content, the current paragraph that has not been understood by storyboard is determined from the N paragraphs.
[0097] Step S603: input the determined current paragraph and the preceding and following paragraphs of the current paragraph in the story outline into the target large language model, so as to use the target large language model to perform storyboard understanding on the current paragraph.
[0098] For example, in one example, based on the story development order of each paragraph in N paragraphs, the two paragraphs adjacent to the current paragraph are determined (corresponding to the context paragraphs mentioned above), and then the current paragraph and the two paragraphs adjacent to the current paragraph are input into the target large language model to understand the current paragraph in a segmented manner. In this way, the current paragraph can be fully understood in combination with the context, for example, from the perspective of background, picture description, and the expression of the target body, and then multiple prompt information of the current paragraph is obtained, which provides strong support for the subsequent improvement of the correlation between the image generated for the current paragraph and the text content of the current paragraph. At the same time, it can also effectively avoid the same angle of the images generated by the previous and next paragraphs, thereby laying the foundation for further improving the user experience.
[0099] Step S604: Get multiple prompt information of the current segment.
[0100] Here, multiple prompt information of the current segment can describe the scene picture of the current segment from multiple dimensions. For example, multiple prompt information of the current segment can describe the scene picture of the current segment from multiple dimensions such as background, picture description and character expression. In this way, it is easy to control the rhythm and theme of the story of each paragraph, laying the foundation for the subsequent accurate generation of images matching each paragraph.
[0101] At this point, by using the above method to understand each of the N paragraphs in detail, multiple prompt information for each paragraph can be obtained.
[0102] Step S605: Based on the multiple prompt information of each paragraph in the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
[0103] In this way, the disclosed scheme uses a large language model to understand the storyboards of each paragraph, so as to obtain prompt information that can accurately describe the scene pictures of each paragraph, and then facilitate the accurate generation of target picture books for the theme content based on the prompt information after the storyboard understanding. In this way, compared with the traditional manual production of picture books, the disclosed scheme uses a large language model to greatly improve the efficiency of picture book creation, and can also accurately control the theme, scene and rhythm of the story in the image, enhance the ability of text to draw pictures, and then can generate higher-quality, diversified and personalized picture book content that meets the user's creation needs, thereby further improving the user experience.
[0104] Furthermore, in a specific example, the disclosed solution can also determine the target style of the subject content, so that the generated image has the target style, thereby further improving the user experience. Specifically, the above-mentioned obtaining an image for each paragraph based on the multiple prompt information of each paragraph in the N paragraphs (for example, the above-mentioned step S605) specifically includes:
[0105] Step S605-1: Determine the target style of the subject content.
[0106] Step S605-2: Determine the image generation model corresponding to the target style.
[0107] In one example, a model capable of outputting an image generation model in a preset style can be pre-trained. For example, multiple preset styles are determined, and a large Stable Diffusion (SD) model is pre-trained for each preset style. In this way, the trained large SD model can generate images in the preset style. Based on this, after the target style is determined, a model matching the target style can be selected from the pre-trained large SD model. In this way, the model matching the target style can be used to generate images in the target style, thereby maintaining consistency in the styles of different images in the target picture book, further improving the user experience.
[0108] Here, in this example, a number of relevant examples of each preset style can be used to fine-tune specific parameters in a preset SD large model for pre-training, so as to obtain a pre-trained SD large model corresponding to the preset style. In this way, the SD large model corresponding to each preset style can be obtained using the above method.
[0109] Furthermore, each pre-trained SD large model can reside in the graphics card memory to increase the speed of picture book production.
[0110] Step S605 - 3 : Inputting a plurality of prompt information of each of the N paragraphs into the image generation model corresponding to the target style to obtain an image for each paragraph, wherein the image of each paragraph has the target style.
[0111] For instance, in one example, when a story outline surrounding the theme content is displayed, in response to a click operation on picture book production, multiple preset styles may be further displayed (for example, keywords of preset styles are displayed). At this time, the target object may select the target style. In this way, the user's customization needs are met through the above-mentioned interactive method, which lays the foundation for the user to obtain the picture book they need in the future. At the same time, it also lays the foundation for further improving and enriching the user experience.
[0112] In this way, the disclosed solution generates a picture book of the style required by the target object according to the needs and preferences of the target object. Moreover, the above process does not require painting skills to produce a personalized picture book that meets the needs of the user, effectively lowering the threshold for picture book production, meeting the user's customization needs, and further improving the user experience.
[0113] The following is a further detailed description of the disclosed solution in conjunction with a specific example. Specifically, as shown in FIG. 7( a ) and FIG. 7( b ), the picture book generation process specifically includes:
[0114] Step S701: In response to a click operation on the application entrance, for example, the client displays an input interface for picture book creation in response to the click operation.
[0115] Here, the application entrance can be specifically the home page, floating ball, editor and welcome card.
[0116] Step S702: In response to a confirmation operation on the content (such as subject content, prompt words, etc.) inputted by the target object, a story outline for the inputted content is displayed.
[0117] For instance, in one example, the client sends an outline generation request to the server in response to a determination operation of the content input by the target object; further, the server sends the outline generation request to the task scheduling center, which creates a task based on the outline generation request and stores the created task in a queue to wait for processing resources to be allocated for the outline generation request.
[0118] Furthermore, based on the tasks in the queue, when the task needs to be executed, the task scheduling center sends an outline generation request to the generation side, so as to call the trained content generation model on the generation side, and use the called content generation model to generate a story outline for the content input by the target object, so as to send the generated story outline to the client through the server.
[0119] Here, the content generation model described above can be specifically a large model, and further, can be specifically a large language model. Alternatively, it can also be other text generation models, as long as the above method can be implemented, and the present disclosure does not limit this.
[0120] Furthermore, when a story outline is displayed on the client, an adjusted story outline can be obtained in response to an editing operation on the displayed story outline.
[0121] Furthermore, in this example, the content input by the target object may be a user instruction, text content (such as plain text), a local document input by the target object, or a document in a database selected by the target object, etc. The present disclosure does not impose any specific restrictions on this.
[0122] Step S703: In response to the submission operation for the story outline (or the adjusted story outline), a style list page containing multiple preset styles (such as colored pencil style, stroke style, American comic style, etc.) is displayed.
[0123] Step S704: In response to a click operation on the style list page, the target style selected by the target object is obtained, a picture book making task is generated based on the obtained story outline and the target style, and the picture book making waiting queue is entered, while the real-time progress of the picture book making task is displayed (for example, the generation waiting page is displayed).
[0124] For example, in one example, as shown in Figure 7(b), first, after obtaining the story outline and the target style selected by the target object, the client generates a picture book creation request and sends it to the server; the server sends the received picture book creation request to the task scheduling center to create a picture book creation task through the task scheduling center. Here, the task scheduling center can return task creation information to the server, for example, if the task is successfully created, it returns the task ID, otherwise, it returns the task status;
[0125] Secondly, when the task is created successfully, the server saves the task ID returned by the task scheduling center, and the task scheduling center sends a picture book generation request to the generation side, so that the picture book can be produced on the generation side; further, the generation side can call the image generation model that matches the target style to generate the image; further, after the picture book production is completed, the picture book data is returned to the task scheduling center for data storage, so that the picture book data can be sent to the client through the server.
[0126] It should be noted here that during the sketchbook generation process, the task scheduling center can also return the task status of the current sketchbook production task (for example, completion percentage, etc.) to the server. At this time, the server generates a prompt message (such as "sketchbook has been generated") based on the received task status and sends it to the client.
[0127] Furthermore, in this example, after receiving the picture book generation request from the task scheduling center, the generation side, on the one hand, uses the large language model to understand the storyline of the story outline (or the adjusted story outline) to obtain multiple prompt information for each paragraph in the story outline; on the other hand, the generation side calls the image generation model corresponding to the target style based on the target style selected by the target object, so as to generate the target picture book required by the target object using the image generation model corresponding to the target style.
[0128] Furthermore, for the relevant content about the large language model and the image generation model, please refer to the above examples, which will not be repeated here.
[0129] Furthermore, in another example, when the server saves the task ID returned by the task scheduling center, if the task scheduling center fails to send a picture book generation request to the generation side, it returns the task status (for example, in a queued state) to the server, and then sends the task status and task ID to the client via the server to show the user that the current picture book production task is in a queued state. At this time, the client will initiate a task polling request based on the task ID to update the current page in real time.
[0130] In addition, in the process of asynchronously generating the picture book in the disclosed solution, the server sends the real-time progress of the picture book production task to the client in the form of a push notification through a message queue mechanism, so that the client can display it.
[0131] Step S705: When a page showing successful generation of the picture book is displayed, in response to a click operation on the preview function of the picture book, a play page for the picture book is displayed.
[0132] Here, in the playback page for the picture book, the target object can play pictures and texts, play voices, turn pages with special page turning effects, and pause and drag pages for the generated picture book.
[0133] For example, in one example, as shown in FIG7(b), in response to a preview operation on a picture book, the client generates a preview request based on the task ID corresponding to the picture book, and sends the preview request to the server. The server determines the task information associated with the task ID based on the task ID included in the received preview request, and generates a data acquisition request for the picture book data based on the determined task information, and sends the request to the task scheduling center. The task scheduling center returns the stored picture book data to the server based on the data acquisition request, and the server directly renders the picture book based on the picture book data, and sends the picture book directly to the client for the user to preview.
[0134] Step S706: In response to the editing operation on the sketchbook, an edited sketchbook is obtained.
[0135] Here, the editing operation can be used to edit the title in the picture book, each page image, and the text (or copywriting) and audio in each page image for regeneration. In this way, a picture book that satisfies the user can be obtained, meeting the user's personalized needs and improving the user's usage experience.
[0136] Thus, the disclosed solution has the following advantages and features in practical application, specifically including:
[0137] (1) Full life cycle automation. The disclosed solution realizes automated processing from user instructions, style selection to content generation, audio and video synthesis and optimization, thus effectively reducing labor costs and time costs.
[0138] (2) High availability. This public solution ensures the stability and availability of the system by introducing key technologies such as distributed microservice architecture, cloud deployment, and load balancing to meet the needs of large-scale, high-efficiency creation.
[0139] (3) Scalability. The disclosed solution supports dynamic expansion and scaling, and can increase or decrease the number of nodes according to actual needs to adapt to the creation needs of different scales, while achieving control over the production cost of the picture book.
[0140] (4) Automatic synthesis of audio and video. The disclosed solution also supports real-time generation of reading audio. After the user freely modifies the text, the corresponding audio data can be generated immediately. In this way, the instant audio generation function greatly facilitates the user to adjust and improve the content during use, thereby improving practicality and convenience.
[0141] (5) Innovation. Compared with the prior art, the disclosed solution supports editing and replacing single-page images and texts, and can generate more high-quality, diversified and user-expected picture book content through prompt word optimization and retry mechanism.
[0142] Furthermore, in practical applications, Figure 8 As shown, the disclosed solution provides a picture book generation system, which adopts a layered architecture, specifically including a data collection layer, a model training layer, a content generation layer, a quality assessment layer, and an optimization and feedback layer, specifically including:
[0143] (1) Data collection layer: used to collect a large amount of painting materials, style samples, user needs and other data, providing a rich data source for subsequent model training (such as image generation model).
[0144] (2) Model training layer: It is used to process and analyze the collected data using deep learning technology to train high-quality picture book models (such as image generation models). At the same time, this layer also has the ability to automatically update and optimize the model to adapt to the ever-changing data and user needs.
[0145] (3) Content generation layer: This layer is used to generate diverse picture book content based on user needs using the trained picture book model. In addition, this layer supports multiple creative styles to meet the different needs of users.
[0146] (4) Quality assessment layer: The quality of the generated picture book content is assessed to ensure that the quality of the content meets expectations. Here, the evaluation indicators may specifically include picture clarity, style consistency, content innovation, etc. In this way, the quality of the generated picture book content can be monitored in real time to ensure that the user experience is not affected.
[0147] (5) Optimization and feedback layer: This layer is used to optimize and adjust the picture book model based on the quality evaluation results (such as evaluation indicators) to improve the accuracy and efficiency of picture book content generation. At the same time, this layer is also responsible for collecting user feedback for further improvement and perfection. In this way, the accuracy of picture book generation of the large model is improved, thereby ensuring the stability and availability of the entire picture book generation system.
[0148] In this way, the disclosed solution can quickly and automatically generate higher-quality picture book content, realize the automated process of picture book generation, lower the threshold for picture book production, and enable users to obtain the picture books they need without spending a lot of time and effort, thereby improving the user's creative experience.
[0149] In summary, compared with the handmade picture books in the prior art, the disclosed solution has the following advantages, including:
[0150] First, automatically generate the content of the picture book. The disclosed solution can automatically identify and generate the content of the picture book through large model technology, effectively avoiding the tedious process of traditional manual production of the picture book and improving the production efficiency of the picture book.
[0151] Second, intelligent typesetting design. The disclosed solution can realize intelligent typesetting design through large model technology, for example, automatically and reasonably layout and adjust the contents of the picture book to make the picture book more beautiful and readable.
[0152] Third, personalized customization: The disclosed solution generates picture book content that meets the user's requirements according to the user's needs and preferences to meet the user's personalized needs.
[0153] Fourth, interactivity is enhanced. The disclosed solution can provide richer playback functions, such as picture and text playback, video playback, etc., through the large model technology to enhance the user's reading experience.
[0154] Fifth, cross-platform adaptability. The disclosed solution can also run on different operating systems and devices, thereby achieving cross-platform adaptability and facilitating users to use it in different environments.
[0155] Furthermore, the disclosed solution can be specifically applied to the following scenarios, including:
[0156] (1) Education. The disclosed solution can be built into a picture book generation tool suitable for children, so as to generate picture books suitable for children to read. The generated picture books cleverly combine vivid and interesting pictures with entertaining and educational texts to help children understand and learn knowledge. In addition, the disclosed solution can also be used as a tool assistant to assist teachers in teaching design, so as to provide students with richer learning resources.
[0157] (2) Publishing industry. The disclosed solution can provide publishing institutions with an automated and intelligent picture book production tool to improve the quality and efficiency of picture books and reduce production costs. At the same time, it can also provide individual creators with a fast and convenient way to create picture books, so that they can quickly complete their works.
[0158] (3) Digital media. The disclosed solution can be applied to digital media platforms to provide users with a personalized picture book reading experience. Moreover, through interactive functions and intelligent question-and-answer methods, the interactivity between users and picture books is enhanced, effectively improving users' reading interest and participation.
[0159] (4) Personal entertainment. The disclosed solution can also be applied to the field of personal entertainment, providing users with an easy and interesting picture book making tool. For example, users can create their own personalized picture books according to their own needs and preferences for reading, sharing and communication.
[0160] The disclosed solution also provides a picture book generating device, such as Fig. 9 As shown, including:
[0161] The text processing unit 901 is used to generate a story outline around the determined theme content based on the determined theme content; wherein the story outline includes N paragraphs around the theme content; wherein N is an integer greater than or equal to 2;
[0162] The multimedia generation unit 902 is used to perform storyboard understanding on each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph; wherein the plurality of prompt information for a paragraph can describe the scene image to which the paragraph corresponds; based on the plurality of prompt information for each of the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
[0163] In a specific example of the disclosed solution, the multimedia generation unit is further used to:
[0164] Based on the text content of each paragraph, generate audio data for each paragraph;
[0165] The audio data of each paragraph is assembled with the images in the target picture book to obtain a target multimedia picture book for the subject content.
[0166] In a specific example of the disclosed solution, the multimedia generation unit is specifically used to:
[0167] According to the story development sequence of the N paragraphs contained in the story outline, the images of each paragraph are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state, wherein the target picture book can dynamically display a book-turning effect around a preset symmetry axis during playback, and each image of the target picture book can be displayed in full size on the display interface.
[0168] In a specific example of the disclosed solution, the multimedia generation unit is specifically used to:
[0169] According to the story development sequence of the N paragraphs included in the story outline, determine the i-th image and the i+1-th image with a front-to-back display sequence;
[0170] According to the following superposition steps, the i-th image and the i+1-th image are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state:
[0171] A blank display area is superimposed on the layer above the i+1th image; wherein the area of the blank display area is the same size as the area of the i-th image;
[0172] Place the sub-image located on one side of the preset symmetry axis in the i-th image on one side of the blank display area;
[0173] Flipping the sub-image located on one side of the preset symmetry axis in the (i+1)th image, and attaching it to the back of the sub-image located on the other side of the preset symmetry axis in the (i)th image after flipping;
[0174] Place the bonded image on the other side of the blank display area to obtain a superimposed image.
[0175] In a specific example of the disclosed solution, the multimedia generation unit is specifically used to:
[0176] Determine the current segment that requires storyboard understanding;
[0177] Inputting the determined current segment and the preceding and following paragraphs of the current segment in the story outline into the target large language model, so as to perform storyboard understanding of the current segment using the target large language model;
[0178] A plurality of prompt information of the current segment is obtained, wherein the plurality of prompt information of the current segment can describe the scene picture of the current segment from multiple dimensions.
[0179] In a specific example of the disclosed solution, the multimedia generation unit is further used to:
[0180] Determine the target style of the subject matter;
[0181] Determine the image generation model corresponding to the target style;
[0182] Input multiple prompt information of each paragraph in the N paragraphs into the image generation model corresponding to the target style to obtain an image for each paragraph, wherein the image of each paragraph has the target style.
[0183] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.
[0184] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0185] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0186] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0187] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0188] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0189] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a picture book generation method. For example, in some embodiments, the picture book generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the picture book generation method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the picture book generating method in any other appropriate manner (eg, by means of firmware).
[0190] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0191] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0192] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0193] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0194] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0195] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0196] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0197] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating a picture book, comprising: Based on the determined theme content, generate a story outline around the theme content; wherein the story outline includes N paragraphs around the theme content; N is an integer greater than or equal to 2; Performing storyboard understanding on each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph; wherein the plurality of prompt information for a paragraph can describe the scene screen to which the paragraph relates; Based on the multiple prompt information of each paragraph in the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
2. The method according to claim 1, further comprising: Based on the text content of each paragraph, generate audio data for each paragraph; The audio data of each paragraph is assembled with the images in the target picture book to obtain a target multimedia picture book for the subject content.
3. The method according to claim 1 or 2, wherein: The step of obtaining an image for each paragraph to obtain a target picture book for the subject content includes: According to the story development sequence of the N paragraphs contained in the story outline, the images of each paragraph are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state, wherein the target picture book can dynamically display a book-turning effect around a preset symmetry axis during playback, and each image of the target picture book can be displayed in full size on the display interface.
4. The method according to claim 3, wherein: According to the story development sequence of the N paragraphs included in the story outline, the images of each paragraph are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state, including: According to the story development sequence of the N paragraphs included in the story outline, determine the i-th image and the i+1-th image with a front-to-back display sequence; According to the following superposition steps, the i-th image and the i+1-th image are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state: A blank display area is superimposed on the layer above the i+1th image; wherein the area of the blank display area is the same size as the area of the i-th image; Place the sub-image located on one side of the preset symmetry axis in the i-th image on one side of the blank display area; Flipping the sub-image located on one side of the preset symmetry axis in the (i+1)th image, and attaching it to the back of the sub-image located on the other side of the preset symmetry axis in the (i)th image after flipping; Place the bonded image on the other side of the blank display area to obtain a superimposed image.
5. The method according to any one of claims 1 to 4, wherein: The step of performing segmentation for each of the N paragraphs included in the story outline to obtain a plurality of prompt information for each paragraph includes: Determine the current segment that requires storyboard understanding; Inputting the determined current segment and the preceding and following paragraphs of the current segment in the story outline into the target large language model, so as to perform storyboard understanding of the current segment using the target large language model; A plurality of prompt information of the current segment is obtained, wherein the plurality of prompt information of the current segment can describe the scene picture of the current segment from multiple dimensions.
6. The method according to claim 5, further comprising: Determine the target style of the subject matter; Determine the image generation model corresponding to the target style; The step of obtaining an image for each paragraph based on the plurality of prompt information of each paragraph in the N paragraphs includes: Input multiple prompt information of each paragraph in the N paragraphs into the image generation model corresponding to the target style to obtain an image for each paragraph, wherein the image of each paragraph has the target style.
7. A picture book generating device, comprising: A text processing unit, configured to generate a story outline around the determined theme content based on the determined theme content; wherein the story outline includes N paragraphs around the theme content; wherein N is an integer greater than or equal to 2; A multimedia generation unit is used to perform storyboard understanding on each of the N paragraphs included in the story outline to obtain multiple prompt information for each paragraph; wherein the multiple prompt information for a paragraph can describe the scene image to which the paragraph corresponds; based on the multiple prompt information for each of the N paragraphs, an image for each paragraph is obtained to obtain a target picture book for the subject content.
8. The device according to claim 7, wherein: The multimedia generating unit is further used for: Based on the text content of each paragraph, generate audio data for each paragraph; The audio data of each paragraph is assembled with the images in the target picture book to obtain a target multimedia picture book for the subject content.
9. The device according to claim 7 or 8, wherein: The multimedia generating unit is specifically used for: According to the story development sequence of the N paragraphs contained in the story outline, the images of each paragraph are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state, wherein the target picture book can dynamically display a book-turning effect around a preset symmetry axis during playback, and each image of the target picture book can be displayed in full size on the display interface.
10. The device according to claim 9, wherein: The multimedia generating unit is specifically used for: According to the story development sequence of the N paragraphs included in the story outline, determine the i-th image and the i+1-th image with a front-to-back display sequence; According to the following superposition steps, the i-th image and the i+1-th image are superimposed to obtain the target picture book in which the images of each paragraph are in a superimposed state: A blank display area is superimposed on the layer above the i+1th image; wherein the area of the blank display area is the same size as the area of the i-th image; Place the sub-image located on one side of the preset symmetry axis in the i-th image on one side of the blank display area; Flipping the sub-image located on one side of the preset symmetry axis in the (i+1)th image, and attaching it to the back of the sub-image located on the other side of the preset symmetry axis in the (i)th image after flipping; Place the bonded image on the other side of the blank display area to obtain a superimposed image.
11. The device according to any one of claims 7 to 10, wherein: The multimedia generating unit is specifically used for: Determine the current segment that requires storyboard understanding; Inputting the determined current segment and the preceding and following paragraphs of the current segment in the story outline into the target large language model, so as to perform storyboard understanding of the current segment using the target large language model; A plurality of prompt information of the current segment is obtained, wherein the plurality of prompt information of the current segment can describe the scene picture of the current segment from multiple dimensions.
12. The device according to claim 11, wherein The multimedia generating unit is further used for: Determine the target style of the subject matter; Determine the image generation model corresponding to the target style; Input multiple prompt information of each paragraph in the N paragraphs into the image generation model corresponding to the target style to obtain an image for each paragraph, wherein the image of each paragraph has the target style.
13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Computer-assisted animation image-text continuity semi-automatic generating method
CN102184200A
Turning-over animation manufacturing method
CN103268625A
Special effect image processing method and device, electronic equipment and storage medium
CN114428573A
Generative artificial intelligence-based novel tweet video generation method and system
CN117078782A
Content generation method and device, computer equipment and storage medium
CN117171369A
Cited By
Picture book video generation method and device based on large model, electronic equipment and medium
CN120264098A
Station group article intelligent broadcasting system and method based on multi-mode emotional speech synthesis
CN121545494A
Picture album generation method
WO2026144565A1
