Consistency generation method of picture book image sequence

Abstract

The invention relates to the technical field of image generation, and discloses a picture book image sequence consistency generation method, which comprises the following steps: receiving picture book customization information input by a user, generating a picture book story text based on a natural language processing technology, and generating a picture book image sequence based on the picture book story text. The method comprises the following steps: respectively carrying out consistency image generation processing on a plurality of picture book roles to generate an initial image image of each role, and when consistency generation of a picture book image sequence is carried out, generating a customized picture book story text by receiving picture book customization information input by a user and based on a natural language processing technology; according to the method, an accurate text basis is provided for subsequent image generation, meanwhile, consistency image generation processing is performed on a plurality of picture book roles on the basis of the story text, iterative training is performed by adopting a LoRA low-rank adaptive fine tuning method, and high consistency of appearances, clothes and proportions of the roles in an image sequence is ensured.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, specifically to a method for generating consistent image sequences for picture books. Background Technology

[0002] Image generation is a computer technology that creates visual content through algorithms. It encompasses digital image encoding and compression processing and is applied in virtual reality, artistic creation, and film and television special effects. Its core principle is based on deep learning models learning the inherent laws of image data. Key technologies include generative adversarial networks, variational autoencoders, and diffusion models. High-quality image output is achieved through generator-discriminator adversarial training.

[0003] Currently, because the picture book generation process involves multiple independent technical modules, the generation model used when automatically generating picture book image sequences cannot effectively maintain the consistency of the same character's image in different scenes when processing a series of images. When the character's appearance, clothing, and proportions change, it will cause a disconnect in the story narrative and fail to guarantee the visual coherence of the picture book.

[0004] Therefore, a consistent generation method for picture book image sequences is proposed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for generating consistent image sequences in picture books, which solves the problem of disjointed story narration and inability to guarantee the visual coherence of picture books mentioned in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for generating consistency of picture book image sequences, the method comprising the following steps: S1. Receive user-inputted picture book customization information and generate picture book story text based on natural language processing technology; S2. Based on the picture book story text, perform consistent image generation processing on multiple picture book characters to generate an initial image image for each character; S3. Perform subject detection and background separation processing on the initial image to generate character segmentation image and background separation data; S4. Based on the separated text and background data of the picture book story, perform consistent background image sequence generation processing to generate a multi-view background image sequence; S5. The character segmentation image and the multi-view background image sequence are stitched together, and a consistent image sequence is generated by combining the image coordination algorithm. S6. Perform multimedia synthesis processing on the consistent image sequence and the picture book story text to generate an interactive multimedia picture book.

[0007] Preferably, generating the picture book story text in step S1 includes the following steps: S11. Receive information from the user regarding the picture book theme, character settings, and style preferences; S12. Use a pre-trained language model to parse user input information and generate a picture book story outline and detailed plot text; S13. Edit and optimize the story text based on user feedback to output customized picture book story text.

[0008] Preferably, generating the initial image of each character in S2 includes the following steps: S21. Based on the customized picture book story text, extract the descriptive text for each character; S22. Input the descriptive text into the diffusion model to generate a preliminary image of each character; S23. The LoRA low-rank adaptive fine-tuning method is used to iteratively train the preliminary image to generate a consistent character image.

[0009] Preferably, generating a consistent character image in step S23 includes the following steps: S231. Using the preliminary image as training data, train the first version of the character-customized LoRA model. S232. Generate multi-angle, multi-pose character images using the current LoRA model; S233. Use the generated character images as new training data to iteratively train the next version of the LoRA model; S234. Repeat the iteration process until the consistency of the character image reaches the preset threshold.

[0010] Preferably, the generation of character segmentation images and background separation data in S3 includes the following steps: S31. Use the SegmentAnything algorithm to perform subject detection on the initial image and identify the character region; S32. Separate the character area from the background to generate a character segmentation image and background mask data; S33. Clean up the background area to ensure that there are no interfering elements in the background after separation.

[0011] Preferably, generating the multi-view background image sequence in S4 includes the following steps: S41. Based on the plot requirements of the picture book story text, determine the background style and perspective requirements; S42. Use a diffusion model to generate a high-resolution base background image; S43. Combining ViewCrafter's new perspective synthesis technology, the basic background image is expanded from multiple perspectives to generate a sequence of distant and near-field background images. S44. Adjust the layout consistency of the background image sequence by cropping and blending methods.

[0012] Preferably, generating a consistent image sequence in step S5 includes the following steps: S51. Align the character segmentation image with the multi-view background image sequence according to the story timeline; S52. Use image stitching technology to blend the character image into the background image to ensure the character's position is reasonable; S53. Use the CDTNet color distribution transmission network to perform color coordination processing on the stitched images to unify the color tone; S54. Apply a shadow generation algorithm to add natural shadows around the character to enhance the realism of the image.

[0013] Preferably, the color coordination processing of the stitched image in S53 includes the following steps: S531. Extract the color distribution features of the stitched image and perform histogram matching; S532, Adjust the brightness, contrast, and saturation parameters of the character and background using CDTNet; S533: Optimize color transitions based on the lighting model to ensure visual consistency.

[0014] Preferably, the generation of the interactive multimedia picture book in step S6 includes the following steps: S61. Synchronize the consistent image sequence with the picture book story text according to page layout; S62. Call the language model to generate audio for the story narration, and use a pre-trained cartoon speech model to synthesize speech; S63. Match background music from the sound effects library and merge it with the narration audio to generate a multimedia audio stream; S64. Combine the image sequence with the audio stream to output an interactive electronic picture book file.

[0015] Preferably, the output of the interactive electronic picture book file in S64 includes the following steps: S641. Add interactive controls to each picture book page to allow users to click to switch pages and play audio; S642, encapsulates image and audio data into standard multimedia formats to generate an executable electronic picture book application; S643. Perform a consistency check to ensure that the image sequence and the story text are aligned in time.

[0016] Compared with existing technologies, the present invention provides a method for generating consistent image sequences in picture books, which has the following beneficial effects: 1. In this invention, when generating consistent image sequences of picture books, customized picture book information input by the user is received and customized picture book story text is generated based on natural language processing technology, providing accurate textual basis for subsequent image generation. At the same time, based on this story text, consistent image generation processing is performed on multiple picture book characters, and iterative training is carried out using the LoRA low-rank adaptive fine-tuning method. This can effectively solve the problem of unstable image features of the same character in different scenes, ensuring a high degree of consistency in the appearance, clothing and proportions of the characters in the image sequence, avoiding narrative discontinuity caused by sudden changes in character image, and improving the visual coherence of the picture book.

[0017] 2. In this invention, when generating consistent picture book image sequences, consistent background image sequence generation is performed based on the separation data of the picture book story text and background. A new perspective synthesis technique is used to generate a multi-perspective background image sequence that matches the development of the story plot. This enables the system to intelligently maintain the spatial logical relationship of the background environment in different story segments, effectively avoiding scene perspective confusion and layout conflicts. At the same time, in the image stitching stage, an image coordination algorithm is combined for fusion processing to ensure the naturalness of the character and scene fusion and improve the overall spatial consistency and realism of the image sequence.

[0018] 3. In this invention, when generating a consistent sequence of picture book images, the character segmentation images are stitched together with a multi-view background image sequence, and an image coordination algorithm is applied to uniformly adjust the color, tone, and lighting and shadow. This achieves coordinated control of multiple elements such as characters, background, and style, enabling the system to effectively avoid style jumps and color mismatches in the final synthesized consistent image sequence. Furthermore, by combining the consistent image sequence with the picture book story text, a high-quality interactive multimedia picture book is generated, enhancing the overall artistic expression and user experience of the picture book. Attached Figure Description

[0019] Figure 1 This is a flowchart of a method for generating a consistent sequence of picture book images according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 The method for generating consistency of picture book image sequences includes the following steps: S1. Receive user-inputted picture book customization information and generate picture book story text based on natural language processing technology; S2. Based on the picture book story text, perform consistent image generation processing on multiple picture book characters to generate an initial image image for each character; S3. Perform subject detection and background separation processing on the initial image to generate character segmentation image and background separation data; S4. Based on the separated data of picture book story text and background, perform consistent background image sequence generation processing to generate a multi-view background image sequence. S5. The character segmentation image is stitched together with the multi-view background image sequence, and a consistent image sequence is generated by combining the image coordination algorithm. S6. Perform multimedia synthesis processing on the consistent image sequence and the picture book story text to generate an interactive multimedia picture book; Generating picture book story text in S1 includes the following steps: S11. Receive information from the user regarding the picture book theme, character settings, and style preferences; S12. A pre-trained language model is used to parse the user input information and generate a picture book story outline and detailed plot text. The specific process is as follows: First, the system receives picture book customization information input by the user. This information usually includes the picture book theme, character settings, and style preferences. The task of the pre-trained language model is to accurately extract these key elements and understand the user's fundamental intention. Based on its massive pre-trained knowledge, the model performs deep semantic analysis on the extracted key information. When the topic is "space exploration", the model will activate the relevant knowledge graph, including scenes, plot elements and related scientific concepts. At the same time, the model will understand the character traits of "bravery" and "intelligence" and infer how these traits are reflected in the story through the character's behavior and dialogue. This stage ensures that the story content not only meets the user's requirements, but also has internal logic and common sense correctness. Based on a deep understanding of user needs, the model begins content creation. It first generates a story outline, outlining the story's development: "Beginning: The astronauts receive a mission; Development: They encounter difficulties during their adventure; Climax: Doraemon uses his intelligence to solve the problem; Ending: They successfully return and gain friendship." Then, the model generates detailed plot text around this outline, filling in specific dialogues, action descriptions, and environmental descriptions to make the story richer and more vivid. After generating the initial story text, the model will perform stylistic adaptation to ensure that the language style meets user preferences and the specific requirements of the picture book. At the same time, the model will optimize the text itself, checking and correcting grammatical errors, logical contradictions and unclear expressions to improve the overall quality of the text. User feedback will be parsed and understood by the model as new input information, thereby driving the model to revise and optimize the story until it outputs a customized picture book story text that fully meets the user's needs. S13. Edit and optimize the story text based on user feedback to output customized picture book story text; Generating the initial image of each character in S2 includes the following steps: S21. Based on the customized picture book story text, extract the descriptive text for each character; S22. Input the descriptive text into the diffusion model to generate a preliminary image of each character; S23. The LoRA low-rank adaptive fine-tuning method is used to iteratively train the initial image to generate a consistent character image. The basic principle of the LoRA method: LoRA is an efficient fine-tuning technique suitable for large-scale pre-trained models. Its core idea is to approximate the update amount of model weights through low-rank decomposition, thereby achieving accurate adaptation while reducing computational resources. Specifically, for pre-trained weight matrices... Its update volume This can be represented as a product of low-rank matrices: ,in , , and rank much smaller and During training, the original weights Frozen, only training low-rank matrices and The parameters in the equation make the forward propagation become This approach reduces the number of trainable parameters and improves fine-tuning efficiency, making it particularly suitable for personalized customization of picture book character images. The specific steps of iterative training are as follows: The first version of LoRA model training: The image of the character generated initially is used as the training dataset. These images are usually generated by the diffusion model based on natural language descriptions, but there will be inconsistencies in details. During training, the images are input into the pre-trained model and its backbone weights are frozen. The character-specific features are learned only through the LoRA module. The training objective is to minimize the difference between the generated images and the expected features. The loss function often uses perceptual loss and feature matching loss. Multi-angle image generation: Using the first version of the trained LoRA model, a set of character images with multiple angles and poses are generated. This step aims to expand the training data, cover various scenes in which the character appears, and provide diverse samples for subsequent iterations. During the generation process, the model will combine text prompts to ensure semantic alignment. Training data update and iteration: The newly generated multi-angle images are used as a new training dataset to replace and supplement the original data. Then, the next version of the LoRA model is trained based on the updated dataset. The LoRA parameters are fine-tuned in each iteration so that the model gradually learns more stable character features. The number of iterations is usually controlled according to the preset convergence conditions. The iteration stops when the change rate of character features is lower than the threshold for several consecutive iterations. Consistency assessment and convergence: After each iteration, the consistency of the generated images is evaluated using both manual and automated metrics. If the convergence criteria are not met, the process returns to the multi-angle image generation step to continue the loop; otherwise, the final LoRA model is output to generate a consistent character image. The generation of a consistent character image in S23 includes the following steps: S231. Using the initial image as training data, train the first version of the character-customized LoRA model. S232. Generate multi-angle, multi-pose character images using the current LoRA model; S233. Use the generated character images as new training data to iteratively train the next version of the LoRA model; S234. Repeat the iterative process until the consistency of the character image reaches the preset threshold. The generation of character segmentation images and background separation data in S3 includes the following steps: S31. Use the SegmentAnything algorithm to perform subject detection on the initial image and identify the character region. The specific process is as follows: First, the system loads the pre-trained SegmentAnything model, which is based on a powerful visual Transformer architecture and can perform a global understanding of the initial input image. When the image is input into the model, the SAM image encoder converts the entire image into a high-dimensional feature embedding vector. This process can be understood as the model performing a "deep understanding" of the image content, extracting the visual features of all potential objects to prepare for subsequent pixel-level recognition. This encoding process only needs to be executed once to support fast segmentation under various prompts. Next, the SAM model needs to be given "cues" to guide it to focus on a specific target in the image—namely, the picture book character. The sources of these cues are: Automatic prompting: Combining text information, a prompt generation module automatically generates a focus box and point of view containing the character on the image; Interactive prompts: In certain areas requiring fine-tuning, users can provide prompts through simple clicks and selections; After receiving these prompts, the SAM model's prompt encoder converts the prompt information into a corresponding feature vector. Then, the prompt features are fused with the previously generated image features in the decoder. During the decoder stage, SAM integrates image features and cue features to generate multiple segmentation masks. Each mask is a binary matrix of the same size as the input image, where white areas represent target objects identified by the model and black areas represent the background. A key feature of SAM is that it can generate multiple valid and reasonable masks at the same time, generating three segmentation results of different granularities for a cue point. Finally, the model evaluates the generated masks and selects the mask with the highest confidence as the final segmentation result. This optimal mask accurately outlines the contour of the character region. The system then uses this mask to accurately "extract" the character pixels from the initial image and generate a transparent character segmentation image while preserving the original image quality. The area outside the mask is processed as background separation data, which is prepared for seamless stitching with the independently generated background image in the future. S32. Separate the character area from the background to generate a character segmentation image and background mask data; S33. Clean up the background area to ensure that there are no interfering elements in the background after separation; The generation of a multi-view background image sequence in S4 includes the following steps: S41. Based on the plot requirements of the picture book story text, determine the background style and perspective requirements; S42. Generate a high-resolution base background image using a diffusion model. The specific process is as follows: The first step is to transform the abstract text description into guiding signals for the model's understanding. The system extracts key descriptions about the background from the picture book story text, and these text cues are converted into a series of high-dimensional text embedding vectors by a text encoder. These text embedding vectors are then fed into the cross-attention layer of the U-Net network, the core component of the diffusion model. In each iteration of the model's image generation, U-Net makes the image features "pay attention" to these text features, thereby ensuring that the final generated image content is highly aligned with the text description, which provides precise semantic control for the generation of background images. In each iteration, the task of the U-Net network is to predict the noise contained in the current image. U-Net is a complex encoder-decoder structure. Its encoder continuously extracts deep features of the image through downsampling, while the decoder gradually reconstructs image details by upsampling combined with skip connections. Its internal ResNet layers and VisionTransformer blocks, especially the self-attention and cross-attention mechanisms, enable it to effectively fuse image information and text conditions. Noise removal: Based on the predicted noise, the model will subtract some of the noise from the current image to obtain a slightly clearer image. This process is similar to a painter first sketching the general outline and then refining the details step by step. The process of "predicting noise - removing noise" is repeated dozens or even hundreds of times. As the iteration progresses, the image becomes clearer and clearer, and the scene described by the text gradually emerges from the chaos, eventually generating a high-resolution, detailed base background image. S43. Combining ViewCrafter's new perspective compositing technology, the base background image is expanded from multiple perspectives to generate a sequence of foreground and background images. The specific process is as follows: First, the system uses a diffusion model to generate a high-resolution base background image based on the picture book story text. Then, the ViewCrafter technology analyzes this base image to extract its implicit depth information and 3D spatial structure. This step is usually achieved through a pre-trained depth estimation model, which can infer the relative distances and layouts of objects in the scene from a 2D image, thus providing a geometric basis for subsequent view synthesis. The generation of the depth map enables the model to understand the spatial relationships of the scene. After acquiring depth information, ViewCrafter's core algorithm begins to work. It uses a novel perspective synthesis technique to simulate the effect of a camera capturing the same scene from different positions and angles. Specifically, the technique uses a differentiable rendering method based on a base image and its depth map to generate a series of images with transformed perspectives. This process relies on a perspective transformation matrix, which defines the trajectory of the virtual camera and ensures that the generated perspectives conform to the perspective rules of the real world. ViewCrafter learns the continuous representation of the scene through a neural network, thereby enabling it to interpolate and extrapolate smoothly transitioned perspective sequences, avoiding distortion and breaks. Based on generating multi-view images, ViewCrafter further combines cropping methods to achieve switching between near and far views. The system will crop regions of different scales from a high-resolution base background image to simulate the effect of camera zoom. At the same time, ViewCrafter will ensure that the cropped image is consistent with the original background in style and lighting to avoid color and texture abrupt changes. This step is usually carried out in conjunction with multi-view expansion. During the cropping process, the model will smooth the edges through image fusion algorithms to enhance visual naturalness. Finally, the system performs consistency checks on the generated multi-view background image sequence. By calculating the structural similarity and feature distance between adjacent images in the sequence, it ensures that the layout and style of background elements remain consistent when the viewpoint is switched. When inconsistent areas are detected, the model automatically performs color correction and texture optimization. In the end, it outputs a multi-view, multi-fragment background image sequence, which can be directly applied to subsequent splicing with character images, providing a dynamic and consistent scene foundation for the picture book. The entire process not only enhances the diversity of the background, but also maintains the spatial logic through intelligent technology, effectively solving the problem of viewpoint chaos in traditional methods. S44. Adjust the layout consistency of the background image sequence through cropping and merging methods; Generating a consistent image sequence in S5 includes the following steps: S51. Align the character segmentation images with the multi-view background image sequence according to the story chronology; S52. Use image stitching technology to blend the character image into the background image to ensure the character's position is reasonable; S53. Use the CDTNet color distribution transmission network to perform color coordination processing on the stitched images to unify the color tone. The specific process is as follows: First, CDTNet performs depth analysis on the two parts of the stitched image separately: Background image color feature extraction: The network takes the background image as the source and inputs it into a feature encoder to extract its global color statistical features, which is equivalent to learning the "color atmosphere" and "style fingerprint" of the background image; Foreground character image color feature analysis: At the same time, the network takes the foreground character image as the target and analyzes its current color attributes. The network will identify different materials and surfaces in the character area and analyze its original lighting and color information. The purpose of this step is to enable the network to fully understand the "target style" of the background and the "original state" of the foreground, so as to establish an accurate correspondence for subsequent color transfer; This is the core step of CDTNet: the network learns the background color distribution features and maps them onto the foreground character image through a learnable transformation. Specifically: Distribution alignment: CDTNet calculates the distribution differences between the background and foreground character images in the color space. Then, it adjusts the color distribution of the foreground character image through a carefully designed transmission module to match its statistical properties with the background image. Local adaptive adjustment: In order to avoid the distortion of the character caused by simple global color overlay, CDTNet will perform local adaptive processing. It will apply different degrees of color adjustment to different areas according to the semantic information of the character to ensure that the color change is natural and in accordance with physical laws. After completing the macroscopic color distribution transmission, CDTNet further processes the details to achieve seamless integration: Edge smoothing and detail preservation: The network prevents the image from becoming blurry during the color migration process by preserving the texture and detail information of the foreground character image. It ensures that the outline of the character and the details of the clothing folds remain clear even when the color is changed. Collaboration with shadow generation algorithms: CDTNet's processing works closely with shadow generation algorithms. After adjusting the colors, it generates soft, natural shadows around the character that match the new color tone environment based on the background lighting direction. This makes the character look like it is truly "inside" the background, enhancing the image's three-dimensionality and realism. Ultimately, the stitched image processed by CDTNet achieves a high degree of color harmony and a natural and unified visual effect. S54. Apply a shadow generation algorithm to add natural shadows around the character to enhance the realism of the image; The color coordination processing of the stitched image in S53 includes the following steps: S531. Extract the color distribution features of the stitched image and perform histogram matching; S532, Adjust the brightness, contrast, and saturation parameters of the character and background using CDTNet; S533: Optimize color transitions based on the lighting model to ensure visual consistency; Generating interactive multimedia picture books in S6 includes the following steps: S61. Synchronize the consistent image sequence with the picture book story text according to page layout; S62. Use the language model to generate audio for the story narration, and use a pre-trained cartoon speech model to synthesize the speech. The specific process is as follows: First, the system will call a large language model to generate a specific narration script based on the generated picture book story text. The LLM model will perform deep semantic analysis on the story text, extract key plot nodes and character dialogues, and convert them into narration text suitable for voice playback. This process focuses on child-friendly language, avoids complex sentence structures, and incorporates interjections to enhance expressiveness. The LLM model uses its pre-trained knowledge base to ensure that the narration text is logically synchronized with the story image sequence and matches the overall style of the picture book. After generating the narration text, the system uses a cartoon speech model pre-trained specifically for children's picture books to synthesize the text into audio. This model achieves high-quality speech output through the following sub-steps: Customizable voice: The cartoon voice model learned the voices of various cartoon characters during the training phase and can automatically match the voice to the story characters; Emotion and rhythm control: The model analyzes the emotional tone of the narration text and simulates the rhythm of real storytelling by adjusting the speech rate, pitch and pauses; Audio generation and optimization: The model converts the text sequence into a Mel spectrogram, and then generates waveform audio through a vocoder. Before output, the system performs noise reduction and volume equalization to ensure audio clarity. The entire process does not require manual recording, achieving efficient and consistent speech production. Based on the generated narration audio, the system will simultaneously integrate background music to enhance the narrative atmosphere. Specific steps include: Background sound matching: The LLM model is used to analyze the emotional tone of the story text and automatically match the most suitable background music segment from the preset sound effects library. The music selection focuses on complementing the spectrum of the narration audio to avoid masking the speech content. Time alignment and blending: The system blends the narration audio and background music in layers according to the time sequence of the picture book pages. The relative volume and fade-in / fade-out effects of the audio tracks are adjusted by audio editing tools to ensure that the narration is clear and prominent, while the background music is faint and indistinct. Output a unified audio stream: The merged audio is exported as a single file, with one audio segment corresponding to each page of the picture book, laying the foundation for subsequent synchronization with the image sequence; Finally, the system encapsulates the audio stream with a consistent image sequence to generate an interactive electronic picture book. This step is achieved using a video compositing tool. Timing synchronization: The system aligns the image sequence with the audio stream frame by frame based on the pagination of the story text and the duration of the narration; Interactive features added: Based on the synthesized video file, interactive controls are embedded to allow children to control audio playback independently while reading. The final output is an electronic picture book application that can run on mobile devices and web pages, completing the transformation from static images to dynamic multimedia experience. S63. Match background music from the sound effects library and merge it with the narration audio to generate a multimedia audio stream; S64. Combine the image sequence with the audio stream to output an interactive electronic picture book file; The steps involved in outputting an interactive electronic picture book file in S64 are as follows: S641. Add interactive controls to each picture book page to allow users to click to switch pages and play audio; S642, encapsulates image and audio data into standard multimedia formats to generate an executable electronic picture book application; S643. Perform a consistency check to ensure that the image sequence and the story text are aligned in time.

[0022] The steps of a method for generating consistency in picture book image sequences are as follows: Step 1: Generation and Semantic Analysis of Story Text Based on natural language processing technology, the system receives customized information input by the user and performs deep semantic parsing through a pre-trained language model. The model first extracts key elements and then generates a logically coherent story outline and detailed text based on a knowledge graph. The principle behind this step is to use the generative capabilities of the language model to transform abstract requirements into a structured narrative, providing accurate textual basis for subsequent image generation and ensuring that the story logic is aligned with the image content. After the text is generated, the system incorporates user feedback through an iterative optimization mechanism to ensure the customization and consistency of the text.

[0023] Step 2: Generating Consistency in Character Image The principle relies on the diffusion model and LoRA low-rank adaptive fine-tuning technology. The system extracts character descriptions from the story text and inputs them into the diffusion model to generate preliminary image images. The principle of LoRA technology is to fine-tune the weights of the pre-trained model through low-rank decomposition, training only a small number of parameters to learn the character-specific features and avoid overfitting. During the iterative training process, the system uses the preliminary image as training data to generate character images from multiple angles and continuously updates the dataset. The model is optimized through the feature matching loss function until the rate of change of the character's appearance, clothing, and proportions is lower than the threshold. This principle ensures the feature stability of the character in different scenes and solves the problem of sudden image changes.

[0024] Step 3: Separating the character from the background The principle is based on the computer vision algorithm, the SegmentAnything model. This model uses a visual Transformer architecture to globally encode the image, identify the character region, and generate a segmentation mask. The separation principle involves pixel-level segmentation: the model locates the character based on the prompts, fuses the image and prompt features through a decoder, and outputs a high-precision mask. Subsequently, the system cleans up the background region, removes interfering elements, and generates a clean character segmentation image and background mask data. The principle of this step is to ensure that the character and the background are processed independently, laying the foundation for subsequent stitching and avoiding artistic distortion during fusion.

[0025] Step 4: Multi-view synthesis of background image sequences The principle combines the diffusion model and the ViewCrafter novel perspective synthesis technology. The diffusion model generates a high-resolution base background image based on the text description. Its principle is to denoise the image content with the text semantics through iterative denoising using the U-Net network. The ViewCrafter technology is based on depth estimation and differentiable rendering: the system infers the three-dimensional spatial structure from the base image, simulates camera movement through the perspective transformation matrix, and generates a sequence of near and far views. In principle, this method maintains the spatial logical consistency of the background, adjusts the layout through cropping and blending, avoids perspective confusion, and ensures a natural transition between scenes.

[0026] Step 5: Image stitching and color coordination The principle involves image stitching algorithms and the CDTNet color distribution transmission network. The system first aligns the character segmentation image with the background sequence in time. Then, it merges the character into the background through stitching technology. The principle of CDTNet is to extract the color distribution features of the background and map them onto the character image through learnable transformations to achieve distribution alignment and local adaptive adjustment. At the same time, it optimizes color transitions based on the lighting model and adds natural shadows to enhance realism. The principle of this step is to unify the tone and lighting through statistical matching and physical simulation to ensure visual harmony between the character and the background.

[0027] Step 6: Multimedia Synthesis and Interactive Output The principle utilizes language models and audio synthesis technology to generate narration audio, which is synchronized with the image sequence. The system calls a pre-trained cartoon voice model to convert the text into emotional speech and matches background music through a sound effects library. The multimedia encapsulation principle is based on a time-series alignment tool to merge image frames with audio streams and add interactive controls. A consistency verification mechanism ensures that the timing of images and text is correct, and finally outputs a standard format electronic picture book. This principle improves the user experience and enhances the immersion of the picture book through multimodal interaction.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0029] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating consistency in a picture book image sequence, characterized in that, The method includes the following steps: S1. Receive user-inputted picture book customization information and generate picture book story text based on natural language processing technology; S2. Based on the picture book story text, perform consistent image generation processing on multiple picture book characters to generate an initial image image for each character; S3. Perform subject detection and background separation processing on the initial image to generate character segmentation image and background separation data; S4. Based on the separated text and background data of the picture book story, perform consistent background image sequence generation processing to generate a multi-view background image sequence; S5. The character segmentation image and the multi-view background image sequence are stitched together, and a consistent image sequence is generated by combining the image coordination algorithm. S6. Perform multimedia synthesis processing on the consistent image sequence and the picture book story text to generate an interactive multimedia picture book.

2. The method for generating consistency of a picture book image sequence according to claim 1, characterized in that, The generation of picture book story text in S1 includes the following steps: S11. Receive information from the user regarding the picture book theme, character settings, and style preferences; S12. Use a pre-trained language model to parse user input information and generate a picture book story outline and detailed plot text; S13. Edit and optimize the story text based on user feedback to output customized picture book story text.

3. The method for generating consistency of picture book image sequences according to claim 1, characterized in that, The initial image of each character in S2 includes the following steps: S21. Based on the customized picture book story text, extract the descriptive text for each character; S22. Input the descriptive text into the diffusion model to generate a preliminary image of each character; S23. The LoRA low-rank adaptive fine-tuning method is used to iteratively train the preliminary image to generate a consistent character image.

4. The method for generating consistency of a picture book image sequence according to claim 3, characterized in that, The generation of a consistent character image in S23 includes the following steps: S231. Using the preliminary image as training data, train the first version of the character-customized LoRA model. S232. Generate multi-angle, multi-pose character images using the current LoRA model; S233. Use the generated character images as new training data to iteratively train the next version of the LoRA model; S234. Repeat the iteration process until the consistency of the character image reaches the preset threshold.

5. The method for generating consistency of a picture book image sequence according to claim 1, characterized in that, The generation of character segmentation images and background separation data in S3 includes the following steps: S31. Use the SegmentAnything algorithm to perform subject detection on the initial image and identify the character region; S32. Separate the character area from the background to generate a character segmentation image and background mask data; S33. Clean up the background area to ensure that there are no interfering elements in the background after separation.

6. The method for generating consistency of a picture book image sequence according to claim 1, characterized in that, The generation of the multi-view background image sequence in S4 includes the following steps: S41. Based on the plot requirements of the picture book story text, determine the background style and perspective requirements; S42. Use a diffusion model to generate a high-resolution base background image; S43. Combining ViewCrafter's new perspective synthesis technology, the basic background image is expanded from multiple perspectives to generate a sequence of distant and near-field background images. S44. Adjust the layout consistency of the background image sequence by cropping and blending methods.

7. The method for generating consistency of a picture book image sequence according to claim 1, characterized in that, The generation of a consistent image sequence in S5 includes the following steps: S51. Align the character segmentation image with the multi-view background image sequence according to the story timeline; S52. Use image stitching technology to blend the character image into the background image to ensure the character's position is reasonable; S53. Use the CDTNet color distribution transmission network to perform color coordination processing on the stitched images to unify the color tone; S54. Apply a shadow generation algorithm to add natural shadows around the character to enhance the realism of the image.

8. The method for generating consistency of a picture book image sequence according to claim 7, characterized in that, The color coordination processing of the stitched image in S53 includes the following steps: S531. Extract the color distribution features of the stitched image and perform histogram matching; S532, Adjust the brightness, contrast, and saturation parameters of the character and background using CDTNet; S533: Optimize color transitions based on the lighting model to ensure visual consistency.

9. The method for generating consistency of a picture book image sequence according to claim 1, characterized in that, The process of generating an interactive multimedia picture book in step S6 includes the following steps: S61. Synchronize the consistent image sequence with the picture book story text according to page layout; S62. Call the language model to generate audio for the story narration, and use a pre-trained cartoon speech model to synthesize speech; S63. Match background music from the sound effects library and merge it with the narration audio to generate a multimedia audio stream; S64. Combine the image sequence with the audio stream to output an interactive electronic picture book file.

10. The method for generating consistency of a picture book image sequence according to claim 9, characterized in that, The output of the interactive electronic picture book file in S64 includes the following steps: S641. Add interactive controls to each picture book page to allow users to click to switch pages and play audio; S642, encapsulates image and audio data into standard multimedia formats to generate an executable electronic picture book application; S643. Perform a consistency check to ensure that the image sequence and the story text are aligned in time.

Images