A story picture book image sequence generation method and system

Abstract

This invention relates to the field of intelligent creation technology and discloses a method and system for generating story picture book image sequences. The method includes: extracting user aesthetic preference features based on personalized image data, environmental visual data, and historical dialogue data; performing semantic analysis on story text prompts, and extracting global character features, action semantic features, and character state features from the story text prompts based on the results of the semantic analysis; constructing cross-frame enhanced character features based on the character state features of three consecutive frames; fusing aesthetic preference features, action semantic features, and cross-frame enhanced character features through multimodal feature fusion to obtain fused features, and injecting the fused features into a diffusion model to generate a visually coherent story picture book image sequence that conforms to the user's aesthetic preferences. This invention solves the problems of excessively high similarity in picture book styles and difficulty in integrating character style and action coherence with the picture book style in existing story picture book image sequence generation methods.

Description

Technical Field

[0001] This invention relates to the field of intelligent creation technology, and in particular to a method and system for generating image sequences for story picture books. Background Technology

[0002] With the rapid development of artificial intelligence, especially generative models, image generation has become a hot research area in multimodal studies. Among these, storybook generation has broad and profound application value in the field of education, particularly in early childhood education, language development, cognitive cultivation, and emotional education. Compared to storybooks created by traditional illustrators, AI-generated content (AIGC) storybooks have the characteristics of low threshold, low cost, and high scalability, and have become a current trend in storybook creation. However, current AIGC-based storybook image sequence generation methods simply process static single images, lacking inter-frame consistency and character state modeling. This often leads to the generated content affecting character consistency. Mainstream storybook generation technologies rely on large language models or pre-trained language models, starting with text and performing multi-stage processing, then using a fine-tuned text-to-image model to generate the final picture book sequence. However, although the concept of "dynamic effect sequence" has been proposed in this generation method, it is essentially still based on the post-processing enhancement of static images. The generated content lacks temporal modeling and action semantic understanding, and cannot realistically depict the continuity of dynamic scenes and the coherence of character behavior in the story. At the same time, it mostly relies on large models to generate text to images, but due to the lack of a deep understanding of the plot structure, character settings and cultural context, the generated images often have inconsistencies with the characters, deviations in scene semantics, and even the loss of cultural elements.

[0003] To address the aforementioned issues, existing technologies have proposed two solutions. The first is to improve the visual adaptation strategy by introducing more refined image priors. This involves using pose features and appearance decomposition modules to alleviate the adapter's shortcomings in fine-grained identity extraction and precise character customization. The second is to optimize the customized modeling mechanism to improve generalization and instruction alignment capabilities. This is achieved through techniques such as regularized concept modeling, data augmentation, and low-rank adaptation to mitigate the overfitting problem inherent in customized methods. However, these solutions all employ additional spatial input to rigorously determine the regional allocation for different characters. When generating picture book image scenes, to ensure scene consistency and reduce layout conflicts between picture book images, a large number of identical images are introduced. This results in excessively high similarity in the style of the generated picture book images, making it difficult to create distinct picture book styles and ensuring deep integration between character styles and picture book scenes. Furthermore, it can easily lead to insufficient continuity in character movements. Therefore, existing methods for generating story picture book image sequences suffer from problems such as excessively high similarity in picture book styles and difficulty in integrating character styles and movement continuity with the overall picture book style. Summary of the Invention

[0004] This invention provides a method and system for generating image sequences for story picture books, in order to solve the problems of excessive similarity in picture book styles and difficulty in integrating character styles and action coherence with the picture book style in existing story picture book creation methods.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a method for generating image sequences for story picture books, comprising: S1. Obtain user-input image data, environmental visual data, historical dialogue data, and story text prompts; S2. Extract the user's aesthetic preference features based on the image data, the environmental visual data, and the historical dialogue data; S3. Perform semantic analysis on the story text prompts, and extract global character features, action semantic features, and character state features from the story text prompts based on the results of the semantic analysis. S4. Construct cross-frame enhanced character features based on the character state features of three consecutive frames; S5. The aesthetic preference features, the action semantic features, and the cross-frame enhanced character features are fused into multimodal features to obtain fused features. A diffusion model is constructed based on the fused features and the global character features. A visually coherent storybook image sequence that conforms to the user's aesthetic preferences is generated through the diffusion model.

[0007] Optionally, in step S2, extracting the user's aesthetic preference features based on the image data, the environmental visual data, and the historical dialogue data includes: S21. A pre-trained visual Transformer model is used to perform convolution calculations on the image data to extract texture features, composition features, and brushstroke style. The extracted texture features, composition features, and brushstroke style are then transformed into high-dimensional style feature vectors. , , where I is a collection of image data containing texture features, composition features, and brushstroke style; S22. A semantic segmentation model is used to extract depth features from the environmental visual data. Based on the depth features, the environmental visual data is segmented to obtain multiple semantic regions. The area weight and spatial coordinates of each semantic region are calculated to generate a spatial layout feature vector. , ,in, This represents the category label of the i-th semantic region. This indicates the area weight of the region in the image. Indicates the relative spatial coordinates of the region; S23. Use a pre-trained BERT model to encode historical dialogue data into historical dialogue data vectors. , ,in, Indicates the user's time Historical dialogue data is used, and a time decay weight model is introduced to apply time weights to the initial aesthetic vectors at different times. The time weights satisfy the following relationship: ; In the formula, Indicates time weighting, Indicates the time decay coefficient. This indicates the time interval between the interaction information at a certain moment and the current time. The user's historical dialogue data vector features are obtained by weighted fusion of historical dialogue data vectors. The weighted fusion satisfies the following relationship: ; In the formula, Represents the vector features of historical dialogue data; For the high-dimensional style feature vector The spatial layout feature vector and the historical dialogue data feature vector Weighted fusion is performed to obtain the user's aesthetic preference characteristics, where the weighted fusion satisfies the following relationship: ; In the formula, This indicates the aesthetic preference characteristics obtained after fusion. , , These are the weights of the high-dimensional style feature vector, the spatial layout feature vector, and the historical dialogue data feature vector, respectively.

[0008] Optionally, semantic parsing is performed on the story text prompts, and based on the results of the semantic parsing, global character features, action semantic features, and character state features are extracted from the story text prompts, including: S31. Input the story text prompts into a pre-trained large language model to identify all character description information in the story text prompts. Construct a structured attribute set based on the character description information, and extract the character global features of cross-segment invariant attributes based on the structured attribute set. The character global features satisfy the following relationship: ; In the formula, For semantic encoders, As a structured attribute set, the text description is compressed into a fixed-dimensional global feature vector. ; S32. Use a named entity recognition model to locate character entities in story text prompts, and extract action triples of character entities using a Transformer-based semantic character annotation model. , ,in, E represents the agent, and E represents the event that occurred. The recipient is represented, and the action triple is encoded using BERT and fused with MLP to generate action semantic features. The action semantic features satisfy the following relationship: ; In the formula, This represents the word embedding mapping function of BERT. For vector concatenation operators, This represents the learnable weight matrix in an MLP. For learnable bias term vectors, It is a non-linear activation function; S33. Extract character entities, attributes, and relationships from story text prompts using a pre-trained scene semantic parser. Construct a character attribute mapping graph based on character entities, attributes, and relationships. Encode the attributes into high-dimensional vectors using a semantic encoder, and then weight and aggregate them according to attribute weights before normalizing them using LayerNorm to obtain character state features. The character state features satisfy the following relationship: ; In the formula, For the j-th attribute word extracted from the mapping graph, For semantic encoders, For attribute importance weights, This represents the linear normalization function.

[0009] Optionally, in S4, constructing cross-frame enhanced character features based on character state features of three consecutive frames includes: S41. Perform vector decomposition on the character state features of three consecutive frames to obtain three sets of appearance feature sub-vectors, posture feature sub-vectors, emotion feature sub-vectors and context semantic feature sub-vectors. S42, Through The function performs feature completion on each sub-vector, resulting in the completed appearance feature sub-vector, the completed posture feature sub-vector, the completed emotion feature sub-vector, and the completed context semantic feature sub-vector. Feature completion of appearance feature subvectors satisfies the following relationship: ; The completion of the emotion feature subvectors satisfies the following relationship: ; The completion of the pose feature subvectors satisfies the following relationship: ; The completion of context semantic feature subvectors satisfies the following relation: ; The completed sub-vectors are concatenated to obtain cross-frame enhanced character features, satisfying the following relationship: ; In the formula, This indicates cross-frame enhancement of character features. This represents the completed appearance feature vector. This represents the completed pose feature vector. This represents the completed emotional feature subvector. This represents the contextual semantic feature subvector after completion.

[0010] Optionally, in step S5, the aesthetic preference features, the action semantic features, and the cross-frame enhanced character features are fused using multimodal features to obtain fused features, including: Aesthetic preference features, action semantic features, and cross-frame enhanced character features are fused and projected to obtain a unified conditional vector that satisfies the following relationship: ; In the formula, Let be the fusion feature of frame t. For the user's aesthetic preference feature vector, For the intention to represent a vector, For cross-frame character feature vectors, , , Let b be the weight matrix and b be the bias vector.

[0011] Optionally, in step S5, a diffusion model is constructed based on the fusion features and the global features of the character. The diffusion model generates a visually coherent sequence of storybook images that conforms to user aesthetic preferences, including: S51, from the fusion feature Mid-cut , , ,in It contains cross-frame information. It includes action triples and character states. It contains local detailed information features; S52. In the low-resolution layer of the convolutional neural network, the global intent g and cross-frame information are combined. As a global conditional injection cross-attention mechanism, the low-resolution layer cross-attention score is obtained, as follows: ; In the formula, low-level query features Query projection matrix , For key vectors, The scaling factor for the key dimension. For normalization function, ; S53. In the intermediate feature layer of the convolutional neural network, the action triples and the character state are combined. An injection cross-attention mechanism is used to obtain the cross-attention score of the intermediate feature layer. The attention score of the intermediate feature layer is as follows: ; In the formula, the mid-level query features Query projection matrix , ; S54. In the high-resolution layer of the convolutional neural network, local detail information features are... By injecting a cross-attention mechanism, the cross-attention score of the high-resolution layer is obtained. The high-resolution layer attention score is as follows: ; In the formula, high-level query features Query projection matrix , ; S55. A diffusion model is constructed based on the cross-attention scores of the low-resolution layer, the poor attention scores of the intermediate feature layer, and the cross-attention scores of the high-resolution layer. The constructed diffusion model is then used to determine the fusion features. Generate a visually coherent sequence of storybook images that aligns with the user's aesthetic preferences.

[0012] Optionally, the method also includes: The image features and text features in the story picture book image sequence are obtained, and the picture book images corresponding to the image features and the picture book text corresponding to the text features are integrated and typed according to user needs to obtain the integrated and typed story picture book.

[0013] Secondly, embodiments of this application provide a personalized story picture book generation system, including a processor and a memory; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps of the method described in the first aspect.

[0014] Beneficial effects: This invention provides a method for generating story picture book image sequences. By acquiring user image data, environmental visual data, and historical dialogue data, this method can deeply mine and extract user aesthetic preference features. This ensures that the generated picture book content can produce different styles of picture book images in terms of visual style and spatial layout based on the image data, environmental visual data, and historical dialogue data, avoiding the problem of highly similar styles in the generated picture books. This allows each created picture book to have its own unique style, thus meeting the personalized needs of users. Simultaneously, by semantically parsing the story text prompts, global character features, action semantic features, and character state features are extracted. Combined with the character state of three consecutive frames, cross-frame enhanced character features are constructed, effectively improving the consistency and coherence of the story characters' performance in the picture book image sequence, preventing obvious logical errors in the actions of story characters in two consecutive picture book images. Furthermore, by multimodal fusion of aesthetic preference features, action semantic features, and cross-frame enhanced character features, and by constructing a diffusion model based on the fused features and global character features, the generated story picture book image sequence is not only visually coherent and smooth, accurately matching the user's aesthetic preferences, but also ensures that the story characters and picture book images are consistent in style and action behavior. This significantly solves the problem in existing technologies where character style and action coherence are difficult to integrate with picture book style. Attached Figure Description

[0015] Figure 1 This is a flowchart of the storybook image sequence generation method of the present invention; Figure 2 This is a schematic diagram of the user aesthetic preference feature extraction process framework in this invention; Figure 3 This is a schematic diagram of the feature extraction process framework of the story semantic parsing module in this invention; Figure 4 This is a schematic diagram of the enhanced role state process framework in this invention; Figure 5 This is a schematic diagram of the conditional fusion and hierarchical injection-diffusion model in this invention. Detailed Implementation

[0016] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0018] Please see Figure 1 This embodiment provides a method for generating image sequences for story picture books, including the following steps: S1. Obtain user-input image data, environmental visual data, historical dialogue data, and story text prompts; S2. Extract the user's aesthetic preference features based on the image data, the environmental visual data, and the historical dialogue data; S3. Perform semantic analysis on the story text prompts, and extract global character features, action semantic features, and character state features from the story text prompts based on the results of the semantic analysis. S4. Construct cross-frame enhanced character features based on the character state features of three consecutive frames; S5. The aesthetic preference features, the action semantic features, and the cross-frame enhanced character features are fused into multimodal features to obtain fused features. A diffusion model is constructed based on the fused features and the global character features. A visually coherent storybook image sequence that conforms to the user's aesthetic preferences is generated through the diffusion model.

[0019] In this embodiment, due to the problems of high consistency in picture book style, poor consistency of character images across images, inaccurate story semantic parsing, and inconsistent clean transitions in current mainstream story picture book image sequence generation methods, a full-chain solution from user data collection to final picture book generation is constructed. This method first lays the foundation for personalization through multi-source data collection, namely image data, environmental visual data, historical dialogue data, and story text prompts. Then, through the synergistic effects of aesthetic preference extraction, semantic parsing, cross-frame state construction, multimodal fusion, and hierarchical injection generation, the automatic generation and layout of the picture book image sequence is finally achieved. This approach, for the first time, deeply couples users' multi-dimensional implicit aesthetic preferences—such as the visual style corresponding to image data, the environmental color corresponding to environmental visual data, and the historical interactions corresponding to historical dialogue data—with the explicit semantic information of the story text—such as the character identities, action logic, and emotional changes in the picture book story. Through the layered control mechanism of the diffusion model, it ensures the dual alignment of the generated results in terms of macro-narrative coherence and micro-detail consistency. This fundamentally solves the industry pain points of existing AIGC picture book generation, such as the inability to determine the overall style of the picture book based on user aesthetic preferences, the difficulty in maintaining consistency between story characters and picture book images in terms of style and action behavior, and the difficulty in integrating character style and action coherence with the picture book style.

[0020] The collection of user multidimensional demand data described in step S1 mainly includes the following data processing steps: 1. Sample collection of user image data, specifically: The system utilizes a client-side upload interface to retrieve user-provided image sample data. These images are selected by the user based on their personal aesthetic preferences and reflect a target visual style. The storage formats include JPG and PNG. The system records the file metadata for each image, forming a personalized aesthetic reference library for each user. This provides raw materials for subsequent style alignment.

[0021] 2. Collection of user room layout and color information, specifically: With user authorization, the surrounding environment is photographed using a mobile terminal. The collected content includes: images of the physical spatial layout of the room obtained using panoramic photos, and images of the overall color distribution of the interior under standard or natural light sources.

[0022] 3. Obtaining user historical dialogue data, specifically: The system retrieves historical interaction records associated with the current user ID from its cloud database. This historical data includes text communications, modification suggestions, and feedback between the user and the system. The fragmented dialogue logs are then reassembled chronologically via an API interface, and interaction information from different points in time is integrated using a time decay weighting model. In the process of AI-generated content (AIGC), the user needs to engage in dialogue with the AI ​​agent. The agent reads and analyzes these dialogues to understand the user's current needs and generates corresponding content accordingly. Therefore, the dialogue between the user and the AI ​​agent most directly reflects the user's needs and aesthetic preferences. Understanding this dialogue is the most important and fastest way to confirm the user's overall ideas and style. The API interface, as a standard information retrieval interface, can not only quickly read the dialogue data between the user and the AI ​​agent but also quickly import dialogues between the user and other AI agents or from other scenarios. By reassembling the retrieved dialogue logs or imported dialogue data from other scenarios chronologically, the system can quickly understand the user's historical needs and pinpoint their aesthetic preferences and style biases.

[0023] In a normal conversation, when performing a task, the goal is usually determined first, followed by communication of details, such as: "We need to create a children's picture book," "The children's picture book needs to teach children to recognize colors," and "We can start by teaching them the three primary colors: red, yellow, and blue." Therefore, the content earlier in the conversation represents the overall direction of the creation and needs to be given higher weight. The later content focuses more on details and has lower importance. Thus, integrating interaction information from different moments using a time decay weighting model can prevent the creation from deviating from the original intention.

[0024] 4. Collection of story text prompts and generation parameters The total number of images N and the N segmented story description texts are obtained by manually inputting the user's input through the text input box in the interactive interface. The prompt word templates for the segmented story text fragments are as follows: "Story text: In a small mountain village, there lived a shepherd boy who went up the mountain to herd sheep every day. This boy liked to joke and always lied. One day, he stood on the hillside and shouted loudly, 'Wolf! Wolf!'..." The above is the text of the story "The Boy Who Cried Wolf." Based on this, create a children's picture book, resulting in 10 illustrations that form a complete visual narrative of the story. In this embodiment, image data is extracted from user-uploaded images using a pre-trained deep learning model with a visual attention mechanism. The user-uploaded images can be photos, paintings, or other images with a clear target visual style tendency. Since different users will upload different images based on their own preferences or desired styles, the extracted image data will also have a strong personal style. The obtained image data can also be used as personalized image data to distinguish different picture book styles.

[0025] To generate unique picture books tailored to each user's personal style, environmental visual data is typically extracted from the user's living environment, such as streets, neighborhoods, residential buildings, and even room layouts. To better highlight the user's personal style and avoid repetition, room layout is the optimal extraction path for environmental visual data. In this embodiment, room layout is used as an example for specific illustration. Using room layout as the environmental visual data extraction path is merely illustrative and not limiting; any data extraction path that can provide color tone and layout information can be used as the source of environmental visual data in this application.

[0026] For historical dialogue data, we mainly obtain users' historical interaction information. Users' historical generation records can better locate the style that users want. Similarly, it can also be obtained from other sources. By inputting dialogue texts from other sources into a pre-trained language model, we can complete the extraction of historical dialogue data from other sources.

[0027] Based on the description in S2, aesthetic preferences are extracted from multidimensional user needs data. Specific details are as follows: Figure 2 As shown, the main processing steps include the following: 1. Image style feature representation, specifically: The Vitality Trace (ViT) visual feature extractor is used to perform convolutional calculations on personalized preference image examples extracted from users to extract deep texture features, composition features, and brushstroke style, which are then transformed into high-dimensional style feature vectors. , where I is the set of image examples collected in S1. The artistic style of the images is quantified by calculating the Gram Matrix to ensure that the visual expressiveness of the subsequently generated images is aligned with the user's preferences.

[0028] 2. Extraction of environmental colors and spatial atmosphere, specifically: The K-means clustering algorithm was used to cluster the colors in the user's room, extracting the main color, secondary color, and accent color of the room to form an environmental color palette set. The acquired room layout image is input into the pre-trained semantic segmentation model DeepLabV3+. A multi-layer convolutional neural network extracts the image's depth features and performs pixel-by-pixel classification, thus segmenting the image into different semantic regions such as walls, floors, furniture, and decorations. Then, the spatial coordinates and area proportions of each segmented semantic region are calculated, and these parameters are used to construct a spatial layout feature vector. ,in This represents the category label of the i-th semantic region. This indicates the area weight of the region in the image. This indicates the relative spatial coordinates of the region.

[0029] 3. Historical dialogue preference mining, specifically: Let the sequence of aesthetic preferences generated by users at different points in time be denoted as . ,in Indicates the user's time The relevant interaction information. First, each text is encoded into a preliminary aesthetic vector using the pre-trained language model BERT. To depict the changing trends of user aesthetics over time, this invention introduces a time decay weight model, applying time weights to aesthetic vectors at different times. The weight decreases as the time of the text occurrence becomes further away from the current time. The weight is defined as:

[0030] in This indicates the interval between the interaction information at a certain moment and the current time. This is the time decay coefficient, used to adjust time sensitivity. The closer the time, the greater the weight; the further back the time, the greater the weight gradually decreases exponentially.

[0031] The user's historical dialogue data vector features are obtained by weighted fusion of historical dialogue data vectors. The weighted fusion satisfies the following relationship: ; In the formula, Represents the vector features of historical dialogue data; For the high-dimensional style feature vector The spatial layout feature vector and the historical dialogue data feature vector Weighted fusion is performed to obtain the user's aesthetic preference characteristics, where the weighted fusion satisfies the following relationship: ; In the formula, This indicates the aesthetic preference characteristics obtained after fusion. , , These are the weights of the high-dimensional style feature vector, the spatial layout feature vector, and the historical dialogue data feature vector, respectively.

[0032] In this embodiment, firstly, deep feature extraction is performed on the personalized image examples provided by the user, and the artistic style is quantified using a Gram matrix. This accurately captures the user's implicit preferences for specific styles, textures, and compositions, ensuring that the visual expressiveness of the generated image highly matches the user's aesthetic. Secondly, the DeepLabV3+ semantic segmentation model is used to perform structured analysis of the user's real-life environment, generating spatial layout feature vectors. This allows the generated picture book scene to naturally integrate into the user's actual life atmosphere, enhancing emotional resonance. Furthermore, by introducing a time decay weight model to weightedly fuse historical dialogue data, the evolution trend of the user's aesthetic preferences can be dynamically tracked, avoiding style misalignment caused by using outdated preferences. Finally, these three types of feature vectors are weighted and fused to form a unified aesthetic preference feature. This ensures that the conditional input of the diffusion model not only includes a single style reference but also incorporates the semantics of the user's living environment and the evolutionary patterns of aesthetics. This generates picture book images that are deeply aligned with the user's personalized needs in terms of visual style, spatial composition, and emotional atmosphere, effectively solving the problems of homogenized picture book styles and lack of personalized customization in existing technologies.

[0033] Based on the textual intent analysis of the story text described in S3, the specific details are as follows: Figure 3 As shown, the main processing steps include the following: 1. Global feature scanning and extraction of the character, specifically: The complete story text collected in S1 is input into GPT-4 to guide the model in extracting core entity attributes, identifying and aggregating character description information across segments, including gender, age, body type, clothing, and facial features. A structured set of textual attributes is generated for each core character. Unlike the character state features within a frame, the structured attribute set is then... Mapping to a high-dimensional latent feature space:

[0034] in, As a semantic encoder, it compresses text descriptions into fixed-dimensional global feature vectors. .

[0035] 2. Dynamic action semantic recognition and vectorized representation, specifically: Using the SpaCy NER named entity recognition model to locate the set of role entities in a text fragment Transformer-based SRL is used to analyze the predicate argument structure. For the t-th segment, the core predicate is identified and the action triple sequence is extracted. ,in E represents the agent, and E represents the event that occurred. This represents the recipient of the action. To transform unstructured actions into features that the model can process, this invention utilizes the pre-trained BERT model to encode each triple as an initial vector, which is then input into an MLP for feature fusion. The action vector calculation formula is as follows:

[0036] in Let represent the action embedding vector of the t-th story segment, which serves as the action constraint when the downstream diffusion model generates images. This represents the word embedding mapping function in BERT, which converts text into numerical vectors in a high-dimensional continuous space. This is a vector concatenation operator that preserves the structured order within the triples. This represents the learnable weight matrix in an MLP, which performs linear transformations and dimensionality reduction on the concatenated long vector. For learnable bias term vectors, It is a non-linear activation function that captures complex non-linear relationships in action semantics.

[0037] 3. Character status analysis, specifically: A pre-trained scene parser is used to perform deep parsing of text fragments. This parser integrates part-of-speech tagging and dependency parsing to extract three core elements from the story text: entity O, attribute A, and relation R. For each identified character entity... The parser dynamically captures descriptive words associated with it, including physical features, emotional states, and identity tags.

[0038] Then, a character feature mapping map is constructed based on the extracted elements. Then, the descriptive words in the image are converted into high-dimensional vectors by an encoder, and feature aggregation is performed to obtain the character state vector of frame t. :

[0039] in For the j-th attribute word extracted from the mapping graph, For semantic encoders, For attribute importance weights, This represents the linear normalization function.

[0040] In this embodiment, firstly, the large language model GPT-4 is used to perform a global scan of the complete story text, extracting invariant attributes of the characters throughout the narrative cycle, such as gender, age, and clothing, to generate a global feature vector. This vector serves as the character's identity anchor, constraining the core appearance of the character to remain unchanged throughout the image sequence and preventing character image drift caused by incomplete text descriptions. Secondly, the character entity is located using spaCy NER, and action triples are extracted using Transformer-based SRL technology and encoded as action semantic features. These features accurately depict the dynamic interaction logic between characters in each frame, ensuring that the actions of the characters in the generated images are strictly consistent with the text descriptions, thus solving the problem of misalignment between text descriptions and image actions that is prone to occur in traditional methods. Furthermore, a character attribute mapping graph is constructed using a scene semantic parser, and weighted aggregation is used to generate character state features. These features integrate the character's emotions, posture, contextual semantics, and other instantaneous attributes, providing a detailed character state description for each frame of the image. The synergistic effect of these three layers of features enables comprehensive semantic constraints, from global features and action features to the character's state, including who the character is, what they are doing, and their state. This lays a solid semantic foundation for the subsequent generation of picture book sequences with high narrative coherence and character consistency.

[0041] Based on the description in S4, the character states are constructed across frames in the story text; specific details are as follows: Figure 4 As shown, the main processing steps include the following: By introducing a feature saliency detection and classification feature fusion strategy, the initial state vector of the character generated by S3 in each frame is... Enhancement is performed. Based on temporally complementary role state feature completion, saliency masks are used to identify missing features, and different compensation strategies are implemented according to the feature type, as shown in the following formula:

[0042] Will The vectors of the preceding and following frames are decomposed into appearance feature sub-vectors according to their dimensions. pose feature subvectors Emotion feature subvectors Contextual semantic feature subvectors .

[0043] Based on the biological and narrative attributes of the features, The function uses three different fusion strategies to generate the completed subvectors: (1) Strong forward inheritance of appearance features; if the current frame does not describe clothing, then Automatically extract the appearance features from the previous frame The formula for fully mapping to the current frame is as follows:

[0044] (2) Bidirectional linear interpolation of emotional features: Emotional changes are continuous and fluctuating, and prediction should be based on the evolution trend of previous and subsequent frames. By calculating the arithmetic mean of the emotional vectors of previous and subsequent frames, a smooth transition estimate is provided for the current frame. The formula is as follows:

[0045] (3) The context-weighted fusion of gesture features and contextual semantic features is performed using a dynamic weighting method, as shown in the following formula:

[0046] in , This is a temporal distance weighting factor used to extract relevant semantic background residuals from neighboring frames.

[0047] The updated subvectors are then concatenated to obtain the enhanced character state vector, as shown in the following formula:

[0048] In the formula, This indicates cross-frame enhancement of character features. This represents the completed appearance feature vector. This represents the completed pose feature vector. This represents the completed emotional feature subvector. This represents the contextual semantic feature subvector after completion.

[0049] Not all of the generated subvectors need to be padded; this can be achieved by using a saliency mask. To determine whether a certain type of feature is valid in frame t, the specific calculation method is to calculate the L2 norm of the subvector. If the value is lower than a set threshold, it is determined that this type of feature is missing in the current text segment, and... ,otherwise During the completion process, only the missing subvectors need to be completed, i.e., when... When, perform the completion operation, and when If so, there is no need to complete it.

[0050] In this embodiment, by decomposing the character's state vector into four sub-vectors—appearance, posture, emotion, and contextual semantics—and employing differentiated temporal completion strategies for different types of features, intelligent compensation for textual jumps in description is achieved. Specifically: appearance features adopt a forward strong inheritance strategy, ensuring that the character's clothing remains stable across consecutive frames when the text does not describe clothing changes, avoiding visual discontinuities caused by sudden changes in clothing style; emotion features employ bidirectional linear interpolation, ensuring that the evolution of the character's emotions conforms to the continuous pattern of human emotional changes and that emotional changes align with normal behavioral logic, preventing unnatural abrupt changes such as crying from joy; posture and contextual semantic features employ dynamic weighted fusion, intelligently extracting relevant information from preceding and following frames to complete the missing action background or semantic context in the current frame. Through adaptive judgment of saliency masks, this module triggers the completion mechanism only when features are missing, avoiding redundant calculations and information overload. The resulting enhanced state vectors enable the appearance, posture, emotions, and semantics of characters in the picture book image sequence to exhibit a smooth, natural, and logical evolution trajectory in the temporal dimension, significantly improving the visual coherence and narrative fluency of the sequence. This solves the core problem of unnatural breaks in character states and disjointed actions between picture book images in existing technologies.

[0051] Based on the multimodal conditional fusion and hierarchical injection diffusion model described in S5, the specific details are as follows: Figure 5 As shown, the main processing steps include the following: By projecting the fusion of aesthetic preferences, intentional features, and role state features, a unified conditional vector is formed:

[0052] in Let be the fusion feature of frame t. This involves creating a feature vector representing the user's aesthetic preferences to capture their personalized style. For the intention to represent a vector, For cross-frame character feature vectors, , , Let be the weight matrix, and b be the bias vector. From Mid-cut , , ,in It contains cross-frame information. It includes action triples and character states. It contains local detailed information features.

[0053] In this embodiment, heterogeneous features belonging to visual style, action logic, and character state are uniformly mapped to the same high-dimensional space through linear projection, forming a fused feature vector that incorporates triple constraints of user preference, textual semantics, and temporal coherence. This approach not only achieves simple concatenation of multimodal information but also automatically learns the contribution weights of different features to the final generated result through a learnable weight matrix. This allows the model to dynamically adjust the constraint strength of each feature based on different story types and user preferences. For example, in stories emphasizing action scenes, the weight of action features may automatically increase; while in stories emphasizing atmosphere creation, the weight of aesthetic features may be more prominent. This adaptive feature fusion mechanism ensures that the generated image comprehensively meets the multiple needs of personalized aesthetics, textual semantics, and temporal coherence, avoiding generation bias caused by a single feature dominating the process.

[0054] Progressive guidance from macro composition to micro details: (1) Global hierarchical injection: In the low-resolution layer of U-Net, the global intent g and cross-frame information are injected. As a global conditional injection into Cross-Attention, the attention scores are as follows:

[0055] Low-level query features Query projection matrix , For key vectors, The scaling factor for the key dimension. For normalization function, .

[0056] (2) Intermediate semantic alignment: In the intermediate feature layer of U-Net, the character action features of the current frame are aligned. After injecting Cross-Attention, the attention scores are as follows:

[0057] Among them, mid-level query features Query projection matrix , .

[0058] (3) Local appearance refinement: In the high-resolution layers of U-Net, appearance embeddings derived from aesthetic preferences are injected as local fine-grained conditions, and the attention scores are as follows:

[0059] High-level query features Query projection matrix , .

[0060] Finally, the Stable Diffusion model is used to generate the final storybook images.

[0061] In this embodiment, by dividing the fused features into three sub-vectors—cross-frame information, action state information, and local detail information—and injecting them into different resolution layers of the U-Net, a progressive generation control from macro to micro is achieved. Injecting global intent and cross-frame information into the low-resolution layer (8×8) establishes the overall composition, scene layout, and approximate character positions in the early stages of image generation, ensuring macroscopic coherence between sequences. Injecting action state features into the intermediate layer (32×32) precisely aligns the relative positions and interaction logic between characters at a medium scale, enabling the generated images to accurately reproduce the storyline described in the text. Injecting local detail features into the high-resolution layer (64×64) allows for refined control of microscopic features such as facial expressions and clothing textures during the final detail reconstruction stage, achieving high-fidelity reproduction of visual appearance. This layered injection strategy simulates the human painting process from sketching to refining details, ensuring that the generated results maintain global structural consistency while preserving the richness and accuracy of local details, significantly improving the overall quality and professional level of the picture book image sequence.

[0062] According to S6, the story picture book image sequence is integrated and laid out based on user needs to obtain the final story picture book, including: The image features and text features in the story picture book image sequence are obtained, and the picture book images corresponding to the image features and the picture book text corresponding to the text features are integrated and typed according to user needs to obtain the integrated and typed story picture book image sequence. The integrated and typed picture book image sequence is arranged sequentially to obtain the final picture book.

[0063] In this embodiment, image and text features are acquired from the image sequence, and then automatically or semi-automatically integrated and formatted according to the user-specified layout style, such as font selection, text and image layout, and decorative elements. The resulting image is arranged in sequence to form a complete picture book. This transforms the previously generated high-quality image sequence into a final product that meets standards or user expectations, enhancing both practicality and commercial value. Furthermore, by allowing users to participate in the layout design, it increases user engagement and satisfaction with the final product. Simultaneously, the automated layout process avoids the inefficiency and inconsistencies of manual layout, achieving full automation from image generation to final output, providing a feasible path for large-scale personalized picture book creation. User requirements in this embodiment can be simple text and image arrangement, such as text above images, text on the left and images on the right, or text within blank spaces in the images. Alternatively, complex requirements can include maintaining consistent character positions across all picture book images. These integrated layout methods can be achieved by extracting image and text features from the image sequence using a feature extraction model and adjusting the positions of the corresponding features. As for the sequential arrangement process, during the generation of the story picture book image sequence, the diffusion model will generate story picture book images in order based on the completeness of the story and the continuity of the character actions. The sequence in the story picture book image sequence is the order in which the story picture book images are generated, with the first generated story picture book image placed at the beginning and the last generated story picture book image placed at the end, thus completing the generation of the story picture book.

[0064] This application also provides a personalized story picture book generation system, including a processor and a memory; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the story picture book image sequence generation method.

[0065] The personalized story picture book generation system described above can implement various embodiments of the story picture book image sequence generation method described above, and can achieve the same beneficial effects, which will not be elaborated here.

[0066] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for generating image sequences for story picture books, characterized in that, include: S1. Obtain user-input image data, environmental visual data, historical dialogue data, and story text prompts; S2. Extract the user's aesthetic preference features based on the image data, the environmental visual data, and the historical dialogue data; S3. Perform semantic analysis on the story text prompts, and extract global character features, action semantic features, and character state features from the story text prompts based on the results of the semantic analysis. S4. Construct cross-frame enhanced character features based on the character state features of three consecutive frames; S5. The aesthetic preference features, the action semantic features, and the cross-frame enhanced character features are fused into multimodal features to obtain fused features. A diffusion model is constructed based on the fused features and the global character features. A visually coherent storybook image sequence that conforms to the user's aesthetic preferences is generated through the diffusion model.

2. The storybook image sequence generation method according to claim 1, characterized in that, In step S2, the user's aesthetic preference features are extracted based on the image data, the environmental visual data, and the historical dialogue data, including: S21. A pre-trained visual Transformer model is used to perform convolution calculations on the image data to extract texture features, composition features, and brushstroke style. The extracted texture features, composition features, and brushstroke style are then transformed into high-dimensional style feature vectors. , , where I is a collection of image data containing texture features, composition features, and brushstroke style; S22. A semantic segmentation model is used to extract depth features from the environmental visual data. Based on the depth features, the environmental visual data is segmented to obtain multiple semantic regions. The area weight and spatial coordinates of each semantic region are calculated to generate a spatial layout feature vector. , ,in, This represents the category label of the i-th semantic region. This indicates the area weight of the region in the image. Indicates the relative spatial coordinates of the region; S23. Use a pre-trained BERT model to encode historical dialogue data into historical dialogue data vectors. , ,in, Indicates the user's time Historical dialogue data is used, and a time decay weight model is introduced to apply time weights to the initial aesthetic vectors at different times. The time weights satisfy the following relationship: ； In the formula, Indicates time weight, Indicates the time decay coefficient. This indicates the time interval between the interaction information at a certain moment and the current time. The user's historical dialogue data vector features are obtained by weighted fusion of historical dialogue data vectors. The weighted fusion satisfies the following relationship: ； In the formula, Represents the vector features of historical dialogue data; For the high-dimensional style feature vector The spatial layout feature vector and the historical dialogue data feature vector Weighted fusion is performed to obtain the user's aesthetic preference characteristics, where the weighted fusion satisfies the following relationship: ； In the formula, This indicates the aesthetic preference characteristics obtained after fusion. , , These are the weights of the high-dimensional style feature vector, the spatial layout feature vector, and the historical dialogue data feature vector, respectively.

3. The method for generating storybook image sequences according to claim 1, characterized in that, In step S3, semantic parsing is performed on the story text prompts. Based on the results of the semantic parsing, global character features, action semantic features, and character state features are extracted from the story text prompts, including: S31. Input the story text prompts into a pre-trained large language model to identify all character description information in the story text prompts. Construct a structured attribute set based on the character description information, and extract the character global features of cross-segment invariant attributes based on the structured attribute set. The character global features satisfy the following relationship: ； In the formula, For semantic encoders, As a structured attribute set, the text description is compressed into a fixed-dimensional global feature vector. ; S32. Use a named entity recognition model to locate character entities in story text prompts, and extract action triples of character entities using a Transformer-based semantic character annotation model. , ,in, E represents the agent, and E represents the event that occurred. The recipient is represented, and the action triple is encoded using BERT and fused with MLP to generate action semantic features. The action semantic features satisfy the following relationship: ； In the formula, This represents the word embedding mapping function of BERT. For vector concatenation operators, This represents the learnable weight matrix in an MLP. For learnable bias term vectors, It is a non-linear activation function; S33. Extract character entities, attributes, and relationships from story text prompts using a pre-trained scene semantic parser. Construct a character attribute mapping graph based on character entities, attributes, and relationships. Encode the attributes into high-dimensional vectors using a semantic encoder, and then weight and aggregate them according to attribute weights before normalizing them using LayerNorm to obtain character state features. The character state features satisfy the following relationship: ； In the formula, For the j-th attribute word extracted from the mapping graph, For semantic encoders, For attribute importance weights, This represents the linear normalization function.

4. The method for generating storybook image sequences according to claim 1, characterized in that, In step S4, cross-frame enhanced character features are constructed based on the character state features of three consecutive frames, including: S41. Perform vector decomposition on the character state features of three consecutive frames to obtain three sets of appearance feature sub-vectors, posture feature sub-vectors, emotion feature sub-vectors and context semantic feature sub-vectors. S42, Through The function performs feature completion on each sub-vector, resulting in the completed appearance feature sub-vector, the completed posture feature sub-vector, the completed emotion feature sub-vector, and the completed context semantic feature sub-vector. Feature completion of appearance feature subvectors satisfies the following relationship: ； The completion of the emotion feature subvectors satisfies the following relationship: ； The completion of the pose feature subvectors satisfies the following relationship: ； The completion of context semantic feature subvectors satisfies the following relation: ； The completed sub-vectors are concatenated to obtain cross-frame enhanced character features, satisfying the following relationship: ； In the formula, This indicates cross-frame enhancement of character features. This represents the completed appearance feature vector. This represents the completed pose feature vector. This represents the completed emotional feature subvector. This represents the contextual semantic feature subvector after completion.

5. The method for generating a storybook image sequence according to claim 1, characterized in that, In step S5, the aesthetic preference features, the action semantic features, and the cross-frame enhanced character features are fused using multimodal features to obtain fused features, including: Aesthetic preference features, action semantic features, and cross-frame enhanced character features are fused and projected to obtain a unified conditional vector that satisfies the following relationship: ； In the formula, Let be the fusion feature of frame t. For the user's aesthetic preference feature vector, For the intention to represent a vector, For cross-frame character feature vectors, , , Let b be the weight matrix and b be the bias vector.

6. The method for generating story picture book image sequences according to claim 5, characterized in that, In step S5, a diffusion model is constructed based on the fusion features and the global features of the character. This diffusion model generates a visually coherent sequence of storybook images that aligns with user aesthetic preferences, including: S51, from the fusion feature Mid-cut , , ,in It contains cross-frame information. It includes action triples and character states. It contains local detailed information features; S52. In the low-resolution layer of the convolutional neural network, the global intent g and cross-frame information are combined. As a global conditional injection cross-attention mechanism, the low-resolution layer cross-attention score is obtained, as follows: ； In the formula, low-level query features Query projection matrix , For key vectors, The scaling factor for the key dimension. For normalization function, ; S53. In the intermediate feature layer of the convolutional neural network, the action triples and the character state are combined. An injection cross-attention mechanism is used to obtain the cross-attention score of the intermediate feature layer. The attention score of the intermediate feature layer is as follows: ； In the formula, the mid-level query features Query projection matrix , ; S54. In the high-resolution layer of the convolutional neural network, local detail information features are... By injecting a cross-attention mechanism, the cross-attention score of the high-resolution layer is obtained. The high-resolution layer attention score is as follows: ； In the formula, high-level query features Query projection matrix , ; S55. A diffusion model is constructed based on the cross-attention scores of the low-resolution layer, the poor attention scores of the intermediate feature layer, and the cross-attention scores of the high-resolution layer. The constructed diffusion model is then used to determine the fusion features. Generate a visually coherent sequence of storybook images that aligns with the user's aesthetic preferences.

7. The method for generating a storybook image sequence according to claim 1, characterized in that, Also includes: The image features and text features in the story picture book image sequence are obtained, and the picture book images corresponding to the image features and the picture book text corresponding to the text features are integrated and typed according to user needs to obtain the integrated and typed story picture book.

8. A personalized story picture book generation system, characterized in that, Including processor and memory; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

Images