Story visualization method supporting fine-grained customization

Abstract

A story visualization method supporting fine-grained customization comprises the steps that a reference image and a narrative text are input, identity features and non-identity editable attributes of the reference image are separated through an identity decoupling encoder, a story semantic parser is utilized to parse the time sequence attribute change trend of the narrative text to generate adjustment parameters, and the adjustment parameters are stored in a database; fine-grained control features are generated through fusion of a cross-modal fusion controller, then a time-sequence coherent image sequence is generated through a cross-frame consistency maintenance module in combination with a diffusion model, and finally a result conforming to fine-grained attribute control and cross-frame identity consistency is output. Compared with an existing method, the method has the remarkable advantages that through four-stage processing, the contradiction between identity fidelity and dynamic attribute adjustment is solved, multi-frame space-time coherence is guaranteed, core indexes comprehensively exceed a baseline method, and a complete solution is provided for high-fidelity story visualization under a single reference image.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and generative artificial intelligence, and in particular to a story visualization method supporting fine-grained customization. Background Art

[0002] Existing customized image generation methods mainly achieve identity (ID)-preserving personalized generation through pre-trained diffusion models. Typical methods include:

[0003] (1) Optimization methods: such as DreamBooth and Textual Inversion, which require fine-tuning the model for the target character, have the disadvantages of high computing resource consumption and high inference latency, making it difficult to meet real-time generation requirements.

[0004] (2) Feedforward method: IPAdapter and PhotoMaker extract identity features through image encoders. Although they achieve single-pass forward generation, they have the following shortcomings: (1) The generated character attributes overfit the reference image, resulting in the inability to dynamically adjust non-identity attributes such as expression and posture according to the text; (2) There is a lack of the ability to parse and control fine-grained features in text descriptions (such as expression and age changes), which leads to the deviation of the generated content from the semantics of the narrative text.

[0005] Existing story visualization methods focus on maintaining identity consistency across frames. The main technical approaches include:

[0006] (1) Attention sharing mechanism: For example, StoryDiffusion ensures character consistency through self-attention interaction of images within a batch, but does not establish an association mapping between text attribute changes and visual generation; ConsiStory uses masked topic attention sharing, which improves local consistency but has limited cross-frame coordination capabilities for global attributes (such as ambient lighting).

[0007] (2) Reference-driven generation: StoryMaker relies on multiple reference images to maintain facial features, but cannot achieve attribute decoupling control based on a single reference image.

[0008] The defects of the prior art mainly include:

[0009] (1) The contradiction between identity preservation and attribute control: Existing customized generation methods over-pursue ID fidelity, resulting in the solidification of character non-identity attributes (expressions, clothing, etc.) and the inability to respond to dynamic changes in text descriptions.

[0010] (2) Lack of fine-grained semantic alignment: Although story visualization technology ensures consistency across frames, it ignores the evolution of temporal attributes implicit in the text (such as changes in character hairstyles and gradual changes in scene lighting), resulting in a fine-grained semantic mismatch between the visual content and the narrative text.

[0011] (3) Generation bottleneck under single reference constraint: Existing methods find it difficult to simultaneously achieve the triple goals of high ID fidelity, cross-frame consistency, and dynamic attribute adjustment in a single reference input scenario.

[0012] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0013] The main purpose of the present invention is to overcome the defects in the above-mentioned background technology and provide a story visualization method that supports fine-grained customization.

[0014] To achieve the above object, the present invention adopts the following technical solutions:

[0015] A story visualization method supporting fine-grained customization includes the following steps:

[0016] Input reference images and narrative text;

[0017] Separate identity features from non-identity editable attributes from the reference image via an identity-decoupled encoder;

[0018] The story semantic parser analyzes the temporal attribute change trends in the narrative text and generates attribute adjustment parameters;

[0019] Through the cross-modal fusion controller, identity features, attribute adjustment parameters and text features are fused to generate fine-grained control features;

[0020] The cross-frame consistency maintenance module is used to perform probabilistic mixing of multi-frame latent features and combine with the diffusion model to generate a temporally coherent image sequence.

[0021] The output is story visualization results that comply with fine-grained attribute control and cross-frame identity consistency.

[0022] Furthermore, the identity decoupling encoder implements feature decoupling based on the StyleGAN latent space, mapping the reference image into an editable identity basis vector.

[0023] Furthermore, the story semantic parser identifies the gradual change type and degree of the character attributes in the text through a large language model, and outputs the attribute offset and dynamic adjustment coefficient.

[0024] Furthermore, the cross-modal fusion controller adopts a residual cross-attention mechanism to adaptively weighted fuse the decoupled identity features and text features through query-key-value mapping.

[0025] Furthermore, the cross-frame consistency maintenance module constrains the character identity and scene element continuity of the generated image by blending the potential features of adjacent frames and calculating attention weights.

[0026] Furthermore, the method further includes: calculating image feature similarity loss for adjacent frames during the diffusion process, and optimizing cross-frame consistency through back propagation.

[0027] Furthermore, the identity features are projected into a stable diffusion compatible space through a mapping network and then fused with the text features.

[0028] Furthermore, the residual cross attention mechanism specifically includes:

[0029] Generate query vector, key vector and value vector through linear transformation;

[0030] Calculate the similarity weight between the query vector and the key vector;

[0031] The value vectors are weighted and summed according to the similarity weight to generate the fused guidance features.

[0032] Furthermore, the method comprises:

[0033] The identity decoupled encoder processes the reference image in parallel:

[0034] Extracting semantic features of narrative text through text encoder;

[0035] Decouple identity features through the CtrlGAN encoder;

[0036] The story semantic parser receives text semantic features, identifies the attribute change type and gradual degree, and outputs a dynamic adjustment coefficient to the feature editing channel;

[0037] The decoupled identity features are projected into the diffusion model compatible space through the mapping network and combined with the attribute adjustment parameters to generate the edited features;

[0038] The cross-modal fusion controller is implemented through a two-level attention mechanism:

[0039] Perform linear projection on the edited features and text features to generate query-key-value pairs;

[0040] Calculate cross attention weights and weighted fusion features;

[0041] The cross-frame consistency maintenance module performs the following operations during the diffusion process:

[0042] Probabilistic sampling and mixing of multi-frame latent features within a batch;

[0043] Constraining character identity continuity in generated images via cross-frame attention interactions.

[0044] A computer program product includes a computer program, which, when executed by a processor, implements the story visualization method supporting fine-grained customization.

[0045] The present invention has the following beneficial effects:

[0046] This paper proposes a story visualization method that supports fine-grained customization. Through an innovative four-stage processing paradigm, it achieves unprecedented fine-grained attribute control while maintaining character identity consistency. A significant technical advantage of this method lies in the use of an identity decoupling encoder (such as CtrlGAN based on StyleGAN) to separate the reference image into editable identity basis vectors and non-identity attributes. This is combined with a story semantic parser (SaM) to accurately capture temporal gradient trends in the text (such as the dynamic coefficient k of lip color from light to dark). The residual cross-attention mechanism of the cross-modal fusion controller (CGF) then deeply aligns attribute parameters, identity features, and text semantics, fundamentally resolving the contradiction between identity fidelity and dynamic attribute adjustment in existing methods. This design enables the generated character to faithfully inherit the core identity features of the reference image (such as face shape) while accurately responding to gradual changes in the text description (such as smile intensity gradient or age evolution). This overcomes the limitations of feed-forward methods such as PhotoMaker that overfit static attributes and cross-frame technologies such as StoryDiffusion that ignore text semantics. Furthermore, the present invention ensures the spatiotemporal coherence of multi-frame sequences with lightweight computation through the probabilistic latent mixing strategy of the inter-frame consistency module (ICA). For example, in the "lip color gradient" task, this method not only achieves a natural transition from light to dark colors, but also simultaneously maintains the cross-frame consistency of clothing accessories; while existing solutions such as StoryMaker maintain high ID fidelity, they cause accessories such as glasses to be lost due to attribute fixation. Quantitative data validates the comprehensive advantages of the present invention: it surpasses all baseline methods in core indicators such as text alignment rate, CLIP face similarity, and overall consistency. User preference experiments further confirm its practical value in complex narrative scenarios, providing a complete solution for high-fidelity story visualization under a single reference image.

[0047] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The overall flow chart of the story visualization method supporting fine-grained customization in the present invention.

[0049] Figure 2 This is a diagram of the overall algorithm architecture of an embodiment of the present invention (Note: the character images in the figure are all AI-generated and are not real people).

[0050] Figure 3This is a qualitative comparison of the results (age, smile) between an embodiment of the present invention and the existing optimal method (Note: the character images in the figure are all AI-generated, not real people).

[0051] Figure 4 This is a qualitative comparison of the results (eye closure, lip color) of an embodiment of the present invention and the existing optimal method (Note: the character images in the figure are all AI-generated, not real people).

[0052] Figure 5 This is the user preference experiment result of the embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0054] This paper proposes a story visualization method that supports fine-grained customization and establishes a customized story visualization framework based on a diffusion model. Through the collaborative mechanism of identity decoupling encoding, text-driven attribute regulation and cross-frame consistency constraints, it achieves the unity of high-fidelity character generation, fine-grained attribute control and cross-frame coherence.

[0055] See Figure 1 and Figure 2 , an embodiment of the present invention provides a story visualization method supporting fine-grained customization, comprising the following steps:

[0056] Step S1, input reference image and narrative text;

[0057] Step S2: Separate identity features and non-identity editable attributes from the reference image through an identity decoupling encoder;

[0058] Step S3: Analyze the temporal attribute change trend in the narrative text through the story semantic parser to generate attribute adjustment parameters;

[0059] Step S4: fusing identity features, attribute adjustment parameters, and text features through a cross-modal fusion controller to generate fine-grained control features;

[0060] Step S5: Probabilistically mixing the multi-frame latent features through the cross-frame consistency maintenance module and generating a temporally coherent image sequence in combination with the diffusion model;

[0061] Step S6: Output story visualization results that comply with fine-grained attribute control and cross-frame identity consistency.

[0062] In some embodiments, the identity decoupling encoder in step S2 implements feature decoupling based on the StyleGAN latent space, mapping the reference image into an editable identity basis vector.

[0063] In some embodiments, the story semantic parser in step S3 uses a large language model to identify the gradual change type and degree of change of character attributes in the text, and outputs attribute offsets and dynamic adjustment coefficients.

[0064] In some embodiments, the cross-modal fusion controller in step S4 employs a residual cross-attention mechanism to adaptively weight the decoupled identity features and text features through query-key-value mapping. In a further preferred embodiment, the residual cross-attention mechanism specifically includes: generating a query vector, a key vector, and a value vector through linear transformation; calculating a similarity weight between the query vector and the key vector; and performing a weighted summation of the value vector based on the similarity weight to generate a fused guiding feature.

[0065] In some embodiments, the identity features in step S4 are projected into a stable diffusion compatible space via a mapping network and then fused with the text features.

[0066] In some embodiments, the cross-frame consistency maintenance module in step S5 constrains the character identity and scene element continuity of the generated image by blending the potential features of adjacent frames and calculating attention weights.

[0067] In some embodiments, image feature similarity loss is calculated for adjacent frames during the diffusion process, and cross-frame consistency is optimized through back-propagation.

[0068] In some embodiments, the story visualization method supporting fine-grained customization specifically includes:

[0069] The identity decoupling encoder processes the reference image in parallel: extracting semantic features of the narrative text through the text encoder; decoupling identity features through the CtrlGAN encoder;

[0070] The story semantic parser receives text semantic features, identifies the attribute change type and gradual degree, and outputs a dynamic adjustment coefficient to the feature editing channel;

[0071] The decoupled identity features are projected into the diffusion model compatible space through the mapping network and combined with the attribute adjustment parameters to generate the edited features;

[0072] The cross-modal fusion controller is implemented through a two-level attention mechanism: linear projection is performed on the edited features and text features to generate query-key-value pairs; cross-attention weights are calculated and weighted fusion features are combined;

[0073] The cross-frame consistency maintenance module performs the following steps during the diffusion process: probabilistically sampling and mixing latent features of multiple frames within a batch; and constraining the character identity continuity of the generated image through cross-frame attention interaction.

[0074] The following further describes specific embodiments of the present invention, its algorithm examples and experimental verification.

[0075] A story visualization method that supports fine-grained customization is established. A customized story visualization framework based on the diffusion model is established. Its overall architecture is as follows Figure 2 As shown. The core of the framework includes: CtrlGAN encoder, story perception module (SaM), customized guided fusion module (CGF) and ID consistency attention mechanism (ICA). Its main working principles include: (1) Identity decoupling and attribute control: decoupling the character identity features and non-identity attributes through the StyleGAN latent space, and establishing a dynamic mapping between text description and attribute parameters. (2) Text semantic guidance: combining the large language model to analyze the temporal attribute change trend of the narrative text, driving the generation process to achieve semantic alignment. (3) Cross-frame consistency maintenance: through the interaction of probabilistic mixing of latent features and cross-frame attention, the temporal coherence of character identity and scene elements is guaranteed.

[0076] The processing flow of this method mainly includes the following stages:

[0077] (1) Input processing: receiving the reference image Ir and narrative text;

[0078] (2) Feature decoupling: The CtrlGAN encoder Ctrl extracts the W+ spatial identity features of Ir;

[0079] (3) Text parsing: SaM parses the text through LLM and outputs the attribute change trend Δw and the adjustment coefficient k;

[0080] (4) Feature Mapping: The mapping network Map projects the features into a stable diffusion compatible space;

[0081] (5) Guided fusion: CGF fuses the mapped features with the text features fs through residual cross attention to generate fine-grained control features;

[0082] (6) Consistency generation: The ICA module mixes latent features across frames and combines with the diffusion model to generate temporally coherent image sequences.

[0083] The specific implementation method of the solution is further described below.

[0084] CtrlGAN Encoder

[0085] The CtrlGAN encoder mainly realizes the decoupled representation of identity features and non-identity attributes. The formula is expressed as:

[0086]

[0087] CtrlCtrl: represents the CtrlGAN encoder, which is a pre-trained image encoding network responsible for converting the reference image into a vector in the StyleGAN latent space.

[0088] Ir: Input reference image, used to extract the identity information of the target character.

[0089] w~r: The representation of the reference image in the W+ latent space of StyleGAN, which contains semantic information related to the identity.

[0090] W+ space: The high-dimensional latent space used in StyleGAN to control the generated images, supporting independent adjustment of character attributes (such as lip color and expression).

[0091] Decoupled semantics: refers to the ability of the CtrlGAN encoder to separate facial attributes (such as identity-related features and identity-irrelevant features), providing conditions for subsequent fine-grained control.

[0092] The CtrlGAN encoder is an image encoding module specifically designed to extract identity (ID) information from a reference image. While traditional methods (such as CLIP-based encoders) can preserve character identity, they have inherent trade-offs when generating diverse facial features (such as makeup and expressions), making it impossible to achieve fine-grained control consistent with text descriptions.

[0093] This method decouples the semantics of facial attributes (such as lip color, eye shape, etc.) to enable the generative model to independently adjust these attributes. Specifically, the CtrlGAN encoder first reverse-maps the visual concepts of the reference image into the StyleGAN latent space (W+ space). This latent space supports high-dimensional control of the character's appearance, allowing independent modification of specific attributes. For example, by adjusting specific directions in the latent space (such as lip color intensity or smile degree), fine-grained editing of the generated image can be achieved.

[0094] The input of the CtrlGAN encoder is the reference image Ir, and the output is the latent vector w~r of this image in the StyleGAN W+ space. This process is completed by a pre-trained encoder network, which can decouple image features into identity-related and identity-independent semantic parts, thus providing a basis for subsequent fine-grained control.

[0095] Story Awareness Module (SaM)

[0096] The Story Awareness Module (SaM) is mainly used to analyze the fine-grained attribute changes in text trends. The formula is expressed as:

[0097]

[0098] Δw: represents the semantic direction of a specific attribute (such as lip color, eye shape) in the latent space, which is used to control the increase or decrease of the attribute.

[0099] k: An adjustment factor that determines the intensity and direction of the attribute change (positive and negative values ​​correspond to strengthening or weakening the attribute, respectively). For example, as the k value increases, the lip color will change from light to dark.

[0100] w~ri: The subspace representation of the reference image in the latent space related to the specific attribute ii, and attribute adjustment is achieved by superimposing Δw·k.

[0101] i∈{0,1,…,N}: index number, representing the N attributes after decoupling that are not related to identity (such as lip color, eye shape, expression).

[0102] W+ space: The latent space of StyleGAN, which supports independent control of character appearance attributes, with each attribute corresponding to an adjustable semantic direction.

[0103] The core goal of the story perception module is to parse the narrative trends in text descriptions through a large language model (LLM), and based on this, dynamically adjust the latent space attributes extracted by the CtrlGAN encoder to achieve fine-grained control of character features.

[0104] The specific processing flow is as follows:

[0105] Text Trend Analysis: LLM analyzes the input story text and identifies the gradual change trend of specific attributes (such as lip color and facial expressions). For example, if the text describes "a woman gradually applying lipstick," LLM will identify the increasing trend of the attribute "lip color intensity."

[0106] Latent Space Adjustment: In the latent space (W+ space) extracted by the CtrlGAN encoder, non-identity-related attributes (such as lip color and eye shape) are directional adjusted. Each attribute corresponds to an independent semantic direction (Δw), and the adjustment coefficient (k) is superimposed to achieve a gradual modification. For example, as the description of "lipstick color darkens" in the text gradually increases, the lip color in the generated image gradually darkens.

[0107] Dynamic Control Mechanism: The Story-Aware Module dynamically adjusts the attribute directions in the latent space to ensure that the appearance changes of the generated character accurately match the text description. For example, in the "woman wearing lipstick" scene, the module gradually increases the lip color intensity parameter, ultimately generating a continuous sequence of images from no lipstick to dark lipstick.

[0108] Customized Guided Fusion (CGF)

[0109] Customization-guided fusion (CGF) is a core strategy that deeply fuses the fine-grained identity information extracted from the reference image (the latent vector wr generated by the CtrlGAN encoder) with the image features (fs) of the text cues, thereby achieving precise control over the character attributes during the generation process.

[0110] The formula is:

[0111]

[0112] The specific processing flow is as follows:

[0113] 1) Input processing:

[0114] Text features: Image features extracted from text cross-attention mechanism

[0115] fs, represents the visual semantics of the text description.

[0116] Identity information: character embeddings generated by the CtrlGAN encoder

[0117] wr, contains the latent space representation related to the character identity.

[0118] 2) Decoupling Cross-Attention Operations:

[0119] Use projection matrices Wqg, Wkg, and Wvg to transform input features into query, key, and value vectors.

[0120] Query vector Qg: generated by multiplying the text feature fs by Wqg, used to match the semantics of the text description.

[0121] Key vector Kg and value vector Vg: generated by multiplying the identity embedding wr with Wkg and Wvg respectively, to express the potential attributes of the role identity.

[0122] 3) Cross-Attention Fusion:

[0123] By calculating the similarity between Qg and Kg (after Softmax normalization), and weightedly aggregating the value vectors in Vg, we generate a character representation fg containing fine-grained control signals. The core of this step is to align the semantics of the text description with the latent attributes of the character identity, ensuring that the generated result not only meets the text requirements but also maintains character consistency.

[0124] 4) Residual fusion and balance control:

[0125] The original text feature fs is combined with the fused role representation fg through a residual connection to generate the final feature fg′=fs+λfgfg′.

[0126] The fusion parameter λ is used to adjust the weight of text prompts and identity information to balance the diversity and controllability of the generated results.

[0127] 5) Injection ratio optimization:

[0128] To avoid over-reliance on identity information or textual cues, customized guidance (fg) is injected only in the final β∈[0,1] steps of the inference process, rather than in all steps. By adjusting β, the priority of maintaining identity matching with textual descriptions can be controlled, for example, focusing on identity generation in the early steps and adjusting attribute details in the later steps.

[0129] Inter-frame latent admixture (ICA)

[0130] The formula is:

[0131]

[0132] While the aforementioned methods can extract identity information from reference images and achieve fine-grained control, the inventors discovered in actual applications that while each generated frame retains the character's identity, the character's appearance is inconsistent across frames (e.g., clothing differences). To address this issue, the present invention introduces the ID Consistency Attention (ICA) module.

[0133] Core design ideas:

[0134] Problem location: When generating the current frame, it is necessary to rely on the information of the previous frame to maintain consistency, but traditional methods lack cross-frame interaction mechanisms, resulting in sudden changes in character features (such as clothing and hairstyle) in different frames.

[0135] Solution: Probabilistic Latent Mixing (Probabilistic Latent Mixing) is used to achieve lightweight and efficient ID consistency preservation by randomly fusing the latent representations of the previous frame and the current frame.

[0136] Technical processing mechanism:

[0137] 1) Batch processing framework: All image features of the current batch (such as clothing, expressions, etc.) are regarded as a whole, and cross-frame interactions are established through the self-attention mechanism.

[0138] 2) Random blending: Randomly select a subset of features from other images within the batch (e.g., clothing texture, color) and fuse them with the current frame's features at a specific ratio (blending probability ρ). For example, if the character in the previous frame was wearing a blue shirt, the blending operation in the current frame might retain some of the blue elements to prevent a sudden change to red.

[0139] 3) Self-attention update: Based on the fused new features, the key and value vectors of the self-attention are recalculated to ensure that the character features remain consistent across frame interactions.

[0140] Specific implementation steps:

[0141] 1) Input features: All image features I of the current batch (including the current frame and the previous frame) are regarded as a multi-dimensional tensor, which contains the batch size B, the number of features per frame N and the channel dimension C.

[0142] 2) Standard self-attention processing: Self-attention (Key-Query-Value matching) is calculated independently for each image feature to generate a preliminary output Oi.

[0143] 3) Cross-frame blending: Randomly select feature subsets Si from other images in the batch (excluding the current image itself), with the blending ratio controlled by ρ. The blended features Si are combined with the current image features Ii to form a new feature set Pi.

[0144] 4) Attention update: Apply linear projection to Pi to generate new key and value vectors (KPi,VPi), keep the query vector Qi unchanged, and finally generate the updated output Oi through self-attention operation.

[0145] Its technical advantages include: Lightweight and efficient: Through random mixing and self-attention mechanisms, cross-frame consistency can be achieved without additional training, making it suitable for real-time generation scenarios. Flexible and controllable: Mixing probability. ρ allows users to adjust the consistency strength (e.g., ρ = 0.5 means 50% of the features come from other frames and 50% are retained in the current frame). Compatible with existing models: The ICA module can be seamlessly integrated into U-Net-based generation models, reusing pre-trained self-attention weights without retraining.

[0146] The four modules of this method achieve a multiplier effect through cascaded control: CtrlGAN provides editable identity basis vectors, SaM perceives attribute trends within text, CGF achieves precise feature fusion, and ICA ensures temporal continuity. This four-stage processing paradigm of "decoupling-regulation-fusion-constraint" fundamentally resolves the conflict between ID fidelity and attribute control in existing methods.

[0147] Experimental results

[0148] Quantitative results

[0149] Compared to existing approaches, this method achieves significant breakthroughs in key metrics such as ID fidelity, fine-grained attribute control, cross-frame consistency, and text semantic alignment. As shown in the quantitative comparison in Table 1, this method achieves optimal performance on the following key metrics:

[0150] Table 1. Quantitative comparison with existing methods

[0151]

[0152] In Table 1, the best results are in bold, and the suboptimal results are underlined. Quantitative data confirm the comprehensive advantages of our method: it surpasses all baseline methods in core metrics such as text alignment rate (30.6%), CLIP face similarity (83.1%), and overall consistency (74.9%).

[0153] Qualitative results

[0154] like Figure 3 and Figure 4 As shown, the present invention achieves consistent character story visualization in complex scenes such as "age gradient" and "lip color gradient", and the attribute changes are continuous and natural without identity distortion.

[0155] like Figure 3 and Figure 4 As shown, existing baseline methods have difficulty achieving fine-grained attribute control. In the age attribute adjustment task (left), the age changes generated by existing methods do not align with the text description. For example, when the character is required to be "old", there is no obvious change in the face. In the smile intensity gradient task (right), the baseline method cannot observe the deepening of the smile. In terms of ID preservation ability: although PhotoMaker can preserve basic identity features (such as face shape), the consistency of clothing style across frames is poor; StoryDiffusion maintains poor facial preservation; although StoryMaker maintains high ID fidelity, it overfits the reference image, resulting in: 1) facial attribute solidification (the glasses attribute marked by the yellow frame is missing) and 2) text semantic deviation (the generated image does not match the description of "smile").

[0156] This verifies the dual advantages of the method of the present invention in terms of fine-grained control and cross-frame consistency.

[0157] User preference experiments

[0158] Figure 5 The results of the user preference experiment are shown. Figure 5 As shown, in user evaluation, the method of the present invention obtains the best evaluation results compared with other methods.

[0159] In summary, the present invention proposes a customized story visualization method and system based on a diffusion model. The system can be understood as comprising the following four main parts: (1) Identity decoupling encoder: separating identity features and editable attributes from reference images; (2) Story semantic parser: parsing the temporal changes of character attributes in text; (3) Cross-modal fusion controller: balancing text prompts and identity feature guidance; (4) Cross-frame consistency maintenance module: ensuring character consistency in multi-image sequences. By collaboratively implementing the four-stage processing paradigm of "decoupling-regulation-fusion-constraint" through the four modules, the present invention breaks through the contradiction between identity fidelity and fine-grained attribute control under single reference image input, while ensuring cross-frame spatiotemporal coherence. The core indicators surpass existing baseline methods, providing a complete solution for high-fidelity story visualization.

[0160] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0161] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0162] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0163] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc or a read-only optical disc (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0164] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0165] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0167] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0168] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0169] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0170] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0171] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0172] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A story visualization method supporting fine-grained customization, characterized in that: The following steps are involved: Input reference images and narrative text; Separate identity features from non-identity editable attributes from the reference image via an identity-decoupled encoder; The story semantic parser analyzes the temporal attribute change trends in the narrative text and generates attribute adjustment parameters; Through the cross-modal fusion controller, identity features, attribute adjustment parameters and text features are fused to generate fine-grained control features; The cross-frame consistency maintenance module is used to perform probabilistic mixing of multi-frame latent features and combine with the diffusion model to generate a temporally coherent image sequence. The output is story visualization results that comply with fine-grained attribute control and cross-frame identity consistency.

2. The method according to claim 1, wherein The identity decoupling encoder implements feature decoupling based on the StyleGAN latent space, mapping the reference image into editable identity basis vectors.

3. The method according to claim 1 or 2, wherein: The story semantic parser uses a large language model to identify the gradual change type and degree of character attributes in the text, and outputs attribute offsets and dynamic adjustment coefficients.

4. The method according to any one of claims 1 to 3, wherein The cross-modal fusion controller adopts the residual cross-attention mechanism to adaptively weight the decoupled identity features and text features through query-key-value mapping.

5. The method according to any one of claims 1 to 4, characterized in that The cross-frame consistency maintenance module constrains the character identity and scene element continuity of the generated image by blending the potential features of adjacent frames and calculating attention weights.

6. The method according to any one of claims 1 to 5, wherein: Also includes: During the diffusion process, the image feature similarity loss is calculated for adjacent frames, and the cross-frame consistency is optimized through back-propagation.

7. The method according to any one of claims 1 to 6, wherein: The identity features are projected into a stable diffusion compatible space through a mapping network and then fused with the text features.

8. The method according to claim 4, wherein The residual cross attention mechanism specifically includes: Generate query vector, key vector and value vector through linear transformation; Calculate the similarity weight between the query vector and the key vector; The value vectors are weighted and summed according to the similarity weight to generate the fused guidance features.

9. The method according to any one of claims 1 to 8, characterized in that: include: The identity decoupled encoder processes the reference image in parallel: Extracting semantic features of narrative text through text encoder; Decouple identity features through the CtrlGAN encoder; The story semantic parser receives text semantic features, identifies the attribute change type and gradual degree, and outputs a dynamic adjustment coefficient to the feature editing channel; The decoupled identity features are projected into the diffusion model compatible space through the mapping network and combined with the attribute adjustment parameters to generate the edited features; The cross-modal fusion controller is implemented through a two-level attention mechanism: Perform linear projection on the edited features and text features to generate query-key-value pairs; Calculate cross attention weights and weighted fusion features; The cross-frame consistency maintenance module performs the following operations during the diffusion process: Probabilistic sampling and mixing of multi-frame latent features within a batch; Constraining character identity continuity in generated images via cross-frame attention interactions.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the story visualization method supporting fine-grained customization is implemented as claimed in any one of claims 1 to 9.

Images