A semantic closed-loop controllable image generation method and device for ancient poetry and prose texts

Abstract

The application discloses a kind of semantic closed loop controllable image generation method and device for classical poetry text, method includes: to the classical poetry text to be visualized is visually semantically analyzed, generates structured semantic information;Image generation control information and consistency verification information are generated based on the same structured semantic information;According to image generation control information, generate candidate image;According to atomic verification problem set, the consistency of candidate image is evaluated, and evaluation result is obtained;According to evaluation result and semantic slot mark, failed semantic unit is positioned to the entity, attribute or relationship in corresponding structured semantic information;According to the failure semantic unit positioned, determine target editing area and execute local image editing;When edited image satisfies preset gate condition, accept and output edited image.The application can improve the semantic fidelity, space controllability and area stability of classical poetry text image generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence image generation, natural language processing, and computer vision, and in particular to a semantically closed-loop controllable image generation method and apparatus for classical Chinese poetry texts. Background Technology

[0002] Classical Chinese poetry is a treasure of Chinese civilization and an important part of language and traditional culture education. Learners often need visual aids to understand the verses. Traditional teaching resources for classical poetry typically rely on manually drawn illustrations, manually compiled explanations of imagery, or manually created courseware, which are time-consuming, costly, and vary greatly in quality. In recent years, text-to-image technology has emerged, capable of generating high-quality images based on natural language prompts, providing a new technological approach for creating illustrations and multimedia resources for classical poetry teaching.

[0003] However, classical Chinese poetry texts are typically characterized by concise language, dense imagery, implicit rhetoric, and abstract emotional expression, making them difficult for image generation systems to reliably recognize and constrain. This often results in semantic discrepancies between the generated images and the original text. Existing text-to-image generation methods can achieve some success with ordinary descriptive texts, but when faced with highly condensed texts like classical poetry, they frequently encounter problems such as missing visual entities, incorrect numbering, attribute mismatches, unrepresented action relationships, spatial relationship shifts, and inconsistencies between the overall scene and the text. In particular, when classical poetry contains ellipsis, metaphors, allusions, and abstract emotions, the implicit visual objects and spatial relationships in the original text must first be structurally analyzed before they can become control conditions that the image generation model can execute.

[0004] To improve image generation results, existing technologies include cue rewriting, region-controlled generation, and image-text consistency evaluation. Cue rewriting methods can expand short texts into descriptive cues more suitable for image generation models, but most methods remain at the one-time cue construction stage, lacking stable semantic evidence that links the entire generation and redrawing process and can be verified and repaired. Region-controlled generation methods can improve the spatial controllability of image generation by utilizing object location or layout information, but usually rely on pre-defined object categories and region layouts. Image-text consistency evaluation methods can judge and provide feedback on the overall matching degree between the generated image and text, but their evaluation results are mostly expressed as overall similarity or overall question-answering score, making it difficult to locate specific failing objects, attributes, or relationships, and also difficult to directly translate into executable local image repair instructions.

[0005] Therefore, existing technologies have at least the following technical problems: First, the entity slots, attribute slots, and spatial relationship slots in the condensed text are unclear, making it difficult for text-to-image generation models to stably bind semantics and regions; Second, the generation control conditions and verification conditions usually have inconsistent sources, making it difficult to trace the evaluation results back to the specific semantic units used in the generation stage; Third, generation failures often only manifest as overall mismatch, making it difficult to pinpoint specific entity missingness, attribute errors, or spatial relationship deviations; Fourth, if local redrawing lacks a gating mechanism, it is easy to damage non-target regions, overall composition, or global semantic consistency while repairing the target region.

[0006] Based on the above problems, it is necessary to provide an automated generation scheme that can convert the implicit visual semantics in ancient poetry texts into machine-readable structured semantic information, and enable this structured semantic information to run through the entire process of generation control, consistency verification, failure localization, and local redrawing gating. Summary of the Invention

[0007] To address the problems in existing technologies, such as unclear semantic slots in image generation due to condensed text, fragmented generation control and consistency verification, difficulty in locating generation failures, and easy damage to non-target areas during local repair, this invention provides a semantically closed-loop controllable image generation method and apparatus for classical Chinese poetry texts. This approach does not rely solely on expanding prompt words; instead, it constructs structured semantic information for image generation and uses this structured semantic information as the unified data foundation for generation, verification, failure location, and repair gating.

[0008] On the one hand, to achieve the above objectives, this invention provides a semantically closed-loop controllable image generation method for classical Chinese poetry texts, comprising: Obtain the text of the ancient poems to be visualized; Visual semantic analysis is performed on the ancient poem text to be visualized to generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. Image generation control information and consistency verification information are generated based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. Based on the image generation control information, candidate images are generated, and the consistency of the candidate images is evaluated based on the atomic verification problem set to obtain the evaluation results corresponding to each semantic slot identifier. Based on the evaluation results and the semantic slot identifier, the failed semantic units are located to the corresponding entities, attributes, or relationships in the structured semantic information; Based on the located failed semantic units, a target editing region is determined in the candidate image, and local image editing is performed on the target editing region to obtain the edited image; When the edited image meets the preset gating conditions, the edited image is accepted and output, and a visualized image of the ancient poem text is output.

[0009] Preferably, the structured semantic information is a machine-readable semantic slot table, graph structure, JSON structure, or vectorized slot set; each semantic slot includes at least a slot identifier, slot type, visualization marker, entity category or relationship type, spatial prior, and validation question pointer; the spatial prior is used to limit the candidate region, relative position, or hierarchical priority of the corresponding visual semantic element in the image plane.

[0010] Preferably, visual semantic analysis is performed on the ancient poem text to be visualized, including: The ancient poem text to be visualized is initially parsed to obtain initial semantic parsing results; The initial semantic parsing results are subjected to semantic checks to identify at least one problem, such as object omission, attribute missing, relationship missing, cultural semantic deviation, incomplete spatial relationship, or non-visualized abstract concepts being mistakenly included in the entity set. A semantic check result containing the problem type, problem slot, and revision suggestions is generated. The initial semantic parsing result is revised based on the semantic check result to generate the structured semantic information.

[0011] Preferably, image generation control information and consistency verification information are generated based on the same structured semantic information, including: The prompt information is generated from the entity information, attribute information, relationship information, and scene information in the structured semantic information; The spatial constraint information is generated from the spatial priors in the structured semantic information; The atomic verification question set is generated from the semantic slot identifiers in the structured semantic information; The spatial constraint information and the atomic verification problem set each carry a semantic slot identifier pointing to the same semantic slot set, which is used to establish a traceability path for the semantic elements on which the evaluation result is based to the image generation control information.

[0012] Preferably, generating candidate images based on the image generation control information includes: The prompt information is input into the image generation model; In the stage where the number of denoising steps or time steps before image generation is greater than a preset time step threshold, the local latent variables corresponding to the target entity are fused with the global latent variables according to the region mask in the spatial constraint information to establish a spatial binding relationship between the target entity and the image region.

[0013] Preferably, the candidate images are evaluated for consistency based on the atomic verification problem set, including global consistency evaluation and local consistency evaluation; The global consistency evaluation is based on the ancient poetry text or the image description generated by the structured semantic information to construct a global verification problem, and the global consistency score of the candidate image is calculated through a multimodal evaluation model. The local consistency evaluation is used to input the candidate image and the set of atomic verification questions into a multimodal evaluation model to obtain the response probability of each atomic verification question. After aligning each response probability with the corresponding expected answer, the local consistency score of atomic verification questions associated with the same semantic slot or the same region is calculated. The expected answer of each atomic verification question includes a positive answer or a negative answer.

[0014] Preferably, based on the evaluation result and the semantic slot identifier, the failed semantic unit is located to the corresponding entity, attribute, or relationship in the structured semantic information, including: Local consistency scores below a preset local consistency threshold, local consistency scores below the average score of the corresponding semantic slot in the same group of candidate images, or comparison verification failure results obtained from forward and reverse atomic verification problems are identified as failed semantic evidence. Based on the semantic slot identifier corresponding to the failed semantic evidence, the failed semantic evidence is mapped to the entity, attribute or relation in the structured semantic information to determine the failed semantic unit to be repaired and the corresponding repair path type. The repair path types include object missing repair, attribute error correction repair, spatial relationship repair, process constraint repair, or hybrid repair.

[0015] Preferably, the preset gating conditions include: The local consistency improvement amount corresponding to the failed semantic unit reaches the preset improvement threshold; The decrease in global consistency did not exceed the preset tolerance; Furthermore, the image differences in non-target regions outside the target editing region do not exceed a preset retention threshold; The differences in the non-target region image include at least one of the following: pixel differences, structural similarity differences, or perceptual feature distances between the edited image and the candidate image outside the target editing region.

[0016] Preferably, obtaining the edited image includes: Local repair control information is generated based on the failed semantic unit and the corresponding spatial prior. The local repair control information includes the target entity or target relationship, repair path type, local editing prompt, negative repair prompt, target area soft mask and local redrawing process parameters. Based on the target region soft mask, noise re-injection or image editing is performed on the corresponding region of the candidate image, while keeping the image content or latent variables of the region outside the target region soft mask unchanged during the editing process.

[0017] On the other hand, to achieve the above objectives, the present invention also provides a semantically closed-loop controllable image generation device for classical Chinese poetry texts, comprising: The text acquisition module is used to acquire the text of ancient poems to be visualized; The semantic parsing module is used to perform visual semantic parsing on the ancient poetry text to be visualized, and generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. The control and verification module is used to generate image generation control information and consistency verification information based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. A candidate image generation module is used to generate candidate images based on the image generation control information; The consistency evaluation module is used to perform consistency evaluation on the candidate images based on the atomic verification problem set, and obtain the evaluation results corresponding to each semantic slot identifier; The failure location module is used to locate the failed semantic unit to the corresponding entity, attribute or relationship in the structured semantic information based on the evaluation result and the semantic slot identifier. The local editing and gating module is used to determine the target editing region in the candidate image based on the located failed semantic units, perform local image editing on the target editing region to obtain the edited image, and accept the edited image when the edited image meets the preset gating conditions; The output module is used to output a visualized image of the ancient poem text.

[0018] Compared with the prior art, the present invention has the following advantages and technical effects: (1) Improved semantic fidelity: By constructing unified structured semantic information and generating control information and consistency verification information on the basis of this, the evaluation results are traceably located to semantic units by using semantic slot pointers, which effectively reduces the binding error between semantic slots and image regions and significantly improves the accuracy of image content in restoring ancient poetry texts.

[0019] (2) Enhanced spatial controllability and regional stability: By constraining the spatial layout of entities through regional planning information and integrating local and global latent variables during the diffusion generation process, a stable binding relationship between entity semantics and image regions is established, thereby improving the spatial controllability of the generated image.

[0020] (3) Improved accuracy of failure localization: By performing fine-grained consistency evaluation on candidate images through the atomic verification problem set, the generation failure can be accurately attributed from the overall mismatch to specific entity missing, attribute error or spatial relationship deviation, providing precise targeting for subsequent local repair.

[0021] (4) Local redrawing disturbance is controllable: By setting the gating conditions, local semantic improvement, global consistency not significantly decreased and non-target areas are kept stable, new deviations caused by unconstrained redrawing are avoided, and the overall coordination and editing stability of the image are guaranteed.

[0022] (5) Strong interpretability of output results: It can simultaneously output a visual educational resource package containing images, semantic mapping, evaluation results and repair records, which improves the transparency of the generation process and the usability of the results, making it convenient for teaching, review and secondary editing. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a semantically closed-loop controllable image generation method for classical Chinese poetry texts according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the construction of structured semantic information and the common origin derivation relationship in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating consistency evaluation, failure location, local redrawing, and gating determination in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] This embodiment proposes a semantically closed-loop controllable image generation method for classical Chinese poetry texts, including: Obtain the text of the ancient poems to be visualized; Visual semantic analysis is performed on the ancient poem text to be visualized to generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. Image generation control information and consistency verification information are generated based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. Based on the image generation control information, candidate images are generated, and the consistency of the candidate images is evaluated based on the atomic verification problem set to obtain the evaluation results corresponding to each semantic slot identifier. Based on the evaluation results and the semantic slot identifier, the failed semantic units are located to the corresponding entities, attributes, or relationships in the structured semantic information; Based on the located failed semantic units, a target editing region is determined in the candidate image, and local image editing is performed on the target editing region to obtain the edited image; When the edited image meets the preset gating conditions, the edited image is accepted and output, and a visualized image of the ancient poem text is output.

[0027] Specifically, see Figure 1 This embodiment provides a semantically closed-loop controllable image generation method for classical Chinese poetry texts. The process takes classical Chinese poetry texts as input and sequentially performs input normalization, structured semantic anchor construction, homology generation control and verification information construction, region-aware image generation, atomic verification, failure attribution, local redrawing, gating judgment, and result output.

[0028] Specifically, it includes: S1: Obtain and standardize the classical Chinese poetry text. Obtain the classical Chinese poetry text to be visualized. The classical Chinese poetry text should at least include the main text of the poem, and may also include the title, author, dynasty, annotations, translation, analytical text, specified visual style, or teaching focus.

[0029] The system performs normalization processing on the input, including identifying the boundaries between the title and the main text, recognizing line breaks in the poem, unifying variant characters or simplified / traditional characters, removing irrelevant punctuation, extracting author and dynasty information, and extracting visual semantic cues from the annotations. For inputs without a translation or annotations, the system can generate auxiliary interpretations, but these interpretations are only used for semantic slot construction and do not replace the original poem text.

[0030] Furthermore, the structured semantic information is a machine-readable semantic slot table, graph structure, JSON structure, or vectorized slot set; each semantic slot includes at least a slot identifier, slot type, visualization marker, entity category or relationship type, spatial prior, and validation question pointer; the spatial prior is used to limit the candidate region, relative position, or hierarchical priority of the corresponding visual semantic element in the image plane.

[0031] Specifically, S2: Constructing structured semantic information.

[0032] The standardized classical Chinese poetry text undergoes processes including sentence decomposition, imagery recognition, visual entity extraction, entity attribute extraction, action relationship extraction, spatial relationship extraction, and emotional atmosphere recognition represented by visual attributes, resulting in structured semantic information. This structured semantic information includes at least scene information, a set of visual entities, a set of relationships between entities, and a set of semantic slots, which can be represented as: (1); in, This indicates the text of the ancient poem to be processed; Represents scene-level information; Represents a collection of visualized entities; Represents a set of action relationships, spatial relationships, and semantic relationships; Represents a set of semantic slots; Represents the i-th entity; Indicates the entity category; Represents a collection of attributes; This describes a partial action or relationship. It represents spatial priors.

[0033] Scene-level information can include time, location, season, weather, perspective, color tone, and emotional atmosphere represented by visual attributes. Visual entities can include people, scenery, objects, celestial bodies, animals, plants, or metaphorical objects that can be expressed through visual attributes. Relationships between entities can include action relationships, spatial relationships, subordinate relationships, contrast relationships, or semantic association relationships.

[0034] Furthermore, visual semantic analysis is performed on the ancient poem text to be visualized, including: The ancient poem text to be visualized is initially parsed to obtain initial semantic parsing results; The initial semantic parsing results are subjected to semantic checks to identify at least one problem, such as object omission, attribute missing, relationship missing, cultural semantic deviation, incomplete spatial relationship, or non-visualized abstract concepts being mistakenly included in the entity set. A semantic check result containing the problem type, problem slot, and revision suggestions is generated. The initial semantic parsing result is revised based on the semantic check result to generate the structured semantic information.

[0035] Specifically, S3: performs semantic anchor-guided parsing, inspection, and revision.

[0036] To reduce parsing bias caused by omissions, metaphors, allusions, and abstract emotions, this embodiment adopts a semantic anchor construction process that combines parsing, checking, and revision, which can be represented as: (2); in, This indicates the initial parsing process, used to generate an initial semantic draft from the ancient poetry text. ; This indicates semantic inspection processing used to output a set of feedback around cultural semantic consistency, visual generativeability, and relational integrity. ; This indicates revision processing, used to combine the original text, the initial semantic draft, and the feedback set to generate revised structured semantic information.

[0037] The initial parsing process is used to identify objects, attributes, actions, spatial orientation, and scene atmosphere in the verses; Semantic inspection processing is used to identify problems such as missing objects, missing attributes, missing relationships, cultural semantic bias, incomplete spatial relationships, or non-visualized abstract concepts being mistakenly included in entity sets; Revision processing is used to remove abstract concepts that are not suitable as image entities and to add visual entities, attributes, or relationships that are necessary for the image to express itself.

[0038] The feedback set can include at least the issue type, issue slot, and revision suggestions.

[0039] For example, for the phrase "bowing down, thinking of my hometown," instead of treating "thinking of my hometown" as a separate visual entity, it is mapped to the character's posture, facial orientation, color tone, compositional white space, or emotional atmosphere slots. For the phrase "it seems like frost on the ground," the "frost" can be treated as a cold white, off-white, low-saturation ground visual effect or a metaphorical attribute, rather than forcibly generating a real frost layer entity.

[0040] Furthermore, image generation control information and consistency verification information are generated based on the same structured semantic information, including: The prompt information is generated from the entity information, attribute information, relationship information, and scene information in the structured semantic information; The spatial constraint information is generated from the spatial priors in the structured semantic information; The atomic verification question set is generated from the semantic slot identifiers in the structured semantic information; The spatial constraint information and the atomic verification problem set each carry a semantic slot identifier pointing to the same semantic slot set, which is used to establish a traceability path for the semantic elements on which the evaluation result is based to the image generation control information.

[0041] Specifically, S4: Image generation control information and consistency verification information are constructed from the same structured semantic information. For example... Figure 2 Based on the same structured semantic information, the synchronous derivation of the hint package (i.e., hint information), regional planning information (i.e., spatial constraint information), and atomic verification question set can be represented as follows: (3); (4); (5); In the formula, This represents a mapping for generating suggestion packets from structured semantic information; This represents a mapping that generates a set of atomic verification questions from structured semantic information. Indicates a prompt packet; This indicates a positive message; This indicates a negative message; Indicates style-related information; This indicates compositional cues; Indicates regional planning information; Represents the set of atomic verification questions; This represents a visual statement or question about the j-th atom; Indicates the expected answer; Indicates the weight of the problem; Indicates semantic hierarchy; This represents a pointer to the corresponding semantic slot in the structured semantic information.

[0042] Positive prompts describe the entities, attributes, actions, spatial relationships, and scene atmosphere that should be presented; negative prompts are used to suppress incorrect objects, duplicate entities, incorrect attributes, incorrect historical context, incorrect spatial relationships, or visual elements that are not suitable for display; area planning information is used to record entity area bounding boxes, area masks, entity area prompts, area priorities, occlusion relationships, and relative position constraints.

[0043] Since the prompt package, regional planning information, and atomic verification question set are all derived from the same structured semantic information, the generated evaluation results can be traced back to the entities, attributes, action relationships, or spatial relationships on which the generation control is based through semantic slot pointers.

[0044] For example, the entity slot "moon" can be used to generate a positive prompt "There is a bright full moon in the night sky" or an atomic verification question "Does the moon exist in the image?"; the spatial relationship slot "egrets above the blue sky" can be used for region planning or to generate a verification question "Is the egret located in the upper sky area of ​​the image?"

[0045] Further, candidate images are generated based on the image generation control information, including: The prompt information is input into the image generation model; In the stage where the number of denoising steps or time steps before image generation is greater than a preset time step threshold, the local latent variables corresponding to the target entity are fused with the global latent variables according to the region mask in the spatial constraint information to establish a spatial binding relationship between the target entity and the image region.

[0046] Specifically, S5: Performs region-aware candidate image generation.

[0047] The prompt package is input into the image generation model, and the regional planning information is used as an entity-level spatial constraint in the generation of candidate images.

[0048] In this embodiment, the image generation model can be a text-to-image model, a latent space diffusion model, a region-controlled generation model, an image editing model, or a combination of the above models. A training-independent region-aware diffusion generation method can be used, binding entity-level cues and spatial priors to image regions through early hard binding and subsequent soft refinement.

[0049] Specifically, a region mask is generated based on prior knowledge of the entity space, and local and global latent variables of the entity region are fused in the early stages of diffusion generation.

[0050] The summation of the masks of each entity region, combined with the global latent variables, can be expressed as: (6); In the formula, Represents the latent variables after the fusion at step t; This represents the mask for the i-th entity region; Represents the local latent variables of the i-th entity region; Represents a global latent variable; ⊙ represents element-wise multiplication; This indicates the number of visualized entities.

[0051] The region mask in equation (6) can be a binary mask or a soft mask with smooth boundaries. If different entity regions overlap, normalization can be performed based on entity priority, occlusion relationship, or relationship slot weight. However, the normalization process does not change the basic technical idea of ​​"complementary fusion of local entity latent variables and remaining global latent variables". This region fusion method enables entity semantics to establish a binding relationship with the image region in the early stage of generation and maintains the consistency of overall illumination, tone, and style in the subsequent denoising process. The system can generate multiple candidate images for the same ancient poem text.

[0052] Furthermore, the candidate images are evaluated for consistency based on the atomic verification problem set, including global consistency evaluation and local consistency evaluation; The global consistency evaluation is based on the ancient poetry text or the image description generated by the structured semantic information to construct a global verification problem, and the global consistency score of the candidate image is calculated through a multimodal evaluation model. The local consistency evaluation is used to input the candidate image and the set of atomic verification questions into a multimodal evaluation model to obtain the response probability of each atomic verification question. After aligning each response probability with the corresponding expected answer, the local consistency score of atomic verification questions associated with the same semantic slot or the same region is calculated. The expected answer of each atomic verification question includes a positive answer or a negative answer.

[0053] Specifically, S6: Perform global consistency evaluation and local consistency evaluation, such as... Figure 3 .

[0054] For each candidate image, a global validation question is constructed based on the ancient poem text or the image description generated from structured semantic information, and a global consistency score is calculated using a multimodal evaluation model, which can be represented as: (7); In the formula, Represents candidate images; This refers to a global validation question constructed from ancient Chinese poetry texts or visual descriptions. This represents the global consistency score of the candidate image relative to the overall semantics of the ancient poem; This represents the probability that the multimodal evaluation model outputs a positive answer.

[0055] For local consistency evaluation, the candidate image and the set of atomic verification questions Q are input into the multimodal evaluation model to obtain the response probability for each atomic verification question, and answer alignment is performed according to the expected answer, which can be represented as: (8); For multiple atomic verification problems associated with the same region or the same semantic slot, the weighted local consistency score can be calculated as follows: (9); in, This represents the probability of a positive answer to the verification question of the m-th atom in the j-th region or slot. This indicates the score after the answers are aligned; Indicates the corresponding weight; This indicates the score for local consistency at the region or slot level.

[0056] The emotional atmosphere slots, characterized by visual attributes, do not directly evaluate poetic beauty, aesthetics, or subjective feelings. Instead, they are transformed into visually determinable questions. For example, atomic verification questions could include "Is the overall image a low-saturation cool color tone?", "Are the figures looking down or gazing into the distance?", "Is there a low-hanging sunset in the image?", and "Does the distant area have a large amount of negative space in the composition?" The answers to these questions can be calculated using multimodal evaluation models or visual feature extraction models.

[0057] Further, based on the evaluation results and the semantic slot identifiers, the failed semantic units are located to the corresponding entities, attributes, or relationships in the structured semantic information, including: Local consistency scores below a preset local consistency threshold, local consistency scores below the average score of the corresponding semantic slot in the same group of candidate images, or comparison verification failure results obtained from forward and reverse atomic verification problems are identified as failed semantic evidence. Based on the semantic slot identifier corresponding to the failed semantic evidence, the failed semantic evidence is mapped to the entity, attribute or relation in the structured semantic information to determine the failed semantic unit to be repaired and the corresponding repair path type. The repair path types include object missing repair, attribute error correction repair, spatial relationship repair, process constraint repair, or hybrid repair.

[0058] Specifically, S7: Semantic location of execution failure.

[0059] The semantic evidence of failure is determined based on the local consistency score, the average score of the corresponding region or slot within the candidate image group, and the comparison of the verification failure results.

[0060] The failure candidate determination can be expressed as: (10); in, This indicates whether the j-th region or slot has been identified as a failure candidate. [·] indicates an indicator function; Indicates the local consistency threshold; This represents the average score of candidate images in the j-th region or slot; This indicates a failure flag for the comparative verification obtained from the forward and reverse atomic verification problem.

[0061] When a failure candidate is valid, the semantic evidence of failure is mapped back to the corresponding entity, attribute, action relationship, or spatial relationship in the structured semantic information through the semantic slot pointer in the atomic verification question. If the failure slot is an entity existence slot, it is determined that the object is missing; if the failure slot is an attribute slot, it is determined that the attribute is incorrect; if the failure slot is an action or spatial relationship slot, it is determined that the action relationship is incorrect or the spatial relationship is off.

[0062] For each entity, the alignment scores of the answers to its associated atomic verification questions can be aggregated to obtain the entity-level consistency score and failure strength. The entity with the highest failure strength is then selected as the target for remediation in the current round, which can be represented as: (11); (12); in, Representation and entity A set of related atomic verification questions; Indicates the entity-level consistency score; Indicates the intensity of entity failure; Represents the set of failed candidate entities; This represents the target entity for repair in the current round. If a failed relationship involves multiple entities, then the region of entities involved in that relationship can be collectively defined as the target repair region.

[0063] S8: Generate local repair control information and perform local redrawing.

[0064] For the target repair object, local repair control information is generated based on failure semantic units, region priors, and local visual evidence, which can be represented as: (13); in, This represents the repair specifications for the target entity in the r-th round; Indicates the repair path type; This indicates a partial editing prompt; Indicates the soft mask of the target entity; This represents the control parameters for the local redrawing process.

[0065] Repair path types include object missing repair, attribute error correction repair, spatial relationship repair, process constraint repair, or mixed repair.

[0066] When performing local redrawing, noise re-injection or image editing is only performed on the area corresponding to the soft mask of the target area, while the original latent variables or original image content are preserved in the non-target areas.

[0067] The redraw initialization and denoising process can be represented as: (14); (15); Where η~N(0,I) represents reinjection noise; Represents the current latent variables of the image; This indicates that the latent variables are redrawn and initialized. This represents the latent variable obtained by the edit branch at step t-1; This represents the latent variable of the original graph branch at step t-1.

[0068] By using equations (14) and (15), repair is performed in the target area while preserving the original image content or latent variables in the non-target area.

[0069] Furthermore, the preset gating conditions include: The local consistency improvement amount corresponding to the failed semantic unit reaches the preset improvement threshold; The decrease in global consistency did not exceed the preset tolerance; Furthermore, the image differences in non-target regions outside the target editing region do not exceed a preset retention threshold; The differences in the non-target region image include at least one of the following: pixel differences, structural similarity differences, or perceptual feature distances between the edited image and the candidate image outside the target editing region.

[0070] Specifically, the gating judgment for the redrawn image includes: calculating the difference in local consistency scores corresponding to the failed semantic units before and after redrawing as the local semantic improvement amount; calculating the difference in global consistency scores before and after redrawing as the global consistency change amount; calculating at least one of the pixel differences, structural similarity differences, or perceptual feature distances of non-target regions before and after redrawing as the non-target region differences; and accepting the redrawn image when the local semantic improvement amount is greater than or equal to a preset improvement threshold, the global consistency change amount is greater than or equal to a negative preset descent tolerance, and the non-target region differences are less than or equal to a preset preservation threshold.

[0071] S9: Perform gating judgment on the redraw result.

[0072] After completing the local redraw, the global consistency evaluation and local consistency evaluation are re-executed, and the local semantic improvement and global consistency change of the target entity are calculated, which can be expressed as: (16); (17); (18); in, This represents the image before the r-th redraw; Indicates the candidate image after redrawing; This represents the local semantic improvement of the target entity; Indicates the amount of change in global consistency; This indicates a localized increase in the threshold. This indicates a decrease in tolerance for global consistency.

[0073] The difference criterion for non-target regions can be further used to quantify the degree of preservation of non-edited regions, and can be expressed as: (19); (20); in, Indicates differences in non-target regions; It can be at least one of pixel difference, structural similarity difference, or perceptual feature distance; This indicates that the non-target region retains a threshold. This indicates whether to accept the current partial redrawing result. If the gating condition is not met, the current redrawing result is rejected, and the original image is retained or the local repair control information is regenerated.

[0074] The closed-loop process can terminate when no new failed entities exist, the overall consistency meets the stopping condition, or the maximum number of iterations is reached. The maximum number of iterations, local threshold, global threshold, and non-target region preservation threshold can be set according to the application scenario, computing resources, or image quality requirements.

[0075] S10: Output the final visualization resources.

[0076] The final image is selected from the set of candidate images and accepted redrawn images. The system can make the selection by combining global consistency and local semantic fidelity based on the atomic verification question set, which can be expressed as: (twenty one); in, This indicates the final output image; This represents the set of candidate images obtained through generation and local redrawing. Indicates the global consistency weight; This represents a set of atomic verification questions generated from structured semantic information. This represents the local semantic fidelity of the candidate image relative to the atomic verification question set.

[0077] When there are multiple candidate images or multiple accepted redrawn images, a comprehensive score is calculated based on the global consistency score, the local consistency score of each semantic slot, the non-target region preservation index, and the semantic slot priority. The image with the highest comprehensive score is selected as the final visualization image.

[0078] When outputting the visualized image, a visualized educational resource package is output simultaneously. The visualized educational resource package includes at least one of the following: image description, structured semantic information, mapping relationship between verses and visualized entities, correspondence between image regions and semantic slots, consistency evaluation results, failure semantic evidence, local repair records, and gating judgment results.

[0079] This embodiment also provides a semantically closed-loop controllable image generation device for classical Chinese poetry texts, including: The text acquisition module is used to acquire the text of ancient poems to be visualized; The semantic parsing module is used to perform visual semantic parsing on the ancient poetry text to be visualized, and generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. The control and verification module is used to generate image generation control information and consistency verification information based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. A candidate image generation module is used to generate candidate images based on the image generation control information; The consistency evaluation module is used to perform consistency evaluation on the candidate images based on the atomic verification problem set, and obtain the evaluation results corresponding to each semantic slot identifier; The failure location module is used to locate the failed semantic unit to the corresponding entity, attribute or relationship in the structured semantic information based on the evaluation result and the semantic slot identifier. The local editing and gating module is used to determine the target editing region in the candidate image based on the located failed semantic units, perform local image editing on the target editing region to obtain the edited image, and accept the edited image when the edited image meets the preset gating conditions; The output module is used to output a visualized image of the ancient poem text.

[0080] Taking "Quiet Night Thoughts" as an example, this device can determine the scene information as nighttime, indoor or semi-indoor, cool white moonlight, and a quiet atmosphere; determine the visual effects of the figure, bed, moon, ground in front of the bed, and moonlight or frost as visual entities or attribute slots; and determine "moonlight shining on the area in front of the bed," "the figure looking up at the moon," and "the figure looking down in thought" as action or spatial relationship slots. Accordingly, atomic verification questions can include "whether the moon exists in the image," "whether the ground in front of the bed is cool white," and "whether the figure is looking up or looking down." If the candidate image lacks the moon, the system locates the "moon" entity slot through the semantic slot pointer and generates a soft mask of the target area in the night sky or outside the window for local redrawing; the redrawing result is accepted only when the moon slot score improves, the global consistency does not decrease significantly, and the difference in non-target areas is less than a threshold.

[0081] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for semantically closed-loop controllable image generation for classical Chinese poetry texts, characterized in that, include: Obtain the text of the ancient poems to be visualized; Visual semantic analysis is performed on the ancient poem text to be visualized to generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. Image generation control information and consistency verification information are generated based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. Based on the image generation control information, candidate images are generated, and the consistency of the candidate images is evaluated based on the atomic verification problem set to obtain the evaluation results corresponding to each semantic slot identifier. Based on the evaluation results and the semantic slot identifier, the failed semantic units are located to the corresponding entities, attributes, or relationships in the structured semantic information; Based on the located failed semantic units, a target editing region is determined in the candidate image, and local image editing is performed on the target editing region to obtain the edited image; When the edited image meets the preset gating conditions, the edited image is accepted and output, and a visualized image of the ancient poem text is output.

2. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 1, characterized in that, The structured semantic information is a machine-readable semantic slot table, graph structure, JSON structure, or vectorized slot set; each semantic slot includes at least a slot identifier, slot type, visual identifier, entity category or relationship type, spatial prior, and validation question pointer; The spatial prior is used to define the candidate region, relative position, or hierarchical priority of the corresponding visual semantic element in the image plane.

3. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 2, characterized in that, Visual semantic analysis of the ancient poems to be visualized includes: The ancient poem text to be visualized is initially parsed to obtain initial semantic parsing results; The initial semantic parsing results are subjected to semantic checks to identify at least one problem, such as object omission, attribute missing, relationship missing, cultural semantic deviation, incomplete spatial relationship, or non-visualized abstract concepts being mistakenly included in the entity set. A semantic check result containing the problem type, problem slot, and revision suggestions is generated. The initial semantic parsing result is revised based on the semantic check result to generate the structured semantic information.

4. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 1, characterized in that, Image generation control information and consistency verification information are generated based on the same structured semantic information, including: The prompt information is generated from the entity information, attribute information, relationship information, and scene information in the structured semantic information; The spatial constraint information is generated from the spatial priors in the structured semantic information; The atomic verification question set is generated from the semantic slot identifiers in the structured semantic information; The spatial constraint information and the atomic verification problem set each carry a semantic slot identifier pointing to the same semantic slot set, which is used to establish a traceability path for the semantic elements on which the evaluation result is based to the image generation control information.

5. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 1, characterized in that, Based on the image generation control information, candidate images are generated, including: The prompt information is input into the image generation model; In the stage where the number of denoising steps or time steps before image generation is greater than a preset time step threshold, the local latent variables corresponding to the target entity are fused with the global latent variables according to the region mask in the spatial constraint information to establish a spatial binding relationship between the target entity and the image region.

6. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 1, characterized in that, The candidate images are evaluated for consistency based on the atomic verification problem set, including global consistency evaluation and local consistency evaluation. The global consistency evaluation is based on the ancient poetry text or the image description generated by the structured semantic information to construct a global verification problem, and the global consistency score of the candidate image is calculated through a multimodal evaluation model. The local consistency evaluation is used to input the candidate image and the set of atomic verification questions into a multimodal evaluation model to obtain the response probability of each atomic verification question. After aligning each response probability with the corresponding expected answer, the local consistency score of atomic verification questions associated with the same semantic slot or the same region is calculated. The expected answer of each atomic verification question includes a positive answer or a negative answer.

7. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 6, characterized in that, Based on the evaluation results and the semantic slot identifiers, the failed semantic units are located to the corresponding entities, attributes, or relationships in the structured semantic information, including: Local consistency scores below a preset local consistency threshold, local consistency scores below the average score of the corresponding semantic slot in the same group of candidate images, or comparison verification failure results obtained from forward and reverse atomic verification problems are identified as failed semantic evidence. Based on the semantic slot identifier corresponding to the failed semantic evidence, the failed semantic evidence is mapped to the entity, attribute or relation in the structured semantic information to determine the failed semantic unit to be repaired and the corresponding repair path type. The repair path types include object missing repair, attribute error correction repair, spatial relationship repair, process constraint repair, or hybrid repair.

8. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 1, characterized in that, The preset gating conditions include: The local consistency improvement amount corresponding to the failed semantic unit reaches the preset improvement threshold; The decrease in global consistency did not exceed the preset tolerance; Furthermore, the image differences in non-target regions outside the target editing region do not exceed a preset retention threshold; The differences in the non-target region image include at least one of the following: pixel differences, structural similarity differences, or perceptual feature distances between the edited image and the candidate image outside the target editing region.

9. The semantically closed-loop controllable image generation method for classical Chinese poetry texts according to claim 8, characterized in that, Obtaining the edited image includes: Local repair control information is generated based on the failed semantic unit and the corresponding spatial prior. The local repair control information includes the target entity or target relationship, repair path type, local editing prompt, negative repair prompt, target area soft mask and local redrawing process parameters. Based on the target region soft mask, noise re-injection or image editing is performed on the corresponding region of the candidate image, while keeping the image content or latent variables of the region outside the target region soft mask unchanged during the editing process.

10. A semantically closed-loop controllable image generation device for classical Chinese poetry texts, used to implement the semantically closed-loop controllable image generation method for classical Chinese poetry texts as described in any one of claims 1-9, characterized in that, include: The text acquisition module is used to acquire the text of ancient poems to be visualized; The semantic parsing module is used to perform visual semantic parsing on the ancient poetry text to be visualized, and generate structured semantic information, which includes visual semantic elements and semantic slot identifiers corresponding to each visual semantic element. The control and verification module is used to generate image generation control information and consistency verification information based on the same structured semantic information. The image generation control information includes at least prompt information for constraining the content of the generated image and spatial constraint information for constraining the spatial layout of the image. The consistency verification information includes a set of atomic verification questions associated with the semantic slot identifier. A candidate image generation module is used to generate candidate images based on the image generation control information; The consistency evaluation module is used to perform consistency evaluation on the candidate images based on the atomic verification problem set, and obtain the evaluation results corresponding to each semantic slot identifier; The failure location module is used to locate the failed semantic unit to the corresponding entity, attribute or relationship in the structured semantic information based on the evaluation result and the semantic slot identifier. The local editing and gating module is used to determine the target editing region in the candidate image based on the located failed semantic units, perform local image editing on the target editing region to obtain the edited image, and accept the edited image when the edited image meets the preset gating conditions; The output module is used to output a visualized image of the ancient poem text.

Images