Intelligent scene image generation control method and system based on natural language

Abstract

The invention provides an intelligent scene image generation control method and system based on a natural language, and relates to the technical field of artificial intelligence and computer vision, and the method comprises the steps: carrying out the semantic analysis of a natural language text, constructing and disambiguating a scene graph structure, and mapping the scene graph structure to a potential feature space of an image generator, and after the candidate image is generated, scene structure consistency verification is carried out, and directional regeneration is carried out on the deviation area. According to the method, the semantic consistency and visual quality of the generated image can be improved, and accurate conversion from the natural language to the image is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, and in particular to a method and system for intelligent scene image generation and control based on natural language. Background Technology

[0002] In the fields of artificial intelligence and computer vision, generating corresponding images based on natural language descriptions is an important research direction. With the development of deep learning technology, it has become possible to directly generate high-quality, semantically accurate images from text descriptions. This technology has broad application prospects in fields such as virtual reality, game design, film and television production, and advertising design. Currently, natural language-based image generation technology mainly uses neural network models to encode text information into feature vectors, and then generates corresponding images through methods such as generative adversarial networks or diffusion models.

[0003] Existing technologies suffer from insufficient semantic understanding when handling complex scene descriptions. When users input descriptions containing multiple objects and their complex spatial relationships, the system often fails to accurately capture the semantic connections and spatial layout between entities, resulting in a significant deviation between the generated image and the user-described scene. Existing technologies lack effective ambiguity resolution mechanisms. Natural language descriptions often contain polysemous words or vague expressions, which existing systems struggle to accurately identify and resolve, leading to generated image content that does not match the user's intent, particularly when dealing with complex descriptions containing multiple referents or implicit semantics. Existing technologies lack effective verification and correction mechanisms after image generation. After generating an image, the system cannot assess the consistency between the image content and the original semantic description, nor does it have the ability to target and correct inconsistencies. This means that even if errors are detected, precise local adjustments cannot be made; the entire image must be regenerated, resulting in inefficiency and inconsistent outcomes. Summary of the Invention

[0004] This invention provides a method and system for intelligent scene image generation and control based on natural language, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a natural language-based intelligent scene image generation and control method, comprising: Perform semantic parsing on natural language text to extract the semantic structure of the scene; A scene graph structure is constructed based on the scene semantic structure. A semantic consistency constraint network is constructed to resolve the nodes and edges of the scene graph structure, generating a disambiguated scene graph structure. The disambiguated scene graph structure is mapped to the latent feature space of the image generator to establish the correspondence between semantic structure and visual features; Based on the correspondence, an overall spatial configuration is generated according to the spatial relationship in the disambiguated scene graph structure. Based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, entity visual features are generated. The overall spatial configuration and the entity visual features are spatially combined and pixel-level fused to obtain candidate images. The candidate image is subjected to scene structure consistency verification. The entity layout and attribute representation in the candidate image are extracted and compared with the disambiguated scene graph structure to generate structural deviation measurement information. Based on the structural deviation measurement information, the deviation region in the candidate image is regenerated in a targeted manner. The regenerated result is fused with the consistent region in the candidate image to generate an intelligent scene image.

[0006] Based on the aforementioned scene semantic structure, a scene graph structure is constructed. A semantic consistency constraint network is then used to resolve the nodes and edges of the scene graph structure, generating a disambiguated scene graph structure, including: The scene entities, spatial relationships, and attribute features are extracted from the scene semantic structure. A node is created for each scene entity and associated with the corresponding attribute feature. An edge is created for each spatial relationship to connect the corresponding node, thus obtaining the scene graph structure. Based on the scene graph structure, a cross-entity semantic dependency propagation graph is constructed. In the semantic dependency propagation graph, a dependency connection is established for each pair of nodes with semantic association, and each dependency connection is labeled with at least one constraint type among co-occurrence constraint, spatial compatibility constraint, and attribute mutual exclusion constraint. For each node and edge in the scene graph structure, the dependency connection and its constraint type associated with the node or edge are found in the semantic dependency propagation graph. Multi-hop constraint propagation is performed along the dependency connection to obtain the constraint propagation result of the node or edge. Based on the constraint propagation results, the scene graph structure is disambiguated. For nodes with unclear entity referents, the referent is determined according to the co-occurrence constraint. For edges with conflicting relationships, the conflict is resolved according to the spatial compatibility constraint. For nodes with ambiguous attributes, the attribute values ​​are clarified according to the attribute mutual exclusion constraint. The disambiguated scene graph structure is then generated.

[0007] Mapping the disambiguated scene graph structure to the latent feature space of the image generator, and establishing the correspondence between semantic structure and visual features, includes: Based on the disambiguated scene graph structure, semantic vector encoding is performed on each node to extract the scene entity and attribute features corresponding to the node, relation vector encoding is performed on each edge to extract the spatial relation semantics corresponding to the edge, and a scene graph semantic vector set is constructed. Obtain the feature dimension structure and spatial distribution characteristics of the latent feature space of the image generator, and construct a cross-modal mapping network from the semantic vector space to the latent feature space; The semantic vector set of the scene graph is input into the cross-modal mapping network. The semantic vector of each node is mapped to the corresponding entity latent feature vector through the cross-modal mapping network, and the relation vector of each edge is mapped to the corresponding spatial layout feature vector. Based on the entity latent feature vector and the spatial layout feature vector, feature position coordinates and feature space range are assigned to each scene entity in the latent feature space, feature constraints are assigned to each spatial relationship, and the mapping position relationship between scene entities and spatial relationships in the latent feature space is established, thus completing the establishment of the correspondence between semantic structure and visual features.

[0008] Based on the aforementioned correspondence, an overall spatial configuration is generated according to the spatial relationships in the disambiguated scene graph structure. Entity visual features are then generated based on the overall spatial configuration and the attribute features of the disambiguated scene graph structure, including: The spatial layout feature vectors in the correspondence are analyzed to obtain the relative positional relationships and spatial constraints between scene entities. Based on the relative positional relationships and spatial constraints, a spatial region is allocated to each scene entity in the image space and a corresponding semantic placeholder is generated. The semantic placeholder records the scene entity identity information and spatial boundary information corresponding to the spatial region, resulting in the overall spatial configuration containing multiple spatial regions and corresponding semantic placeholders. Extract the entity latent feature vector from the correspondence and extract the corresponding attribute features from the disambiguated scene graph structure to construct the mapping relationship between scene entities and entity latent feature vectors and attribute features; Obtain the semantic placeholder identifier for each spatial region, query the mapping relationship based on the scene entity identity information in the semantic placeholder identifier to obtain the corresponding entity potential feature vector and attribute features, and generate the entity visual features of the scene entity within the spatial boundary information of the spatial region based on the entity potential feature vector and attribute features.

[0009] The overall spatial configuration and the entity visual features are spatially combined and fused at the pixel level to obtain candidate images, including: Extract spatial boundary information and semantic placeholders for each spatial region from the overall spatial configuration, determine the scene entity corresponding to the spatial region based on the semantic placeholders, obtain the entity visual features of the scene entity, and establish a spatial mapping relationship from the spatial region to the entity visual features. Based on the spatial mapping relationship, each entity visual feature is placed at the target position in the image space according to the spatial boundary information, keeping the spatial position of the entity visual feature consistent with the position of the spatial region, to obtain the spatial combination result; Boundary fusion is performed on the boundary regions between adjacent spatial regions to eliminate pixel discontinuities between visual features of adjacent entities and generate a boundary fusion region; pixel values ​​of multiple entity visual features in the overlapping region of the spatial combination result are extracted; based on the spatial relationship describing the occlusion relationship between scene entities in the overlapping region in the disambiguated scene graph structure, pixel depth priority is determined; and pixel values ​​in the overlapping region are hierarchically fused according to the pixel depth priority to generate an overlapping fusion region. The candidate image is obtained by embedding the boundary fusion region and the overlapping fusion region into the spatial combination result.

[0010] The candidate images are subjected to scene structure consistency verification. Entity layout and attribute representation in the candidate images are extracted and compared with the structure of the disambiguated scene graph to generate structural deviation metric information, including: Identify visual entities in the candidate images, extract the position coordinates and size information of each visual entity in the image space, and construct an entity layout representation of the candidate images; extract visual attributes for each visual entity, obtain shape features, color features and texture features, and construct an attribute representation description of the candidate images; Spatial relationships and attribute features are extracted from the disambiguated scene graph structure. The spatial relationships are converted into reference position constraints and reference size constraints, and the attribute features are converted into reference shape constraints, reference color constraints, and reference texture constraints. Based on the entity layout representation and visual entity, the positional deviation between the position coordinates and the reference position constraint of the corresponding scene entity is calculated, and the size deviation between the size information and the reference size constraint of the corresponding scene entity is calculated to obtain the layout deviation metric; based on the attribute representation description and visual entity, the shape deviation between the shape feature and the reference shape constraint is calculated, the color deviation between the color feature and the reference color constraint is calculated, and the texture deviation between the texture feature and the reference texture constraint is calculated to obtain the attribute deviation metric. The layout deviation metric and the attribute deviation metric are fused to obtain the structural deviation metric information.

[0011] Based on the structural deviation metric information, the deviation regions in the candidate image are regenerated in a targeted manner. The regenerated results are then fused with the consistent regions in the candidate image to generate an intelligent scene image, including: The structural deviation measurement information is compared with a preset deviation judgment threshold to identify deviation entities and consistency entities; in the candidate image, the image area occupied by the deviation entity is marked as the deviation area, and the image area occupied by the consistency entity is marked as the consistency area. For each deviation region, semantic description information and spatial relationship information of the deviation entity corresponding to the deviation region are extracted from the disambiguated scene graph structure. The semantic description information is converted into regenerated semantic guidance features, and the spatial relationship information is converted into regenerated spatial constraint features. Based on the regenerated semantic guidance features and the regenerated spatial constraint features, the deviation region is regenerated in a targeted manner to generate regenerated region content that meets the requirements of the disambiguated scene graph structure. Replace the corresponding deviation regions in the candidate image with the regenerated region content of all deviation regions, while keeping the consistent regions in the candidate image unchanged, to obtain the image after region replacement; perform boundary smoothing processing on the boundary between the regenerated region content and the consistent region content in the image after region replacement to eliminate pixel discontinuities, and generate the intelligent scene image.

[0012] A second aspect of the present invention provides an intelligent scene image generation and control system based on natural language, comprising: The first unit is used to perform semantic parsing of natural language text and extract the semantic structure of the scene; The second unit is used to construct a scene graph structure based on the scene semantic structure, and to resolve the nodes and edges of the scene graph structure by constructing a semantic consistency constraint network to generate a disambiguated scene graph structure. The third unit is used to map the disambiguated scene graph structure to the latent feature space of the image generator, and establish the correspondence between semantic structure and visual features; The fourth unit is used to generate an overall spatial configuration based on the correspondence and the spatial relationship in the disambiguated scene graph structure, generate entity visual features based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, and perform spatial combination and pixel-level fusion of the overall spatial configuration and the entity visual features to obtain candidate images. The fifth unit is used to perform scene structure consistency verification on the candidate image, extract the entity layout and attribute representation in the candidate image, compare it with the disambiguated scene graph structure, generate structural deviation measurement information, and perform directional regeneration of the deviation region in the candidate image based on the structural deviation measurement information. The regeneration result is then fused with the consistent region in the candidate image to generate an intelligent scene image.

[0013] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0014] Fourth aspect of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] The beneficial effects of this application are as follows: By performing semantic parsing on natural language text and constructing a scene graph structure, a precise conversion from natural language to a visualized scene is achieved, solving the problem of semantic ambiguity in text description and improving the matching degree between generated images and user intent.

[0016] By establishing a correspondence between semantic structure and visual features, the generated images are more reasonable in terms of spatial layout and entity visual representation, enhancing the logic and realism of the image content and meeting the generation needs of complex scenes.

[0017] By introducing a scene structure consistency verification mechanism, deviation areas in candidate images are regenerated in a targeted manner. This effectively avoids common problems in traditional image generation methods, such as missing entities and incorrect attributes, thereby improving the quality and accuracy of the generated images and making the final generated intelligent scene images more consistent with the intent of the original language description. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the intelligent scene image generation and control method based on natural language according to an embodiment of the present invention. Figure 2 A flowchart illustrating the process of establishing the correspondence between semantic structure and visual features. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0021] Figure 1 This is a flowchart illustrating the intelligent scene image generation and control method based on natural language according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: Perform semantic parsing on natural language text to extract the semantic structure of the scene; A scene graph structure is constructed based on the scene semantic structure. A semantic consistency constraint network is constructed to resolve the nodes and edges of the scene graph structure, generating a disambiguated scene graph structure. The disambiguated scene graph structure is mapped to the latent feature space of the image generator to establish the correspondence between semantic structure and visual features; Based on the correspondence, an overall spatial configuration is generated according to the spatial relationship in the disambiguated scene graph structure. Based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, entity visual features are generated. The overall spatial configuration and the entity visual features are spatially combined and pixel-level fused to obtain candidate images. The candidate image is subjected to scene structure consistency verification. The entity layout and attribute representation in the candidate image are extracted and compared with the disambiguated scene graph structure to generate structural deviation measurement information. Based on the structural deviation measurement information, the deviation region in the candidate image is regenerated in a targeted manner. The regenerated result is fused with the consistent region in the candidate image to generate an intelligent scene image.

[0022] In one optional implementation, a scene graph structure is constructed based on the scene semantic structure, and the nodes and edges of the scene graph structure are resolved by constructing a semantic consistency constraint network to generate a disambiguated scene graph structure, including: The scene entities, spatial relationships, and attribute features are extracted from the scene semantic structure. A node is created for each scene entity and associated with the corresponding attribute feature. An edge is created for each spatial relationship to connect the corresponding node, thus obtaining the scene graph structure. Based on the scene graph structure, a cross-entity semantic dependency propagation graph is constructed. In the semantic dependency propagation graph, a dependency connection is established for each pair of nodes with semantic association, and each dependency connection is labeled with at least one constraint type among co-occurrence constraint, spatial compatibility constraint, and attribute mutual exclusion constraint. For each node and edge in the scene graph structure, the dependency connection and its constraint type associated with the node or edge are found in the semantic dependency propagation graph. Multi-hop constraint propagation is performed along the dependency connection to obtain the constraint propagation result of the node or edge. Based on the constraint propagation results, the scene graph structure is disambiguated. For nodes with unclear entity referents, the referent is determined according to the co-occurrence constraint. For edges with conflicting relationships, the conflict is resolved according to the spatial compatibility constraint. For nodes with ambiguous attributes, the attribute values ​​are clarified according to the attribute mutual exclusion constraint. The disambiguated scene graph structure is then generated.

[0023] Receive scene description data containing semantic structure, such as "a man wearing a red coat is standing next to a blue car, and a woman holding a mobile phone is sitting in a coffee shop seat". From this description, extract scene entities ("man", "car", "woman", "mobile phone", "coffee shop", "seat"), spatial relationships ("standing next to", "sitting in", "holding"), and attribute features ("red coat", "blue") through semantic analysis.

[0024] For each extracted scene entity, create a corresponding node. For example, create node N1 to represent "man" and associate it with the attribute "wearing a red coat"; create node N2 to represent "car" and associate it with the attribute "blue"; and so on, create nodes N3 ("woman"), N4 ("phone"), N5 ("coffee shop"), and N6 ("seat"). For each spatial relationship, create an edge connecting the corresponding node. For example, create edge E1 connecting N1 and N2 to represent the relationship of "standing next to..."; create edge E2 connecting N3 and N6 to represent the relationship of "sitting on..."; and create edge E3 connecting N3 and N4 to represent the relationship of "holding". Through the above steps, the initial scene graph structure G is constructed.

[0025] Based on the initial scene graph structure G, a semantic dependency propagation graph D is constructed. In the semantic dependency propagation graph D, dependency connections are established for nodes that are semantically related, and the constraint types are labeled. Specifically, for entities that frequently co-occur, co-occurrence constraints are established; for entities with spatial relationships, spatial compatibility constraints are established; and for entities with mutually exclusive attributes, attribute mutual exclusion constraints are established.

[0026] For example, analyzing historical data reveals that "coffee shop" and "seat" frequently co-occur. Therefore, a dependency connection DC1 is established between nodes N5 and N6 in D, and labeled as a co-occurrence constraint. It is determined that there is no conflict between the spatial relationships "standing next to" and "sitting on" in D. Therefore, a dependency connection DC2 is established between edges E1 and E2 in D, and labeled as a spatial compatibility constraint. It is found that "red" and "blue" are mutually exclusive attribute values. Therefore, a dependency connection DC3 is established between the "red" attribute of node N1 and the "blue" attribute of node N2 in D, and labeled as an attribute mutual exclusion constraint.

[0027] For each node and edge in the scene graph G, multi-hop constraint propagation is performed in the semantic dependency propagation graph D. Starting from the target node or edge, a breadth-first or depth-first search is performed along the dependency connections in D to collect all constraint information related to the node or edge. The maximum number of propagation hops is set to 3, that is, constraint information is allowed to be transmitted through a maximum of 3 dependency connections.

[0028] Taking node N3 ("woman") as an example, starting from N3, we search for its related dependency connections in D. Through one-hop propagation, we find the relationship between N3 and N4, "holding a mobile phone"; through two-hop propagation, we find the relationship between N3 and N6, "sitting in a seat," and the co-occurrence constraint between N6 and N5; through three-hop propagation, we find other constraints related to N5 ("coffee shop"). Summarizing these constraint information, we obtain the constraint propagation result R3 for node N3.

[0029] Based on the results of constraint propagation, disambiguation processing is performed on the scene graph G. For nodes with unclear entity referents, co-occurrence constraints are used to determine the referent. For example, if "she" appears in the original description, the constraint information in R3 ("holding a mobile phone", "sitting in a seat") determines that "she" refers to "woman" rather than other entities.

[0030] For edges with conflicting relationships, the conflict is resolved based on spatial compatibility constraints. For example, if the description contains two relationships, "the man is standing next to the car" and "the man is standing on top of the car", and the spatial compatibility constraint indicates that these two relationships cannot be true at the same time, then the more reasonable relationship "the man is standing next to the car" is selected based on the context information.

[0031] For nodes with ambiguous attributes, the attribute values ​​are determined based on attribute mutual exclusion constraints. For example, if the original description only mentions "colored car" without specifying the color, the specific color of the car is determined to be "blue" based on the context and attribute mutual exclusion constraints (such as red and blue being mutually exclusive).

[0032] Through the above disambiguation process, a disambiguated scene graph structure G' is generated. In G', each node has a clear entity reference, each edge represents a non-conflicting spatial relationship, and each node's attributes have clear values. For example, G' explicitly indicates "a man in a red coat standing next to a blue car" and "a woman holding a mobile phone sitting in a coffee shop," eliminating the problems of ambiguous references, conflicting relationships, and vague attributes.

[0033] This scene graph disambiguation method can effectively handle semantic ambiguity in complex scenes, improve the accuracy of scene understanding, and provide reliable semantic representations for downstream tasks such as scene reasoning, visual question answering, and automatic navigation.

[0034] In one optional implementation, mapping the disambiguated scene graph structure to the latent feature space of the image generator to establish the correspondence between semantic structure and visual features includes: Based on the disambiguated scene graph structure, semantic vector encoding is performed on each node to extract the scene entity and attribute features corresponding to the node, relation vector encoding is performed on each edge to extract the spatial relation semantics corresponding to the edge, and a scene graph semantic vector set is constructed. Obtain the feature dimension structure and spatial distribution characteristics of the latent feature space of the image generator, and construct a cross-modal mapping network from the semantic vector space to the latent feature space; The semantic vector set of the scene graph is input into the cross-modal mapping network. The semantic vector of each node is mapped to the corresponding entity latent feature vector through the cross-modal mapping network, and the relation vector of each edge is mapped to the corresponding spatial layout feature vector. Based on the entity latent feature vector and the spatial layout feature vector, feature position coordinates and feature space range are assigned to each scene entity in the latent feature space, feature constraints are assigned to each spatial relationship, and the mapping position relationship between scene entities and spatial relationships in the latent feature space is established, thus completing the establishment of the correspondence between semantic structure and visual features.

[0035] like Figure 2 As shown, the method includes: Semantic vector encoding is performed based on the disambiguated scene graph structure. For nodes, a 512-dimensional semantic embedding vector is used to represent scene entities and attribute features. For example, for the node "red sofa," the "sofa" entity is encoded as a 256-dimensional vector, the "red" attribute is encoded as a 128-dimensional vector, and then combined with 128-dimensional position and size encodings to form a complete node semantic vector. For edges, a 256-dimensional vector is used to encode spatial relationship semantics. For example, for the relationship edge "sofa - located in - center of room," the "located in" relationship is encoded as a 128-dimensional vector, the "center of room" position feature is encoded as a 128-dimensional vector, and then combined to form the edge semantic vector. The semantic vector sets of nodes and edges together constitute the scene graph semantic vector set. Each scene graph typically contains vector representations of 5-20 nodes and 8-30 edges.

[0036] Feature space analysis was performed on the pre-trained image generator. Through reverse engineering, a latent feature space of 1024 dimensions was determined, where the first 512 dimensions primarily control content features, and the latter 512 dimensions primarily control style features. The distribution of semantic regions in the latent feature space was identified; for example, dimensions 60-120 were found to be related to object shape, dimensions 121-180 to texture, and dimensions 320-380 to color. Based on these analyses, a cross-modal mapping network was constructed, containing six fully connected layers, each with 1024, 2048, 2048, 1024, 1024, and 1024 neurons respectively. The GELU activation function was used, and a residual connection was added between layers 3 and 5. This network receives scene graph semantic vectors as input and outputs a mapping vector of the latent feature space.

[0037] The semantic vector set of the scene graph is input into a cross-modal mapping network, which processes the semantic vectors of nodes and edges separately. For node semantic vectors, such as the 512-dimensional vector of "red sofa," the mapping network transforms it into a 1024-dimensional entity latent feature vector. The first 512 dimensions represent content features such as the shape and material of the sofa, while the last 512 dimensions represent visual style features such as the color red. For edge relation vectors, such as the 256-dimensional vector of "sofa - located - in the center of the room," a dedicated relation mapping sub-network transforms it into a 512-dimensional spatial layout feature vector, which encodes positional constraints and spatial relationship information. For a complex scene graph containing 15 nodes and 25 edges, 15 entity latent feature vectors and 25 spatial layout feature vectors are generated.

[0038] Based on the latent feature vectors and spatial layout feature vectors of entities, feature positions and ranges are assigned to each scene entity in the latent feature space. A feature space grid is constructed, dividing the 1024-dimensional space into 16×16×4 logical blocks. Each scene entity is assigned to a specific block according to its semantic characteristics. For example, the "red sofa" is assigned to the position at coordinates (5, 8, 2), with a feature range of 3×2×1, occupying approximately 0.23% of the total feature space volume. The spatial relationship "sofa - located in - center of the room" is transformed into feature constraints, requiring that the Manhattan distance between the feature center point of the sofa and the center point of the feature space does not exceed 20% of the total spatial size. The spatial relationships between adjacent entities are transformed into feature flow fields through feature interpolation algorithms, guiding a smooth transition between features. Finally, a complete feature position mapping table is constructed, recording the feature coordinates, range, and constraints of each scene entity, forming a correspondence between semantic structure and visual features. This mapping relationship allows the image generator to precisely control the visual representation and spatial layout of each entity in the scene when generating images, ensuring that the generated images conform to the semantic structure described by the scene graph.

[0039] Through the above technical steps, a high-precision mapping from the semantic structure of the scene graph to the latent feature space of the image generator was achieved, establishing a direct correspondence between semantic and visual features, and providing a technical foundation for subsequent accurate image generation.

[0040] In one optional implementation, based on the correspondence, an overall spatial configuration is generated according to the spatial relationships in the disambiguated scene graph structure, and entity visual features are generated based on the overall spatial configuration and attribute features in the disambiguated scene graph structure, including: The spatial layout feature vectors in the correspondence are analyzed to obtain the relative positional relationships and spatial constraints between scene entities. Based on the relative positional relationships and spatial constraints, a spatial region is allocated to each scene entity in the image space and a corresponding semantic placeholder is generated. The semantic placeholder records the scene entity identity information and spatial boundary information corresponding to the spatial region, resulting in the overall spatial configuration containing multiple spatial regions and corresponding semantic placeholders. Extract the entity latent feature vector from the correspondence and extract the corresponding attribute features from the disambiguated scene graph structure to construct the mapping relationship between scene entities and entity latent feature vectors and attribute features; Obtain the semantic placeholder identifier for each spatial region, query the mapping relationship based on the scene entity identity information in the semantic placeholder identifier to obtain the corresponding entity potential feature vector and attribute features, and generate the entity visual features of the scene entity within the spatial boundary information of the spatial region based on the entity potential feature vector and attribute features.

[0041] The spatial layout feature vectors in the correspondence are analyzed to extract the relative positional relationships and spatial constraints between scene entities. These feature vectors are typically stored as high-dimensional vectors, such as 512-dimensional vectors, containing information such as relative distance, direction, and size ratio between entities. For example, for the relationship "table-above-vase," the spatial layout feature vector would contain encoded information indicating that the vase should be located above the table. A pre-trained decoder then converts these feature vectors into standardized positional relationship descriptors, such as "located above and in contact" or "located on the left and separated."

[0042] Spatial constraints include physical constraints (such as gravity constraints and support constraints) and scene consistency constraints. For example, the constraints obtained from the analysis indicate that "the vase must be supported by the tabletop" and "the chair cannot be suspended in the air." These constraints are represented as a set of rules to guide the subsequent spatial area allocation process.

[0043] Based on the relative positional relationships and spatial constraints obtained from the analysis, a specific spatial region is assigned to each scene entity in a standardized image space (such as a 1024×1024 pixel canvas). The spatial region assignment adopts a top-down approach, first determining the position of the main entities (such as rooms, large furniture), and then determining the position of the secondary entities. For each assigned spatial region, a corresponding semantic placeholder is generated. This placeholder is a data structure containing: a unique entity identifier (such as "entity_001"), an entity category (such as "table", "vase"), spatial boundary information (represented by bounding box coordinates, such as [x_min, y_min, x_max, y_max] = [256, 348, 768, 512]), depth information (used to handle occlusion relationships), and entity orientation information.

[0044] In this way, an overall spatial configuration containing multiple spatial regions and corresponding semantic placeholders is generated. This configuration can be regarded as the "skeleton" of the scene, providing a spatial framework for subsequent visual feature generation.

[0045] The system extracts latent feature vectors from the correspondences. These feature vectors are typically high-dimensional vectors, such as 256-dimensional vectors, encoding the basic visual appearance features of the entities, such as shape, texture, and material. Simultaneously, corresponding attribute features, such as color, material, and style, are extracted from the disambiguated scene graph structure. For example, for the entity "red leather sofa," the extracted attribute features would include key-value pairs such as "color: red" and "material: leather."

[0046] This information is organized into a mapping table, which maps each scene entity to its corresponding entity latent feature vector and attribute features. For example, the entity "Sofa_001" is mapped to a 256-dimensional feature vector and attribute set {"Color":"Red", "Material":"Leather", "Style":"Modern"}.

[0047] With the overall spatial configuration and entity mapping relationship established, we begin to generate specific visual features for each scene entity, obtain semantic placeholder identifiers for each spatial region, query the mapping relationship through entity identity information (such as "chair_002"), and obtain the corresponding entity potential feature vectors and attribute features.

[0048] Using a feature generation network, the entity's latent feature vector and attribute features are taken as conditional inputs to generate visual features of the entity within the spatial boundaries specified by the semantic placeholder. This process employs conditional generation techniques to ensure that the generated visual features conform to attribute constraints. For example, for "red leather sofa," visual features with a distinct red hue and leather texture are generated. The generated visual features are typically represented as feature maps (such as tensors of size 64×64×512), encoding the detailed appearance features of the entity within the specified spatial region.

[0049] In practical applications, if a scene contains three entities: a "wooden table," a "red vase," and a "green plant," the process begins by allocating spatial regions on a standard 1024×1024 canvas based on their relative positions (e.g., "vase on the table," "plant beside the table") and spatial constraints. These regions are designated as follows: table region [200, 300, 800, 500], vase region [450, 200, 550, 300], and plant region [820, 350, 950, 600]. Semantic placeholders are generated for each region, recording entity and spatial information. Potential feature vectors and attribute features (e.g., the "material: wood" of the wooden table, the "color: red" of the vase) are extracted from the correspondence and scene graph. A mapping relationship is established, and based on this information, visual features conforming to these attribute features are generated within their respective spatial regions. Examples include a wood-grain textured tabletop, a red vase, and a green plant. This completes the visual feature generation process for the entire scene.

[0050] In one optional implementation, the overall spatial configuration and the entity visual features are spatially combined and fused at the pixel level to obtain candidate images, including: Extract spatial boundary information and semantic placeholders for each spatial region from the overall spatial configuration, determine the scene entity corresponding to the spatial region based on the semantic placeholders, obtain the entity visual features of the scene entity, and establish a spatial mapping relationship from the spatial region to the entity visual features. Based on the spatial mapping relationship, each entity visual feature is placed at the target position in the image space according to the spatial boundary information, keeping the spatial position of the entity visual feature consistent with the position of the spatial region, to obtain the spatial combination result; Boundary fusion is performed on the boundary regions between adjacent spatial regions to eliminate pixel discontinuities between visual features of adjacent entities and generate a boundary fusion region; pixel values ​​of multiple entity visual features in the overlapping region of the spatial combination result are extracted; based on the spatial relationship describing the occlusion relationship between scene entities in the overlapping region in the disambiguated scene graph structure, pixel depth priority is determined; and pixel values ​​in the overlapping region are hierarchically fused according to the pixel depth priority to generate an overlapping fusion region. The candidate image is obtained by embedding the boundary fusion region and the overlapping fusion region into the spatial combination result.

[0051] Spatial boundary information and semantic placeholders for each spatial region are extracted from the overall spatial configuration. Spatial boundary information includes the boundary coordinates of the spatial region on a two-dimensional plane. For example, the boundary coordinates of region A are [(10, 10), (10, 110), (110, 110), (110, 10)], representing a rectangular region. Semantic placeholders are used to mark the scene entity type corresponding to the spatial region, such as "table," "chair," or "window." Based on the semantic placeholders, the scene entities corresponding to the spatial regions are determined. For example, when the semantic placeholder is "chair," entities of the "chair" category are searched in the scene entity library to obtain the entity's visual features, including color, texture, and shape. These features are typically stored in pixel matrix form, such as a 512×512 RGB image. A spatial mapping relationship between spatial regions and entity visual features is established, recording the entity visual features corresponding to each spatial region and their placement positions.

[0052] Based on the established spatial mapping relationship, each entity visual feature is placed at its target position in the image space according to its spatial boundary information. Specifically, a blank canvas of the same size as the final target image is created, for example, 1024×768 pixels. For each spatial region, the position and size of the entity visual feature on the canvas are calculated based on its boundary information. If the original size of the entity visual feature is inconsistent with the target spatial region, its size is adjusted to ensure that the visual feature completely covers the target region. For example, for spatial region A, the corresponding "chair" visual feature is adjusted to a size consistent with the boundary of region A and placed at the position of region A. In this way, the spatial position of the entity visual feature is kept consistent with the spatial region position, resulting in a spatial combination result.

[0053] To eliminate pixel discontinuities between visual features of adjacent entities, boundary fusion is performed on the boundary regions between adjacent spatial areas. Adjacent spatial regions are identified; for example, regions A and B share a boundary line. A 5-pixel-wide strip is taken on each side of this shared boundary to form a boundary fusion region. Within this region, pixel interpolation techniques are used to smoothly transition the pixel values ​​of the visual features of the entities on both sides. For example, a weighted averaging method can be used, where the pixel weights on both sides of the boundary decrease with increasing distance from the boundary. For a point 2 pixels from the boundary, its pixel value can be a weighted average of the pixel values ​​of the visual features of the entities on both sides in a ratio of 0.7:0.3. In this way, abrupt changes at the boundary are visually eliminated, generating a smooth boundary fusion region.

[0054] For overlapping regions in the spatial composition results, it is necessary to handle pixel value conflicts of multiple entity visual features and extract the pixel values ​​of each entity's visual features within the overlapping region. For example, in the overlapping region of region C and region D, the visual features of "table" and "book" coexist. Based on the spatial relationships described in the disambiguated scene graph structure, pixel depth priority is determined. If the scene graph indicates "book is on table," then the pixel depth priority of "book" is higher than that of "table." Pixel values ​​within the overlapping region are then hierarchically fused according to pixel depth priority, assigning a depth value to each entity. Entities with smaller depth values ​​(such as the foreground object "book") will cover entities with larger depth values ​​(such as the background object "table"). In some cases, semi-transparent fusion can be performed based on the transparency attributes of the entities. For example, for transparent objects like "glass window," their pixel values ​​can be mixed with the pixel values ​​of the objects behind them in a 7:3 ratio. Through this hierarchical fusion method, visually reasonable overlapping fused regions are generated.

[0055] By embedding the boundary fusion region and the overlapping fusion region into the spatial combination result, the final candidate image is obtained. In specific implementation, the pixels of the corresponding boundary region and overlapping region in the original spatial combination result are replaced with the pixel values ​​after fusion processing. For example, for the boundary fusion region, the original pixel values ​​of 5 pixels on each side of the boundary are replaced with the pixel values ​​after fusion; for the overlapping fusion region, the original pixel values ​​of the overlapping part are replaced with the pixel values ​​after hierarchical fusion. Through this pixel-level replacement operation, a visually coherent candidate image with a reasonable spatial structure is generated. This image accurately reflects the scene structure described by the overall spatial configuration, while preserving the details of the visual features of each entity, and achieving a natural transition between the boundaries and overlapping regions of entities.

[0056] In one optional implementation, the candidate image is subjected to scene structure consistency verification. Entity layout and attribute representation in the candidate image are extracted and compared with the disambiguated scene graph structure to generate structural deviation metric information, including: Identify visual entities in the candidate images, extract the position coordinates and size information of each visual entity in the image space, and construct an entity layout representation of the candidate images; extract visual attributes for each visual entity, obtain shape features, color features and texture features, and construct an attribute representation description of the candidate images; Spatial relationships and attribute features are extracted from the disambiguated scene graph structure. The spatial relationships are converted into reference position constraints and reference size constraints, and the attribute features are converted into reference shape constraints, reference color constraints, and reference texture constraints. Based on the entity layout representation and visual entity, the positional deviation between the position coordinates and the reference position constraint of the corresponding scene entity is calculated, and the size deviation between the size information and the reference size constraint of the corresponding scene entity is calculated to obtain the layout deviation metric; based on the attribute representation description and visual entity, the shape deviation between the shape feature and the reference shape constraint is calculated, the color deviation between the color feature and the reference color constraint is calculated, and the texture deviation between the texture feature and the reference texture constraint is calculated to obtain the attribute deviation metric. The layout deviation metric and the attribute deviation metric are fused to obtain the structural deviation metric information.

[0057] Visual entities in candidate images are identified using computer vision algorithms. For each identified visual entity, its position coordinates and size information in image space are extracted. The position coordinates are typically represented as two-dimensional coordinates (x, y), where x and y represent the pixel positions of the visual entity's center point in the horizontal and vertical directions of the image, respectively. The size information includes the width w and height h of the visual entity. For example, for an image containing "a red apple on a blue plate," the position coordinates of the apple entity are extracted as (320, 240), and its size is (50, 50) pixels; the position coordinates of the plate entity are extracted as (320, 300), and its size is (150, 20) pixels. This position and size information constitutes the entity layout representation of the candidate image.

[0058] Visual attributes are extracted for each visual entity. Shape features are obtained through edge detection and contour analysis, and can be represented as a set of shape descriptors, such as the textual expression of roundness, convexity, or Fourier descriptors. Color features are extracted by calculating the color histogram or dominant hue within the entity region, and can be represented as the color distribution in RGB or HSV space. Texture features are extracted using methods such as local binary mode or gray-level co-occurrence matrix, and can be represented as texture roughness, regularity, etc. Taking the apple entity in the previous example as an example, its shape features are represented as "round, roundness value of 0.92"; its color features are "dominant hue is red, RGB values ​​are concentrated in the range of (200-255, 0-50, 0-50)"; and its texture features are "smooth surface, high texture regularity". These attribute information constitute the attribute representation description of the candidate image.

[0059] Spatial relationships and attribute features are extracted from the disambiguated scene graph structure. Spatial relationships such as "above" and "next to" are converted into reference position constraints and reference size constraints. For example, "apple above plate" can be converted into the position constraint "the y-coordinate of the apple should be less than the y-coordinate of the plate" and the size constraint "the size of the apple should be less than the size of the plate". Attribute features such as "red" and "circular" are converted into corresponding reference shape constraints, reference color constraints, and reference texture constraints. For example, "red apple" is converted into the color constraint "RGB values ​​should be within the red range, mainly (200-255, 0-50, 0-50)".

[0060] Based on entity layout representation and visual entities, the positional deviation between the position coordinates and the reference position constraints of the corresponding scene entities is calculated. This can be used to calculate the distance between the actual coordinates and the desired coordinate region, or to check whether spatial relationships are satisfied. For example, if the scene diagram specifies "the apple should be on top of the plate," it checks whether the y-coordinate of the apple entity is less than the y-coordinate of the plate entity. If not, the positional deviation is increased. Similarly, the dimensional deviation between the size information and the reference size constraints is calculated, for example, to check whether the constraint "the apple should be smaller than the plate" is satisfied. These two deviations together constitute the layout deviation metric.

[0061] Based on attribute representation descriptions and visual entities, the shape deviation between shape features and reference shape constraints is calculated. For example, for a "round apple," the extracted roundness value is compared to the ideal circle. The color deviation between color features and reference color constraints is calculated, for example, comparing the extracted RGB color distribution to a specified red range. The texture deviation between texture features and reference texture constraints is calculated, for example, comparing the extracted surface smoothness to the desired smooth texture. These three deviations together constitute the attribute deviation metric.

[0062] In practical applications, layout deviation can be represented by a weighted sum of position deviation and size deviation. For example, if the actual position of the apple differs from the expected position by 30 pixels, and the actual size of the plate differs from the expected size by 20%, then the layout deviation is 0.6×30+0.4×20=22. Attribute deviation can also be represented by a weighted sum of shape deviation, color deviation, and texture deviation. For example, if the roundness deviation of the apple is 0.15, the color deviation is 0.25, and the texture deviation is 0.10, then the attribute deviation is 0.3×0.15+0.5×0.25+0.2×0.10=0.19.

[0063] By fusing layout deviation metrics and attribute deviation metrics, structural deviation metrics are obtained. The fusion method can be a simple weighting, such as 0.5 × layout deviation metric + 0.5 × attribute deviation metric, or a more complex fusion strategy can be adopted, such as considering the user's sensitivity to different types of deviations. This structural deviation metrics provide a quantitative basis for the system to evaluate the degree of matching between candidate images and the desired scene structure, thereby assisting the system in selecting or generating image content that better meets the user's expectations.

[0064] In one optional implementation, the process of regenerating the deviation regions in the candidate image based on the structural deviation metric information, and then fusing the regenerated results with the consistent regions in the candidate image to generate an intelligent scene image includes: The structural deviation measurement information is compared with a preset deviation judgment threshold to identify deviation entities and consistency entities; in the candidate image, the image area occupied by the deviation entity is marked as the deviation area, and the image area occupied by the consistency entity is marked as the consistency area. For each deviation region, semantic description information and spatial relationship information of the deviation entity corresponding to the deviation region are extracted from the disambiguated scene graph structure. The semantic description information is converted into regenerated semantic guidance features, and the spatial relationship information is converted into regenerated spatial constraint features. Based on the regenerated semantic guidance features and the regenerated spatial constraint features, the deviation region is regenerated in a targeted manner to generate regenerated region content that meets the requirements of the disambiguated scene graph structure. Replace the corresponding deviation regions in the candidate image with the regenerated region content of all deviation regions, while keeping the consistent regions in the candidate image unchanged, to obtain the image after region replacement; perform boundary smoothing processing on the boundary between the regenerated region content and the consistent region content in the image after region replacement to eliminate pixel discontinuities, and generate the intelligent scene image.

[0065] The structural deviation metric is compared with a preset deviation threshold to identify deviating and consistent entities. For example, if the structural deviation metric of an entity exceeds the preset threshold of 0.7, it is identified as a deviating entity; otherwise, it is identified as a consistent entity. In a practical case, if the structural deviation metric of the "dog" entity is 0.85, while the structural deviation metric of the "tree" entity is 0.45, and the preset threshold is 0.7, then "dog" is identified as a deviating entity, and "tree" is identified as a consistent entity.

[0066] In the candidate image, the regions occupied by each entity are identified. By using image segmentation techniques and entity location information, the pixel region occupied by each entity can be accurately identified in the candidate image. Continuing the above example, the pixel region occupied by the "dog" entity is marked as the deviation region, and the pixel region occupied by the "tree" entity is marked as the consistency region. Specifically, a binary mask image is generated, where the pixel values ​​of the deviation region are set to 1, and the pixel values ​​of the consistency region are set to 0.

[0067] For each identified deviation region, semantic description information and spatial relationship information of the deviation entity are extracted from the disambiguated scene graph structure. Semantic description information includes the entity's category, attributes, state, etc., such as "a brown puppy"; spatial relationship information includes the relative positional relationship between the entity and other entities, such as "the dog is on the left side of the tree" and "the dog is standing on the grass".

[0068] The extracted semantic description information is converted into regenerated semantic guidance features, and the text description is converted into a feature vector that can guide the image generation model. A pre-trained text encoder is used to encode the semantic description text into a high-dimensional feature vector. For example, for the description "a brown puppy", a 512-dimensional feature vector is generated, which contains semantic information about the dog's appearance, color, size and other semantic information.

[0069] Simultaneously, spatial relationship information is converted into regenerated spatial constraint features, and a spatial constraint graph is constructed based on the spatial relationships. This graph is used to guide the position and layout of the regenerated content. For example, for the relationship "the dog is on the left side of the tree," corresponding spatial constraint features are generated to ensure that the regenerated dog is indeed located on the left side of the tree. These spatial constraint features are usually represented in the form of coordinates, bounding boxes, or spatial heatmaps.

[0070] Based on the aforementioned semantic guidance features and spatial constraint features for regeneration, the biased region is regenerated in a targeted manner. A conditional generation model is used, taking the biased region mask, semantic guidance features, and spatial constraint features as conditional inputs to generate new region content that meets the requirements. In practice, generation techniques such as diffusion models can be used, and high-quality image content can be generated through multi-step iterative optimization. For example, for the "dog" region that needs to be regenerated, image content of a dog that meets these conditions is generated based on the semantic description of "a brown puppy" and the spatial relationship of "the dog is on the left side of the tree."

[0071] After regenerating all the deviation regions, the content of the regenerated regions replaces the corresponding deviation regions in the candidate image, while keeping the consistent regions in the candidate image unchanged, resulting in the image with the replaced regions. This step is achieved through image synthesis technology, specifically by using the previously generated binary mask image to guide the pixel-level replacement operation.

[0072] The boundary between the regenerated and consistent regions in the replaced image is smoothed using a feathering blending technique. This creates a transition region at the boundary, allowing the regenerated and original content to blend naturally. Specifically, a transition band with a width of 10-20 pixels can be defined around the boundary. Within this region, the original and regenerated images are weighted and blended, with the weights gradually changing from the boundary outwards. For example, near the boundary point inside the regenerated region, the weight of the regenerated content is 0.9, and the weight of the original content is 0.1; while near the boundary point of the consistent region, the weight of the regenerated content is 0.1, and the weight of the original content is 0.9. Through this smooth transition, the final generated smart scene image is visually more coherent and natural, with no obvious stitching marks.

[0073] The above describes in detail the complete technical implementation process of regenerating the deviation region in the candidate image based on structural deviation measurement information and fusing it with the consistent region to generate an intelligent scene image. This method can accurately correct the parts of the candidate image that are inconsistent with the structure of the disambiguated scene image, while retaining the regions that meet the requirements, and finally generating an intelligent scene image that is both consistent with the user's intention and visually coherent.

[0074] This invention relates to a natural language-based intelligent scene image generation and control system, the system comprising: The first unit is used to perform semantic parsing of natural language text and extract the semantic structure of the scene; The second unit is used to construct a scene graph structure based on the scene semantic structure, and to resolve the nodes and edges of the scene graph structure by constructing a semantic consistency constraint network to generate a disambiguated scene graph structure. The third unit is used to map the disambiguated scene graph structure to the latent feature space of the image generator, and establish the correspondence between semantic structure and visual features; The fourth unit is used to generate an overall spatial configuration based on the correspondence and the spatial relationship in the disambiguated scene graph structure, generate entity visual features based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, and perform spatial combination and pixel-level fusion of the overall spatial configuration and the entity visual features to obtain candidate images. The fifth unit is used to perform scene structure consistency verification on the candidate image, extract the entity layout and attribute representation in the candidate image, compare it with the disambiguated scene graph structure, generate structural deviation measurement information, and perform directional regeneration of the deviation region in the candidate image based on the structural deviation measurement information. The regeneration result is then fused with the consistent region in the candidate image to generate an intelligent scene image.

[0075] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0076] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0077] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural language-based intelligent scene image generation and control method, characterized in that, include: Perform semantic analysis on natural language text to extract the semantic structure of the scene; A scene graph structure is constructed based on the scene semantic structure. A semantic consistency constraint network is constructed to resolve the nodes and edges of the scene graph structure, generating a disambiguated scene graph structure. The disambiguated scene graph structure is mapped to the latent feature space of the image generator to establish the correspondence between semantic structure and visual features; Based on the correspondence, an overall spatial configuration is generated according to the spatial relationship in the disambiguated scene graph structure. Based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, entity visual features are generated. The overall spatial configuration and the entity visual features are spatially combined and pixel-level fused to obtain candidate images. The candidate image is subjected to scene structure consistency verification. The entity layout and attribute representation in the candidate image are extracted and compared with the disambiguated scene graph structure to generate structural deviation measurement information. Based on the structural deviation measurement information, the deviation region in the candidate image is regenerated in a targeted manner. The regenerated result is fused with the consistent region in the candidate image to generate an intelligent scene image.

2. The method according to claim 1, characterized in that, Based on the aforementioned scene semantic structure, a scene graph structure is constructed. A semantic consistency constraint network is then used to resolve the nodes and edges of the scene graph structure, generating a disambiguated scene graph structure, including: The scene entities, spatial relationships, and attribute features are extracted from the scene semantic structure. A node is created for each scene entity and associated with the corresponding attribute feature. An edge is created for each spatial relationship to connect the corresponding node, thus obtaining the scene graph structure. Based on the scene graph structure, a cross-entity semantic dependency propagation graph is constructed. In the semantic dependency propagation graph, a dependency connection is established for each pair of nodes with semantic association, and each dependency connection is labeled with at least one constraint type among co-occurrence constraint, spatial compatibility constraint, and attribute mutual exclusion constraint. For each node and edge in the scene graph structure, the dependency connection and its constraint type associated with the node or edge are found in the semantic dependency propagation graph. Multi-hop constraint propagation is performed along the dependency connection to obtain the constraint propagation result of the node or edge. Based on the constraint propagation results, the scene graph structure is disambiguated. For nodes with unclear entity referents, the referent is determined according to the co-occurrence constraint. For edges with conflicting relationships, the conflict is resolved according to the spatial compatibility constraint. For nodes with ambiguous attributes, the attribute values ​​are clarified according to the attribute mutual exclusion constraint. The disambiguated scene graph structure is then generated.

3. The method according to claim 1, characterized in that, Mapping the disambiguated scene graph structure to the latent feature space of the image generator, and establishing the correspondence between semantic structure and visual features, includes: Based on the disambiguated scene graph structure, semantic vector encoding is performed on each node to extract the scene entity and attribute features corresponding to the node, relation vector encoding is performed on each edge to extract the spatial relation semantics corresponding to the edge, and a scene graph semantic vector set is constructed. Obtain the feature dimension structure and spatial distribution characteristics of the latent feature space of the image generator, and construct a cross-modal mapping network from the semantic vector space to the latent feature space; The semantic vector set of the scene graph is input into the cross-modal mapping network. The semantic vector of each node is mapped to the corresponding entity latent feature vector through the cross-modal mapping network, and the relation vector of each edge is mapped to the corresponding spatial layout feature vector. Based on the entity latent feature vector and the spatial layout feature vector, feature position coordinates and feature space range are assigned to each scene entity in the latent feature space, feature constraints are assigned to each spatial relationship, and the mapping position relationship between scene entities and spatial relationships in the latent feature space is established, thus completing the establishment of the correspondence between semantic structure and visual features.

4. The method according to claim 1, characterized in that, Based on the aforementioned correspondence, an overall spatial configuration is generated according to the spatial relationships in the disambiguated scene graph structure. Entity visual features are then generated based on the overall spatial configuration and the attribute features of the disambiguated scene graph structure, including: The spatial layout feature vectors in the correspondence are analyzed to obtain the relative positional relationships and spatial constraints between scene entities. Based on the relative positional relationships and spatial constraints, a spatial region is allocated to each scene entity in the image space and a corresponding semantic placeholder is generated. The semantic placeholder records the scene entity identity information and spatial boundary information corresponding to the spatial region, resulting in the overall spatial configuration containing multiple spatial regions and corresponding semantic placeholders. Extract the entity latent feature vector from the correspondence and extract the corresponding attribute features from the disambiguated scene graph structure to construct the mapping relationship between scene entities and entity latent feature vectors and attribute features; Obtain the semantic placeholder identifier for each spatial region, query the mapping relationship based on the scene entity identity information in the semantic placeholder identifier to obtain the corresponding entity potential feature vector and attribute features, and generate the entity visual features of the scene entity within the spatial boundary information of the spatial region based on the entity potential feature vector and attribute features.

5. The method according to claim 1, characterized in that, The overall spatial configuration and the entity visual features are spatially combined and fused at the pixel level to obtain candidate images, including: Extract spatial boundary information and semantic placeholders for each spatial region from the overall spatial configuration, determine the scene entity corresponding to the spatial region based on the semantic placeholders, obtain the entity visual features of the scene entity, and establish a spatial mapping relationship from the spatial region to the entity visual features. Based on the spatial mapping relationship, each entity visual feature is placed at the target position in the image space according to the spatial boundary information, keeping the spatial position of the entity visual feature consistent with the position of the spatial region, to obtain the spatial combination result; Boundary fusion is performed on the boundary regions between adjacent spatial regions to eliminate pixel discontinuities between visual features of adjacent entities and generate a boundary fusion region; pixel values ​​of multiple entity visual features in the overlapping region of the spatial combination result are extracted; based on the spatial relationship describing the occlusion relationship between scene entities in the overlapping region in the disambiguated scene graph structure, pixel depth priority is determined; and pixel values ​​in the overlapping region are hierarchically fused according to the pixel depth priority to generate an overlapping fusion region. The candidate image is obtained by embedding the boundary fusion region and the overlapping fusion region into the spatial combination result.

6. The method according to claim 1, characterized in that, The candidate images are subjected to scene structure consistency verification. Entity layout and attribute representation in the candidate images are extracted and compared with the structure of the disambiguated scene graph to generate structural deviation metric information, including: Identify visual entities in the candidate images, extract the position coordinates and size information of each visual entity in the image space, and construct an entity layout representation of the candidate images; extract visual attributes for each visual entity, obtain shape features, color features and texture features, and construct an attribute representation description of the candidate images; Spatial relationships and attribute features are extracted from the disambiguated scene graph structure. The spatial relationships are converted into reference position constraints and reference size constraints, and the attribute features are converted into reference shape constraints, reference color constraints, and reference texture constraints. Based on the entity layout representation and visual entity, the positional deviation between the position coordinates and the reference position constraint of the corresponding scene entity is calculated, and the size deviation between the size information and the reference size constraint of the corresponding scene entity is calculated to obtain the layout deviation metric; based on the attribute representation description and visual entity, the shape deviation between the shape feature and the reference shape constraint is calculated, the color deviation between the color feature and the reference color constraint is calculated, and the texture deviation between the texture feature and the reference texture constraint is calculated to obtain the attribute deviation metric. The layout deviation metric and the attribute deviation metric are fused to obtain the structural deviation metric information.

7. The method according to claim 1, characterized in that, Based on the structural deviation metric information, the deviation regions in the candidate image are regenerated in a targeted manner. The regenerated results are then fused with the consistent regions in the candidate image to generate an intelligent scene image, including: The structural deviation measurement information is compared with a preset deviation judgment threshold to identify deviation entities and consistency entities; in the candidate image, the image area occupied by the deviation entity is marked as the deviation area, and the image area occupied by the consistency entity is marked as the consistency area. For each deviation region, semantic description information and spatial relationship information of the deviation entity corresponding to the deviation region are extracted from the disambiguated scene graph structure. The semantic description information is converted into regenerated semantic guidance features, and the spatial relationship information is converted into regenerated spatial constraint features. Based on the regenerated semantic guidance features and the regenerated spatial constraint features, the deviation region is regenerated in a targeted manner to generate regenerated region content that meets the requirements of the disambiguated scene graph structure. Replace the corresponding deviation regions in the candidate image with the regenerated region content of all deviation regions, while keeping the consistent regions in the candidate image unchanged, to obtain the image after region replacement; perform boundary smoothing processing on the boundary between the regenerated region content and the consistent region content in the image after region replacement to eliminate pixel discontinuities, and generate the intelligent scene image.

8. A natural language-based intelligent scene image generation and control system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The first unit is used to perform semantic parsing of natural language text and extract the semantic structure of the scene; The second unit is used to construct a scene graph structure based on the scene semantic structure, and to resolve the nodes and edges of the scene graph structure by constructing a semantic consistency constraint network to generate a disambiguated scene graph structure. The third unit is used to map the disambiguated scene graph structure to the latent feature space of the image generator, and establish the correspondence between semantic structure and visual features; The fourth unit is used to generate an overall spatial configuration based on the correspondence and the spatial relationship in the disambiguated scene graph structure, generate entity visual features based on the overall spatial configuration and the attribute features in the disambiguated scene graph structure, and perform spatial combination and pixel-level fusion of the overall spatial configuration and the entity visual features to obtain candidate images. The fifth unit is used to perform scene structure consistency verification on the candidate image, extract the entity layout and attribute representation in the candidate image, compare it with the disambiguated scene graph structure, generate structural deviation measurement information, and perform directional regeneration of the deviation region in the candidate image based on the structural deviation measurement information. The regeneration result is then fused with the consistent region in the candidate image to generate an intelligent scene image.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images