Image generation method and device based on cross-modal model and spatial topology

Abstract

The application discloses an image generation method based on a cross-modal model and spatial topology, relates to the technical field of artificial intelligence, and particularly relates to an image generation method and device based on a cross-modal model and spatial topology; the method comprises the following steps: obtaining a candidate design scheme text and a real scene base map of a target land plot, inputting a first cross-modal visual understanding model to form a design point array, and inputting a second cross-modal visual understanding model to generate a basic rendering prompt word; extracting, normalizing and deduplicating a user negative description, and translating the user negative description into a negative constraint; calculating an original width-height ratio of the real scene base map, determining a target output ratio closest to the original width-height ratio from a ratio set supported by a visual generation model, and constructing a combined rendering instruction and generating a rendering image according to the target output ratio. The method also updates the negative constraint or the basic rendering prompt word through a stateful multi-round conversation, generates a local redrawing constraint when carrying a mask image, and generates an edited rendering image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image generation method and apparatus based on cross-modal models and spatial topology. Background Technology

[0002] In the fields of urban and rural planning, urban renewal, and landscape design, the rapid and intuitive transformation of textual conceptual design schemes into visual renderings is crucial for communication, public participation, and decision support. Traditional rendering production relies on professional designers manually creating renderings using computer-aided design (CAD) software, 3D modeling, and rendering tools. This process is time-consuming, labor-intensive, costly, and difficult to respond quickly to multiple rounds of revisions.

[0003] With the development of artificial intelligence technology, using AI models to transform text-based design schemes into realistic visual renderings can greatly improve the efficiency of design expression and scheme communication. However, in practical application scenarios with strong constraints and high interactivity, such as urban renewal and landscape design, existing scheme generation systems based on text-to-image models face the following core challenges: High barriers to human-computer interaction and complex prompting technology: High-quality image generation heavily relies on precise and professional rendering prompts. Existing systems typically require users to directly input or for the system to simply concatenate prompts, demanding that users translate design intent into model jargon. For ordinary planners or public participants, it is difficult to drive the model to generate the desired effect using conversational natural language instructions (such as "It's too dark here, add some warm light"), resulting in significant barriers to human-computer interaction and limiting the technology's widespread adoption and application depth.

[0004] The mismatch between editing intent and constraint strength leads to insufficient flexibility: When modifying generated renderings, users have diverse intentions, ranging from minor adjustments to local materials to exploring entirely new compositions or perspectives. Existing methods typically employ fixed, global, and strong constraint strategies (such as completely locking the background and terrain), failing to intelligently adapt to diverse user editing intents. Constraints may be insufficient for high-fidelity local modifications, while becoming too rigid for global creative exploration. The system lacks the ability to dynamically adjust constraint strength based on interaction context (such as whether a local mask is provided).

[0005] Therefore, there is an urgent need in this field for an intelligent image generation technology that can understand natural language interaction intentions, has multi-turn dialogue state management capabilities, and can adaptively adjust generation constraints, so as to achieve truly efficient, smooth, safe and reliable human-computer collaborative design iteration. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent image generation technology that can understand natural language interaction intentions, has multi-turn dialogue state management capabilities, and can adaptively adjust generation constraints.

[0007] The objective of this invention is achieved through the following technical solution: Therefore, according to one aspect of the present invention, an image generation method based on a cross-modal model and spatial topology is provided, comprising the following steps: obtaining candidate design scheme text and real-scene base map of a target plot; inputting the candidate design scheme text and real-scene base map into a first cross-modal visual understanding model, extracting feasible design elements from the real-scene base map, and performing structured processing on the design elements to form a design point array; wherein, the design point array includes object information, modification processing description information, and attribute information; inputting the candidate design scheme text and real-scene base map into a second cross-modal visual understanding model to generate basic rendering prompts; wherein, the basic rendering prompts include spatial anchor point descriptions and effect-oriented descriptions; obtaining negative descriptions input by the user, and normalizing and deduplicating the negative descriptions to translate them into negative constraints; constructing combined rendering instructions based on the basic rendering prompts, design point array, negative constraints, and target output ratio determined according to the original aspect ratio of the real-scene base map; calling a visual generation model, and generating a rendering image of the design scheme based on the combined rendering instructions and the real-scene base map.

[0008] Specifically, it also includes the following steps: calling the third cross-modal visual understanding model, performing inverse spatial mapping on the rendering image based on the design point array, and identifying the design facilities in the rendering image; extracting the spatial anchor points of the design facilities and the corresponding design element descriptions, and generating a normalized spatial coordinate map.

[0009] The above process involves inputting candidate design scheme text and real-scene base map into the second cross-modal visual understanding model to generate basic rendering prompts. Specifically, this includes the following steps: inputting candidate design scheme text and real-scene base map into the second cross-modal visual understanding model, generating a Chinese scene description based on the candidate design scheme text and real-scene base map; performing regular expression cleaning on the Chinese scene description to remove redundant content and generate basic rendering prompts; where redundant content includes thought chain prefixes, redundant quotation marks, and / or opening remarks.

[0010] Specifically, before constructing the combined rendering instructions based on the basic rendering prompts, design point arrays, negative constraints, and the target output ratio determined according to the original aspect ratio of the real-world base map, the following steps are also included: using a regular expression cleaning operator to perform terrain risk word cleaning on the basic rendering prompts, replacing the terrain risk words in the basic rendering prompts with the corresponding terrain safety words in the preset safety dictionary; parsing the pixel array of the real-world base map, calculating the original aspect ratio of the real-world base map, and determining the target output ratio closest to the original aspect ratio according to the set of ratios supported by the visual generation model; merging the negative constraints with the baseline negative prompt word library to generate negative semantic constraints; and constructing the combined rendering instructions based on the cleaned basic rendering prompts, design point arrays, negative semantic constraints, and target output ratio.

[0011] The above also includes the following steps: creating or invoking a stateful multi-turn session corresponding to the target plot, and generating or reading the corresponding session identifier; generating a session context record associated with the session identifier; if a negative description carrying the session identifier is received, then the session context record is read based on the session identifier, the negative description is extracted, normalized, and deduplicated, and the original language is retained or converted to the target language according to the input requirements of the visual generation model, generating negative constraints, and updating the negative constraints in the session context record based on the negative constraints; if an edit request carrying the session identifier is received, then the session context record is read based on the session identifier, and the basic rendering prompt words in the session context record are generated or updated.

[0012] More specifically, if an edit request carrying a session identifier is received, the session context record is read based on the session identifier, and basic rendering prompts in the session context record are generated or updated. This includes the following steps: obtaining the current rendering image corresponding to the edit request, and determining whether the edit request carries a mask image; if the edit request carries a mask image, the edit request, the current rendering image, and the mask image are input into the second cross-modal visual understanding model to generate corresponding basic rendering prompts for local editing, and the basic rendering prompts in the session context record are updated based on the basic rendering prompts for local editing.

[0013] More specifically, after updating the basic rendering prompts in the session context record based on the local editing of the basic rendering prompts, the following steps are also included: determining the area to be redrawn and the non-redrawn area based on the mask image; generating local redrawing constraints based on the area to be redrawn and the non-redrawn area; updating the combined rendering instructions based on the updated basic rendering prompts and local redrawing constraints, and calling the visual generation model to generate the edited rendering image.

[0014] According to another aspect of the present invention, an image generation apparatus based on a cross-modal model and spatial topology is provided, comprising: a design point array generation module, used to input candidate design scheme text and a real-scene base map into a first cross-modal visual understanding model, extract feasible design elements from the real-scene base map, and perform structured processing on the design elements to form a design point array; a prompt word generation module, used to input candidate design scheme text and a real-scene base map into a second cross-modal visual understanding model, and generate basic rendering prompt words; an interaction module, used to acquire negative descriptions and / or editing requests input by the user, and update negative constraints and / or basic rendering prompt words in the context record based on the negative descriptions and / or editing requests; and a rendering instruction generation module. The module is divided into three parts: a rendering module and a spatial coordinate map generation module. The former is based on the basic rendering prompts, design point array, negative constraints, and target output ratio determined by the original aspect ratio of the real-world base map, which constructs combined rendering instructions; the latter is used to call the visual generation model and generate a rendering of the design scheme based on the combined rendering instructions and the real-world base map; the latter is used to call the third cross-modal visual understanding model, perform inverse spatial mapping on the rendering based on the design point array, identify the design facilities in the rendering, and extract the spatial anchor points of the design facilities and the corresponding design element descriptions to generate a normalized spatial coordinate map.

[0015] According to another aspect of the present invention, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement the steps of the above-described image generation method based on a cross-modal model and spatial topology.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided that stores computer instructions which, when executed by a processor, implement the steps of the image generation method based on a cross-modal model and spatial topology as described above.

[0017] The beneficial effects of this invention are: 1. By using the first cross-modal visual understanding model, implementable design elements are extracted from candidate design scheme texts and real-world base maps, and a design point array is formed. This can eliminate invalid design content in invisible areas, occluded areas, or blind spots, ensuring that subsequent rendered objects maintain a correspondence with the visible space in the real-world base map, and reducing the situation where unimplementable content is incorrectly generated.

[0018] 2. By generating basic rendering prompts including spatial anchor point descriptions and effect-oriented descriptions through the second cross-modal visual understanding model, and combining negative constraints, terrain risk word cleaning, target output ratio closest to the original aspect ratio and design point array to construct combined rendering instructions, it can simultaneously constrain design objects, spatial orientation, screen ratio, excluded content and terrain stability, reducing the probability of the visual generation model producing terrain abrupt changes, scale imbalance, viewpoint drift or unwanted objects.

[0019] 3. By recording basic rendering prompts, negative constraints, combined rendering instructions, and the current rendering image through stateful multi-turn sessions corresponding to the target plot, and updating the corresponding context when receiving additional negative descriptions or user editing requests, it can support users to continuously modify rendering results, avoiding the loss of context and discontinuity of rendering style caused by generating from scratch every time an edit is made.

[0020] 4. By inputting the user's editing request, the current rendering image, and the mask image into the second cross-modal visual understanding model when the user's editing request carries a mask image, the model generates basic rendering prompts for local editing and generates local redrawing constraints based on the mask image. This limits the visual generation model to reconstruct only the areas to be redrawn, keeping the non-redrawn areas, existing fixed objects, building outlines, and ground morphology stable, thereby improving the controllability and consistency of local editing. Attached Figure Description

[0021] The invention can be better understood by describing exemplary embodiments disclosed herein in conjunction with the accompanying drawings, in which: Figure 1 The diagram shown is a schematic flowchart of an image generation method based on a cross-modal model and spatial topology according to a first embodiment of the present invention. Figure 2 The diagram shown is a schematic diagram of the program modules of an image generation device based on a cross-modal model and spatial topology according to Embodiment 1 of the present invention; Figure 3 The diagram shown is a hardware structure schematic of a computing device according to Embodiment 1 of the present invention. Detailed Implementation

[0022] The following describes specific embodiments of the present invention. It should be noted that, in order to provide a concise description, this specification cannot exhaustively describe all features of the actual embodiments. It should be understood that, in the actual implementation of any embodiment, just as in any engineering or design project, various specific decisions are often made to achieve the developer's specific goals and to meet system-related or business-related constraints, and this can change from one embodiment to another. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this invention, some design, manufacturing, or production modifications based on the technical content disclosed in this invention are merely conventional technical means and should not be construed as insufficient content of this invention.

[0023] Unless otherwise defined, the technical or scientific terms used in the claims and description shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including" and similar terms mean that the element or object preceding "comprising" or "including" encompasses the element or object listed following "comprising" or "including" and its equivalents, and do not exclude other elements or objects. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, nor are they limited to direct or indirect connections.

[0024] Example 1 Please see Figure 1 This embodiment proposes an image generation method based on cross-modal models and spatial topology, including the following steps: S1: Obtain the candidate design scheme text and real-scene base map of the target plot, and input the candidate design scheme text and real-scene base map into the first cross-modal visual understanding model. In this embodiment, the first cross-modal visual understanding model adopts a text and multimodal understanding model (e.g., doubao-seed-2-0-lite-260215) to extract feasible design elements from the real-scene base map.

[0025] Among them, the candidate design scheme text is the design scheme text generated or selected for the target plot, and the real scene base map is the on-site photo of the target plot or the current image used to carry the design effect.

[0026] In this embodiment, step S1 specifically includes the following steps S11 to S13: S11: Based on the candidate design scheme text, identify the object to be modified and the corresponding modification treatment description.

[0027] S12: Based on the real-scene base map, identify the visible areas in the real-scene base map.

[0028] Specifically, the real-scene base map is input into the first cross-modal visual understanding model to identify visual areas such as roads, ground, walls, bridge pillars, building facades, green belts, open spaces, obstructions, and blind spots; at the same time, visual occlusion filtering detection is performed to mark areas deep under bridges, the back of buildings, areas obscured by walls, areas beyond the image boundary, or other areas that cannot be seen from the current base map. Figure 2 The region confirmed by the visible plane.

[0029] S13: Determine whether the object to be modified is located in the visible area.

[0030] If so, the object to be modified and its corresponding modification description shall be retained as an implementable design element. If not, then remove the modification description corresponding to the object to be modified.

[0031] By simultaneously analyzing candidate design scheme text and real-world base map, and eliminating design content that cannot be implemented in the visible area of ​​the current base map, it is possible to avoid the subsequent image generation process from incorrectly rendering invisible areas, occluded areas, or objects outside the image into the image, thereby improving the spatial consistency between design elements and the real-world base map.

[0032] S2: Structuralize the design elements to form a design point array; the design point array includes object information, modification description information, and attribute information. Specifically, this includes the following steps S21 to S22: S21: Extract the object information, modification and processing description information, and attribute information of each implementable design element, and encapsulate them into corresponding design points.

[0033] In this embodiment, the attribute information is an English description carrying material and size characteristics.

[0034] S22: Sort the design points according to their spatial location on the real-world base map or their order of appearance in the candidate design scheme text to form a design point array.

[0035] The sorted design point array retains object information, modification processing description information, attribute information, and optional spatial location index.

[0036] By converting feasible design elements into arrays of design points containing object information, modification processing description information, and attribute information, natural language schemes can be decomposed into structured data that can be called by subsequent compilation, constraints, and rendering, reducing the loss of correspondence between design objects, actions, and attributes.

[0037] S3: Input the candidate design scheme text and real-world base map into the second cross-modal visual understanding model to generate basic rendering prompts; among them, the basic rendering prompts include spatial anchor descriptions and effect-oriented descriptions.

[0038] In this embodiment, step S3 specifically includes the following steps S31 to S33: S31: Input the candidate design scheme text and the real scene base map into the second cross-modal visual understanding model, and generate a Chinese scene description based on the candidate design scheme text and the real scene base map.

[0039] In this embodiment, the second cross-modal visual understanding model uses a text and multimodal understanding model (e.g., doubao-seed-2-0-lite-260215) to generate basic rendering prompts based on candidate design scheme text and real-world background images.

[0040] Specifically, the second cross-modal visual understanding model is called through the / generate-prompt interface, taking the candidate design scheme text and the real scene base map as input to generate a Chinese scene description of 150-250 characters.

[0041] The first sentence of the Chinese scene description is spatially anchored to form a spatial anchor point description, which is used to indicate the orientation of the site. Subsequent Chinese scene descriptions are effect-oriented descriptions, which adopt the aesthetic guidance of renderings and are used to express the warm-toned twilight atmosphere, natural human activities, site interface updates and the integration of landscape facilities.

[0042] More specifically, when inputting candidate design scheme text and real-scene base map into the second cross-modal visual understanding model, corresponding constraint prompts are also input at the same time. The constraint prompts are used to restrict the creation of buildings or terrains that do not exist in the real-scene base map, the movement of distant features to nearby locations, and the alteration of the main building outlines and terrain relationships in the real-scene base map.

[0043] By simultaneously reading candidate design text and real-world base map using a second cross-modal visual understanding model to generate basic rendering prompts, the prompts can both contain the intent of the design and be constrained by the spatial relationships of the real-world base map.

[0044] S32: Perform regular expression cleaning on the Chinese scene description, remove redundant content, and generate basic rendering prompts; among which, redundant content includes thought chain prefixes, redundant quotation marks, and / or opening remarks.

[0045] Specifically, regular expression cleaning is performed on the Chinese scene descriptions output by the second cross-modal visual understanding model to remove opening phrases such as "The following is", "Okay", and "I will", as well as redundant quotation marks, Markdown tags, and thought chain prefixes. The main text content related to spatial anchors, design effects, material atmosphere, and visible objects is retained as basic rendering prompts.

[0046] By using regular expressions to clean up redundant content, we can reduce the amount of irrelevant natural language entering the visual generation model and improve the stability of rendering instructions.

[0047] S33: Use regular expression cleaning operators to perform terrain risk word cleaning on the basic rendering prompts, and replace the terrain risk words in the basic rendering prompts with the corresponding terrain safety words in the preset safety dictionary.

[0048] Specifically, by concatenating two layers of regular expression cleaning operators sanitizeDangerousWords, two-layer risk word cleaning is performed. The first layer of risk word cleaning performs forced semantic replacement, and the second layer of risk word cleaning performs runtime marking and alarms.

[0049] Based on the danger dictionary (DANGEROUS_WORDS), a probe is used to sniff out terrain risk words, and then the following first-level and second-level risk word cleaning are performed sequentially: The first layer of risk word cleaning: The regular expression cleaning operator sanitizeDangerousWords is called to traverse the filtered terrain risk words. For risk words already recorded in the pre-set safety dictionary SAFE_REPLACEMENTS, forced semantic replacement is performed to directly replace them with terrain safety words; for example, raised planters is replaced with ground-level planting beds, and terraced is replaced with flat ground-level layered landscape.

[0050] The second layer of risk word cleaning: For residual risk words such as excavate and sunken that cannot be directly replaced and may cause terrain excavation or sudden changes in elevation, runtime marking and warnings are performed. Only runtime warnings are recorded, and they are avoided from being written into the combined rendering command as a positive rendering action.

[0051] S4: Obtain the negative description input by the user and synthesize negative semantic constraints. Specifically, this includes the following steps S41 to S42: S41: Obtain the negative description input by the user, extract, normalize and deduplicate the negative description, and select to retain the original language or convert it to the target language according to the input requirements of the visual generation model to generate negative constraints.

[0052] By reading user input of avoidable or prohibited items, negative descriptions are extracted based on trigger words such as "don't," "avoid," "must not," "prohibit," "do not set," "remove," and "weaken." For example, if the user inputs "don't have a basketball court" or "avoid steel structures," the corresponding content is identified as a negative description. The extracted results are then normalized and deduplicated; subsequently, a translation operator or language model is used to translate the Chinese negative descriptions into negative constraints, which are then written into a negative constraint object. If no negative description is obtained, an empty negative constraint object is generated.

[0053] S42: Merge the negative constraints with the baseline negative cue lexicon to generate negative semantic constraints.

[0054] In this embodiment, the baseline negative prompt vocabulary BASE_NEGATIVE_PROMPTS includes high-frequency distortion risk words such as CGI render, 3Dvisualization, illustration, digital art, and viewpoint shift. The negative constraints obtained after extraction, normalization, deduplication, and optional language conversion are combined with the baseline negative prompt vocabulary and subjected to set union and deduplication processing to generate negative semantic constraints, which are then written into the combined rendering instructions as AVOID semantic paragraphs.

[0055] S5: Analyze the pixel array of the real-scene base map, calculate the original aspect ratio of the real-scene base map, and determine the target output ratio that is closest to the original aspect ratio based on the set of ratios supported by the visual generation model.

[0056] Specifically, the underlying image processing engine is called to parse the reference base map pixel array, obtain the image width W and height H and calculate the original aspect ratio r=W / H; obtain the set of proportions supported by the visual generation model, and determine the candidate proportion with the smallest difference from r as the target output proportion.

[0057] S6: Construct combined rendering instructions based on basic rendering prompts, design point arrays, negative constraints, and the target output ratio of the real-world base map.

[0058] In this embodiment, a combined rendering instruction is constructed based on the cleaned base rendering prompts, design point array, negative semantic constraints, target output ratio, terrain hard lock constraint declaration, and optical anchor points.

[0059] Among them, the terrain hardlock constraint declaration is used to restrict the visual generation model from changing the relative positions and distances between buildings, roads and infrastructure.

[0060] Optical anchor points are used to record the shooting parameters and viewpoint characteristics of the real-world base map, and to set the viewpoint constraint strength according to the subsequently determined image generation mode.

[0061] S7: Calls the visual generation model and generates a rendering of the design scheme based on the combined rendering instructions and the real-world base map.

[0062] In this embodiment, when generating the rendered image for the first time in step S7, a normal single-image generation mode is used, which allows the visual generation model to select a target perspective that is more suitable for displaying the design scheme, while keeping the relative positions and distances between buildings, roads, and infrastructure unchanged. The visual generation model uses a visual image generation model (e.g., gemini-3-pro-image-preview).

[0063] S8: Identify the design facilities in the rendered image, generate a normalized spatial coordinate map, and perform layer overlay. Specifically, this includes the following steps S81 to S83: S81: Invoke the third cross-modal visual understanding model.

[0064] In this embodiment, the third cross-modal visual understanding model adopts a text and multimodal understanding model (such as doubao-seed-2-0-lite-260215), and performs reverse spatial mapping on the rendered image based on the design point array through interface routing for reverse spatial mapping, and identifies the design facilities in the rendered image; and matches the identified design facilities with the object information and modification processing description information in the design point array.

[0065] S82: Extract the spatial anchor points of the design facilities and the corresponding design element descriptions, and generate a normalized spatial coordinate map (that is, strictly constrain the x and y coordinate domains within the closed interval [0, 1].

[0066] Each record in the normalized spatial coordinate map includes the design facility identifier, normalized spatial coordinates, corresponding design element description, and optional object information, modification treatment description information, and attribute information.

[0067] S83: Based on the spatial coordinate map, perform automated calculations and absolute positioning on the layers of the rendered image to generate overlay, interactive design annotation hotspots.

[0068] Furthermore, in some embodiments, an image generation method based on a cross-modal model and spatial topology of this application further includes the following step S9: S9: Redraw the rendered graph through a stateful multi-turn session. This includes the following steps: S91: Create or invoke a stateful multi-turn session corresponding to the target land parcel, and generate or read the corresponding session identifier; generate a session context record associated with the session identifier; In this embodiment, when the rendered image is generated for the first time or when the editing stage is entered for the first time, a stateful multi-turn session corresponding to the target plot is created, a session identifier (sessionId) is generated, and a session context record is generated.

[0069] The session context record includes at least the target plot identifier, session identifier, current base rendering prompt, current negative constraint, current combined rendering instruction, current rendering graph access information, and session liveness status. The length of the session context record is kept within the context range supported by the text and multimodal understanding model.

[0070] If a valid session already exists, read the existing session identifier and session context record.

[0071] S92: Receive content input by the user through a stateful multi-turn session, identify the input content, and return and execute the corresponding steps based on the identification results.

[0072] If a negative description carrying a session identifier is received, return to step S4, read the session context record based on the session identifier, extract, normalize and deduplicate the negative description, and select to retain the original language or convert it to the target language according to the input requirements of the visual generation model, generate negative constraints, and update the negative constraints in the session context record.

[0073] If an edit request carrying a session identifier is received, return to step S3, read the session context record based on the session identifier, generate or update the basic rendering prompt words in the session context record, and re-execute steps S6 to S8.

[0074] Furthermore, if an edit request carrying a session identifier is received, step S3 also performs the following steps S34 and S35: S34: Obtain the current rendered image corresponding to the edit request, and determine the image generation mode based on whether the edit request carries a mask image.

[0075] The mask image is used to mark the areas to be modified, while the current render image is used to provide the existing style, lighting, perspective, and relationship with the surrounding environment.

[0076] If the editing request includes a mask image, the annotation or local redrawing mode is adopted. The editing request, the current rendered image, and the mask image are input into the second cross-modal visual understanding model to generate corresponding basic rendering prompts for local editing, and then proceed to step S35. In this mode, the original viewpoint, building location, and road relationship of the current rendered image remain unchanged.

[0077] After inputting the user's editing request, the current rendered image, and the mask image into the second cross-modal visual understanding model, basic rendering prompts for local editing are generated. These basic rendering prompts retain the overall style, lighting, and perspective relationships in the current rendered image and describe the content that needs to be modified only for the corresponding area in the mask image.

[0078] If the user's edit request does not include a mask image, the normal single-image generation mode is used to generate basic rendering prompts for global adjustments based on the user's edit request and the current rendered image; the visual generation model is allowed to select a target perspective that is more suitable for displaying the design scheme, while keeping the relative positions and distances between buildings, roads and infrastructure unchanged.

[0079] S35: In annotation or local redraw mode, based on the basic rendering prompts in the session context record, update the basic rendering prompts in the local editing, specifically including the following steps: determine the area to be redrawn and the non-redrawn area based on the mask map, and generate local redraw constraints based on the area to be redrawn and the non-redrawn area.

[0080] Among them, the local redraw constraint requires that the updated base rendering prompts be applied only to the area to be redrawn, while the non-redrawn areas and existing fixed objects remain unchanged.

[0081] After generating local redraw constraints, step S6 updates the combined rendering instructions based on the updated base rendering prompts and local redraw constraints, and then step S7 calls the visual generation model to generate the edited rendering image.

[0082] In one embodiment, the mask image is a mask overlay. The pixel markers of the overlay are read, the area where the mask overlay is located is determined as the area to be redrawn, and the area outside the mask overlay is determined as the non-redrawn area; a first local redraw constraint is generated, which requires clearing the mask overlay, performing local editing only in the mask overlay area, and performing seamless blending processing on the edges of the area to be redrawn.

[0083] In another embodiment, the mask image is the mask boundary. The pixel-level mask boundary is read, and the area within the boundary is determined as the area to be redrawn, while the area outside the boundary is determined as the non-redrawn area. A second local redraw constraint is generated. The second local redraw constraint first extracts the average color of the surrounding environment based on the mask boundary and fills the area to be redrawn. Then, high-frequency noise is injected into the filled area to be redrawn and Gaussian blur is applied, making the area to be redrawn a noisy area that needs to be realistically reconstructed.

[0084] By saving the correspondence between basic rendering prompts, negative constraints, combined rendering instructions and the current rendering map through stateful multi-turn sessions, it can support users to continuously add negative descriptions or make editing requests. By distinguishing between local redrawing areas and non-redrawing areas through mask maps, and combining high-frequency noise, Gaussian blur and complete terrain hard lock constraint declarations, it can keep non-redrawing areas stable when making local modifications, reducing style drift and spatial misalignment caused by full map redrawing.

[0085] Please continue reading. Figure 2 This invention discloses an image generation apparatus based on a cross-modal model and spatial topology. In this embodiment, the image generation apparatus based on a cross-modal model and spatial topology may include or be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the invention and implement the aforementioned image generation method based on a cross-modal model and spatial topology. The program module referred to in this invention is a series of computer program instruction segments capable of performing a specific function, and is more suitable than the program itself for describing the execution process of an image generation apparatus based on a cross-modal model and spatial topology in a storage medium. The following description will specifically introduce the functions of each program module in this embodiment: The design point array generation module is used to input candidate design scheme text and real scene base map into the first cross-modal visual understanding model, extract feasible design elements from the real scene base map, and perform structured processing on the design elements to form a design point array.

[0086] The prompt word generation module is used to input candidate design scheme text and real-scene base map into the second cross-modal visual understanding model to generate basic rendering prompt words.

[0087] The interaction module is used to create or invoke stateful multi-turn sessions corresponding to the target plot, generate or read session identifiers, obtain negative descriptions and / or edit requests input by the user, and update negative constraints and / or basic rendering prompts in the context record based on the negative descriptions and / or edit requests.

[0088] The rendering instruction generation module constructs combined rendering instructions based on basic rendering prompts, design point arrays, negative constraints, and the target output ratio determined according to the original aspect ratio of the real-world base map. When the editing request carries a mask image, it generates local redraw constraints based on the mask image and updates the combined rendering instructions based on the updated basic rendering prompts and local redraw constraints.

[0089] The visual generation module is used to call the visual generation model and generate a rendering of the design scheme based on the combined rendering instructions and the real scene base map, as well as generate an edited rendering based on the updated combined rendering instructions.

[0090] The spatial coordinate map generation module is used to call the third cross-modal visual understanding model, perform inverse spatial mapping on the rendered image based on the design point array, identify the design facilities in the rendered image, and extract the spatial anchor points of the design facilities and the corresponding design element descriptions to generate a normalized spatial coordinate map.

[0091] This embodiment also provides a computing device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple servers) capable of executing programs. The computing device 20 in this embodiment includes, but is not limited to, a memory 21 and a processor 22 that can communicate with each other via a system bus. Figure 3 As shown. It should be noted that, Figure 3 Only a computing device 20 with components 21-22 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0092] In this embodiment, the memory 21 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the computing device 20, such as the hard disk or memory of the computing device 20. In other embodiments, the memory 21 may also be an external storage device of the computing device 20, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computing device 20. Of course, the memory 21 may include both the internal storage unit and the external storage device of the computing device 20. In this embodiment, the memory 21 is typically used to store the operating system and various application software installed on the computing device 20, such as the program code of an image generation device based on a cross-modal model and spatial topology in Embodiment 1. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.

[0093] In some embodiments, processor 22 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 22 is typically used to control the overall operation of computing device 20. In this embodiment, processor 22 is used to run program code stored in memory 21 or process data, for example, to run an image generation device based on a cross-modal model and spatial topology to implement an image generation method based on a cross-modal model and spatial topology as described in Embodiment 1.

[0094] This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The computer-readable storage medium of this embodiment is used to store an image generation device based on a cross-modal model and spatial topology. When executed by a processor, it implements the image generation method based on a cross-modal model and spatial topology of Embodiment 1.

[0095] In summary, according to the exemplary embodiments, the image generation method and apparatus based on cross-modal models and spatial topology of the present invention have the following beneficial effects: 1. By using the first cross-modal visual understanding model, implementable design elements are extracted from candidate design scheme texts and real-world base maps, and a design point array is formed. This can eliminate invalid design content in invisible areas, occluded areas, or blind spots, ensuring that subsequent rendered objects maintain a correspondence with the visible space in the real-world base map, and reducing the situation where unimplementable content is incorrectly generated.

[0096] 2. By generating basic rendering prompts including spatial anchor point descriptions and effect-oriented descriptions through the second cross-modal visual understanding model, and combining negative constraints, terrain risk word cleaning, target output ratio closest to the original aspect ratio and design point array to construct combined rendering instructions, it can simultaneously constrain design objects, spatial orientation, screen ratio, excluded content and terrain stability, reducing the probability of the visual generation model producing terrain abrupt changes, scale imbalance, viewpoint drift or unwanted objects.

[0097] 3. By recording basic rendering prompts, negative constraints, combined rendering instructions, and the current rendering image through stateful multi-turn sessions corresponding to the target plot, and updating the corresponding context when receiving additional negative descriptions or user editing requests, it can support users to continuously modify rendering results, avoiding the loss of context and discontinuity of rendering style caused by generating from scratch every time an edit is made.

[0098] 4. By inputting the user's editing request, the current rendered image, and the mask image into the second cross-modal visual understanding model when the user's editing request carries a mask image, basic rendering prompts for local editing are generated, and local redrawing constraints are generated based on the mask image. This can limit the visual generation model to only reconstruct the area to be redrawn. The annotation or local redrawing mode maintains the original viewpoint, building position, and road relationship unchanged, and keeps the non-redrawing area, existing fixed objects, building outlines, and ground morphology stable, thereby improving the controllability and consistency of local editing.

[0099] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0101] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0102] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0104] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image generation method based on cross-modal models and spatial topology, characterized in that, Includes the following steps: Obtain candidate design scheme texts and real-world base maps for the target site; The candidate design scheme text and the real-scene base map are input into the first cross-modal visual understanding model. The feasible design elements in the real-scene base map are extracted, and the design elements are structured to form a design point array. The design point array includes object information, modification processing description information, and attribute information. The candidate design scheme text and real-scene base map are input into the second cross-modal visual understanding model to generate basic rendering prompts; wherein, the basic rendering prompts include spatial anchor descriptions and effect-oriented descriptions; Obtain the negative description input by the user, and normalize and deduplicate the negative description, translating it into a negative constraint; Based on the aforementioned basic rendering prompts, design point arrays, negative constraints, and the target output ratio determined according to the original aspect ratio of the real-world base map, a combined rendering instruction is constructed. The visual generation model is invoked, and a rendering of the design scheme is generated based on the combined rendering instructions and the real-world base map. The third cross-modal visual understanding model is invoked, and based on the design point array, inverse spatial mapping is performed on the rendered image to identify the design facilities in the rendered image; Extract the spatial anchor points and corresponding design element descriptions of the designed facilities to generate a normalized spatial coordinate map.

2. The image generation method based on cross-modal model and spatial topology according to claim 1, characterized in that, The step of inputting the candidate design scheme text and the real-scene base map into the second cross-modal visual understanding model to generate basic rendering prompts includes the following steps: The candidate design scheme text and the real-scene base map are input into the second cross-modal visual understanding model, and a Chinese scene description is generated based on the candidate design scheme text and the real-scene base map. Regular expression cleaning is performed on the Chinese scene description to remove redundant content and generate basic rendering prompts; The redundant content includes thought chain prefixes, redundant quotation marks, and / or opening remarks.

3. The image generation method based on cross-modal model and spatial topology according to claim 2, characterized in that, Before constructing the combined rendering instructions based on the basic rendering prompts, design point arrays, negative constraints, and the target output ratio determined according to the original aspect ratio of the real-world base map, the following steps are also included: A regular expression cleaning operator is used to perform terrain risk word cleaning on the basic rendering prompt words, replacing the terrain risk words in the basic rendering prompt words with the corresponding terrain safety words in the preset safety dictionary; The pixel array of the real-scene base image is analyzed, the original aspect ratio of the real-scene base image is calculated, and the target output ratio closest to the original aspect ratio is determined according to the set of ratios supported by the visual generation model. The negative constraints are merged with the baseline negative prompt lexicon to generate negative semantic constraints; Based on the cleaned basic rendering prompts, the design point array, negative semantic constraints, and the target output ratio, the combined rendering instructions are constructed.

4. The image generation method based on cross-modal model and spatial topology according to claim 1, characterized in that, It also includes the following steps: Create or invoke a stateful multi-turn session corresponding to the target land parcel, and generate or read the corresponding session identifier; Generate a session context record associated with the session identifier; If a negative description carrying the session identifier is received, the session context record is read based on the session identifier, the negative description is extracted, normalized and deduplicated, and the original language is retained or converted into the target language according to the input requirements of the visual generation model, generating negative constraints, and updating the negative constraints in the session context record based on the negative constraints. If an edit request carrying the session identifier is received, the session context record is read based on the session identifier, and the basic rendering prompt words in the session context record are generated or updated.

5. The image generation method based on cross-modal model and spatial topology according to claim 4, characterized in that, If an edit request carrying the session identifier is received, the session context record is read based on the session identifier, and the basic rendering prompt words in the session context record are generated or updated. This specifically includes the following steps: Obtain the current rendered image corresponding to the edit request, and determine whether the edit request carries a mask image; If the edit request carries a mask image, then the edit request, the current rendered image, and the mask image are input into the second cross-modal visual understanding model to generate corresponding basic rendering prompts for local editing, and the basic rendering prompts in the session context record are updated based on the basic rendering prompts for local editing.

6. The image generation method based on cross-modal model and spatial topology according to claim 5, characterized in that, After updating the base rendering tooltip in the session context record based on the locally edited base rendering tooltip, the following steps are also included: The regions to be redrawn and the regions not to be redrawn are determined based on the mask image. Based on the region to be redrawn and the region not to be redrawn, local redrawing constraints are generated. The combined rendering instructions are updated based on the updated base rendering prompts and the local redraw constraints, and the visual generation model is invoked to generate the edited rendered image.

7. An image generation apparatus based on a cross-modal model and spatial topology, employing the image generation method based on a cross-modal model and spatial topology as described in any one of claims 1 to 6, characterized in that, include: The design point array generation module is used to input candidate design scheme text and real scene base map into the first cross-modal visual understanding model, extract feasible design elements from the real scene base map, and perform structured processing on the design elements to form a design point array. The prompt word generation module is used to input the candidate design scheme text and the real scene base map into the second cross-modal visual understanding model to generate basic rendering prompt words; An interaction module is used to acquire negative descriptions and / or edit requests input by the user, and update negative constraints and / or basic rendering prompts in the context record based on the negative descriptions and / or edit requests; The rendering instruction generation module constructs combined rendering instructions based on the basic rendering prompts, design point arrays, negative constraints, and the target output ratio determined according to the original aspect ratio of the real scene base map. The visual generation module is used to call the visual generation model and generate a rendering of the design scheme based on the combined rendering instructions and the real-world base map. The spatial coordinate map generation module is used to call the third cross-modal visual understanding model, and perform inverse spatial mapping on the rendered map based on the design point array to identify the design facilities in the rendered map; Additionally, the spatial anchor points and corresponding design element descriptions of the designed facilities are extracted to generate a normalized spatial coordinate map.

8. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the instructions, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer instructions, characterized in that, When executed by a processor, this instruction implements the steps of the method according to any one of claims 1 to 6.

Images