An indoor layout semantic constraint generation method based on multi-modal alignment

Abstract

The application discloses a kind of indoor layout semantic constraint generation methods based on multimodal alignment, it is related to artificial intelligence and indoor layout technical field, specific steps include as follows: the multiple modal data corresponding to each functional area of target indoor scene is collected;Obtain modal feature vector, cross attention calculation is carried out to modal feature vector, to obtain each modal correlation value;Based on each modal correlation value, the initial alignment of multiple modal data of each functional area is realized;According to loss function minimization, the multiple modal data after initial alignment is iteratively optimized, based on the preset indoor layout generation model, to obtain target indoor layout initial scheme;Construct semantic constraint condition and parameterize, to obtain parameterized semantic constraint rule tree;Based on parameterized semantic constraint rule tree, target indoor layout initial scheme is modified, based on the target indoor layout scheme after modification, update preset knowledge base, to obtain dynamic updating knowledge base.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and interior layout technology, specifically a method for generating semantic constraints for interior layout based on multimodal alignment. Background Technology

[0002] With the development of Building Information Modeling (BIM), 3D reconstruction technology, and smart homes, the automated generation and optimization of interior space layouts has gradually become a research hotspot. Traditional interior layout intent parsing and constraint generation systems often rely on regular expression matching and manually predefined keyword templates to process text instructions, and the constraint rules for historical layout cases require manual annotation. Specifically, from a system architecture perspective, they are characterized by single-modal parsing, static rules, and module independence. Single-modal parsing: Relies on regular expression matching of text instructions, can only recognize preset keywords, and cannot integrate geometric features such as room polygons and door / window positions in CAD base maps, resulting in a lack of spatial semantics; Static rule base: Constraint generation relies entirely on a manually predefined rule base, unable to dynamically adapt to personalized user needs. New rules require recoding, resulting in poor scalability; Independent intent parsing and constraint generation modules: The layout generation module and the intent parsing module operate independently, with parsing results transmitted as discrete parameters, losing complex logical connections and causing the layout scheme to deviate from the user's intent. From the perspective of data processing methods, manual templates have poor generalization ability: designers need to predefine keyword templates, they cannot parse unregistered descriptions, and the template library needs to be reconstructed for each new furniture type, resulting in high maintenance costs; constraint annotation is inefficient: constraint rules for historical layout cases need to be manually annotated, the average time per case is long, and the subjective error rate is heavily dependent on the skill level of workers.

[0003] Chinese invention application CN104050722B discloses an image content-driven method for generating indoor 3D scene layouts and colors. The method comprises five steps: 1. Preprocessing images from an indoor image scene dataset to obtain a semantic tag map and a layout top view for each image; 2. Defining four layout rules and four color rules, as well as the relationship between layout and color rules; 3. Performing semantic segmentation on the input image and establishing a relationship between the input image and the input indoor 3D scene through semantic tagging; 4. Generating a layout for the indoor 3D scene based on the distribution of layout rules and the layout constraints of the input image; 5. Obtaining an indoor 3D scene with a similar color style to the input image based on the distribution of color rules, the relationship between layout and color rules, and the layout and color constraints of the input image. This method is widely used in scene generation in the fields of computer vision and computer graphics.

[0004] Chinese invention application CN107590852B discloses an automatic construction method for indoor 3D scenes with behavioral constraints. Specifically, it includes: determining a room, room parameters, and a first behavioral semantic based on input information; obtaining a corresponding first object category based on the first behavioral semantic; determining a corresponding first reference layout diagram and a 3D model of an object in the room based on the first object category and the room parameters; and guiding the generation of an indoor 3D scene of the room based on the first reference layout diagram and the 3D model. The step of determining the corresponding first reference layout diagram and the 3D model of the object based on the first object category and the room parameters includes: determining a first scene category to which the first reference layout diagram belongs based on the first object category and the room parameters; determining the first reference layout diagram from the candidate layout diagrams under the first scene category, the first object category and its objects, and the room parameters, and selecting one of the first object categories; and selecting a 3D model of an object from a 3D model library based on the blank area in the first reference layout diagram and the object in the selected first object category.

[0005] Existing technologies suffer from the following problems: First, they rely solely on single-modal data for functional area identification, resulting in a lack of unified representation among different modal information and making it difficult to achieve accurate alignment of multimodal data. Second, existing multimodal alignment methods do not consider iterative optimization processes, which can easily lead to semantic bias and feature loss, making it difficult to guarantee the stability and accuracy of data fusion. Third, users' semantic requirements are often vague and unstructured, and existing methods lack a mechanism to transform them into computable parameterized constraints. Fourth, they lack the ability to structure and dynamically update the knowledge base of generated and corrected schemes, hindering knowledge accumulation and reuse, and affecting the continuous optimization and intelligence level of the system. Therefore, there is an urgent need for a method or system for generating semantic constraints for indoor layouts based on multimodal alignment to solve the problems existing in the current technologies. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide a method for generating semantic constraints for indoor layouts based on multimodal alignment, which includes the following steps: Step 1: Based on the target interior design drawings uploaded by the user, obtain multiple functional areas of the target interior and collect multimodal data corresponding to each functional area of ​​the target interior scene; Step 2: Extract modal features of each functional region from the multimodal data and vectorize them to obtain modal feature vectors. Perform cross-attention calculation on the modal feature vectors to obtain the modal association values. Based on the modal association values, perform initial alignment of the multimodal data of each functional region to obtain the initially aligned multimodal data. Step 3: Based on minimizing the loss function, iteratively optimize the initially aligned multimodal data to obtain optimized aligned multimodal data; Step 4: Obtain user browsing behavior data, and based on the optimized aligned multimodal data, historical layout data of interior layout cases, and the preset interior layout generation model, obtain the initial target interior layout scheme. Step 5: Based on the user requirement description data of each functional area of ​​the target indoor scene input in real time, construct semantic constraints, and parameterize the semantic constraints to obtain a parameterized semantic constraint rule tree; Step 6: Based on the parametric semantic constraint rule tree, modify the initial target interior layout scheme to obtain the modified target interior layout scheme; Step 7: Based on the revised target interior layout scheme, update the preset knowledge base to obtain a dynamically updated knowledge base.

[0007] In a preferred embodiment, the multimodal data specifically includes: two-dimensional planar image data of each functional area of ​​the target indoor scene, three-dimensional point cloud data of each functional area of ​​the target indoor scene, attribute annotation data of each functional area of ​​the target indoor scene, panoramic image data of each functional area of ​​the target indoor scene, and user requirement description data of each functional area of ​​the target indoor scene.

[0008] In a preferred embodiment, the modal features specifically include: geometric features, visual features, semantic features, and spatial topological features; the modal feature vectors specifically include: geometric feature vectors, visual feature vectors, semantic feature vectors, and spatial topological feature vectors.

[0009] In a preferred embodiment, cross-attention calculation is performed on modal feature vectors to obtain modal correlation values; initial alignment of multimodal data in each functional region is achieved based on the modal correlation values; the specific implementation process for obtaining initially aligned multimodal data includes: constructing a neural network model, which specifically includes: an input layer, a feature mapping layer, an attention calculation layer, a threshold comparison layer, and an output layer; inputting the modal feature vectors into the neural network model, further mapping the modal feature vectors to the same spatial dimension by the feature mapping layer, and calculating the correlation values ​​between modal feature vectors in the same spatial dimension by the attention calculation layer; further sorting according to the correlation values ​​between modal feature vectors, and presetting a correlation value threshold range, removing correlation values ​​between modal feature vectors that are not within the preset correlation value threshold range by the threshold comparison layer, retaining correlation values ​​between modal feature vectors within the preset correlation value threshold range, obtaining the multimodal data corresponding to the modal feature vectors for the retained correlation values, performing initial alignment, and further outputting the initially aligned multimodal data through the output layer.

[0010] In a preferred embodiment, the specific process of iteratively optimizing the initially aligned multimodal data to obtain optimized aligned multimodal data includes: constructing a loss function, inputting the initially aligned multimodal data into the loss function to obtain an alignment loss value; and setting a preset alignment loss value threshold, comparing and analyzing the obtained alignment loss value with the preset alignment loss value threshold. The specific comparison and analysis logic includes: if the obtained alignment loss value is less than or equal to the preset alignment loss value threshold, then there is no need to iteratively optimize the initially aligned multimodal data, and the initially aligned multimodal data is directly used as the optimized aligned multimodal data; if the obtained alignment loss value is greater than the preset alignment loss value threshold, then the initially aligned multimodal data is iteratively optimized.

[0011] In a preferred embodiment, the specific process of iteratively optimizing the initially aligned multimodal data includes: S301: If the obtained alignment loss value is greater than the preset alignment loss value threshold, then the initially aligned multimodal data will be used as the input for the current iteration. S302: Dynamically update the preset correlation value threshold range of the threshold comparison layer of the neural network model using gradients to obtain the updated neural network model; S303: Using the updated neural network model, the updated initial aligned multimodal data is obtained, and the updated initial aligned multimodal data is further input into the loss function to obtain the updated alignment loss value; S304: If the updated alignment loss value is still greater than the preset alignment loss value threshold, then return to S301 for continuous iteration until the updated alignment loss value is less than the preset alignment loss value threshold, then stop the iteration. S305: If the updated alignment loss value is less than the preset alignment loss value threshold, the iteration ends directly to obtain the optimized aligned multimodal data.

[0012] In a preferred embodiment, the specific process of acquiring user browsing behavior data and obtaining the initial target interior layout scheme based on optimized aligned multimodal data, historical layout data of interior layout cases, and a preset interior layout generation model includes: Pre-setting an interior layout generation model; inputting the acquired user browsing behavior data, optimized aligned multimodal data, and historical layout data of interior layout cases into the interior layout generation model; the interior layout generation model performs deep correlation analysis on the three types of input data; converting the user browsing behavior data into a behavior preference vector through a time series encoding layer; mapping the optimized aligned multimodal data into a multimodal alignment feature matrix through a cross-modal attention layer; selecting the top k historical layout schemes with high similarity from the historical layout data of interior layout cases; fusing the behavior preference vector, multimodal alignment feature matrix, and the top k historical layout schemes by a fusion layer; and further refining the generator layer through a U-Net architecture, generating hierarchical constraints from overall functional zoning to local furniture placement to obtain the initial target interior layout scheme.

[0013] In a preferred embodiment, the specific process of parameterizing the semantic constraints to obtain a parameterized semantic constraint rule tree includes: semantically parsing the user requirement description data of each functional area of ​​the target indoor scene input in real time, and then constructing semantic constraints; parameterizing the semantic constraints, specifically including: converting fuzzy semantic constraints into specific values ​​or value ranges to obtain parameterized semantic constraints; further, taking all objects in each functional area of ​​the target indoor scene and each functional area of ​​the target indoor scene as nodes; and using the parameterized semantic constraints as edges connecting the nodes, connecting all nodes and edges to obtain a structured parameterized semantic constraint rule tree.

[0014] In a preferred embodiment, the semantic constraints specifically include: room size constraints, wall and door / window location constraints, area division constraints, furniture category and quantity constraints, spatial relationship constraints, layout style constraints, and color constraints.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention obtains multiple functional areas within a target interior based on user-uploaded interior design drawings, and collects multimodal data corresponding to each functional area of ​​the target interior scene; further, it extracts modal features from the multimodal data of each functional area and vectorizes them to obtain modal feature vectors; it then performs cross-attention calculation on the modal feature vectors to obtain modal association values; further, it achieves initial alignment of the multimodal data of each functional area based on the modal association values; and finally, iteratively optimizes the initially aligned multimodal data according to the minimization of the loss function to obtain optimized aligned multimodal data. Modal data; further acquiring user browsing behavior data, and obtaining an initial target interior layout scheme based on optimized aligned multimodal data, historical layout data of interior layout cases, and a preset interior layout generation model; further constructing semantic constraints based on real-time input user requirement description data for each functional area of ​​the target interior scene, and parameterizing the semantic constraints to obtain a parameterized semantic constraint rule tree; further revising the initial target interior layout scheme based on the parameterized semantic constraint rule tree to obtain a revised target interior layout scheme; and further updating the preset knowledge base based on the revised target interior layout scheme to obtain a dynamically updated knowledge base. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the steps of a method for generating semantic constraints for indoor layout based on multimodal alignment, according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram showing the connection of various modules in an indoor layout semantic constraint generation system based on multimodal alignment, according to an embodiment of this application. Detailed Implementation

[0018] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of the present invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of the present invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. Example 1

[0020] Please see Figure 1 As shown in the figure, this application provides a method for generating semantic constraints for indoor layout based on multimodal alignment. The method includes the following steps: Step 1: Based on the target interior design drawings uploaded by the user, obtain multiple functional areas of the target interior and collect multimodal data corresponding to each functional area of ​​the target interior scene; Based on the above embodiments, the multimodal data specifically includes: two-dimensional planar image data of each functional area of ​​the target indoor scene, three-dimensional point cloud data of each functional area of ​​the target indoor scene, attribute annotation data of each functional area of ​​the target indoor scene, panoramic image data of each functional area of ​​the target indoor scene, and user requirement description data of each functional area of ​​the target indoor scene. Specifically, the two-dimensional planar image data of each functional area of ​​the target indoor scene is acquired through high-definition cameras from multiple preset perspectives, including visual information such as the planar layout, wall outlines, and the location and size of doors and windows in each functional area; the three-dimensional point cloud data of each functional area of ​​the target indoor scene is acquired using LiDAR scanning technology, which can accurately obtain the three-dimensional spatial coordinates, geometric shapes, and other three-dimensional information of objects in each functional area; the attribute annotation data of each functional area of ​​the target indoor scene is obtained through user-uploaded CAD drawings, including attribute annotation information of each functional area (such as furniture type, furniture name, corridor, balcony, orientation of doors and windows, living room, kitchen, bathroom, etc.); the target indoor scene... Panoramic image data of each functional area in the interior scene is obtained through panoramic cameras or laser scanners deployed within the target interior scene. This includes high-resolution RGB panoramic images and depth images, which can realistically reproduce the overall visual effect of the space. User-required description data is obtained through keyboard text input or voice input. This includes design requirements based on user input, descriptions of the purpose of functional areas (such as "the master bedroom needs to be equipped with a 1.8m wide double bed and two bedside tables" and "the living room needs to reserve space for a 6-seater sofa combination"), as well as relevant semantic information in the interior design specifications (such as "the length of the kitchen countertop is not less than 2.4m" and "the area of ​​the wet and dry separation area in the bathroom is not less than 1.5㎡"). Specifically, the process of obtaining multiple functional areas of the target interior based on the target interior design drawings uploaded by the user involves parsing the uploaded target interior design drawings and reading the functional zoning layers to obtain multiple functional areas of the target interior. The purpose is to collect targeted multimodal data for each functional area to ensure the refinement of the collected multimodal data and the accuracy of its application. Step 2: Extract modal features of each functional region from the multimodal data and vectorize them to obtain modal feature vectors. Perform cross-attention calculation on the modal feature vectors to obtain the modal association values. Based on the modal association values, perform initial alignment of the multimodal data of each functional region to obtain the initially aligned multimodal data. Based on the above embodiments, modal features specifically include: geometric features, visual features, semantic features, and spatial topological features; modal feature vectors specifically include: geometric feature vectors, visual feature vectors, semantic feature vectors, and spatial topological feature vectors. Specifically, the purpose of extracting geometric features from the multimodal data for each functional area is as follows: Geometric features focus on "the three-dimensional shape, size, relative position, spatial boundary, and volume information of objects / areas," and their extraction is necessary because interior design needs to first clarify "what can be placed in the interior space and how large it should be"; the purpose of extracting visual features from each functional area is as follows: Visual features focus on "color, texture, local details, global tone, and lighting information," and their extraction is necessary because interior design needs to meet users' sensory needs such as "visual aesthetics and style harmony"; the purpose of extracting semantic features from the visual features of each functional area is as follows: Semantic features focus on "object attributes, user needs, style preferences, etc.", and their extraction is necessary because interior design needs to meet users' sensory needs such as "visual aesthetics and style harmony"; Design needs to be "user-centric," transforming abstract needs into concrete design constraints. This can further compensate for the limitations of visual and geometric features in functional discrimination and achieve semantic consistency between different modalities. The purpose of extracting the spatial topological features of the visual features of each functional area is to focus on the "relationships between objects and areas" (such as nesting, support, connectivity, occlusion, relative positional relationships, and adjacency). The necessity of its extraction stems from the fact that interior design needs "smooth circulation and functional continuity," avoiding spatial structural contradictions, maintaining the rationality of the interior spatial structure, and constraining the relative positional relationships of different functional areas and interior objects to prevent alignment results or layout generation from violating the actual interior spatial layout logic. Specifically, cross-attention calculation is performed on modal feature vectors to obtain modal correlation values; initial alignment of multimodal data in each functional region is achieved based on the correlation values; the specific implementation process for obtaining initially aligned multimodal data includes: constructing a neural network model, which specifically includes: an input layer, a feature mapping layer, an attention calculation layer, a threshold comparison layer, and an output layer; inputting the modal feature vectors into the neural network model, further mapping the modal feature vectors to the same spatial dimension by the feature mapping layer, and calculating the correlation values ​​between modal feature vectors in the same spatial dimension by the attention calculation layer; further sorting according to the correlation values ​​between modal feature vectors, and setting a threshold range for correlation values; removing modal feature vectors whose correlation values ​​are not within the preset threshold range by the threshold comparison layer, and retaining the correlation values ​​between modal feature vectors within the preset threshold range; obtaining the multimodal data corresponding to the modal feature vectors for the retained correlation values ​​and performing initial alignment; and further outputting the initially aligned multimodal data through the output layer. The specific process of the attention computation layer calculating the correlation value between modal feature vectors in the same spatial dimension includes: assuming the first... Each functional region modal feature vector is mapped to the same spatial dimension and represented as a visual feature vector. Geometric eigenvectors semantic feature vector Spatial topological feature vectors ;No. Each functional region modal feature vector is mapped to the same spatial dimension and represented as a visual feature vector. Geometric eigenvectors semantic feature vector Spatial topological feature vectors ; and Indicates the functional area number; [The text abruptly ends here, likely due to an incomplete sentence or a formatting error The modal feature vector of the first functional region and the first The similarity of the modal feature vectors of each functional region is calculated, and a scaling factor is introduced to obtain the corresponding association value. Specifically, the reason for performing similarity calculation on modal feature vectors and introducing a scaling factor is to alleviate the problem of excessively large correlation values ​​in the similarity calculation of modal feature vectors due to excessively high vector dimensions. The value of the scaling factor is determined by those skilled in the art based on the actual situation, so as to ensure that the scaled correlation value can effectively measure the similarity between modal feature vectors and maintain numerical stability.

[0021] Step 3: Based on minimizing the loss function, iteratively optimize the initially aligned multimodal data to obtain optimized aligned multimodal data; Based on the above embodiments, the specific process of iteratively optimizing the initially aligned multimodal data to obtain optimized aligned multimodal data includes: constructing a loss function, inputting the initially aligned multimodal data into the loss function to obtain an alignment loss value; and setting a preset alignment loss value threshold, comparing and analyzing the obtained alignment loss value with the preset alignment loss value threshold. The specific comparison and analysis logic includes: if the obtained alignment loss value is less than or equal to the preset alignment loss value threshold, then there is no need to iteratively optimize the initially aligned multimodal data, and the initially aligned multimodal data is directly used as the optimized aligned multimodal data; if the obtained alignment loss value is greater than the preset alignment loss value threshold, then iterative optimization is performed on the initially aligned multimodal data. The specific process of iteratively optimizing the initially aligned multimodal data includes: S301: If the obtained alignment loss value is greater than the preset alignment loss value threshold, then the initially aligned multimodal data will be used as the input for the current iteration. S302: Dynamically update the preset correlation value threshold range of the threshold comparison layer of the neural network model using gradients to obtain the updated neural network model; S303: Using the updated neural network model, the updated initial aligned multimodal data is obtained, and the updated initial aligned multimodal data is further input into the loss function to obtain the updated alignment loss value; S304: If the updated alignment loss value is still greater than the preset alignment loss value threshold, then return to S301 for continuous iteration until the updated alignment loss value is less than the preset alignment loss value threshold, then stop the iteration. S305: If the updated alignment loss value is less than the preset alignment loss value threshold, the iteration is terminated directly to obtain the optimized aligned multimodal data; Specifically, the purpose of dynamically updating the preset correlation value threshold range of the threshold comparison layer of the neural network model is to filter out more accurate correlation values ​​to be eliminated and retained by updating the preset correlation value threshold range, so as to improve multimodal data alignment and reduce alignment error; the preset alignment loss value threshold is obtained by those skilled in the art through experiments based on historical multimodal data alignment. Step 4: Obtain user browsing behavior data, and based on the optimized aligned multimodal data, historical layout data of interior layout cases, and the preset interior layout generation model, obtain the initial target interior layout scheme. Based on the above embodiments, the specific process of obtaining the initial target interior layout scheme by acquiring user browsing behavior data, optimizing and aligning multimodal data, historical layout data of interior layout cases, and a preset interior layout generation model includes: Pre-setting an interior layout generation model; inputting the acquired user browsing behavior data, optimized and aligning multimodal data, and historical layout data of interior layout cases into the interior layout generation model; the interior layout generation model performs deep correlation analysis on the three types of input data; converting the user browsing behavior data into a behavior preference vector through a time series encoding layer; mapping the optimized and aligning multimodal data into a multimodal alignment feature matrix through a cross-modal attention layer; selecting the top k historical layout schemes with high similarity from the historical layout data of interior layout cases; fusing the behavior preference vector, multimodal alignment feature matrix, and the top k historical layout schemes by a fusion layer; and further refining the generator layer through a U-Net architecture, generating hierarchical constraints from overall functional zoning to local furniture placement to obtain the initial target interior layout scheme. Specifically, user browsing behavior data includes: layout style tags that users frequently browse / favorite (such as "modern minimalist style", "Japanese wood style", "light luxury style", etc.), the types of houses viewed, and the time spent on the site. By acquiring user browsing behavior data, we can uncover users' implicit layout preferences, which can help generate initial solutions that meet users' style and functional needs. Furthermore, the interior design platform can acquire user browsing behavior data. Specifically, the historical layout data for interior design cases is obtained from publicly available historical layout data obtained by design companies or public design resource databases. This includes: two-dimensional layout information (including floor plans, geometric boundaries and area ratios of functional zones (living room, bedroom, kitchen, etc.); three-dimensional spatial data (including interior 3D models, spatial coordinates of furniture placement, and point cloud data); functional attribute annotations (functional descriptions of each spatial area, furniture types, style tags (such as modern, Nordic, Chinese)); and design semantic constraint data: adjacency constraints (such as bedrooms usually need to be adjacent to bathrooms, and kitchens often need to be adjacent to dining rooms), spatial size and proportion constraints (minimum / maximum area constraints: such as bedrooms not less than 6 square meters, and master bedrooms should be larger than secondary bedrooms), and location constraints (such as the TV background wall should face the sofa, and the dining table should be close to the kitchen area), etc. Specifically, the historical layout data of interior layout cases are filtered to select the top k historical layout schemes with high similarity, where k is an integer greater than 1. The specific value can be set by those skilled in the art and may be the top 5 or 6 historical layout schemes with high similarity. No specific limitation is made here. Specifically, the preset indoor layout generation models include: layout generation models based on generative adversarial networks (GAN), layout generation models based on variational autoencoders (VAE), and layout optimization models based on reinforcement learning (RL). Step 5: Based on the user requirement description data of each functional area of ​​the target indoor scene input in real time, construct semantic constraints, and parameterize the semantic constraints to obtain a parameterized semantic constraint rule tree; Based on the above embodiments, the specific process of parameterizing the semantic constraints to obtain a parameterized semantic constraint rule tree includes: semantically parsing the user requirement description data of each functional area of ​​the target indoor scene input in real time, and then constructing semantic constraints; parameterizing the semantic constraints, specifically including: converting fuzzy semantic constraints into specific values ​​or value ranges to obtain parameterized semantic constraints; further, taking all objects in each functional area of ​​the target indoor scene and each functional area of ​​the target indoor scene as nodes; and using the parameterized semantic constraints as edges connecting the nodes, connecting all nodes and edges to obtain a structured parameterized semantic constraint rule tree; Specifically, the semantic constraints include: room size constraints (e.g., 8m long × 4m wide), wall and door / window location constraints (e.g., door / window coordinate range), area division constraints (e.g., living room / bedroom functional zoning), furniture category and quantity constraints (e.g., 1 sofa, 1 TV, 3 beds, etc.), spatial relationship constraints (e.g., sofa located in front of TV), layout style constraints (e.g., modern, Nordic, Chinese, etc.), and color constraints (e.g., gray sofa, gray living room floor tiles, warm wall colors, etc.). Specifically, fuzzy semantic constraints are transformed into specific numerical values ​​or numerical ranges to obtain parameterized semantic constraints. For example, if the semantic constraint given by the user is "the bedroom should be close to the bathroom", it can be parameterized into "the distance between the center points of the bedroom and the bathroom is ≤5m", which is in line with the parameters of the interior layout design. The purpose of parameterizing semantic constraints is to transform the user's fuzzy semantic constraint description into a unified, calculable numerical or range form, so that the semantic constraints can be directly processed in the computer system. Step 6: Based on the parametric semantic constraint rule tree, modify the initial target interior layout scheme to obtain the modified target interior layout scheme; Specifically, the parametric semantic constraint rule tree is embedded in the preset interior layout generation model. The semantic constraints of the initial target interior layout scheme are extracted, and the semantic constraints of the initial scheme are corrected based on the parametric semantic constraint rule tree. Schemes that do not meet the user's semantic constraints are corrected to meet the parametric semantic constraints. After correction, the preset interior layout generation model outputs the corrected target interior layout scheme, which satisfies the parametric semantic constraints. A layered rendering of the layout and a detailed list of parametric semantic constraints are generated through visualization. This list includes the dimensions of each room, the precise coordinates of doors and windows, and the types and quantities of furniture, allowing for further verification and adjustment by the user or in subsequent design processes. Step 7: Based on the revised target interior layout scheme, update the preset knowledge base to obtain a dynamically updated knowledge base; Specifically, the revised target interior layout scheme is exported as a standardized layout JSON file. The system reads this standardized layout JSON file and stores it in a preset knowledge base, updating the preset knowledge base to obtain a dynamically updated knowledge base. This dynamically updated knowledge base adopts a distributed storage architecture, supporting real-time layout scheme updates and multi-dimensional retrieval. It can automatically associate layout schemes of similar apartment types, semantic constraint rule trees, and user preference data. Simultaneously, the system standardizes the data in this dynamically updated knowledge base, unifying data formats and units of measurement, and uses a data cleaning mechanism to remove redundant information and outliers, ensuring data accuracy and consistency. The construction of this dynamically updated knowledge base not only provides data support for the rapid iterative optimization of subsequent interior layout schemes but also enables the discovery of potential layout design patterns through machine learning algorithms, providing experiential references for the intelligent generation of new apartment types.

[0022] Example 2 Please see Figure 2 As shown, in another embodiment of the present invention, the present invention also discloses an indoor layout semantic constraint generation system based on multimodal alignment, the system comprising the following modules: Multimodal data acquisition module; based on the target interior design drawings uploaded by the user, to obtain multiple functional areas of the target interior, and collect multimodal data corresponding to each functional area of ​​the target interior scene; A multimodal data alignment module extracts modal features from each functional region of the multimodal data and vectorizes them to obtain modal feature vectors. Cross-attention calculation is performed on the modal feature vectors to obtain modal association values. Initial alignment of the multimodal data in each functional region is achieved based on these modal association values ​​to obtain initially aligned multimodal data. Multimodal data optimization module: Based on minimizing the loss function, iteratively optimizes the initially aligned multimodal data to obtain optimized aligned multimodal data; Solution generation module: Acquire user browsing behavior data, and obtain the initial solution for the target interior layout based on optimized and aligned multimodal data, historical layout data of interior layout cases, and preset interior layout generation model; Semantic constraint parameterization module: Based on the user requirement description data of each functional area of ​​the target indoor scene input in real time, construct semantic constraint conditions, and parameterize the semantic constraint conditions to obtain a parameterized semantic constraint rule tree; Scheme correction module: Based on the parametric semantic constraint rule tree, the initial scheme of the target indoor layout is corrected to obtain the corrected target indoor layout scheme; Knowledge base construction module: Based on the revised target indoor layout scheme, update the preset knowledge base to obtain a dynamically updated knowledge base; The modules described above are connected via wired and / or wireless means to enable data transmission between them.

[0023] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0024] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating semantic constraints for indoor layout based on multimodal alignment, characterized in that, The method includes the following steps: Step 1: Based on the target interior design drawings uploaded by the user, obtain multiple functional areas of the target interior and collect multimodal data corresponding to each functional area of ​​the target interior scene; Step 2: Extract modal features of each functional region from the multimodal data and vectorize them to obtain modal feature vectors. Perform cross-attention calculation on the modal feature vectors to obtain the modal association values. Based on the modal association values, perform initial alignment of the multimodal data of each functional region to obtain the initially aligned multimodal data. Step 3: Based on minimizing the loss function, iteratively optimize the initially aligned multimodal data to obtain optimized aligned multimodal data; Step 4: Obtain user browsing behavior data, and based on the optimized aligned multimodal data, historical layout data of interior layout cases, and the preset interior layout generation model, obtain the initial target interior layout scheme. Step 5: Based on the user requirement description data of each functional area of ​​the target indoor scene input in real time, construct semantic constraints, and parameterize the semantic constraints to obtain a parameterized semantic constraint rule tree; Step 6: Based on the parametric semantic constraint rule tree, modify the initial target interior layout scheme to obtain the modified target interior layout scheme; Step 7: Based on the revised target interior layout scheme, update the preset knowledge base to obtain a dynamically updated knowledge base; Specifically, the multimodal data includes: two-dimensional planar image data of each functional area of ​​the target indoor scene, three-dimensional point cloud data of each functional area of ​​the target indoor scene, attribute annotation data of each functional area of ​​the target indoor scene, panoramic image data of each functional area of ​​the target indoor scene, and user requirement description data of each functional area of ​​the target indoor scene. Modal features specifically include: geometric features, visual features, semantic features, and spatial topological features; modal feature vectors specifically include: geometric feature vectors, visual feature vectors, semantic feature vectors, and spatial topological feature vectors. The specific process of acquiring user browsing behavior data and obtaining the initial target interior layout scheme based on optimized and aligned multimodal data, historical layout data of interior layout cases, and a preset interior layout generation model includes: Pre-setting an interior layout generation model; inputting the acquired user browsing behavior data, optimized and aligned multimodal data, and historical layout data of interior layout cases into the interior layout generation model; the interior layout generation model performs deep correlation analysis on the three types of input data; converting user browsing behavior data into a behavior preference vector through a time series encoding layer; mapping the optimized and aligned multimodal data into a multimodal alignment feature matrix through a cross-modal attention layer; selecting the top k historical layout schemes with high similarity from the historical layout data of interior layout cases; fusing the behavior preference vector, multimodal alignment feature matrix, and the top k historical layout schemes by a fusion layer; and further refining the generator layer through a U-Net architecture, generating hierarchical constraints from overall functional zoning to local furniture placement to obtain the initial target interior layout scheme. The specific process of parameterizing the semantic constraints to obtain a parameterized semantic constraint rule tree includes: semantically parsing the user requirement description data of each functional area of ​​the target indoor scene input in real time, and then constructing semantic constraints; parameterizing the semantic constraints, specifically including: converting fuzzy semantic constraints into specific values ​​or value ranges to obtain parameterized semantic constraints; further, taking all objects in each functional area of ​​the target indoor scene and each functional area of ​​the target indoor scene as nodes; and using the parameterized semantic constraints as edges connecting the nodes, connecting all nodes and edges to obtain a structured parameterized semantic constraint rule tree.

2. The method for generating semantic constraints for indoor layout based on multimodal alignment according to claim 1, characterized in that, Cross-attention calculation is performed on the modal feature vectors to obtain the association values ​​of each modality; the initial alignment of multimodal data in each functional area is achieved based on the association values ​​of each modality. The specific implementation process for obtaining initially aligned multimodal data includes: constructing a neural network model, which specifically includes an input layer, a feature mapping layer, an attention calculation layer, a threshold comparison layer, and an output layer; inputting modal feature vectors into the neural network model, further mapping the modal feature vectors to the same spatial dimension by the feature mapping layer, and calculating the correlation values ​​between modal feature vectors in the same spatial dimension by the attention calculation layer; further sorting according to the correlation values ​​between modal feature vectors and setting a preset correlation value threshold range; removing modal feature vectors whose correlation values ​​are not within the preset correlation value threshold range by the threshold comparison layer, and retaining the correlation values ​​between modal feature vectors within the preset correlation value threshold range; obtaining the multimodal data corresponding to the modal feature vectors with the retained correlation values ​​and performing initial alignment; and further outputting the initially aligned multimodal data through the output layer.

3. The method for generating semantic constraints for indoor layout based on multimodal alignment according to claim 2, characterized in that, The specific process of iteratively optimizing the initially aligned multimodal data to obtain optimized aligned multimodal data includes: constructing a loss function, inputting the initially aligned multimodal data into the loss function to obtain the alignment loss value; and setting a preset alignment loss value threshold, comparing and analyzing the obtained alignment loss value with the preset alignment loss value threshold. The specific comparison and analysis logic includes: if the obtained alignment loss value is less than or equal to the preset alignment loss value threshold, then there is no need to iteratively optimize the initially aligned multimodal data, and the initially aligned multimodal data is directly used as the optimized aligned multimodal data; if the obtained alignment loss value is greater than the preset alignment loss value threshold, then the initially aligned multimodal data is iteratively optimized.

4. The method for generating semantic constraints for indoor layout based on multimodal alignment according to claim 3, characterized in that, The specific process of iteratively optimizing the initially aligned multimodal data includes: S301: If the obtained alignment loss value is greater than the preset alignment loss value threshold, then the initially aligned multimodal data will be used as the input for the current iteration. S302: Dynamically update the preset correlation value threshold range of the threshold comparison layer of the neural network model using gradients to obtain the updated neural network model; S303: Using the updated neural network model, the updated initial aligned multimodal data is obtained, and the updated initial aligned multimodal data is further input into the loss function to obtain the updated alignment loss value; S304: If the updated alignment loss value is still greater than the preset alignment loss value threshold, then return to S301 for continuous iteration until the updated alignment loss value is less than the preset alignment loss value threshold, then stop the iteration. S305: If the updated alignment loss value is less than the preset alignment loss value threshold, the iteration ends directly to obtain the optimized aligned multimodal data.

5. The method for generating semantic constraints for indoor layout based on multimodal alignment according to claim 1, characterized in that, The semantic constraints specifically include: room size constraints, wall and door / window location constraints, area division constraints, furniture category and quantity constraints, spatial relationship constraints, layout style constraints, and color constraints.

Images