Multi-modal interactive AI design generation method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of artificial intelligence aided design, in particular to a multi-modal interactive AI design generation method and device, equipment, a storage medium and a product. According to the method, an initial scheme is generated on the basis of the multi-modal requirement of a user, then the modification intention of the user for the scheme is collected, and through deep semantic analysis and implicit intention reasoning, the multi-modal interactive AI design is generated; and dynamically extracting modification preference characteristics of the user, and gradually learning the design preference required by the user and adjusting and displaying the scheme in real time in each round of interaction until the user is satisfied. According to the method, a traditional one-way generation mode is upgraded into a continuous learning enhanced generation mode, the number of times of repeated modification and regeneration is remarkably reduced, and therefore the design efficiency of man-machine cooperation is greatly improved; meanwhile, the model can more accurately understand and respond to the deep design appeal of the user, so that the finally generated design result is more matched with the psychological expectation of the user, and the design quality and the user satisfaction are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-aided design technology, and in particular to multimodal interactive AI design generation methods, apparatus, devices, storage media, and products. Background Technology

[0002] As digitalization accelerates, artificial intelligence (AI) is driving profound changes across industries. In design, creative, and content production, generative AI is seen as central to next-generation productivity tools, with potential extending far beyond automating simple tasks. The industry's shared vision is to develop it into intelligent creative tools capable of deep, natural collaboration with humans, thereby liberating creators' productivity and inspiring more diverse and personalized innovation.

[0003] Currently, most AI design tools in the industry are built on a one-way, static generation paradigm. Their standard workflow can be summarized as follows: the user first inputs a descriptive prompt or provides several reference materials; then, the AI ​​model generates a solution based on this initial input. When the output does not meet the user's expectations, the entire interaction process stalls: the user has to return to the starting point, painstakingly rethinking and modifying the input information in hopes of obtaining a better result in the next generation. If the user is not satisfied with the solution, they need to readjust the input information and generate again. This approach is inefficient and cannot effectively capture and utilize the user's dynamically changing deep design intentions during the interaction process, resulting in low human-machine collaboration efficiency and a gap between the design outcome and the user's psychological expectations.

[0004] Therefore, how to enable generative AI tools to dynamically understand users' design styles and intentions, and to enable them to continuously learn and adapt to users' personalized design preferences, has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] The main objective of this invention is to provide a multimodal interactive AI design generation method, apparatus, device, storage medium, and product, aiming to solve the technical problem in the prior art of how to enable generative AI tools to dynamically understand the user's design style and intent, and enable them to continuously learn and adapt to the user's personalized design preferences.

[0006] To achieve the above objectives, the present invention provides a multimodal interactive AI design and generation method, the method comprising the following steps: The multimodal design requirements information is input into the cross-modal design generation model to obtain the initial design scheme; The design scheme is presented to obtain information on the user's intention to modify the design scheme; Feature extraction is performed on the modification intent information to obtain user modification preference features; After each round of scheme presentation, the design scheme is adjusted based on the user's modification preference characteristics, and the adjusted design scheme is presented to the user until the optimization termination condition is met.

[0007] Optionally, the step of inputting multimodal design requirement information into a cross-modal design generation model to obtain an initial design scheme includes: The multimodal design requirements information is classified according to data categories to obtain design reference data and user design instruction information; Based on the domain knowledge base corresponding to the design reference data, a design reference vector is obtained; Semantic analysis is performed based on the user design instruction information to obtain the design intent vector; Using the design intent vector and the design reference vector as conditional inputs, content is generated through a cross-modal design generation model to obtain the initial design scheme.

[0008] Optionally, the step of extracting features from the modification intent information to obtain user modification preference features includes: If the modification intent information is a user-initiated editing operation, then the design element to be operated and the operation type are determined based on the user-initiated editing operation; Based on the manipulated design elements, operation types, and modification case library, the user's implicit intent to modify is derived. Based on the modified implicit intent, construct a user potential preference vector; The user's potential preference vector is used as the user's modified preference feature.

[0009] Optionally, the step of extracting features from the modification intent information to obtain user modification preference features further includes: If the modification intent information is a user's natural language modification instruction, then deep semantic analysis is performed based on the user's natural language modification instruction to determine the modification content and target style information; Based on the modifications, determine the editing tools and the initial editing operation sequence; The initial editing operation sequence is corrected based on the design constraint information to obtain an optimized editing operation sequence; Based on the target style information, editing tools, and optimized editing operation sequence, user modification preference characteristics are obtained.

[0010] Optionally, the design constraint information includes at least one of functional constraints, structural constraints, and compliance constraints; The step of revising the initial editing operation sequence based on design constraint information to obtain an optimized editing operation sequence includes: Based on the design constraint information and the initial editing operation sequence, a feasibility check is performed using a predefined design rule engine; Based on the verification results, identify the operation steps in the initial editing operation sequence that do not conform to the design constraints; Based on the domain knowledge integrated into the design rule engine, the parameters of the operation steps that do not conform to the design constraints are adjusted; Based on the adjusted operation steps, an optimized editing operation sequence that satisfies all design constraints is generated.

[0011] Optionally, the multimodal interactive AI design and generation method further includes: Based on the user's initial and final design schemes during the same design scheme generation process, the user's global design modification preference information during this design generation is obtained; Update the user's design style vector based on the global design modification preference information and the multimodal design requirement information; Based on the updated style vector, the design style weight matrix in the cross-modal design generation model is adjusted so that the subsequently generated initial scheme is closer to the user's design style preference.

[0012] Furthermore, to achieve the above objectives, the present invention also proposes a multimodal interactive AI design generation device, the multimodal interactive AI design generation device comprising: The design scheme generation module is used to input multimodal design requirement information into the cross-modal design generation model to obtain the initial design scheme; The modification intent acquisition module is used to display the design scheme and obtain the user's modification intent information for the design scheme; The modification intent analysis module is used to extract features from the modification intent information to obtain user modification preference features; The design scheme optimization module is used to adjust the design scheme according to the user's modification preference characteristics after each round of scheme display, and then display the adjusted design scheme to the user until the optimization termination condition is met.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a multimodal interactive AI design generation device, which includes: a memory, a processor, and a multimodal interactive AI design generation program stored in the memory and executable on the processor. The multimodal interactive AI design generation program is configured to implement the steps of the multimodal interactive AI design generation method described above.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a multimodal interactive AI design generation program, which, when executed by a processor, implements the steps of the multimodal interactive AI design generation method described above.

[0015] In addition, to achieve the above objectives, the present invention also proposes a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multimodal interactive AI design and generation method described above.

[0016] The technical solutions proposed in this application, including one or more, possess at least the following technical effects: By constructing a dynamic perception and continuously optimized interactive generation closed loop, this application fundamentally changes the unidirectional and static working mode of traditional AI design tools. This application uses the user's modification intent information in multiple iterations as key input, and through feature extraction technology, quantifies and models the user's modification preference features—features that even the user themselves may not be able to clearly express. This mechanism enables the AI ​​model to transcend the limitations of a single interaction and gradually understand and adapt to the user's unique style and deep intent. Because the model can intelligently adjust subsequent solutions based on historical feedback, users no longer need to fall into the inefficient cycle of modification prompts and regeneration, significantly reducing the number of iterations required to achieve a satisfactory solution, accelerating the design process, and significantly improving design quality and user satisfaction. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the multimodal interactive AI design and generation method of the present invention; Figure 2 This is a flowchart illustrating the second embodiment of the multimodal interactive AI design and generation method of the present invention; Figure 3 This is a flowchart illustrating the third embodiment of the multimodal interactive AI design and generation method of the present invention; Figure 4 This is a schematic diagram illustrating the entire process of the multimodal interactive AI design and generation method of the present invention; Figure 5 This is a structural block diagram of the first embodiment of the multimodal interactive AI design and generation device of the present invention; Figure 6 This is a schematic diagram of the structure of the multimodal interactive AI design and generation device of the hardware operating environment involved in the embodiments of the present invention.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods. The main solution of this application embodiment is as follows: inputting multimodal design requirement information into a cross-modal design generation model to obtain an initial design scheme; displaying the design scheme to obtain user modification intention information; extracting features from the modification intention information to obtain user modification preference features; after each round of scheme display, adjusting the design scheme according to the user modification preference features, and displaying the adjusted design scheme to the user until the optimization termination condition is met.

[0023] Currently, most AI design tools in the industry are built on a one-way, static generative paradigm. Their standard workflow can be summarized as follows: the user first inputs a descriptive prompt or provides several reference materials; then, the AI ​​model generates a solution based on this initial input. When the output does not match the user's expectations, the entire interaction process stalls: the user has to return to the starting point, painstakingly rethinking and modifying the input information in hopes of obtaining a better result in the next generation. If the user is not satisfied with the solution, they need to readjust the input information and generate again. This approach is inefficient and cannot effectively capture and utilize the user's dynamically changing deep design intentions during the interaction process, resulting in low human-machine collaboration efficiency and a gap between the design outcome and the user's psychological expectations. Therefore, how to enable generative AI tools to dynamically understand the user's design style and intentions, and to enable them to continuously learn and adapt to the user's personalized design preferences, is a pressing technical problem that needs to be solved.

[0024] This application fundamentally changes the traditional one-way, static working mode of AI design tools by constructing a dynamic perception and continuous optimization interactive generation closed loop. It uses the user's modification intent information across multiple iterations as key input and, through feature extraction technology, quantifies and models the user's modification preferences—features that even the user themselves may not clearly express. This mechanism enables the AI ​​model to transcend the limitations of a single interaction, gradually understanding and adapting to the user's unique style and deep intent. Because the model can intelligently adjust subsequent solutions based on historical feedback, users are no longer trapped in the inefficient cycle of modification prompts and regeneration. This significantly reduces the number of iterations required to achieve a satisfactory solution, accelerates the design process, and significantly improves design quality and user satisfaction.

[0025] It should be noted that the executing entity of this invention can be a multimodal interactive AI design and generation device, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a thermal management device capable of realizing the above functions of a multimodal interactive AI design and generation device, etc. This embodiment does not specifically limit it in this way. The following uses a multimodal interactive AI design and generation device as the executing entity to describe this embodiment and the following embodiments.

[0026] Based on this, embodiments of this application provide a multimodal interactive AI design and generation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multimodal interactive AI design and generation method of this application.

[0027] In this embodiment, the multimodal interactive AI design and generation method includes steps S10 to S50: Step S10: Input the multimodal design requirements information into the cross-modal design generation model to obtain the initial design scheme.

[0028] It's important to note that when performing this step, users need to provide their initial design concepts. These concepts can exist in multiple forms, known as "multimodal design requirements information." Therefore, design requirements information here does not refer to a single type of input, but rather encompasses various media that users may use to express their design intentions. Specifically, this can include, but is not limited to, textual descriptions of design goals and styles, images or sketches providing visual references, explanatory documents specifying functions and structures, and even user-specified design specifications and constraints.

[0029] Understandably, this multimodal input method greatly lowers the barrier for users to express their ideas, enabling even users who are not good at precise descriptions to convey their thoughts to the system in a more intuitive way.

[0030] It should be understood that the system inputs the processed multimodal information into a cross-modal design generation model for content generation. This model is an artificial intelligence model trained on a large-scale dataset, possessing content generation and cross-modal understanding capabilities. Its core function is to integrate scattered design requirement information in different forms into a unified and complete design concept. Through this integration process, the model ultimately generates an "initial design scheme" that meets the user's initial needs. For example, in an interior design scenario, a user can upload a picture of a living room style they like (image modality) and add the text "I hope the color scheme is brighter and I want to add a reading corner" (text modality). The model will then generate a preliminary interior layout rendering that integrates these elements. This initial scheme provides a crucial starting point and foundation for subsequent human-computer interaction and collaborative optimization. In a preferred embodiment, the cross-modal design generation model of this invention adopts a hierarchical hybrid architecture, with the core consisting of an autoregressive model and a diffusion model working together. At the model front end, an autoregressive model based on the Transformer architecture is used to handle the multimodal input alignment task, specifically through a cross-attention mechanism to achieve spatial mapping and semantic fusion of feature vectors from different modalities such as text, images, and sketches. The model backend uses the DiT (Diffusion Transformer) diffusion model as the generation engine, which transforms the multimodal conditional vectors fused from the frontend into high-quality design solutions through an iterative denoising process. This front-end and back-end separation architecture ensures both the flexibility of multimodal understanding and the professionalism and stability of the generated results.

[0031] Furthermore, in the multimodal alignment stage, the system first equips each input modality with a dedicated encoder: the text modality uses a pre-trained language model to extract semantic features, while the image / sketching modality uses a convolutional neural network to extract visual features. Subsequently, feature alignment is achieved through a cross-modal attention layer. Specifically, the feature vectors of each modality are projected into a shared latent space, and attention weights between modalities are calculated to achieve adaptive feature fusion. For conflicting multimodal inputs, such as an input image displaying a modern style while the required text describes a traditional style, the system dynamically adjusts the contribution weights of each modality through a learnable gating mechanism, ultimately outputting a unified conditional vector as a guide for the generation of the cross-modal model.

[0032] Step S20: Display the design scheme and obtain information on the user's intention to modify the design scheme.

[0033] It should be noted that, depending on the application scenario, this design scheme may take the form of a two-dimensional image, a three-dimensional model, a design document, or a piece of program code. The purpose of this demonstration is to provide users with a concrete, reviewable, and evaluable design prototype, enabling them to offer suggestions for modification based on their professional judgment or personal preferences.

[0034] Understandably, after a user reviews a design proposal, the system will proactively acquire the user's intention to modify the design. This intention primarily includes two types of actions: First, proactive user actions, such as dragging and dropping a sofa on an interior design rendering to change its layout, or selecting and altering the fabric's color and pattern on a clothing design draft. These direct physical interactions precisely convey the user's explicit instructions to adjust the form, position, or attributes of specific elements within the design, characterized by their intuitiveness and strong directionality. Second, natural language modification instructions, such as, when faced with a preliminary web design prototype, a user might input via voice or text, "Adjust the navigation bar to a transparent background and enlarge and bold the main title font." These instructions carry higher-level, semantic-based modification goals and aesthetic preferences, requiring the system not only to understand the literal meaning but also to accurately map and execute them onto the corresponding components of the design. The system will comprehensively capture both types of interaction data as the core basis for understanding the user's true intentions and driving the continuous evolution of the design, thereby gradually narrowing the gap between the initial design and the user's ideal effect.

[0035] It should be understood that the purpose of this step is to capture the user's subjective feedback. At this stage, the system comprehensively records all interaction data that may reflect the user's preferences, and then organizes and analyzes this seemingly scattered information about modification intentions to obtain user preferences. This becomes the key data foundation for optimizing the model's output and is the core driving force for the evolution of the design solution from "preliminary usability" to "user satisfaction."

[0036] Step S30: Extract features from the modification intent information to obtain user modification preference features.

[0037] It's important to note that the goal of feature extraction here is not simply to record the user's commands themselves, but rather to uncover the underlying, quantifiable stylistic tendencies or functional requirements behind these fragmented, superficial interactions. This process typically involves data analysis models used to transform diverse input information into a structured, machine-interpretable representation of user modification preferences.

[0038] Understandably, user modification preferences are a fundamental summary of a user's aesthetic or functional needs. For example, if a user repeatedly changes the color of buttons on a webpage from blue to green while reviewing different webpage designs, the feature extraction process might identify a "preference for green tones" as a color preference. Similarly, when a user repeatedly uses terms like "minimalist" and "white space" to describe their modification expectations, the system might extract a higher-level style preference feature like "leaning towards minimalism." These features constitute a digital profile of a user's unique design taste.

[0039] Understandably, because the intent to modify information is inherently multimodal, the feature extraction process also requires corresponding multimodal understanding techniques. For direct user modification operations, such as dragging, smearing, or selecting, the system analyzes the design elements affected by these operations, the direction and degree of their attribute changes, and summarizes the user's adjustment patterns. For natural language commands, natural language processing techniques are needed to parse the text, identify key actions, target objects, and modifiers, thereby understanding the semantic connotation of the user's intent.

[0040] It should be understood that this step is crucial in transforming explicit user feedback into a computer-interpretable implicit user model. The resulting user modification preference feature is an integrated, vectorized representation of user preferences. This feature vector encapsulates the personalized needs demonstrated by the user in this and previous interactions. It provides clear and quantitative guidance for the efficient and accurate optimization of design solutions in subsequent steps, and is the foundation for the entire system to achieve personalized adaptation and continuous learning.

[0041] Step S40: After each round of scheme presentation, adjust the design scheme according to the user's modification preference characteristics, and present the adjusted design scheme to the user until the optimization termination condition is met.

[0042] It's important to note that this step is a dynamic, multi-round, closed-loop process. The new solution generated after each adjustment is immediately shown to the user to obtain further feedback. This cycle of "show-feedback-adjustment-show again" allows the design to gradually approach its ideal state under continuous user guidance. Each round of interaction deepens the system's understanding of user preferences, making subsequent adjustments more precise and efficient. The process terminates when the user explicitly expresses satisfaction and confirms the solution.

[0043] Understandably, the user modification preference features obtained in the preceding steps are used to precisely and automatically adjust the design schemes displayed in the current round. This adjustment process is a creative regeneration based on a deep understanding of user preferences by the generative model. For example, when the user modification preference features indicate that the user prefers a "minimalist modern style," the model, when adjusting an interior layout scheme, will not only remove the complex decorative items specified by the user, but may also proactively adjust the lines of the furniture to be cleaner and unify the color scheme to a more elegant neutral tone, thereby achieving intelligent optimization from local modifications to overall style coordination.

[0044] It should be understood that one of the core characteristics of this step is the system's ability to continuously learn. This means that the system does not view each round of modification instructions in isolation, but rather accumulates and integrates the user's modification preference features extracted from each round of interaction, forming a continuously evolving, global user preference model. For example, in the first round of interaction, the user may have expressed a preference for a certain color; in subsequent rounds, the user may have adjusted the density of the layout multiple times. The system will integrate this preference information, which is scattered across different rounds and for different design dimensions, so that in subsequent automatic adjustments, it can generate a solution that meets both color requirements and layout preferences in one go, demonstrating an increasingly comprehensive understanding of the user's intent.

[0045] In a feasible embodiment, the multimodal interactive AI design and generation method further includes steps S501 to S503.

[0046] Step S501: Based on the initial and final design schemes of the user in the same design scheme generation process, obtain the user's global design modification preference information in this design generation.

[0047] It's important to note that after each design proposal is finalized and selected by the client, the system summarizes the entire design process, extracting the key evolutionary path and decision-making nodes from the initial proposal generation to the final version confirmation. The system automatically analyzes and records the user's feedback throughout all interaction rounds, the system's corresponding adjustment strategies, and the visual and parameter differences between each version. This process aims to build a structured, reusable design knowledge case study, deeply interpreting how user intent is gradually concretized into the final design outcome, thereby forming a valuable design experience archive and a record of the user's design preferences.

[0048] Understandably, the core value of this end-to-end summary lies in its powerful retrospective and learning capabilities. By analyzing the entire interaction history, the system can identify the design elements that users are most sensitive to, the most frequently modified types, and the most effective optimization directions. For example, the summary might reveal that the user's pursuit of a "sense of spaciousness" is a core demand that persists throughout, even though its expression varies in different rounds (e.g., initially requesting the removal of redundant furniture, then suggesting an adjustment to the wall color). The system will identify this as a key implicit need and elevate it to the strategic level to guide the design of similar scenarios in the future.

[0049] It should be understood that the findings of this summary will be transformed into highly structured knowledge and injected into the system's experience base. This knowledge can be used to optimize user models, enabling the system to be more forward-looking in serving these users in the future. Furthermore, after anonymization and summarization, the design logic and solutions from real-world projects can serve as valuable case studies, enriching the system's general design knowledge and improving its initialization solutions and intelligent adjustment capabilities when facing diverse design tasks, thereby achieving continuous evolution of the entire system.

[0050] Step S502: Update the user's design style vector based on the global design modification preference information and the multimodal design requirement information.

[0051] It's important to note that the purpose of this step is to deeply integrate the macro-level preferences extracted from specific design projects with the abstract requirements initially proposed by the user. The design style vector here is a quantitative model used to digitally represent the user's comprehensive preferences, which dynamically evolves with each successful design collaboration. The system compares and analyzes the global design modification preference information obtained (for example, in this project, the user ultimately showed a stable preference for low-saturation colors, minimalist layouts, and natural materials) with the initially input multimodal design requirement information (which may include text describing the user as "tranquil and Zen-like," or a reference image with a similar atmosphere) to identify the intrinsic connection between the initial requirements and actual modification behaviors, thus serving as the basis for updating the vector.

[0052] Understandably, the essence of this update process is to make the system's user model more accurate and comprehensive. It not only records the user's initial needs but, more importantly, learns the user's actual global modification preferences. The combination of these two aspects reveals the user's true taste more profoundly. For example, the user's initial need might be "fashionable and modern," while global modification preferences show that the final design heavily utilizes cool materials like metal and glass, as well as straight lines. Through this update, the system can concretize the vague concept of "fashionable and modern" into a stronger preference weight for "cool colors," "industrial-style materials," and "geometric lines," integrating this into the user's design style vector, making its connotation more precise. This significantly reduces the communication costs associated with repeated modifications. This mechanism ensures that the system can grow alongside the user, continuously accumulating consensus, and ultimately achieving a leap from passively responding to commands to proactively understanding and meeting the user's deep-seated needs.

[0053] Step S503: Adjust the design style weight matrix in the cross-modal design generation model according to the updated style vector so that the initial scheme generated subsequently is closer to the user's design style preference.

[0054] It's important to note that the updated user preferences are directly applied to the core engine of the solution generation. The design style weight matrix is ​​a key set of parameters within the cross-modal design generation model. It determines how the model transforms requirement information from different modalities, such as text and images, into combinations of specific design elements and rendering styles. Adjusting this matrix fundamentally fine-tunes the model's "aesthetic sense," ensuring that its output inherently bears the user's stylistic imprint.

[0055] Understandably, the adjustments here are not applied to a globally applicable model, but rather to a unique design style weight matrix created and maintained exclusively for each user. Every user in the system possesses a unique "design style record." The preferences learned in this project will be incrementally recorded into the user's style matrix as long-term, personalized knowledge, ensuring that their personal aesthetic preferences are continuously accumulated and inherited without interfering with data from other users.

[0056] It should be understood that this "one-to-one correspondence between users and style matrices" mechanism allows each subsequent passive response instruction to be transformed into an accumulation that actively carries and evolves each user's unique design style. In any future design task, each generated design solution will, from its very source, deeply integrate the user's historical preferences, making it possible for the initial solution to closely align with user expectations, thus greatly improving collaboration efficiency and user experience.

[0057] In this embodiment, a closed-loop system is constructed, encompassing initial generation, multi-round interactive optimization, and long-term preference learning. The system first generates an initial solution based on the user's multimodal design requirements. Then, by displaying the solution and capturing the user's modification intentions (including direct operations and natural language commands), it extracts the user's real-time modification preference features. Based on this, the solution is iteratively optimized until the user is satisfied. Crucially, after each design task is completed, the system reviews the global modification preferences and integrates them with the initial requirements to update the user's long-term design style vector. This vector is then used to fine-tune the user-specific design style weight matrix, thereby achieving personalized evolution of the model itself.

[0058] In summary, this technical solution constructs a closed loop of "interaction optimization - preference extraction - model update," which can dynamically adjust the user-specific style generation matrix by utilizing the real-time and long-term user preferences extracted in each design interaction. This makes the initial solution generated by the system increasingly aligned with the user's personal taste, thereby significantly reducing the number of solution iterations and improving design efficiency. At the same time, by continuously learning user preferences, it achieves a personalized design experience that becomes smarter with use.

[0059] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 In the multimodal interactive AI design and generation method, step S10 includes steps S101 to S104: Step S101: Classify the multimodal design requirement information according to data categories to obtain design reference data and user design instruction information.

[0060] It's important to note that the purpose of this step is to perform structured preprocessing of the raw, multimodal information input by the user. The data categorization here includes not only classification by file format but also functional differentiation based on the different roles these inputs play in the solution generation process. Specifically, design reference data mainly refers to materials that provide visual, stylistic, or pattern examples for the user's design intent, such as user-uploaded images of intent, similar style samples, or real-life photographs. These provide specific visual anchors and style imitation objects for the model. User design instructions, on the other hand, focus on containing user-initiated functional requirements, explicit modification instructions, or abstract style descriptions. For example, text instructions like "Change the main color scheme to warm colors" or voice instructions like "Add a bookshelf to the living room" directly express the user's proactive design intentions and specific constraints. This categorization aims to distribute input information carrying different functions to the corresponding processing modules of the model.

[0061] Understandably, this classification helps the system handle mixed information flows. For example, when a user simultaneously provides a picture of a modern minimalist restaurant (design reference data) and a voice message stating "I hope the table can accommodate six people" (user design instruction), the system identifies the picture as a style reference and the voice message as a functional constraint. The model can then prioritize extracting stylistic features such as lines, colors, and materials from the picture, while using "accommodating six people" as a hard requirement to generate or adjust the size and layout of the table, thus ensuring that the output solution meets functional requirements while maintaining the user's preferred aesthetic style. This approach reduces the requirements for the standardization of user input formats and improves the system's robustness in understanding complex and arbitrary input.

[0062] It should be understood that once the boundaries between "design reference data" and "user design instruction information" are clearly defined, the cross-modal design generation model can grasp the key points in the understanding phase. For example, it can use a specialized visual encoder to deeply interpret the implicit style in the reference data, and at the same time use natural language processing technology to accurately analyze the key parameters and intentions in the design instructions. This avoids the model understanding bias or contradictions in the generated results that may be caused by mixing information of different importance together, and provides a reliable structured input for generating high-quality initial design solutions.

[0063] Step S102: Obtain the design reference vector based on the domain knowledge base corresponding to the design reference data.

[0064] It's important to note that this step aims to transform unstructured design reference data into a standardized numerical form that is machine-recognizable and processable, namely, a "design reference vector." The core of this transformation lies in utilizing a pre-built domain knowledge base. This knowledge base is not simply a resource library, but a structured data system integrating professional knowledge within specific design domains (such as interior design, graphic design, and 3D design). Its content covers recognized design style feature libraries, color theory systems, material texture databases, spatial layout rules, and aesthetic evaluation standards. Through the professional rules and models of this knowledge base, the system can deeply analyze the deep features in the reference data. For example, it can accurately extract key dimensions from a furniture image, such as its style (e.g., "New Chinese style"), dominant color scheme, line usage characteristics, and material texture, quantifying these abstract features into a series of semantically meaningful values, ultimately forming a high-dimensional "design reference vector."

[0065] Understandably, the key role of this step is to achieve a precise quantitative expression of a user's vague aesthetic preferences. For example, in an interior design scenario, when a user provides an image of a "Mediterranean style" living room as reference data, the system will access an interior design knowledge base. The model in the knowledge base will perform multi-level analysis on the image, identifying a series of features such as "blue and white main color scheme," "arched elements," "natural stone texture," and "cotton and linen materials." Subsequently, the system maps these features to the corresponding dimensions defined in the knowledge base and assigns them corresponding intensity values, ultimately generating a comprehensive design reference vector. This vector is a language that the generative model can directly understand and calculate, providing a precise and calculable basis for subsequently generating design schemes that conform to this style tone.

[0066] It should be understood that the quality of generated design reference vectors directly depends on the completeness and professionalism of the domain knowledge base. A high-quality domain knowledge base ensures that the extracted feature vectors not only comprehensively capture the visual appearance of the reference data, but also reflect the underlying design logic and aesthetic principles that conform to industry standards. This gives the generated vectors better generalization ability and guiding significance.

[0067] Step S103: Perform semantic analysis based on the user design instruction information to obtain the design intent vector.

[0068] It's important to note that this step converts the design instructions expressed by the user in natural language and other forms into a structured, machine-understandable numerical representation—the design intent vector. This process relies on natural language processing technology for deep semantic analysis. Its goal is not simply to understand the literal meaning, but to accurately capture the user's core design intent, specific functional requirements, and potential preferences. The semantic analysis process typically includes word segmentation, part-of-speech tagging, and syntactic analysis of the text or transcribed speech. Furthermore, it uses a pre-trained language model to understand the deeper meaning of words in a specific design context, identify key entities (such as "sofa," "warm colors"), actions (such as "add," "replace"), and modifying conditions (such as "large," "near the window"). Finally, these semantic elements are deconstructed and mapped into a high-dimensional vector space, which is the design intent vector.

[0069] Step S104: Using the design intent vector and the design reference vector as conditional inputs, content is generated through cross-modal design generation model to obtain the initial design scheme.

[0070] It should be noted that fusing the design intent vector and the design reference vector into the input is not simply adding the values ​​of the two vectors together. Instead, the functional and spatial semantics carried by the design intent vector and the aesthetic semantics of style, color, and material contained in the design reference vector are understood separately. Then, the model searches for the correlation and compatibility between these two sets of semantic information in its internal high-dimensional feature space to establish a unified and comprehensive conditional representation, thereby constituting a generation constraint that guides the model in content generation and combines functional constraints with style guidance.

[0071] In a preferred embodiment, the cross-modal design generation model employs a hierarchical conditional control strategy. At the input stage, the model receives aligned and fused multimodal conditional vectors as global style guidance. Additionally, the model receives structured parameters extracted from design reference data (such as color codes, material indexes, and layout templates) as local constraints. This hierarchical control ensures that the generation process grasps both the overall style direction and precisely meets specific functional requirements. Furthermore, to enhance the professional compliance of the generated results, this invention employs a two-stage training strategy: the first stage uses a large-scale cross-domain design dataset to pre-train the base model to acquire general design capabilities; the second stage performs expert knowledge fine-tuning for specific vertical domains (such as interior design and industrial design). The fine-tuning data includes modification trajectories annotated by professional designers and matched multimodal inputs, enabling the model to learn the mapping rules from initial requirements to professional solutions. An adversarial loss term is specifically added during the training process to ensure that the generated solutions meet user needs while also satisfying industry standards and physical feasibility.

[0072] Understandably, the design intent vector representing user functional needs and the design reference vector representing stylistic aesthetics, obtained from previous steps, are fused and used as common constraints, then input into a specially trained cross-modal design generation model. This generation model is typically based on a deep learning architecture. By receiving the conditional information encoded by these two vectors, it performs deep feature alignment and semantic fusion internally, and based on the knowledge learned from massive amounts of design data, it generates a completely new design outcome that meets all the constraints—the initial design scheme.

[0073] It should be understood that this step achieves a calculable transformation from abstract requirement parameters to a concrete design solution. For example, the design intent vector includes instructions such as "a kitchen countertop with an L-shaped layout is needed" and "space should be reserved for a double-door refrigerator," while the design reference vector defines aesthetic characteristics of "modern industrial style" (such as exposed pipes and dark-toned metal materials). The cross-modal generative model will consider both aspects of information simultaneously. It will not simply piece together elements, but will creatively generate a 3D model or floor plan of the kitchen that satisfies the requirements of the L-shaped countertop and refrigerator space layout, while strictly adhering to the characteristics of industrial style in terms of overall visual effects (such as color matching, material selection, and lighting design). The model ensures the organic unity of functionality and aesthetics in the generated solution, rather than their separation.

[0074] This embodiment categorizes the multimodal design requirements provided by the user into data categories (function and style), and transforms them into precise design intent vectors and design reference vectors respectively. Then, the two are used as collaborative conditions and input into the cross-modal generation model, driving the model to generate an initial design scheme that meets the user's complex needs based on a deep integration of functional constraints and aesthetic features.

[0075] In summary, this embodiment, through precise classification and vectorization representation strategies based on data function categories, enables the system to clearly separate and strengthen the processing of specific user functional instructions and abstract style preferences. This ensures that core requirements are not overwhelmed by vague reference information. Furthermore, by employing a cross-modal generation model for deep semantic fusion and conditional generation of the two types of vectors, the final output design not only faithfully reproduces the functional requirements (such as spatial layout and item configuration) in the user's instructions but also naturally incorporates the desired artistic style (such as color, material, and form). This effectively resolves the inherent contradiction in existing technologies where style transfer and functional implementation are difficult to reconcile. This process significantly reduces reliance on the standardization of user input and enhances the system's ability to understand and execute complex and arbitrary instructions, thereby improving the overall quality of design generation and user satisfaction.

[0076] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 In the multimodal interactive AI design and generation method, step S20 includes steps S201 to S204: Step S201: If the modification intent information is a user-initiated editing operation, then determine the design element to be operated and the operation type based on the user-initiated editing operation.

[0077] Step S202: Based on the design element being operated on, the operation type, and the modification case library, reason to obtain the user's implicit intention to modify.

[0078] It's important to note that the system first needs to determine whether the user's intended modification constitutes a proactive editing operation. A user-proactive editing operation refers to the user's direct intervention in specific components of the initially generated design scheme, such as dragging a furniture model on the visual interface, deleting a decorative element, or adjusting the color of a certain area. The core task of this step is to accurately identify the specific target of such operations, i.e., the manipulated design element, and the nature of the user's action, i.e., the operation type. For example, when a user drags a sofa model from the center of the living room to a position near the window, the "sofa" is the manipulated design element, and "moving its position" is the operation type. This step completes the transformation from the user's intuitive interactive behavior to machine-parseable structured information (elements and operations).

[0079] Understandably, the core idea of ​​this step is to combine the identified design elements and operation types with a pre-built modification case library containing a wealth of design modification experience and common sense. Through inference mechanisms, the system deciphers the user's implicit intentions, which are not explicitly expressed. The modification case library stores a wealth of design adjustment cases and their corresponding intent explanations. The system infers by comparing the similarity between the current operation and historical cases in the library. For example, if a user performs the operation "replace the dark wood dining table with a light-colored glass dining table," the system will identify the operation element as "dining table" and the operation type as "replace material." By querying the case library, the system may find that similar operations are often associated with intents such as "improving the brightness of the space" or "creating a modern minimalist feel," thus inferring that the user's implicit intent might be to make the dining area appear more open and stylish, rather than simply replacing a piece of furniture. This approach allows the system to understand the complex aesthetic preferences or functional optimization needs that may be hidden behind a user's simple operation, providing crucial directional guidance for generating more accurate modification solutions and enhancing the intelligence and efficiency of human-computer interaction.

[0080] Step S203: Construct a user potential preference vector based on the modified implicit intent.

[0081] It should be noted that the implicit user modification intent obtained from the previous step of reasoning is transformed into a numerical representation that can be directly used by the generative model, namely the user latent preference vector. This vector can quantify the deep tendencies that users do not explicitly state but reveal through their operational behavior, such as implicit preferences for specific styles, functions or aesthetic attributes.

[0082] Understandably, the construction process involves feature extraction and quantification of implicit intentions, mapping them into a vector space defined with various design attribute dimensions. For example, if the inferred implicit intention is "to enhance the sense of spatial transparency," the system will decompose this intention into quantifiable design attributes such as "increasing natural light," "using light colors," and "reducing visual obstruction," assigning corresponding intensity values ​​to these attributes, and finally synthesizing a preference vector that can be used to guide subsequent adjustments to the design.

[0083] Step S204: Use the user's potential preference vector as the user's modified preference feature.

[0084] It should be noted that the system formally defines the generated user potential preference vector as the user's modified preference feature, which is used to accurately guide the generation model in the subsequent scheme optimization process.

[0085] Understandably, establishing a latent preference vector as a feature for modifying preferences aims to create a stable and efficient information transmission mechanism. For example, this vector might contain quantified preferences for multiple dimensions such as "color brightness," "material reflectivity," and "layout openness." When the model generates new solutions, these feature values ​​serve as strong constraints, ensuring that the output aligns with the user's deeper expectations in the corresponding dimensions.

[0086] In a feasible embodiment, step S20 further includes steps S211 to S214.

[0087] Step S211: If the modification intent information is a user's natural language modification instruction, then perform deep semantic analysis based on the user's natural language modification instruction to determine the modification content and target style information.

[0088] It should be noted that when a user makes a modification request through natural language instructions, the core task of this step is to perform deep semantic analysis on the instruction. The purpose is to accurately identify and extract two types of key information contained in the instruction: first, the specific content of the modification, that is, which elements or attributes of the design scheme need to be adjusted; second, the target style information, that is, the aesthetic style or atmosphere that the user expects the overall scheme to present after modification.

[0089] Understandably, deep semantic analysis needs to overcome the ambiguity and diversity of natural language, transforming colloquial descriptions into structured, unambiguous design parameters. For example, a user instruction like "Make the living room brighter and more modern" can be analyzed to determine that the modification is to increase the space's brightness, with a target style of modern. This process may leverage natural language processing technology to analyze the semantics, syntactic structure, and contextual relationships of words, mapping them to a predefined design knowledge base to accurately interpret the user's true design needs. Therefore, this step requires converting the user's subjective and emotional descriptions into a set of objective and quantifiable design constraints. By clearly defining the modification content and target style information, the system provides clear and explicit input for subsequent solution generation and optimization, ensuring the directionality and accuracy of design adjustments and improving user experience and system intelligence.

[0090] Step S212: Based on the modified content, determine the editing tool and the initial editing operation sequence.

[0091] It should be noted that, based on the specific modifications parsed in the previous step, the system automatically determines the editing tools required to perform the modifications and plans the initial sequence of editing operations. The editing tools referred to here are functional modules within the software system used to implement specific design changes, such as color adjustment tools, layout reconstruction tools, or model replacement tools. The initial sequence of editing operations refers to a series of ordered steps initially devised to achieve the modification goals.

[0092] Understandably, this step establishes a mapping mechanism from abstract modification requirements to specific operational instructions. For example, if the modification is "change the wall color from light yellow to light gray," the system will select a color editing tool and generate an initial operation sequence including steps such as "selecting the target wall," "calling the color palette," and "applying the new color value." This process relies on the system's internal pre-built design rules and operation logic library to ensure that the selected tools and operations accurately match the user's intent.

[0093] It should be understood that this step translates semantic modification instructions into a computer-executable action flow, providing a clear technical path for specific changes to the design scheme. This not only ensures the accuracy and effectiveness of modification operations but also lays a solid foundation for achieving automated and intelligent design adjustments, significantly improving the efficiency and reliability of scheme iteration.

[0094] Step S213: Correct the initial editing operation sequence according to the design constraint information to obtain an optimized editing operation sequence.

[0095] It should be noted that before executing any specific modification operations, design constraint information needs to be introduced to review and revise the initial editing operation sequence. The design constraint information mentioned here refers to various limiting conditions that affect the rationality and feasibility of the design, such as the physical rules of spatial layout, the compatibility of material properties, safety standards of design specifications, and specific user preferences. The purpose is to ensure that the operation results executed by the initial sequence not only meet the modification intent but also conform to the actual design logic and constraints, thereby generating a technically reasonable and user-satisfactory optimized operation sequence.

[0096] In one embodiment, the design constraint information includes at least one of functional constraints, structural constraints, and compliance constraints.

[0097] It's important to note that functional constraints focus on whether a design solution meets its intended purpose and performance requirements. For example, in an office space design, it's essential to ensure adequate lighting and good ventilation in workstation areas, and that corridor widths meet fire evacuation requirements. These conditions ensuring the design's usability fall under functional constraints. Structural constraints involve the physical stability and material properties of the design object itself. For instance, in interior design, removing load-bearing walls or excessively adding weight to partition walls on floors may exceed the building's structural bearing capacity. Such physical limitations related to building safety constitute structural constraints. Compliance constraints refer to the laws, regulations, industry standards, or specific norms that a design solution must follow. For example, restroom designs must comply with national accessibility design standards, and the decoration materials used in commercial spaces must meet the corresponding fire resistance ratings. These externally mandated regulations constitute compliance constraints.

[0098] Understandably, functional, structural, and compliance constraints differ across design fields. For instance, when designing a new smartphone, functional constraints require the device to ensure clear call quality, long battery life, and a smooth touch experience. This dictates the antenna layout, battery capacity selection, and processor tuning during the design process. Structural constraints involve the physical stacking of internal components, such as the motherboard size, camera module thickness, and screen-to-casing assembly tolerances. Any modifications to the external dimensions or internal structure must be made within this framework to avoid component interference or insufficient structural strength. Compliance constraints require the device to pass electromagnetic radiation safety certifications and wireless communication network access licenses in various countries, and restrict the use of hazardous substances. These mandatory standards determine the key parameters for material selection and circuit design.

[0099] It should be understood that this classification method systematically covers the main dimensions affecting the rationality of a design. Functional constraints, starting from user activities and spatial efficiency, ensure the practical value of the design; structural constraints, starting from the inherent properties of the physical entity, guarantee the engineering feasibility of the design; and compliance constraints, starting from social consensus and legal regulations, ensure the legality and standardization of the design. Together, they constitute a multi-layered constraint system, enabling the system to consider and weigh different aspects when revising the initial editing operation sequence.

[0100] In one embodiment, the step of modifying the initial editing operation sequence according to design constraint information to obtain an optimized editing operation sequence includes: performing a feasibility check using a predefined design rule engine based on the design constraint information and the initial editing operation sequence; identifying operation steps in the initial editing operation sequence that do not conform to the design constraints based on the check result; adjusting the parameters of the operation steps that do not conform to the design constraints based on the domain knowledge integrated by the design rule engine; and generating an optimized editing operation sequence that satisfies all design constraints based on the parameter-adjusted operation steps.

[0101] It should be noted that, in this embodiment, based on the design constraint information, a predefined design rule engine performs feasibility verification on the initial sequence. This design rule engine is a core computational module that embeds domain-specific knowledge and logical judgment rules. Its function is to transform abstract design constraints (such as "load-bearing walls must not be demolished") into executable verification logic and systematically scan each operation step in the initial sequence to evaluate whether it meets all preset constraints.

[0102] Understandably, after the feasibility verification is completed, the system will accurately locate the operational steps in the initial sequence that do not conform to design constraints based on the verification results. For example, if the initial sequence includes the operation of "expanding the window width to three meters," and the rule engine determines based on structural constraints that this dimension exceeds the maximum width requirement of the wall between windows, then this step will be identified as non-compliant. Subsequently, the system will invoke the domain knowledge integrated in the design rule engine to adjust the parameters of these problematic steps. Domain knowledge may include a library of alternative solutions, optimization algorithms, or empirical parameters. For example, regarding the window issue mentioned above, the engine may automatically adjust its width to the maximum value allowed by the current wall structure, or recommend a solution using multiple combined windows.

[0103] It should be understood that after parameter adjustments are completed, the system will integrate all the verified original steps with the adjusted new steps, reassemble them, and generate an optimized editing sequence that meets all design constraints. This process automates the transformation from an initial, potentially flawed expression of intent to a final, professionally sound, and feasible solution. These interconnected steps ensure the systematic, automated, and intelligent nature of the correction process, significantly improving the accuracy and reliability of design assistance.

[0104] Step S214: Based on the target style information, editing tools, and optimized editing operation sequence, obtain the user's modification preference characteristics.

[0105] It's important to note that the core of this step lies in the deep integration and abstract analysis of the various concrete design data generated in the preceding steps. Target style information indicates the aesthetic direction of the design, the choice of editing tools reflects the specific technical path the user takes to achieve modifications, and the optimized editing operation sequence precisely records a series of substantive operations taken to achieve a specific design goal. Combining and analyzing these three elements allows us to uncover the user's potential and stable design tendencies and habits.

[0106] Understandably, the formation of user modification preference characteristics is an information extraction process. By performing correlation analysis on the three types of information mentioned above, the system can identify user patterns and tendencies in design behavior, and the resulting user modification preference characteristics have significant application value. It is not only a summary of the current design session but also serves as a key basis for personalized design assistance. This characteristic can be recorded by the system and used to optimize future interaction experiences. When a user initiates a new design request, the system can pre-recommend tools or style solutions that match their habits based on historical preference characteristics, thereby achieving intelligent and personalized design processes and improving the efficiency and satisfaction of human-computer collaboration. This step is a crucial link in achieving continuous learning and adaptive capabilities in this method.

[0107] like Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the entire process generated for the design scheme of an embodiment of the present invention.

[0108] Understandably, this process is a user-centric, data-driven intelligent design generation and iterative optimization system. Its core process begins with the user intent input stage: users express their initial design ideas and needs through diverse input interfaces, such as natural language descriptions, sketches, uploaded reference images, or parameter adjustment sliders. Upon receiving this potentially ambiguous or cross-modal intent information, the system first initiates an intelligent parsing and preliminary generation stage. In this stage, the system deeply integrates an advanced cross-modal generation model and a structured domain knowledge base. The generation model is responsible for understanding the deep semantics of the user input and transforming it into visual design elements, while the domain knowledge base ensures that the generated solution conforms to industry standards, physical feasibility, and basic aesthetic principles, thus outputting an initial design solution that both aligns with the user's initial intent and possesses professional rationality. The preliminary solution is then presented to the user for review and interactive editing. All user operations, whether direct drag-and-drop modifications, attribute adjustments, or selection of system-provided options, are captured in real time. This interactive data, along with the user's explicit feedback, is fed into a user preference database and a personalized preference learning module. This module utilizes machine learning algorithms to continuously extract and update users' unique preference characteristics (such as a preference for specific styles or a particular function) from their historical behavior, forming a dynamically evolving user profile. Based on this profile, the system can provide increasingly precise and detailed editing suggestions in subsequent interactions, achieving deep personalization and customization of the design. Users can save the output once they are satisfied with the optimized design. Simultaneously, the system proactively invites users to provide feedback (such as satisfaction ratings or specific modification suggestions). This feedback information is then fed back into the system database to continuously optimize the generation model and preference learning algorithm, thus forming a self-improving positive cycle. In summary, this flowchart depicts not only a one-time design generation but also a design ecosystem that emphasizes deep user participation, achieves personalized customization through continuous interaction, and possesses self-evolving capabilities.

[0109] In this embodiment, the system executes corresponding parsing paths based on the type of modification intent information: for proactive editing operations, the system identifies the manipulated element and operation type, infers the implicit intent based on the modification case library, and constructs a potential preference vector; for natural language instructions, deep semantic parsing is performed to determine the modification content and target style, and then the editing operation sequence is planned and optimized. Finally, the solution integrates the target style, editing tools, and optimized operation sequence to extract user modification preference features that can quantify the user's deep-seated tendencies.

[0110] In summary, this embodiment transforms different types of potentially ambiguous user inputs into structured operation sequences and introduces design constraint information and a rule engine for feasibility verification and optimization, ensuring the rationality and feasibility of the modification scheme. It achieves accurate capture and digital representation of implicit design preferences behind explicit user instructions, and the user modification preference characteristics it produces provide the system with continuous learning and personalized adaptation capabilities. Therefore, this scheme can significantly improve the intelligence level of computer-aided design systems, making the generation and adjustment of design schemes more in line with the real needs of users, effectively reducing the number of repeated modifications, thereby improving overall design efficiency and user satisfaction.

[0111] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multimodal interactive AI design and generation method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0112] This application also provides a multimodal interactive AI design generation device, please refer to... Figure 5 The multimodal interactive AI design generation device includes: The design scheme generation module 10 is used to input multimodal design requirement information into the cross-modal design generation model to obtain an initial design scheme; Modification intent acquisition module 20 is used to display the design scheme and obtain the user's modification intent information for the design scheme; Modification intent analysis module 30 is used to extract features from the modification intent information to obtain user modification preference features; The design scheme optimization module 40 is used to adjust the design scheme according to the user's modification preference characteristics after each round of scheme display, and then display the adjusted design scheme to the user until the optimization termination condition is met.

[0113] In one embodiment, the design scheme generation module 10 is further configured to classify the multimodal design requirement information according to data categories to obtain design reference data and user design instruction information; obtain a design reference vector based on the domain knowledge base corresponding to the design reference data; perform semantic analysis based on the user design instruction information to obtain a design intent vector; and use the design intent vector and the design reference vector as conditional inputs to generate content through a cross-modal design generation model to obtain the initial design scheme.

[0114] In one embodiment, the modification intent analysis module 30 is further configured to: if the modification intent information is a user-initiated editing operation, determine the manipulated design element and operation type based on the user-initiated editing operation; infer the user's implicit modification intent based on the manipulated design element, operation type, and modification case library; construct a user latent preference vector based on the implicit modification intent; and use the user latent preference vector as the user's modification preference feature.

[0115] In one embodiment, the modification intent analysis module 30 is further configured to: if the modification intent information is a user's natural language modification instruction, perform deep semantic analysis based on the user's natural language modification instruction to determine the modification content and target style information; determine the editing tool and initial editing operation sequence based on the modification content; modify the initial editing operation sequence based on design constraint information to obtain an optimized editing operation sequence; and obtain user modification preference features based on the target style information, editing tool, and optimized editing operation sequence.

[0116] In one embodiment, the modification intent analysis module 30 is further configured to perform feasibility verification using a predefined design rule engine based on the design constraint information and the initial editing operation sequence; identify operation steps in the initial editing operation sequence that do not conform to the design constraints based on the verification result; adjust the parameters of the operation steps that do not conform to the design constraints based on the domain knowledge integrated by the design rule engine; and generate an optimized editing operation sequence that satisfies all design constraints based on the operation steps with adjusted parameters.

[0117] In one embodiment, the modification intent acquisition module 20 is further configured to obtain the user's global design modification preference information in the current design generation based on the initial and final design schemes of the user in the same design scheme generation process; update the user's design style vector based on the global design modification preference information and the multimodal design requirement information; and adjust the design style weight matrix in the cross-modal design generation model based on the updated style vector so that the subsequently generated initial scheme is closer to the user's design style preference.

[0118] This embodiment fundamentally changes the traditional one-way, static working mode of AI design tools by constructing a dynamic perception and continuous optimization interactive generation closed loop. This application uses the user's modification intent information in multiple iterations as key input, and through feature extraction technology, quantifies and models the user's modification preferences—features that even the user themselves may not be able to clearly express. This mechanism enables the AI ​​model to transcend the limitations of a single interaction, gradually understanding and adapting to the user's unique style and deep intent. Because the model can intelligently adjust subsequent solutions based on historical feedback, users no longer need to get stuck in the inefficient cycle of modification prompts and regeneration, significantly reducing the number of iterations required to achieve a satisfactory solution, accelerating the design process, and significantly improving design quality and user satisfaction.

[0119] The multimodal interactive AI design generation apparatus provided in this application, employing the multimodal interactive AI design generation method described in the above embodiments, can solve the technical problem of how to enable generative AI tools to dynamically understand users' design styles and intentions, and to enable them to continuously learn and adapt to users' personalized design preferences. Compared with the prior art, the beneficial effects of the multimodal interactive AI design generation apparatus provided in this application are the same as those of the multimodal interactive AI design generation method described in the above embodiments, and other technical features in the multimodal interactive AI design generation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0120] This application provides a multimodal interactive AI design generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multimodal interactive AI design generation method in Embodiment 1 above.

[0121] The following is for reference. Figure 6 This document illustrates a structural schematic diagram suitable for implementing the multimodal interactive AI design generation device of the embodiments of this application. The multimodal interactive AI design generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6The multimodal interactive AI design generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0122] like Figure 6 As shown, the multimodal interactive AI design generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multimodal interactive AI design generation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multimodal interactive AI design generation device to communicate wirelessly or wiredly with other devices to exchange data. Although a multimodal interactive AI design generation device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0123] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0124] The multimodal interactive AI design generation device provided in this application, employing the multimodal interactive AI design generation method described in the above embodiments, solves the technical problem of how to enable generative AI tools to dynamically understand users' design styles and intentions, and to enable them to continuously learn and adapt to users' personalized design preferences. Compared with the prior art, the beneficial effects of the multimodal interactive AI design generation device provided in this application are the same as those of the multimodal interactive AI design generation method described in the above embodiments, and other technical features in this multimodal interactive AI design generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0125] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multimodal interactive AI design and generation method described in the above embodiments.

[0128] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0129] The aforementioned computer-readable storage medium may be included in the multimodal interactive AI design generation device; or it may exist independently and not assembled into the multimodal interactive AI design generation device.

[0130] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the multimodal interactive AI design generation device, the multimodal interactive AI design generation device: inputs multimodal design requirement information into a cross-modal design generation model to obtain an initial design scheme; displays the design scheme and obtains user modification intention information; extracts features from the modification intention information to obtain user modification preference features; after each round of scheme display, adjusts the design scheme according to the user modification preference features, and displays the adjusted design scheme to the user until the optimization termination condition is met.

[0131] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0134] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described multimodal interactive AI design generation method. This addresses the technical problem of how generative AI tools can dynamically understand users' design styles and intentions, enabling them to continuously learn and adapt to users' personalized design preferences. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multimodal interactive AI design generation method provided in the above embodiments, and will not be repeated here.

[0135] The computer program product provided in this application can solve the technical problem of multimodal interactive AI design generation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multimodal interactive AI design generation method provided in the above embodiments, and will not be repeated here.

[0136] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A multimodal interactive AI design and generation method, characterized in that, The multimodal interactive AI design and generation method includes: The multimodal design requirements information is input into the cross-modal design generation model to obtain the initial design scheme; The design scheme is presented to obtain information on the user's intention to modify the design scheme; Feature extraction is performed on the modification intent information to obtain user modification preference features; After each round of scheme presentation, the design scheme is adjusted based on the user's modification preference characteristics, and the adjusted design scheme is presented to the user until the optimization termination condition is met.

2. The multimodal interactive AI design and generation method according to claim 1, characterized in that, The step of inputting multimodal design requirement information into the cross-modal design generation model to obtain an initial design scheme includes: The multimodal design requirements information is classified according to data categories to obtain design reference data and user design instruction information; Based on the domain knowledge base corresponding to the design reference data, a design reference vector is obtained; Semantic analysis is performed based on the user design instruction information to obtain the design intent vector; Using the design intent vector and the design reference vector as conditional inputs, content is generated through a cross-modal design generation model to obtain the initial design scheme.

3. The multimodal interactive AI design and generation method according to claim 1, characterized in that, The step of extracting features from the modification intent information to obtain user modification preference features includes: If the modification intent information is a user-initiated editing operation, then the design element to be operated and the operation type are determined based on the user-initiated editing operation; Based on the manipulated design elements, operation types, and modification case library, the user's implicit intent to modify is derived. Based on the modified implicit intent, construct a user potential preference vector; The user's potential preference vector is used as the user's modified preference feature.

4. The multimodal interactive AI design and generation method according to claim 1, characterized in that, The step of extracting features from the modification intent information to obtain user modification preference features also includes: If the modification intent information is a user's natural language modification instruction, then deep semantic analysis is performed based on the user's natural language modification instruction to determine the modification content and target style information; Based on the modifications, determine the editing tools and the initial editing operation sequence; The initial editing operation sequence is corrected based on the design constraint information to obtain an optimized editing operation sequence; Based on the target style information, editing tools, and optimized editing operation sequence, user modification preference characteristics are obtained.

5. The multimodal interactive AI design and generation method according to claim 4, characterized in that, The design constraint information includes at least one of functional constraints, structural constraints, and compliance constraints; The step of revising the initial editing operation sequence based on design constraint information to obtain an optimized editing operation sequence includes: Based on the design constraint information and the initial editing operation sequence, a feasibility check is performed using a predefined design rule engine; Based on the verification results, identify the operation steps in the initial editing operation sequence that do not conform to the design constraints; Based on the domain knowledge integrated into the design rule engine, the parameters of the operation steps that do not conform to the design constraints are adjusted; Based on the adjusted operation steps, an optimized editing operation sequence that satisfies all design constraints is generated.

6. The multimodal interactive AI design and generation method according to any one of claims 1 to 5, characterized in that, The multimodal interactive AI design and generation method further includes: Based on the user's initial and final design schemes during the same design scheme generation process, the user's global design modification preference information during this design generation is obtained; Update the user's design style vector based on the global design modification preference information and the multimodal design requirement information; Based on the updated style vector, the design style weight matrix in the cross-modal design generation model is adjusted so that the subsequently generated initial scheme is closer to the user's design style preference.

7. A multimodal interactive AI design generation device, characterized in that, The multimodal interactive AI design generation device includes: The design scheme generation module is used to input multimodal design requirement information into the cross-modal design generation model to obtain the initial design scheme; The modification intent acquisition module is used to display the design scheme and obtain the user's modification intent information for the design scheme; The modification intent analysis module is used to extract features from the modification intent information to obtain user modification preference features; The design scheme optimization module is used to adjust the design scheme according to the user's modification preference characteristics after each round of scheme display, and then display the adjusted design scheme to the user until the optimization termination condition is met.

8. A multimodal interactive AI design generation device, characterized in that, The multimodal interactive AI design generation device includes: a memory, a processor, and a multimodal interactive AI design generation program stored in the memory and executable on the processor, wherein the multimodal interactive AI design generation program is configured to implement the multimodal interactive AI design generation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a multimodal interactive AI design generation program, which, when executed by a processor, implements the multimodal interactive AI design generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multimodal interactive AI design and generation method as described in any one of claims 1 to 6.

Images