3D design auxiliary method based on multi-modal input and intelligent optimization

Abstract

The invention discloses a 3D design assisting method based on multi-modal input and intelligent optimization. The method comprises the steps that to-be-processed text data and to-be-processed image data are acquired; respectively carrying out feature extraction on the to-be-processed text data and the to-be-processed image data to obtain text features and image features; the text features comprise size description, functional description, material description and structure description; the image features comprise image structure features; performing feature fusion on the text features and the image features to obtain fusion features; constructing a plurality of 3D models based on the fusion features; and constructing a knowledge graph based on a preset design rule and the text features, evaluating each 3D model based on the knowledge graph, and determining an optimal 3D model based on an evaluation result. According to the technical scheme, the generation efficiency and accuracy of the 3D model are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D design assistance, and in particular relates to a 3D design assistance method based on multi-modal input and intelligent optimization. Background Art

[0002] In current 3D design, how to efficiently and accurately generate models that meet user needs is still a problem that needs to be solved urgently. Traditional methods mostly rely on single modal input (such as text or images), which has limitations when dealing with complex design requirements: text input is difficult to accurately express design intent, and image input is limited by conditions such as quality, angle and lighting, resulting in unstable recognition results. In addition, the model evaluation standards lack consistency, making it difficult to ensure the quality of the generated model and the design efficiency. The present invention combines multimodal input of text and images, and adopts multimodal learning technology and knowledge graphs to significantly improve the flexibility and accuracy of 3D model generation, and optimize the intelligence and efficiency of the model screening process. Summary of the invention

[0003] The purpose of the present invention is to provide a 3D design assistance method based on multimodal input and intelligent optimization to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a 3D design assistance method based on multimodal input and intelligent optimization, comprising:

[0005] Obtaining text data to be processed and image data to be processed;

[0006] Extracting features from the text data to be processed and the image data to be processed respectively to obtain text features and image features; the text features include size description, functional description, material description and structure description; the image features include image structure features;

[0007] Performing feature fusion on the text feature and the image feature to obtain a fused feature;

[0008] constructing a plurality of 3D models based on the fused features;

[0009] A knowledge graph is constructed based on preset design rules and the text features, each of the 3D models is evaluated based on the knowledge graph, and an optimal 3D model is determined based on the evaluation results.

[0010] Optionally, the process of acquiring the text features specifically includes:

[0011] The text data to be processed is parsed based on the GPT-4O model to obtain the text features.

[0012] Optionally, the image structural features include edge features, texture features and shape features of objects in the image data to be processed.

[0013] Optionally, the process of acquiring the image structural features specifically includes:

[0014] Inputting the image data to be processed into the CLIP-ViT model, extracting the contour and boundary of the object in the image data to be processed, and obtaining the edge feature;

[0015] Encoding the local pixel difference of the image data to be processed according to a local binary pattern algorithm to obtain the texture feature;

[0016] The shape feature is obtained by performing frequency domain representation on the contour and boundary of the object based on Fourier transform.

[0017] Optionally, the fusing the text features and the image features specifically includes:

[0018] The image features and the text features are projected into a unified dimensional space through an adaptation matrix, and the image features and the text features are fused based on a multi-head attention mechanism to obtain the fused features.

[0019] Optionally, constructing a plurality of 3D models based on the fusion features specifically includes:

[0020] The fusion features are input into the Point-E model, a number of initial models are generated by using point cloud technology, and each initial model is rendered and optimized to obtain a number of 3D models.

[0021] Optionally, the constructing of a knowledge graph based on preset design rules and the text features specifically includes:

[0022] The preset industry basic design rules, size descriptions and the functional descriptions are converted into screening rules, and a knowledge graph is constructed based on the screening rules.

[0023] Optionally, the evaluating each of the 3D models based on the knowledge graph specifically includes:

[0024] Each of the 3D models is scored based on the knowledge graph, and the 3D model with the highest score is selected as the optimal 3D model.

[0025] The technical effects of the present invention are:

[0026] The present invention provides a method that combines multimodal learning technology, pre-trained models and knowledge graphs to effectively fuse text and image inputs, and through feature fusion, improves adaptability to complex design requirements and enhances the comprehensive performance of the model. Building a knowledge graph that supports dynamic updates enables the system to flexibly integrate the design rules and functional requirements required by users and guide the model screening process. Combining the above advantages, this method significantly improves the generation efficiency and accuracy of 3D models, and provides strong intelligent support for design innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is a flow chart of a 3D design assistance method based on multi-modal input and intelligent optimization in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a 3D model generation process driven by multimodal input in an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the fusion of image and text feature vectors in the feature fusion process in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the embodiment of the present invention in which GPT-4O and the knowledge graph jointly assist in model screening. DETAILED DESCRIPTION

[0033] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0034] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.

[0035] Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art generally understood by the present invention. Although the present invention describes only preferred methods, any method similar or equivalent to that described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.

[0036] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0037] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] Embodiment 1

[0040] like Figure 1 - Figure 4 As shown, in this embodiment, a 3D design assistance method based on multimodal input and intelligent optimization is provided, including: obtaining text data to be processed and image data to be processed; performing feature extraction on the text data to be processed and the image data to be processed respectively to obtain text features and image features; the text features include size description, functional description, material description and structure description; the image features include image structure features; performing feature fusion on the text features and the image features to obtain fusion features; constructing several 3D models based on the fusion features; constructing a knowledge graph based on preset design rules and the text features, evaluating each of the 3D models based on the knowledge graph, and determining the optimal 3D model based on the evaluation results.

[0041] This embodiment discloses a 3D design assistance method based on multimodal input and intelligent optimization, including the following steps: S100: multimodal input processing; parse text descriptions through pre-trained GPT-4O, extract design intent and functional requirements, and use CLIP-ViT to extract image structure features. S200: multimodal fusion; use BLIP-2 to fuse text and image features, and generate a unified feature vector as 3D generation input. S300: preliminary 3D model generation; use Point-E and other models to generate 10 preliminary 3D models, and use Blender for preliminary rendering to ensure that they meet the design intent. S400: demand analysis and model screening; use GPT-4O to parse user descriptions, convert semantics into screening rules, build knowledge graphs to assist screening, use semantic similarity analysis to evaluate the match between models and requirements, and score and screen the models with the highest scores according to design rules. This embodiment improves the efficiency and accuracy of 3D design and supports design innovation through multimodal input fusion and intelligent optimization.

[0042] This embodiment provides a 3D design assistance method and system based on multimodal input and intelligent optimization. The method integrates text and image input and uses advanced multimodal learning technology to achieve efficient 3D model generation and screening. Specifically, the method includes the following steps:

[0043] like Figure 1 As shown, this embodiment provides a 3D design assistance method based on multimodal input and intelligent optimization. The method combines image and text input, uses multimodal fusion technology and intelligent generation algorithm to achieve efficient 3D model generation and optimization. The specific steps are as follows:

[0044] Step S100: Multimodal input processing: Through the processing of multimodal input image and text data, the system can simultaneously obtain visual features and language descriptions. Image input is processed by the CLIP-ViT model to extract structural features in the image, and text input is parsed by GPT-4O to obtain design intent and functional requirements. Specifically, the following steps S101-S102 are included:

[0045] Step S101: Image input; use the CLIP-ViT model to extract features from the input image. The main goal is to obtain geometric features and structural information from the image for subsequent multimodal fusion and 3D model generation. The extraction of image features specifically includes the following aspects:

[0046] S101-1: Edge features: The CLIP-ViT model first analyzes the edge information in the image and extracts the outline and boundary of the object through convolution operation. These edge features are represented by vectors as E(e1,e2,...,e n ), where e nRepresents the edge strength value at the nth pixel. Edge features are the core information that describes the geometric shape of an object.

[0047] S101-2: Texture features: In addition to edge information, CLIP-ViT also supports the extraction of image surface texture features. The local binary pattern (LBP) algorithm is used to encode the local pixel differences of the image and generate a texture vector T (t1, t2, ..., t m ), where t m Indicates the texture mode within the local window.

[0048] S101-3: Shape features: By further analyzing the edges and contours of the image, the overall geometric shape features of the object are extracted. The object contour is represented in the frequency domain using Fourier transform to obtain the shape descriptor S (s1, s2, ..., s i ), where s i The frequency components that represent the outline of an object.

[0049] Furthermore, the image features extracted by CLIP-ViT can be expressed as:

[0050] G={E,T,S}

[0051] Step S102: Text input: GPT-4O parses the text input provided by the user, mainly extracting functional requirements and design constraints related to 3D model generation from the design description. The specific text feature extraction process is as follows:

[0052] S102-1: Size description: GPT-4O parses size-related information in the text, such as length, width, height, or diameter. It performs word segmentation and entity recognition on sentences containing size descriptions and converts the size values ​​D (d1, d2, ..., d j ) extracted, d j Represents the j-th size parameter.

[0053] S102-2: Functional description: Identify the functional requirements of the model from the text, such as the purpose of the design object or its operating characteristics in the actual scenario. The functional requirements are generated through GPT-4O to generate a function vector F (f1, f2, ..., f k ), f k Indicates a specific functional requirement, such as strength, bearing capacity, etc.

[0054] S102-3: Material and structure description: Further analyze the material selection and structure requirements in the text. Use word vectorization technology to convert the material properties in the text (such as steel, aluminum, etc.) into vectors M (M1, M2, ..., M l ), and generate the constraint vector C(C1,C2,...,Cq ).

[0055] Furthermore, the text features are expressed as:

[0056] T={D,F,M,C}

[0057] The above features are used together with image features for multimodal fusion to ensure that the 3D model generation meets the user's design requirements.

[0058] Step S200: multi-modal fusion; Figure 2 As shown in the figure, the image features and text features are fused through the BLIP-2 model to generate a unified feature vector. The specific fusion steps include:

[0059] S200-1: Feature adaptation: First, the image feature G and the text feature T are dimensionally matched to ensure that the two feature vectors can be aligned in the same dimensional space. Assume that the image feature dimension is n and the text feature dimension is m. The conversion of the feature dimension is completed through the adaptation matrix A, which is expressed as:

[0060] R=A·G+A·T

[0061] Where A is the adaptation matrix, which is used to project image and text features into a unified dimensional space.

[0062] S200-2: Attention mechanism fusion: Through the multi-head attention mechanism, the image features and text features are fused. Let the attention weight be α i , the fused feature vector R is expressed as:

[0063]

[0064] where α i is the attention weight calculated based on the feature importance, G i and T i are the i-th components of image and text features respectively.

[0065] S200-3: Fusion feature output: Further, the generated fusion vector R (r1, r2, ..., r s ) represents a unified multimodal feature. This feature will be used as input to generate a 3D model in the subsequent steps.

[0066] Step S300: Preliminary 3D model generation: After multimodal fusion is completed, the system uses the Point-E model to generate multiple preliminary 3D models. These models are generated based on the unified feature vector after fusion, and the Point-E model is particularly good at generating high-quality point cloud models, which are suitable for the representation of complex geometric structures. Specifically, the following steps S301-S302 are included:

[0067] Step S301: Generate a 3D model using Point-E; Figure 3 As shown, the fused feature vector $R$ is input into the Point-E model, and the model generates multiple preliminary 3D models through point cloud technology. Each 3D model is represented as a set of point sets:

[0068] R = A·G + A·TP = {(x 1, y1,z1),(x 12 y2,z2),...,(x n, y n ,z n )}

[0069] Where (x n, y n ,z n ) is the coordinate of each point in the point cloud, representing the geometric structure of the model.

[0070] During the model generation process, attention is paid to details and structural integrity to ensure that the generated 3D model meets the user's design requirements.

[0071] Step S302: Use Blender to render the 3D model; after generating the preliminary point cloud model, use Blender to further render and optimize the model. Rendering includes setting lighting effects, applying material mapping, and adjusting the camera perspective to ensure that the generated model visually meets the design requirements. The rendering result is:

[0072] M=R(P)+texture map+lighting

[0073] Among them, R(P) is the rendered 3D model, which combines lighting and material information.

[0074] Step S400: User demand analysis and model screening: The system analyzes the user's design requirements through GPT-4O, converts them into screening rules, and intelligently screens the generated 3D models in combination with the design rules in the knowledge graph. Specifically, the following steps S401-S403 are included:

[0075] Step S401: GPT-4O analyzes user requirements: The system analyzes the functional requirements and geometric constraints input by the user and converts them into actionable screening rules. These rules include wing load capacity, streamlined design of the fuselage, material requirements, etc.

[0076] Step S402: construct a knowledge graph; Figure 4 As shown in the figure, a knowledge graph containing basic design rules, functional requirements and geometric constraints is constructed, and the graph is used to assist in matching the generated model with the user's needs during the model screening process. The specific process is as follows:

[0077] S402-1: Basic design rules: Extract basic design rules from industry standards and existing design experience, covering the physical parameters, material specifications and basic functional requirements required by the model. These rules are used as nodes in the graph to perform basic compliance verification on the generated model.

[0078] S402-2: Functional requirement matching: Match the functional requirements input by the user with the historical design experience and rules in the knowledge graph. For example, if the user wants the generated model to have the function of "wind resistance", the knowledge graph will return a model with the corresponding functional design for comparison.

[0079] Expressed as: Function matching matrix {M f1 ,M f2 ,...,M fn}, the matrix compares the functional characteristics of each model with user requirements to ensure that the design goals are met.

[0080] S402-3: Geometric constraint verification: Through the geometric constraint nodes, the knowledge graph can verify whether the generated model meets the geometric constraints such as size and shape. It is represented as: Geometric constraint node network {G n1 ,G n2 ,...,G nm}, used to determine whether the generated model meets the user's geometric requirements.

[0081] Step S403: Model scoring and selection; Figure 4 As shown, according to the above rules, the generated 3D models are scored, and the scoring criteria include structural integrity, functional matching and innovation. The system automatically evaluates each model and selects the model with the highest score as the further recommended design solution. The specific scoring steps are as follows:

[0082] S403-1: Structural integrity: First, the generated 3D model is scored for its geometric structural integrity, mainly to assess whether the model meets the user's geometric constraints and ensure that all geometric features of the model are complete.

[0083] Scoring rules: The geometric deviation calculation formula ΔG = |actual geometric value - ideal geometric value|, and the scoring standard is 0-10 points.

[0084] S403-2: Functional matching: By matching the user's functional requirements (such as strength, durability, etc.), the system scores the functional characteristics of each model. The score is based on the functional matching matrix in the knowledge graph.

[0085]

[0086] Scoring rules: Function matching scoring formula F s=∑User demand characteristics*model characteristics, ensuring the adaptability of each model to functional requirements.

[0087] S403-3: Innovation: Evaluate the innovative design of the model. The system will score the model's innovation in terms of function, form or material.

[0088] Scoring rules: The innovation score is based on the difference between the historical design in the knowledge graph and the existing model. The greater the difference, the higher the score. The standard is 0-10 points.

[0089] Furthermore, the system will give an overall score to all generated models according to the above scoring criteria, and recommend the model with the highest score to the user as a further design solution.

[0090] The following is an embodiment of this embodiment (combined with Figure 3 and Figure 4 ).

[0091] Step S101: Image input; in this embodiment, the user first provides an image of the aircraft's appearance. The system extracts key geometric features in the image through the CLIP-ViT model. These features include:

[0092] S101-1: Geometry: Contour information of the wing, fuselage, and tail: The system identifies the chord length and curvature of the wing, the streamlined design of the fuselage, and the angle and layout of the tail as the basic geometric features for subsequent 3D generation.

[0093] S101-2: Detailed features: The system further extracts information about important design parts such as the engine air intake and landing gear position.

[0094] S101-3: Texture and Material Features: The texture details and reflective properties of the aircraft surface are extracted. This information will reflect the visual realism in the subsequent rendering process.

[0095] The extracted features are converted into image feature vectors G(g1,g2,...,g n ), where g1 represents the geometric features of the wing, g2 represents the length features of the fuselage, and so on. This vector is used for the next step of multimodal fusion.

[0096] Step S102: Text input: The user also provides a detailed text description of the aircraft design. The system parses the text through GPT-4O to extract specific information about the design goals and functional requirements. The extracted text features include:

[0097] S102-1: Functional requirement: Increase the load-bearing capacity of the wing while maintaining aerodynamic performance.

[0098] S102-2: Geometric constraints: Maintain the streamlined design of the aircraft and avoid significant geometric changes in the wings and tail.

[0099] S102-3: Structural optimization: Reduce the overall weight of the aircraft while ensuring structural strength, especially in the wing area.

[0100] S102-4: Material selection: The wings are made of carbon fiber and the fuselage is made of lightweight aluminum alloy.

[0101] S102-5: Special requirements: High temperature resistant materials must be used in the air inlet area to ensure safety under high temperature airflow.

[0102] GPT-4O converts the above information into a text feature vector, denoted as T(t1,t2,...,t m ), where t1 represents the functional requirement, t2 represents the geometric constraint, and so on.

[0103] Step S200: multi-modal fusion; Figure 3 As shown in the figure, the system fuses the image features with the text features through the BLIP-2 model to generate a unified feature vector R (r1, r2, ..., r s ). The specific fusion steps include:

[0104] S200-1: Feature adaptation processing: In order to ensure that image features and text features are processed in the same vector space, BLIP-2 first converts the image features G(g1,g2,...,g n ) and text features T(t1,t2,...,t m ) is normalized through an adaptation layer to ensure that the two feature vectors have the same dimension.

[0105] S200-2: Feature fusion generation: After feature concatenation and weight adjustment, a unified feature vector R (r1, r2, ..., r s ). This vector includes the geometry of the aircraft and also reflects the user's design goals and functional requirements.

[0106] Step S300: 3D model generation: After the fused unified feature vector is generated, the system uses the Point-E model to generate a 3D model. The specific steps are as follows:

[0107] Step S301: Point-E generates a 3D model: The Point-E model receives a unified feature vector R (r1, r2, ..., r s) as input to generate multiple 3D point cloud models. The focus of model generation is to accurately present complex geometric structures such as wings, fuselage, and tail, while paying attention to the load-bearing capacity and structural optimization of the wings. The model reflects the streamlined design and lightweight fuselage structure.

[0108] Step S302: Blender rendering: The generated point cloud model is imported into Blender for further rendering. The Blender rendering process includes:

[0109] S302-1: Lighting settings: simulate the reflection effect of real light to ensure the realistic appearance of the model under different lighting conditions.

[0110] S302-2: Material Application: Apply the specified carbon fiber or aluminum alloy material to different parts of the model.

[0111] S302-3: Viewing angle adjustment: generating rendering images of multiple 3D models according to different viewing angle settings.

[0112] Step S400: User demand analysis and model screening, such as Figure 4 As shown in the figure, after generating a preliminary 3D model, the system uses GPT-4O to analyze user needs and screens the generated model in combination with the knowledge graph. The specific steps are as follows:

[0113] Step S401: GPT-4O analyzes user requirements: The system analyzes the user's design description through GPT-4O and converts it into screening rules. These rules cover the structural integrity, functional matching and geometric constraints of the aircraft, and are converted into computer-executable screening criteria in text form.

[0114] Step S402: Knowledge graph construction: The system constructs a knowledge graph containing basic design rules, functional requirements and geometric constraints. During the model screening process, the matching process of the knowledge graph can further demonstrate the specific matching method and judgment criteria.

[0115] Through various data sources such as design documents, industry databases, standard documents and expert interviews, we systematically collect information related to basic design rules, functional requirements and geometric constraints to ensure the comprehensiveness and accuracy of the knowledge graph. Furthermore, we use named entity recognition and relationship extraction technology to perform in-depth knowledge extraction on the collected text data, identify specific design parameters and their relationships, and model the information in the form of nodes and edges to form a structured knowledge graph. Then, after the graph is established, we regularly collect data from the latest industry standards, research results and user feedback to dynamically update the knowledge graph, ensure that its content is consistent with actual design requirements, and introduce new design standards and domain knowledge. Furthermore, in the process of model scoring and selection, the constructed knowledge graph is used to judge the compliance and adaptability of the generated model, and assist in screening out the 3D model that best meets user needs.

[0116] Step S403: Model scoring and selection: The system scores the generated 3D model according to the design rules. The scoring criteria include:

[0117] S403-1: Structural integrity: Based on specific quantitative parameters such as the geometric continuity of the model, stress distribution between nodes, and stiffness index.

[0118] S403-2: Functional matching: Introduce quantitative evaluation criteria for specific functions, such as load bearing capacity, numerical analysis of aerodynamic performance, etc., to judge whether the model meets the functional requirements of users.

[0119] S403-3: Innovation: This part can analyze the uniqueness and improvement of the model in design by comparing it with the existing knowledge graph model.

[0120] It is feasible that in step S403, the specific operation steps are as follows: First, set the scoring criteria according to the design rules, including evaluation indicators such as structural integrity, functional matching and innovation, among which structural integrity evaluates the completeness and accuracy of the model in geometric structure, functional matching evaluates the ability of the model to meet the user's design needs, and innovation evaluates the uniqueness of the model in design ideas and forms. Further, using the set scoring criteria, the 10 preliminary 3D models generated in step S300 are comprehensively scored. The specific method is to conduct quantitative evaluation for each indicator, for example, using a scoring scale of 0 to 10 to ensure the objectivity and consistency of the evaluation process. Then, the scoring results of each indicator are combined to calculate the total score of each model, ensuring that each indicator is weighted according to its importance in the design to reflect the comprehensive performance of the model. Further, based on the scoring results, the model with the highest score is selected as the basis for subsequent optimization and adjustment.

[0121] This embodiment proposes an innovative 3D design assistance method and system by combining multimodal learning technology, pre-trained GPT-4O model, CLIP-ViT model and knowledge graph. The system effectively integrates text and image input, uses GPT-4O to extract design intent and functional requirements, and analyzes image structure features through CLIP-ViT to lay the foundation for subsequent 3D model generation. The feature fusion achieved by the BLIP-2 model improves the adaptability to complex design requirements and enhances the comprehensive performance of the model. In the generation stage, the Point-E model is used to generate a high-quality preliminary 3D model, and it is rendered through Blender to ensure that the model meets the design intent. In addition, the construction of a knowledge graph that supports dynamic updates and the use of the GPT-4O model enable the system to flexibly integrate the design rules and functional requirements required by users and guide the model screening process. Combining the above advantages, this method significantly improves the generation efficiency and accuracy of 3D models, and provides strong intelligent support for design innovation.

[0122] The present embodiment provides a 3D design assistance system based on multimodal input and intelligent optimization, the system comprising:

[0123] Multimodal input module: The pre-trained GPT-4O is used to parse the user's text input to extract the design intent and functional requirements, while CLIP-ViT is used to extract structural features in the image for subsequent processing;

[0124] Feature fusion module: Use BLIP-2 to fuse text and image features to generate a unified feature vector, which provides input for 3D model generation;

[0125] 3D model generation module: Generate a preliminary 3D model using the Point-E model and render it through Blender to ensure that the generated model meets the design intent;

[0126] User demand analysis and rule matching module: Use GPT-4O to analyze the semantics of design input, build and maintain a knowledge graph containing basic design rules, functional requirements and geometric constraints, dynamically update the knowledge graph to ensure the validity of its content, and assist in judging the compliance and adaptability of the generated model during the model screening process;

[0127] Rule-based intelligent model scoring and optimization module: Based on the set scoring criteria, the model's structural integrity, functional matching and innovation are evaluated. After comprehensive scoring, the model with the highest score is selected for further optimization and adjustment.

[0128] Any process or method description in the flowchart of this embodiment or described in other ways herein can be understood as a module, fragment or part of a code including one or more executable instructions for implementing the steps of a specific logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, device or equipment, and the computer-readable medium can be any medium containing storage, communication, propagation or transmission programs for use by execution systems, devices or equipment, including read-only memories, disks or optical disks, etc.

[0129] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A 3D design assistance method based on multimodal input and intelligent optimization, characterized in that: include: Obtaining text data to be processed and image data to be processed; Extracting features from the text data to be processed and the image data to be processed respectively to obtain text features and image features; the text features include size description, functional description, material description and structure description; the image features include image structure features; Performing feature fusion on the text feature and the image feature to obtain a fused feature; constructing a plurality of 3D models based on the fused features; A knowledge graph is constructed based on preset design rules and the text features, each of the 3D models is evaluated based on the knowledge graph, and an optimal 3D model is determined based on the evaluation results.

2. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The process of acquiring the text features specifically includes: The text data to be processed is parsed based on the GPT-4O model to obtain the text features.

3. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The image structural features include edge features, texture features and shape features of objects in the image data to be processed.

4. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 3, characterized in that: The process of acquiring the image structural features specifically includes: Inputting the image data to be processed into the CLIP-ViT model, extracting the contour and boundary of the object in the image data to be processed, and obtaining the edge feature; Encoding the local pixel difference of the image data to be processed according to a local binary pattern algorithm to obtain the texture feature; The shape feature is obtained by performing frequency domain representation on the contour and boundary of the object based on Fourier transform.

5. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The feature fusion of the text feature and the image feature specifically includes: The image features and the text features are projected into a unified dimensional space through an adaptation matrix, and the image features and the text features are fused based on a multi-head attention mechanism to obtain the fused features.

6. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The constructing of a plurality of 3D models based on the fusion features specifically includes: The fusion features are input into the Point-E model, a number of initial models are generated by using point cloud technology, and each initial model is rendered and optimized to obtain a number of 3D models.

7. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The constructing of the knowledge graph based on the preset design rules and the text features specifically includes: The preset industry basic design rules, size descriptions and the functional descriptions are converted into screening rules, and a knowledge graph is constructed based on the screening rules.

8. The 3D design assistance method based on multimodal input and intelligent optimization according to claim 1, characterized in that: The evaluating each of the 3D models based on the knowledge graph specifically includes: Each of the 3D models is scored based on the knowledge graph, and the 3D model with the highest score is selected as the optimal 3D model.

Images