A method and system for generating 3D models by combining text and images

Abstract

This invention discloses a method and system for generating 3D models by combining text and images, belonging to the field of 3D modeling and artificial intelligence technology. The method includes: acquiring a user's text description, 2D image, and user identifier; retrieving the user's unique 3D user preference profile; determining the dominant text-image fusion mode based on the profile history and calculating text modality weight parameters; extracting text and image features; performing cross-modal fusion according to the text modality weight parameters to generate fused features; generating model geometry and texture through a 3D generation network and performing differentiated optimization based on industry category; outputting editable or non-editable formats, while simultaneously updating the user preference profile and storing it in a closed loop. This invention reduces the computational load of a single modality and improves generation accuracy by combining text and images, and ensures data integrity through a user preference storage and confirmation-retransmission protocol. It is applicable to 3D content creation fields such as film, architecture, and product design.

Description

Technical Field

[0001] This invention relates to the fields of 3D modeling and artificial intelligence technology, specifically to a method and system for generating 3D models by combining text and images. Background Technology

[0002] Traditional 3D modeling relies on manual operation, which is inefficient and has a high barrier to entry. In recent years, artificial intelligence-based 3D generation technology has made progress, mainly divided into two categories: text-to-3D generation and image-to-3D generation. However, the following technical shortcomings still exist:

[0003] 1. Pure text-driven methods lack geometric constraints, requiring extensive iterative optimization and resulting in significant computational redundancy. Pure image-driven methods are limited by monocular depth ambiguity, maintaining geometric accuracy only on the visible surface (approximately 85%-92%), while invisible areas rely on inference, easily leading to ambiguity. Existing technologies, due to their limited modal information, struggle to balance generation efficiency and accuracy.

[0004] 2. Existing methods (such as DreamReward and DreamDPO) can only perform global preference alignment during the model training phase. All users share the same parameters, making it impossible to dynamically adapt to individual differences at runtime or continuously learn the evolution of user preferences over multiple uses. Users need to reset parameters such as output format and fusion mode each time they use the platform, resulting in low interaction efficiency.

[0005] 3. Existing generation technologies mainly output lightweight mesh formats such as STL and OBJ, which do not include parametric modeling history, feature trees, dimensional constraints, or assembly relationships, making them unsuitable for direct use in engineering design software such as CAD / SolidWorks. Even if some methods can output STEP format, they lack complete feature editing capabilities, preventing users from modifying dimensions, adjusting views, or re-annotating engineering drawings, thus failing to meet the iterative needs of industrial design.

[0006] To address the aforementioned problems, we propose a method and system for generating 3D models by combining text and graphics. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for generating three-dimensional models by combining text and images, thus overcoming the deficiencies of existing technologies.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] This invention provides a method for generating a three-dimensional model by combining text and images, comprising the following steps:

[0010] Obtain the user's text description information, at least one two-dimensional image, and user identification information; perform integrity verification on the obtained text, image, and identification information; if any necessary information is missing or the format is incorrect, return an error message to the user and wait for retransmission.

[0011] Based on the user identification information that has passed the integrity verification, a retrieval request is sent to the user storage platform; the user preference profile of the user returned by the user storage platform is received.

[0012] If the search is successful, the user preference profile will be loaded.

[0013] If the search fails, the default user preference profile is initialized and stored in the user storage platform;

[0014] Based on the historical records in the loaded user preference profile, the dominant mode of the generated image-text fusion is automatically determined, and the text modality weight parameters are output to subsequent steps;

[0015] The text description information and the two-dimensional image are input into the text encoder and the image encoder, respectively, to obtain the text semantic feature vector and the image visual feature vector. Null value detection is performed on the extracted feature vector. If it is empty, it is extracted again. The retry is not more than a preset number of times. Otherwise, the process is terminated.

[0016] The text semantic feature vector and the image visual feature vector, along with the text modality weight parameters, are input into the image-text fusion module to generate a fused feature vector. The fused feature vector is then subjected to dimensionality and numerical range verification to ensure that it meets the input requirements of subsequent networks; otherwise, re-fusion is triggered.

[0017] The validated fusion feature vectors are input into the 3D generation network to generate the geometric structure and texture information of the 3D model; based on the industry application category in the user preference profile, the corresponding optimization sub-network is automatically selected for differentiated optimization; after generation, the 3D model is self-checked to repair or regenerate defective models.

[0018] Based on user preference profiles or real-time user commands, the generated 3D model is sent to the output adaptation module, generating editable and non-editable formats in parallel; all formats are temporarily stored in the output buffer, and a selection interface is displayed to the user; after the user selects, the corresponding format data is delivered, and unselected formats are automatically cleared after a preset retention period;

[0019] Record the user's final adopted output format, style parameters, text modality weight parameters, and industry category; compare this with historical user preference profiles.

[0020] If they match, the confidence level is increased; if they differ, a new branch is created and a version tag is added; the updated file is encrypted and written back to the user storage platform, awaiting confirmation.

[0021] If no response is received, a preset number of retries will be made; once completed, the process loop ends.

[0022] Preferably, the integrity check in step S1 is performed using the following expression:

[0023] The input integrity judgment criteria are:

[0024]

[0025] in, This indicates whether the text description information satisfies the conditions of being non-empty and having a length not exceeding a preset limit. This indicates whether a two-dimensional image is a valid image format and has a resolution not lower than the minimum threshold. This indicates whether the user identification information contains a valid token or user ID.

[0026] Preferably, the validity of the file in step S2 is determined by the following expression:

[0027]

[0028] in, This indicates whether the user preference profile exists in the user storage platform; This indicates whether the archive contains necessary fields such as output format history, text modality weight parameter history, and industry category. Indicates whether the file version is compatible with the current processing engine; This indicates whether the file's checksum is correct.

[0029] Preferably, the text modality weight parameter in step S3 The number of text-dominated tasks and image-dominated tasks recorded in the user preference profile are calculated using the following formula:

[0030]

[0031] in, This indicates the number of times a user has explicitly selected the text-dominant mode in their user preference profile and the output has been adopted. This represents the number of times a user has explicitly selected the dominant image mode in their user preference profile and the output has been adopted; the constants 1 and 2 in the denominator are Laplacian smoothing parameters, used to smooth the image. and When both are 0, =0.5.

[0032] Preferably, the fused feature vector generated in step S5 passes through a feature cache queue before being input into the 3D generation network; the queue temporarily stores the fused features and their user identifiers of the most recent preset number of times; when a subsequent step detects that the quality of the generated 3D model is lower than a threshold, the corresponding features are automatically extracted from the cache queue and a backup generation network is called to regenerate the model.

[0033] The fused feature vector is calculated using the following formula:

[0034]

[0035] in, Represents the fused feature vector; This is a text semantic feature vector; It is the image visual feature vector; Indicates For query vectors, The result is the text-guided image features obtained by the cross-attention algorithm for key-value pairs; Indicates For query vectors, This is the result of image-guided text features obtained by using a cross-attention algorithm on key-value pairs.

[0036] Preferably, the quality of the 3D model generated in step S6 is judged by the following geometric completeness index:

[0037]

[0038] in, This indicates the number of vertices in the generative model; Indicates the number of faces in the generated model; This is a preset minimum vertex / face number threshold; Indicates whether there are holes in the model; This indicates whether there are non-manifold edges or vertices in the model.

[0039] Preferably, the management of the output buffer in step S7 includes: allocating an independent buffer namespace for each user; attaching a timestamp and retention period to each stored data; and periodically cleaning up expired data in the background.

[0040] The recommended priority of output formats is calculated using the following formula:

[0041]

[0042] in, Indicate the output format Recommended rating; This indicates the format of the user's historical selections in the user preference profile. The number of times; This indicates the total number of times the user has output data in history. This is a function for matching industry scenarios.

[0043] Preferably, the confidence level of the preference branch in step S8 is updated according to the following incremental formula:

[0044]

[0045] in, The confidence level of this preference branch before the update is initially set to 1; if the output format, style parameters, and text modality weight parameters ultimately adopted by the user are consistent with the current branch record, then... =1, otherwise =0; If the adoption result is inconsistent with the current branch record, then =0.5, otherwise =0; when If the confidence level is below a preset threshold (e.g., 0.5), the branch is marked as a low-confidence branch and its priority is reduced in subsequent recommendations.

[0046] The present invention also provides a system for generating three-dimensional models by combining text and images, for implementing the above-described method, comprising:

[0047] The input integrity verification module is configured to: obtain text description information, at least one two-dimensional image and user identification information, perform integrity verification, package the input data packet into a data packet and generate a unique identifier after the verification is successful, and transmit it to the subsequent module through the data bus; if the verification fails, return the error reason and request retransmission.

[0048] The user preference management module is connected to the input integrity verification module via a data channel with an acknowledgment signal. It is configured to: retrieve and load the user preference profile from the user storage platform based on the user identification information; if the retrieval fails, initialize the default profile; this module is equipped with a read-write lock.

[0049] The image-text fusion control and execution module, connected to the user preference management module, includes: a dominant mode decision unit that determines text modality weight parameters based on preference profiles; a text encoder and an image encoder, the output of which is equipped with a null value detector; and an image-text fusion unit that outputs fused feature vectors and sets a dimension validator.

[0050] The 3D generation and optimization module, connected to the image and text fusion control and execution module, includes: a 3D generation network; an industry optimization sub-network selector that dynamically loads the corresponding sub-network according to the industry category; and a geometric self-checking unit that detects and repairs model defects.

[0051] The output adaptation and buffering module, connected to the 3D generation and optimization module, includes: a multi-format parallel generator; a user-level output buffer for storing data in various formats; and a user confirmation interface for receiving selection instructions and delivering data.

[0052] The user storage platform is connected to the user preference management module and the output adapter and buffer module through an encrypted channel, and is configured to: encrypt and store user preference files, provide write confirmation signals, periodically perform integrity checks and restore from the previous version;

[0053] The data flow monitoring and alarm unit connects to all modules via the data bus and is configured to: monitor the data packet transmission status, receive confirmation messages after each module has finished processing; if no confirmation is received within a timeout, it determines that the data is lost and instructs retransmission, while generating an alarm log.

[0054] This invention provides a method and system for generating 3D models by combining text and images. It offers the following advantages: it achieves bidirectional complementary fusion of text and images through a cross-attention algorithm, dynamically adjusts the contribution weights of text and images, and utilizes complementary information to constrain the 3D generation space;

[0055] Create an independent user preference profile for each user, recording output format, style parameters, fusion mode and industry category, and supporting cross-device synchronization and encrypted storage;

[0056] It generates editable 2D engineering drawings for CAD (supporting angle correction) and editable 3D solid models for SolidWorks (preserving the complete feature tree) in parallel. Users can directly modify feature dimensions, add constraints, and rotate the viewpoint; compared to the existing "non-editable" mesh output;

[0057] The system automatically switches and optimizes sub-networks based on industry category: the film and television production mode focuses on materials and lighting, while the architectural design mode focuses on outlines and structural lines, making the generated model more professionally applicable in their respective fields.

[0058] In each stage, including input acquisition, feature extraction, fusion output, model generation, and output adaptation, integrity verification, null value detection, dimension verification, and geometric self-checking are implemented to ensure that signals or data are transmitted in a closed loop from input to output, thus avoiding data loss due to temporary calculation errors or transmission anomalies. Attached Figure Description

[0059] Figure 1 This is a flowchart of the overall process of the method of the present invention.

[0060] Figure 2 This is a flowchart illustrating the user preference profile loading and updating process of this invention.

[0061] Figure 3 This is a detailed flowchart of the image-text fusion and 3D generation method of the present invention.

[0062] Figure 4 This is a system module and data flow diagram of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, in one embodiment, a method for generating a 3D model by combining text and images includes the following steps:

[0065] Step S1: Input Acquisition and Integrity Verification

[0066] The system acquires the user's text description information (which is a natural language sentence or keyword sequence, such as generating a wooden chair with armrests), at least one two-dimensional image (which is an RGB color image, grayscale image, or depth image, such as a photograph of an object taken from a single perspective), and user identification information (which is at least one of username, user ID, session token, or device fingerprint). The system performs integrity checks on the acquired text, image, and identification information (integrity checks include checking data existence, format validity, and content within a preset range). If any necessary information is missing or formatted incorrectly, an error message is returned to the user, and the system waits for retransmission. The system ensures the input data is complete before proceeding to the next step.

[0067] Specifically, when a user initiates a 3D model generation request through a client, web page, industrial design terminal, or boiler equipment management platform, this method receives text description information uploaded by the user, at least one 2D image, and user identification information. The text description information may include boiler component names, structural features, size ranges, application scenarios, material requirements, connection relationships, and output requirements; the 2D image may be a site photograph, 2D design drawing, boiler component sketch, maintenance image, or reference appearance drawing; the user identification information may be a user ID, a valid token, or a company account.

[0068] Upon receiving data, this method first checks whether the text is empty, whether the length exceeds the preset limit, and whether it contains a valid engineering description field; secondly, it checks whether the image can be decoded, whether the image format is a allowed format such as JPG, PNG, BMP, or TIFF, and whether the image resolution is not lower than the minimum threshold; finally, it checks whether the user identification information exists, whether the token is valid, and whether the user ID conforms to the system encoding rules.

[0069] If any necessary data is missing or incorrectly formatted, this method does not pass the data to subsequent steps. Instead, it generates an error response containing the error field, error type, and correction suggestions, and returns it to the user interface, waiting for the user to resubmit or resubmit. Only after the text, image, and user identifier have all passed validation is this method encapsulated into an input data packet and a unique data packet identifier generated, allowing subsequent steps to track the processing status based on this identifier.

[0070] Step S2: User Preference Profile Retrieval and Loading

[0071] Based on the user identification information that has passed the integrity verification, a retrieval request is sent to the user storage platform (the user-specific storage platform is a cloud object storage service, a local encrypted database, or a distributed file system); the user preference profile of the user returned by the user storage platform is received (the user preference profile is a structured file in JSON, XML, or key-value pair format, used to record the output format, style parameters, image and text fusion mode, and industry category preferred by the user in historical generation activities).

[0072] If the search is successful, the user preference profile will be loaded.

[0073] If the search fails (either on first use or due to version incompatibility, verification failure, or other reasons), then initialize the default user preference profile (with the output format set to STL and the image / text fusion mode set to balanced mode in the default profile). =0.5, industry category set to general) and stored in the user storage platform to ensure that there is always available preference data for subsequent steps;

[0074] Specifically, after the input data packet passes integrity verification, this method transmits the user identification information and the unique identifier of the data packet to the user preference management step. This step initiates a file retrieval request to the user's dedicated storage platform based on the user identification information. This request includes the user ID, request time, data packet identifier, and access token.

[0075] After receiving a retrieval request, the user-specific storage platform queries the user's preference profile based on the user ID. This preference profile includes at least the user's historical output formats, commonly used industry categories, historical image-text fusion weights, model style parameters, structural line enhancement levels, texture styles, model accuracy levels, cache retention periods, and preference branch confidence scores.

[0076] If the storage platform successfully returns a preference profile, this method verifies the profile's integrity, including field integrity, version number, timestamp, and checksum. If the verification passes, the preference profile is loaded, and concurrent tasks are prevented from modifying the profile during the reading process. If the retrieval fails, such as in the case of first-time user use, corrupted profile, lost profile, or version incompatibility, this method creates a default preference profile. In the default preference profile, the text-image fusion weight is set to balanced, the default output format is set to OBJ and CAD 2D engineering drawings, the default industry category is set to general industrial modeling, and the default profile is immediately written to the user's dedicated storage platform.

[0077] Step S3: Determine the dominant mode of image and text fusion

[0078] Based on the historical records in the loaded user preference profile, the dominant mode of the generated image-text fusion is automatically determined, and the text modality weight parameters (within the range of [0,1], where 1 indicates complete dependence on text and 0 indicates complete dependence on image) are output to subsequent steps;

[0079] Specifically, after loading the preference profile, this method reads the historical text-image fusion records in the preference profile and determines whether the user relied more on text descriptions, two-dimensional images, or a balance of both in previous generation tasks.

[0080] In boiler equipment modeling scenarios, if a user consistently specifies furnace dimensions, tube bundle arrangement, interface direction, and material parameters through detailed text descriptions, this method determines that the user is text-driven. If a user consistently uploads on-site photos, structural diagrams, or sketches and rarely inputs detailed text, this method determines that the user is image-driven. If the contributions of the two types of inputs are similar in the historical records, a balanced fusion mode is adopted.

[0081] This method sets text modality weight parameters based on the judgment result. If the preference profile indicates text-dominant, then Set to greater than 0.5; if the indicator image is dominant, then Set to less than 0.5; if there are no valid historical records, then Set it to 0.5. (After confirming) As a control signal passed to subsequent image-text fusion steps, it is used to control the contribution ratio of text semantic features and image visual features in the fusion process.

[0082] Step S4: Multimodal Feature Extraction

[0083] The text description information and two-dimensional image that passed the integrity check in step S1 are input into the text encoder (the text encoder is a pre-trained language model based on the Transformer architecture, such as BERT, RoBERTa, or CLIP) and the image encoder (the image encoder is a pre-trained visual model based on convolutional neural network or VisionTransformer, such as ResNet, ViT, or CLIP) respectively, to obtain the text semantic feature vector (the text semantic feature vector is a fixed-dimensional real number vector with a dimension range of 256 to 1024) and the image visual feature vector (the image visual feature vector is a fixed-dimensional real number vector with a dimension range of 256 to 1024); null value detection is performed on the extracted feature vector (null value detection includes checking whether the vector is a null pointer, whether there is a NaN or Inf value, and whether each component is within the valid numerical range). If it is null, it is extracted again, and the retry is not exceeded by a preset number of times (the preset number of retry is 3 times), otherwise the process is terminated.

[0084] Specifically, this method receives the input data packet and After that, input the text description information into the text encoder and the two-dimensional image into the image encoder.

[0085] The text encoder performs text cleaning, word segmentation, entity recognition, and semantic encoding on the text description information. For text in the boiler field, the text encoder focuses on identifying component names, spatial relationships, dimensional parameters, material types, interface types, assembly locations, and industry applications. For example, when the text contains "the burner interface is set on the front side of the boiler furnace, the flue gas outlet is arranged at the top, and the tube bundle is arranged longitudinally," the text encoder will extract structural semantics such as the furnace, burner interface, flue gas outlet, and tube bundle arrangement direction, and output a text semantic feature vector.

[0086] The image encoder preprocesses two-dimensional images, including image decoding, size normalization, noise suppression, edge enhancement, and target region extraction. For boiler photographs or engineering drawings, the image encoder focuses on extracting outer contours, hole regions, pipe orientation, structural edges, texture distribution, and local connectivity features, and outputs an image visual feature vector.

[0087] Each encoder output is configured with null value detection. If the text semantic feature vector is empty, the image visual feature vector is empty, or there are outliers, all zeros, or infinite values ​​in the feature vector, this method sends a re-encoding instruction to the corresponding encoder. The system will retry a maximum of 3 times; if it still cannot obtain valid features, it will return the reason for feature extraction failure to the user and terminate the current generation process.

[0088] Step S5: Image-text fusion and feature output

[0089] The text semantic feature vector and image visual feature vector obtained in step S4, together with the text modality weight parameters determined in step S3, are input into the image-text fusion module (the image-text fusion module uses a cross-attention network or a gated fusion network to achieve an adaptive weighted combination of text features and image features) to generate a fusion feature vector (the dimension of the fusion feature vector is the same as that of the text feature vector or image feature vector, or is aligned with the input dimension of the 3D generation network after linear transformation); the fusion feature vector is subjected to dimension verification (checking whether the vector dimension is consistent with the input layer dimension of the 3D generation network) and numerical range verification (checking whether each component of the vector is within a preset range such as [-1,1] or [0,1]) to ensure that it meets the input requirements of subsequent networks; otherwise, re-fusion is triggered.

[0090] Specifically, after both the text semantic feature vector and the image visual feature vector pass the validity check, this method performs the image-text fusion step, receiving the text semantic feature vector, the image visual feature vector, and the feature vector determined in step S3. The image-text fusion step is based on a cross-attention algorithm to perform fusion processing, enabling text features to constrain local regions of the image, while simultaneously allowing image features to supplement appearance details not explicitly expressed in the text description.

[0091] Specifically, this method first maps text semantic features to a text query space and image visual features to an image key-value space, then calculates the text's attention response to image regions; simultaneously, it maps image visual features to an image query space and text semantic features to a text key-value space, calculating the image's attention response to text semantics. Subsequently, according to... The ratio of the text-dominant component to the image-dominant component is dynamically adjusted to obtain a fused feature vector.

[0092] In boiler modeling scenarios, if the text explicitly describes "cylindrical furnace body, bottom support frame, and side flange interface," and the image contains corresponding outlines, the fusion step will enhance the weight of these consistent regions. If the image shows support ribs, inspection ports, or pipe transition structures not described in the text, image features will be added to the fused features. If there is a conflict between the text and the image, for example, the text describes a rectangular furnace body while the image outline is cylindrical, then... The value determines whether text constraints or image constraints should be used first.

[0093] After fusion is complete, this method checks whether the dimension of the fused feature vector matches the input layer of the 3D generative network and checks for outliers. If the verification fails, the fusion is re-executed; if fusion fails consecutively, an alarm is triggered and the process is terminated. The fused features that pass the verification are written to a feature cache queue. This queue adopts a first-in, first-out strategy, temporarily storing the five most recent fused features and their corresponding data packet identifiers for each user, for rollback in case of subsequent model quality anomalies.

[0094] Step S6: 3D Model Generation and Industry-Specific Optimization

[0095] The validated fusion feature vectors are input into a 3D generation network (which is a diffusion-based generation network, a NeRF-based implicit field generation network, or a Gaussian-based generation network) to generate the geometric structure of the 3D model (the geometric structure is represented by a polygonal mesh composed of vertex coordinates and face indices, or by point clouds and voxels) and texture information (texture information is UV mapping or per-vertex color). Based on the industry application category in the user's preference profile (the industry application category includes at least one of film and television production, architectural design, product design, game assets, and printing), the corresponding optimization sub-network is automatically selected for differentiated optimization (optimizing materials and lighting in film and television production mode, and optimizing contour lines and structural lines in architectural design mode). After generation, the 3D model is self-checked (self-checking includes checking whether the model is closed, whether there are non-manifold edges, whether it is self-intersecting, and whether it contains isolated vertices), and defective models are repaired or regenerated.

[0096] Specifically, this method receives the fused feature vector output in step S5 and inputs it into the main 3D generation network. The main 3D generation network generates a basic 3D model based on the fused features, including geometric structure, vertex coordinates, patch topology, texture mapping, and basic material information.

[0097] In the field of boiler technology, the master 3D generative network can generate initial models of boiler structures such as the boiler shell, burner interface, flue outlet, heat exchange tube bundle, support frame, inspection door, and flange connection. For dimensions, positional relationships, or connection methods explicitly given in the text, the master 3D generative network treats them as strong constraints; for the external contours and local structures shown in the image, the master 3D generative network treats them as morphological constraints.

[0098] After the basic model is generated, this method reads the industry application category from the user's preference profile and loads the corresponding optimized sub-network. If the category is boiler engineering or architectural design, this method loads the outline and structural line reinforcement sub-network to enhance the model's boundary lines, assembly lines, pipe centerlines, interface edges, and support structures, making the model more suitable for engineering review and secondary design. If the category is film and television production or demonstration, it loads the material and lighting optimization sub-network to enhance metal materials, reflection effects, surface textures, and display rendering effects. If the category is teaching and training, it loads the structural layer display sub-network, allowing the furnace, tube bundles, shell, and interface components to be viewed in layers.

[0099] Subsequently, this method performs quality checks on the generated model, including hole detection, non-manifold surface detection, self-intersection detection, duplicate surface detection, normal consistency detection, and connection boundary integrity detection. If minor defects are found, an automatic repair algorithm is invoked to fill holes, reconstruct normals, merge duplicate vertices, and correct the topology. If the defects exceed the repair threshold, this method extracts the corresponding fusion features from the feature cache queue in step S5 and calls a backup 3D generation network to regenerate the model, avoiding unusable results due to single-generation anomalies.

[0100] Step S7: Output adaptation and user confirmation

[0101] Based on user preference profiles or real-time user commands, the generated 3D model is sent to the output adaptation module, generating in parallel editable formats (including 2D engineering drawings in DXF / DWG format supported by CAD, and 3D solid models in STEP / SLDPRT format supported by SolidWorks) and non-editable formats (including lightweight triangular mesh formats such as STL, OBJ, and GLB). All formats are temporarily stored in the output buffer (the output buffer is a temporary storage area in memory or a disk cache directory), and a selection interface is displayed to the user. After the user selects a format, the corresponding format data is delivered. Unselected formats are automatically cleared after a preset retention period (the preset retention period is 24 hours or a user-defined period).

[0102] Specifically, after the 3D model passes the geometric self-check, this method transfers the model data to the output adaptation step. The output adaptation step first reads the output format preference record from the user's preference profile, and simultaneously receives the user's immediate selection instruction. If the user specifies an output format, the immediate selection instruction is used first; otherwise, the default output format is automatically determined based on historical preferences.

[0103] This method generates multiple formats in parallel. For editable formats, it generates CAD 2D engineering drawings and SolidWorks 3D solid models. The CAD 2D engineering drawings are projected from the 3D model and include front view, top view, side view, key section lines, dimensions, and part numbers; the SolidWorks 3D solid model is encapsulated through solidification, surface closure, and feature tree encapsulation, allowing users to continue editing within the engineering software. For non-editable or lightweight display formats, it generates at least one of STL, OBJ, or GLB formats for printing, quick preview, web display, or loading on digital twin platforms.

[0104] The generated data in various formats is temporarily stored in a user-level output buffer. Each user is allocated an independent buffer namespace, and each file includes a data packet identifier, generation timestamp, format type, retention period, and access status. This method presents a format selection interface to the user. After the user selects the target format, the corresponding file is retrieved from the buffer and delivered to the user. Unselected formats continue to be retained for a preset time (e.g., 24 hours), after which they are automatically deleted by the background. Users can also actively query, download, or delete data in the buffer using commands.

[0105] Step S8: Updating and Closed-Loop Storage of User Preference Profiles

[0106] Record the user's final adopted output format, style parameters (including rendering style, line width, color mapping, LOD level, etc.), text modal weight parameters, and industry category; compare this with historical user preference profiles.

[0107] If they match, increase the confidence level (increment the confidence level by 1 or multiply by a coefficient of 1.1). If they differ, create a new branch (the new branch contains the complete parameter configuration and timestamp for this update) and attach a version tag. Encrypt the updated file and write it back to the user storage platform, waiting for an acknowledgment signal (the acknowledgment signal is an HTTP 200 status code or a custom ACK message returned by the storage platform).

[0108] If no response is received, a preset number of retries will be made; once completed, the process loop ends.

[0109] Specifically, after the user confirms and downloads or adopts a certain output format, this method sends the result record of this generation task to the user preference management step. This record includes the user's final adopted output format, output style parameters, and the actual output format used. Information such as value, industry application category, model quality score, whether repair was triggered, whether a backup generation network was called, and whether the user deleted other file formats.

[0110] This method compares the current record with the historical preference profile loaded in step S2. If the current output format, style parameters, and industry category are consistent with existing preference branches, the confidence score of that preference branch is increased. If the current output format is different, or the style parameter difference exceeds a preset threshold, a new preference branch is created, and a version number, timestamp, and applicable scenario label are added to that branch. For boiler industry users, if they repeatedly select CAD 2D engineering drawings and SolidWorks solid models, editable engineering formats will be prioritized in subsequent tasks; if they repeatedly select GLB format, lightweight display output will be preferred in subsequent tasks.

[0111] After the preference profile is updated, this method writes the complete profile back to the user's dedicated storage platform via an encrypted channel. The storage platform returns an acknowledgment signal upon successful writing. If this method does not receive an acknowledgment signal within a preset time, it will automatically retry the write operation, up to a preset number of times; if it still fails, the update record will be written to a local secure queue, and an alarm log will be sent to the data flow monitoring and alarm unit, waiting for the network to recover before continuing to write.

[0112] The integrity check in step S1 is performed using the following expression:

[0113] The input integrity judgment criteria are:

[0114]

[0115] in, This indicates whether the text description information is non-empty and its length does not exceed a preset limit (the preset limit is 1024 characters; 1 indicates a value if it meets the requirement, and 0 indicates a value otherwise). Indicates whether the 2D image is a valid image format (valid formats include JPEG, PNG, BMP, TIFF) and its resolution is not lower than the minimum threshold (the minimum threshold is 32×32 pixels, 1 if satisfied, 0 otherwise); Indicates whether the user identification information contains a valid token or user ID (a valid token is a JWT or SessionID issued by the system; it is 1 if valid, otherwise it is 0).

[0116] when When the value is 1, this method determines that the input is complete; otherwise, it determines that the input is incomplete and returns an error response containing information about the missing fields (the error response includes the field name and expected format).

[0117] like Figure 2 As shown, the validity of the file in step S2 is determined by the following expression:

[0118]

[0119] in, Indicates whether the user preference profile exists in the user storage platform (1 if it exists, 0 if it does not exist). Indicates whether the file contains necessary fields such as output format history, text modality weight parameter history, and industry category (1 for complete, 0 for incomplete). Indicates whether the file version is compatible with the current processing engine (1 for compatibility, 0 for incompatibility); This indicates whether the file's checksum is correct (the checksum is calculated using the MD5 or SHA256 algorithm; 1 indicates correct and 0 indicates incorrect).

[0120] when When the value is 1, this method loads the historical user preference profile; otherwise, it initializes the default preference profile.

[0121] The text modality weight parameter in step S3 The number of text-dominated tasks and image-dominated tasks recorded in the user preference profile are calculated using the following formula:

[0122]

[0123] in, This indicates the number of times a user has explicitly selected text-based mode in their user preference profile and the output has been adopted (text-based mode refers to the number of times a user explicitly specifies "text-based" or actively selects text during the generation process). (Set to greater than 0.5). This indicates the number of times a user has explicitly selected the image-dominated mode in their user preference profile and the output has been adopted (image-dominated mode refers to the user explicitly specifying "image-centric" or actively selecting images during the generation process). (Set to less than 0.5); the constants 1 and 2 in the denominator are Laplace smoothing parameters, used to smooth the data. and When both are 0, =0.5;

[0124] when When the value is greater than 0.5, this invention employs text-driven fusion. When the value is less than 0.5, image-driven fusion is used. When the value is 0.5, balanced fusion is used.

[0125] like Figure 3As shown, the fused feature vector generated in step S5 passes through a feature cache queue (the feature cache queue is a first-in-first-out circular buffer with a capacity of 5) before being input into the 3D generation network; the queue temporarily stores the fused features and their user identifiers from the most recent preset number of times (the preset number of times is 5); when subsequent steps detect that the quality of the generated 3D model is lower than a threshold (the quality threshold is, for example, Q_valid=0 or the number of model faces is less than 100), the present invention automatically extracts the corresponding features from the cache queue and calls the backup generation network (the backup 3D generation network is a generation model with a different structure but the same function, such as a lightweight network or a higher precision network) to regenerate;

[0126] The fused feature vector in step S5 is calculated using the following formula:

[0127]

[0128] in, Represents the fused feature vector; This is a text semantic feature vector; It is the image visual feature vector; Indicates For query vectors, The result is the text-guided image features obtained by the cross-attention algorithm for key-value pairs; Indicates For query vectors, This is the result of image-guided text features obtained by using a cross-attention algorithm on key-value pairs.

[0129] The quality of the 3D model generated in step S6 is judged by the following geometric completeness indicators:

[0130]

[0131] in, This indicates the number of vertices in the generative model; Indicates the number of faces in the generated model; This is a preset minimum vertex / face number threshold; This indicates whether there are holes in the model (holes are determined by checking the number of boundary edges; if the number of boundary edges is greater than 0, there are holes, 1 indicates the presence of holes, and 0 indicates the absence of holes). Indicates whether there are non-manifold edges or vertices in the model (non-manifold edges are edges shared by more than two faces, and non-manifold vertices are vertices whose neighborhoods are not connected; 1 indicates existence, and 0 indicates non-existence).

[0132] when When the value is 0, this invention determines that the model has defects and automatically calls the repair algorithm or regenerates it.

[0133] The management of the output buffer in step S7 includes: allocating an independent buffer namespace for each user; attaching a timestamp and retention period to each stored data (the retention period can be set to 1 hour, 24 hours, 7 days or permanent); and periodically cleaning up expired data in the background (the background is a system daemon process or a scheduled task).

[0134] The recommended priority of the output format in step S7 is calculated using the following formula:

[0135]

[0136] in, Indicate the output format Recommended rating; This indicates the format of the user's historical selections in the user preference profile. The number of times; This indicates the total number of times the user has output data in history. For industry scenario matching functions,

[0137] when CAD 2D engineering drawings and When using 0.8 for architectural design, For SolidWorks 3D solid models and Use 0.8 for product design purposes, and 0.2 for all other cases; and For the preset weighting coefficients, and + =1; This invention uses the format with the highest score as the default recommended option.

[0138] The confidence level of the preference branch in step S8 is updated according to the following incremental formula:

[0139]

[0140] in, The confidence level of this preference branch before the update is initially set to 1; if the output format, style parameters, and text modality weight parameters ultimately adopted by the user are consistent with the current branch record (consistent means the format is exactly the same, and the style parameter similarity is ≥95%), If the value deviation is ≤0.05, then =1, otherwise =0; If the adoption result is inconsistent with the current branch record (inconsistency means different format or style parameter similarity <80%), then the adoption result is inconsistent with the current branch record. If the value deviation is >0.1, then =0.5, otherwise =0; when When the confidence level is below a preset threshold (e.g., 0.5), the present invention marks the branch as a low-confidence branch and reduces its priority in subsequent recommendations (reducing priority means multiplying the weight of the branch in the recommendation formula by 0.5).

[0141] The following describes a system for generating three-dimensional models by combining text and images, provided by the present invention. The system for generating three-dimensional models by combining text and images described below can be referred to in correspondence with the method for generating three-dimensional models by combining text and images described above.

[0142] like Figure 4 As shown, in one embodiment, a system for generating a 3D model by combining text and images includes:

[0143] The input integrity verification module (consisting of a data receiving unit and a verification logic unit) is configured to: acquire the user's text description information, at least one two-dimensional image, and user identification information; perform format and integrity verification; if the verification passes, package the data into an input data packet and generate a unique identifier for the data packet (the unique identifier is a UUID or an auto-incrementing serial number); and transmit the packet to subsequent modules via the data bus (the data bus is a PCIe bus, USB bus, or on-chip network NoC); if the verification fails, return the error reason and request retransmission.

[0144] The user preference management module (which consists of a user identification unit, a file retrieval unit, and a file update unit) is connected to the input integrity verification module via a data channel with an acknowledgment signal. It is configured to: send a retrieval request to the user storage platform based on user identification information and receive the returned user preference file; if the retrieval fails, initialize and store the default file; the management module internally has a read-write lock (used to prevent multiple threads from simultaneously reading and writing the same user file, leading to data inconsistency).

[0145] The image-text fusion control and execution module (comprising a dominant mode decision unit, a text encoder, an image encoder, and an image-text fusion unit) is connected to the user preference management module and includes: a dominant mode decision unit that determines text modality weight parameter values ​​based on user preference profiles; text encoders and image encoders, each with a null value detector at its output (the null value detector is hardware logic or software function that checks whether the output tensor is valid), triggering re-encoding when a null value is detected; and an image-text fusion module that outputs a fused feature vector and sets a dimension validator at its output (the dimension validator compares the vector dimension with a preset value, triggering an alarm if they do not match).

[0146] The 3D generation and optimization module (which consists of a main 3D generation network, an industry-optimized sub-network selector, and a geometric self-checking unit) is connected to the image-text fusion control and execution module and includes: a 3D generation network; an industry-optimized sub-network selector that dynamically loads the corresponding sub-network according to the industry category (the sub-network is stored in the form of a plug-in and loaded into the GPU memory as needed); and a geometric self-checking unit that detects and repairs model defects.

[0147] The output adaptation and buffering module (which consists of a multi-format parallel generator, a user-level output buffer, and a user confirmation interface) is connected to the 3D generation and optimization module and includes: a multi-format parallel generator (which uses multi-threading or GPU multi-task flow to generate different output formats simultaneously); a user-level output buffer that independently stores data in each format and includes a timestamp; and a user confirmation interface that receives selection instructions and delivers data.

[0148] The user storage platform (which is an object storage service such as AWS S3, Alibaba Cloud OSS, or a local distributed database) is connected to the user preference management module and the output adaptation and buffering module via an encrypted channel (the encrypted channel uses TLS 1.2 or higher protocol). It is configured to: encrypt and store the user preference files of each user, and provide a write confirmation signal method (i.e., the storage platform returns a confirmation signal after completing the write), and only after receiving the confirmation signal is the update considered successful; periodically perform integrity verification on the files (verify once every 24 hours), and restore from the previous version when corruption is found (the previous version retains the 3 most recent backups).

[0149] The data flow monitoring and alarm unit (which is an independent hardware monitoring chip or software watchdog) is connected to all the above modules via a data bus. It is configured to: monitor the data packet transmission status in real time; after each module completes processing, it sends an acknowledgment message (including the data packet identifier and processing status) to this unit; if no acknowledgment is received within a preset timeout period (the timeout period can be set to 100 milliseconds to 2 seconds, which is configurable), it determines that the data is lost and instructs the previous module to retransmit, while generating an alarm log (the alarm log records the timestamp, the identifier of the lost data packet, and the name of the module involved).

[0150] The data transmission between the above modules all adopts the acknowledgment-retransmission protocol (the acknowledgment-retransmission protocol means that after the sender sends data, it starts a timer and waits for the receiver to return an acknowledgment frame. If it does not receive the acknowledgment frame after the timeout, it retransmits the same data. The maximum number of retransmissions is 3). After the sender sends data, it waits for the receiver to return an acknowledgment signal. If no acknowledgment is received, it will automatically retransmit to ensure that there is no data loss during hardware connection and signal transmission.

[0151] In one implementation, taking the architectural design industry as an example, the user provides a hand-drawn sketch of a building and a text description "Modern style villa, two floors, with large floor-to-ceiling windows", and the system generates an editable SolidWorks 3D solid model.

[0152] Input and verification: The user uploads a hand-drawn sketch (JPEG format) and a text description, along with the user identification. The system checks that the text is not empty, the image format is correct, and the user identification is legal, and confirms that the input is complete.

[0153] Load the user preference profile: The system retrieves the user preference profile of this user from the cloud storage platform according to the user identification. The profile record shows that the user's historical preference output format is STEP, the industry category is "architectural design", and the historical Chinese text-dominated mode has been used 3 times and the image-dominated mode 0 times. The profile is valid, and the system loads and uses it.

[0154] Determine the dominant mode of text-image fusion: Since the user has a higher number of historical Chinese text-dominant times, the system automatically sets the text weight parameter of this text-image fusion to a higher value (0.8), that is, this generation is mainly guided by the text description and the hand-drawn sketch is used as an auxiliary reference.

[0155] Extract multi-modal features: The system inputs the text description into the text encoder to obtain a text semantic feature vector; inputs the hand-drawn sketch into the image encoder to obtain an image visual feature vector. Both vectors are valid.

[0156] Text-image fusion: The system follows =0.8 ratio, and fuses the text features and image features through the cross-attention algorithm to generate a unified fused feature vector. This feature vector enters the next step after dimension and value verification. At the same time, this fused feature is stored in the cache queue for backup.

[0157] Generate a 3D model and optimize for the industry: The fused feature vector is input into the 3D generation network, and an initial 3D mesh model containing vertices and faces is output. Since the industry category in the user preference profile is "architectural design", the system automatically loads the contour line and structure line enhancement sub-network to enhance the contours such as the roof line, floor line, and window frame of the model, making the lines clear and sharp. After generation, it is checked that the model has no holes and no non-manifold edges, and the model is qualified.

[0158] Output adaptation and user confirmation: The system generates three formats in parallel: editable CAD 2D engineering drawing (DXF), editable SolidWorks 3D solid model (STEP), and non-editable lightweight format (OBJ). All formats are temporarily stored in the output buffer and reserved for the user for 24 hours.

[0159] Based on user history preferences (most commonly STEP format) and industry scenarios (architectural design has a high demand for CAD engineering drawings), the system calculates a recommendation score and sets the STEP format as the default recommendation. Once the user confirms their selection of the STEP format, the system delivers the download.

[0160] Update user preference profile: The system records the output format (STEP) adopted by the user this time, and the settings used. Value (0.8) and industry category (architectural design). After comparing with historical archives, add the STEP format count and add the new count. The value is added to the history. The updated file is encrypted and written back to the cloud storage platform. The process ends after receiving a successful storage confirmation signal.

[0161] At this point, the user has obtained a modern-style villa 3D solid model that can be directly edited in SolidWorks and has a freely rotating viewpoint.

[0162] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a three-dimensional model by combining text and images, characterized in that, Includes the following steps: Obtain the user's text description information, at least one two-dimensional image, and user identification information; perform integrity verification on the obtained text, image, and identification information; if any necessary information is missing or the format is incorrect, return an error message to the user and wait for retransmission. Based on the user identification information that has passed the integrity verification, a retrieval request is sent to the user storage platform; the user preference profile of the user returned by the user storage platform is received. If the search is successful, the user preference profile will be loaded. If the search fails, the default user preference profile is initialized and stored in the user storage platform; Based on the historical records in the loaded user preference profile, the dominant mode of the generated image-text fusion is automatically determined, and the text modality weight parameters are output to subsequent steps; The text description information and the two-dimensional image are input into the text encoder and the image encoder, respectively, to obtain the text semantic feature vector and the image visual feature vector. Null value detection is performed on the extracted feature vector. If it is empty, it is extracted again. The retry is not more than a preset number of times. Otherwise, the process is terminated. The text semantic feature vector and the image visual feature vector, along with the text modality weight parameters, are input into the image-text fusion module to generate a fused feature vector; The feature vectors are fused and their dimensions and numerical ranges are checked to ensure that they meet the input requirements of subsequent networks; otherwise, re-fusion is triggered. The validated fused feature vectors are input into the 3D generative network to generate the geometric structure and texture information of the 3D model; based on the industry application category in the user preference profile, the corresponding optimization sub-network is automatically selected for differentiated optimization; After generation, the 3D model undergoes self-checking to repair or regenerate defective models. Based on user preference profiles or real-time user commands, the generated 3D model is sent to the output adaptation module, generating editable and non-editable formats in parallel; all formats are temporarily stored in the output buffer, and a selection interface is displayed to the user. After the user selects the corresponding format, the data will be delivered. Data in unselected formats will be automatically deleted after a preset retention period. Record the user's final adopted output format, style parameters, text modality weight parameters, and industry category; compare this with historical user preference profiles. If they match, the confidence level is increased; if they differ, a new branch is created and a version tag is added; the updated file is encrypted and written back to the user storage platform, awaiting confirmation. If no response is received, a preset number of retries will be made; once completed, the process loop ends.

2. The method according to claim 1, characterized in that, The integrity check in step S1 is performed using the following expression: The input integrity judgment criteria are: in, This indicates whether the text description information satisfies the conditions of being non-empty and having a length not exceeding a preset limit. This indicates whether a two-dimensional image is a valid image format and has a resolution not lower than the minimum threshold. This indicates whether the user identification information contains a valid token or user ID.

3. The method according to claim 1, characterized in that, The validity of the file in step S2 is determined by the following expression: in, This indicates whether the user preference profile exists in the user storage platform; This indicates whether the archive contains necessary fields such as output format history, text modality weight parameter history, and industry category. Indicates whether the file version is compatible with the current processing engine; This indicates whether the file's checksum is correct.

4. The method according to claim 1, characterized in that, The text modality weight parameter in step S3 The number of text-dominated tasks and image-dominated tasks recorded in the user preference profile are calculated using the following formula: in, This indicates the number of times a user has explicitly selected the text-dominant mode in their user preference profile and the output has been adopted. This represents the number of times a user has explicitly selected the dominant image mode in their user preference profile and the output has been adopted; the constants 1 and 2 in the denominator are Laplacian smoothing parameters, used to smooth the image. and When both are 0, =0.

5.

5. The method according to claim 1, characterized in that, The fused feature vector generated in step S5 passes through a feature cache queue before being input into the 3D generation network. The queue temporarily stores the fused features and their user identifiers from the most recent preset number of times. When a subsequent step detects that the quality of the generated 3D model is lower than a threshold, the corresponding features are automatically extracted from the cache queue and a backup generation network is called to regenerate the model. The fused feature vector is calculated using the following formula: in, Represents the fused feature vector; This is a text semantic feature vector; It is the image visual feature vector; Indicated by For query vectors, The result is the text-guided image features obtained by the cross-attention algorithm for key-value pairs; Indicated by For query vectors, This is the result of image-guided text features obtained by using a cross-attention algorithm on key-value pairs.

6. The method according to claim 1, characterized in that, The quality of the 3D model generated in step S6 is judged by the following geometric completeness indicators: in, This indicates the number of vertices in the generated model; Indicates the number of facets in the generated model; This is a preset minimum vertex / face number threshold; Indicates whether there are holes in the model; This indicates whether there are non-manifold edges or vertices in the model.

7. The method according to claim 1, characterized in that, The management of the output buffer in step S7 includes: allocating an independent buffer namespace for each user; attaching a timestamp and retention period to each stored data; and periodically cleaning up expired data in the background. The recommended priority of output formats is calculated using the following formula: in, Indicate the output format Recommended rating; This indicates the format of the user's historical selections in the user preference profile. The number of times; This indicates the total number of times the user has output data in history. This is a function for matching industry scenarios.

8. The method according to claim 1, characterized in that, The confidence level of the preference branch in step S8 is updated according to the following incremental formula: in, The confidence level of this preference branch before the update is initially set to 1; if the output format, style parameters, and text modality weight parameters ultimately adopted by the user are consistent with the current branch record, then... =1, otherwise =0; If the adoption result is inconsistent with the current branch record, then =0.5, otherwise =0; when If the confidence level is below a preset threshold (e.g., 0.5), the branch is marked as a low-confidence branch and its priority is reduced in subsequent recommendations.

9. A system for generating three-dimensional models by combining text and images, used to implement the method described in any one of claims 1 to 8, characterized in that, include: The input integrity verification module is configured to: obtain text description information, at least one two-dimensional image and user identification information, perform integrity verification, package the input data packet into a data packet and generate a unique identifier after the verification is successful, and transmit it to the subsequent module through the data bus; if the verification fails, return the error reason and request retransmission. The user preference management module is connected to the input integrity verification module via a data channel with an acknowledgment signal. It is configured to: retrieve and load the user preference profile from the user storage platform based on the user identification information; if the retrieval fails, initialize the default profile; this module is equipped with a read-write lock. The image-text fusion control and execution module, connected to the user preference management module, includes: a dominant mode decision unit that determines text modality weight parameters based on preference profiles; a text encoder and an image encoder, the output of which is equipped with a null value detector; and an image-text fusion unit that outputs fused feature vectors and sets a dimension validator. The 3D generation and optimization module, connected to the image and text fusion control and execution module, includes: a 3D generation network; an industry optimization sub-network selector that dynamically loads the corresponding sub-network according to the industry category; and a geometric self-checking unit that detects and repairs model defects. The output adaptation and buffering module, connected to the 3D generation and optimization module, includes: a multi-format parallel generator; a user-level output buffer for storing data in various formats; and a user confirmation interface for receiving selection instructions and delivering data. The user storage platform is connected to the user preference management module and the output adapter and buffer module through an encrypted channel, and is configured to: encrypt and store user preference files, provide write confirmation signals, periodically perform integrity checks and restore from the previous version; The data flow monitoring and alarm unit connects to all modules via the data bus and is configured to: monitor the data packet transmission status, receive confirmation messages after each module has finished processing; if no confirmation is received within a timeout, it determines that the data is lost and instructs retransmission, while generating an alarm log.

Images