Three-dimensional model reconstruction method and device, equipment and storage medium
By extracting visual features and text information from two-dimensional images and using visual language models to identify and retrieve component models, the problem of ignoring labeled layer information in the prior art is solved, and the accurate and flexible reconstruction of the three-dimensional model is achieved.
Patent Information
- Application Number
- CN202411568087.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-11-04
AI Technical Summary
The prior art ignores the label layer information of the two-dimensional drawings during the three-dimensional model reconstruction process, resulting in the reconstruction results being inaccurate and flexible enough.
By extracting visual features from two-dimensional images, using visual language models to identify component information with prompt information, and reconstructing component models are retrieved to adjust and reconstruct three-dimensional models, using visual encoder and multi-layer perceptron to extract image features, and combining language models to process text Features, realize accurate reconstruction of three-dimensional models.
It improves the accuracy and flexibility of three-dimensional model reconstruction, and can effectively utilize image and text information to quickly and accurately reconstruct three-dimensional models of complex objects.
Smart Images

Figure CN119068126A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to the field of three-dimensional model reconstruction and three-dimensional modeling technology. Background Art
[0002] Drawings designed using computer-aided design software, such as CAD drawings, usually include geometry layers and annotation layers. The geometry layer contains orthogonal projections that describe the geometry of the 3D model, and the annotation layer can contain dimension annotations, functional symbols, etc. In the process of using design drawings to rebuild a 3D model, the geometry layer is usually used, while the annotation information in the annotation layer is ignored. Summary of the invention
[0003] The present disclosure provides a three-dimensional model reconstruction method, device, equipment and storage medium to solve or alleviate one or more technical problems in the prior art.
[0004] In a first aspect, the present disclosure provides a three-dimensional model reconstruction method, comprising: Extract visual features of an object to be reconstructed from a two-dimensional image; use a visual language model to identify component information of the object based on the visual features and prompt information; wherein the prompt information is used to indicate a three-dimensional reconstruction task that needs to be performed by the visual language model; retrieve a component model of the object based on the component information of the object; adjust the component model of the object based on the component information of the object to reconstruct a three-dimensional model of the object.
[0005] In a second aspect, the present disclosure provides a three-dimensional model reconstruction device, comprising: An extraction module, used for extracting visual features of an object to be reconstructed from a two-dimensional image; A recognition module, used to use a visual language model to identify component information of the object based on the visual features and prompt information; wherein the prompt information is used to indicate a three-dimensional reconstruction task that needs to be performed by the visual language model; A calling module, used for calling a component model of the object based on the component information of the object; An adjustment module is used to adjust the component model of the object based on the component information of the object to reconstruct a three-dimensional model of the object.
[0006] In a third aspect, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.
[0007] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.
[0008] In a fifth aspect, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.
[0011] Figure 1 is a flowchart of a three-dimensional model reconstruction method according to an embodiment of the present disclosure; Figure 2 is a flowchart of a three-dimensional model reconstruction method according to another embodiment of the present disclosure; Figure 3 is a flowchart of a three-dimensional model reconstruction method according to another embodiment of the present disclosure; Figure 4 is an example diagram of the default values of component-specific parameters; Figure 5 and Figure 6 This is an example diagram after changing the number of cabinet grids in the component-specific parameters; Figure 7 and Figure 8 This is an example image after changing the grid width in the component-specific parameters; Fig. 9 and Fig.10 This is an example diagram after changing the position of the door baffle in the component-specific parameters; Fig.11 is a flowchart of a three-dimensional model reconstruction method according to another embodiment of the present disclosure; Fig.12 is a schematic diagram of an example of a model network structure; Fig.13 This is an example of an input drawing; Fig.14 yes Fig.13 Example of the reconstruction result; Fig.15 This is an example of an input drawing; Fig.16 yes Fig.15 Example of the reconstruction result; Fig.17 It is an example diagram of expressing a shape program using a programming language; Fig.18 This is an example diagram of the model's dialogue; Fig.19 is a structural schematic diagram of a three-dimensional model reconstruction device according to an embodiment of the present disclosure; Fig. 20 is a structural schematic diagram of a three-dimensional model reconstruction device according to another embodiment of the present disclosure; Fig.21 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0012] The present disclosure will be further described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0013] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.
[0014] Figure 1 FIG. 1 is a flow chart of a three-dimensional model reconstruction method according to an embodiment of the present disclosure. The method may include: S110, extracting visual features of the object to be reconstructed from the two-dimensional image; S120, using the visual language model to identify component information of the object based on the visual features and the prompt information, so as to reconstruct a three-dimensional model of the object; wherein the prompt information is used to indicate a three-dimensional reconstruction task that needs to be performed by the visual language model; S130, retrieving a component model of the object based on the component information of the object; S140: Adjust the component model of the object based on the component information of the object to reconstruct a three-dimensional model of the object.
[0015] In the disclosed embodiments, a two-dimensional image may be a plane image composed of two-dimensional elements and does not contain depth information, and may also be referred to as a two-dimensional picture, a plane picture, etc. A two-dimensional image may include bitmaps such as photos and pictures, and may also include vector graphics such as CAD plane drawings. Visual features may be extracted based on the geometric layer and / or annotation layer of the two-dimensional image, rather than only based on the geometric layer. There are many ways to extract image features. In one example, the visual features of an object to be modeled can be extracted from the geometric layer and / or annotation layer of a two-dimensional image through image extraction technology such as OCR technology. In another example, visual features can be extracted from an image through an artificial intelligence model such as a visual language model. For example, a two-dimensional CAD image contains a geometric layer and an annotation layer of a cabinet. From the two-dimensional CAD drawing, not only the visual features corresponding to the geometric layer of the cabinet can be extracted, but also the visual features corresponding to the annotation layer of the cabinet can be extracted.
[0016] In the disclosed embodiment, the visual language model can understand and interpret the association between the image and the text, and generate an accurate and vivid natural language description based on the image. The visual language model may include a visual encoder, a multilayer perceptron (MLP), and a language model part. For example, a visual language model is a multimodal large model. The multimodal large model may include a visual encoder, an MLP, and a large language model using a vision transformer (ViT). For example, the multimodal large model may be a Mini-InternVL-1.5-2B model, which may include a visual encoder, an MLP, and a language model of InternViT-300M. The encoder, MLP, language model, etc. in the above-mentioned visual language model may also be of other types, which are not limited in the disclosed embodiment. The prompt information may include the content of the input model used to describe the model task. The prompt information may also be referred to as a prompt word, a prompt word template, a prompt, etc.
[0017] In the disclosed embodiment, the visual features extracted from the two-dimensional image and the prompt information used to describe the three-dimensional reconstruction task can be input into the visual language model, and the component information of each object in the two-dimensional image can be obtained after being processed by the visual language model. An object may include one or more components. For example, a cabinet may include component information such as a cabinet body and a door. The component information of an object can be used to reconstruct the three-dimensional model of the object. For example, from the two-dimensional image of a bicycle, the information of multiple components of the target object "bicycle" can be extracted, namely "body, handlebars, seat, pedals and two wheels". Based on the information of multiple components of the "bicycle", the three-dimensional models of each component can be constructed separately, thereby forming a three-dimensional model of the entire object "bicycle".
[0018] In the disclosed embodiment, the component model of the object to be reconstructed can be retrieved from the database according to the component information obtained from the visual language model, the parameters of the component model retrieved from the database can be adjusted according to the recognized component information, and the 3D model of the object can be established based on the adjusted component model.
[0019] According to the embodiments of the present disclosure, the model of the two-dimensional image is reconstructed in combination with the visual language model and the prompt information, thereby improving the flexibility of the input format and the accuracy of the model reconstruction result.
[0020] Figure 2 3D model reconstruction method according to another embodiment of the present disclosure is a flow chart, the method may include one or more features of the above-mentioned 3D model reconstruction method. In one embodiment, step S110 extracts visual features of the object to be reconstructed from the 2D image, including: S210, using the visual language model to segment the two-dimensional image into multiple image blocks; S220, inputting the multiple image blocks into the visual encoder and the multi-layer perceptron MLP of the visual language model to obtain visual features of the object; wherein the visual features include vector representations obtained based on the mapping of the multiple image blocks.
[0021] In the disclosed embodiment, a two-dimensional image is input into a visual language model, and the visual language model divides the two-dimensional image into a plurality of image patches. For example, image A is input into a visual language model, and image A can be divided into a plurality of image patches such as A1, A2, and A3. Each image patch can represent a local area of the two-dimensional image. There are many ways to extract image patches, such as a sliding window method, a method based on a convolution operation, and a grid method. Among them, the sliding window method can slide a window of a fixed size on the original image, and each time the sliding step length is set, so as to extract a local area of the image as an image patch. The method based on the convolution operation can use the convolution operation to extract the image patch, and can use the sliding of the convolution kernel on the image and the convolution operation. The convolution operation can not only extract the local features of the image, but also control the size and position of the extracted image patch by adjusting the size and step length of the convolution kernel. The grid method can divide the image into a plurality of image patches evenly according to the number of rows and columns of the grid.
[0022] In the disclosed embodiment, the segmented multiple image blocks are input into the visual encoder part of the visual language model to extract image features, and the image features are aligned with the text features using a multi-layer perceptron (MLP). The image features are processed to obtain the visual features of each image block, and then the visual features of the object are obtained. The visual features may include visual tokens. The visual tokens may be vector representations mapped to the image blocks into which the image is segmented, and these vector representations may be regarded as basic processing units in the model.
[0023] According to the embodiments of the present disclosure, a two-dimensional image is segmented using a visual language model, and then visual features are extracted through a visual encoder and MLP, thereby improving the accuracy of the visual feature extraction results.
[0024] Figure 3 3D model reconstruction method according to another embodiment of the present disclosure is a flow chart of the method, which may include one or more features of the above-mentioned 3D model reconstruction method. In one embodiment, step S120 uses a visual language model to identify component information of the object based on the visual features and prompt information, including: S310, inputting the visual feature and the prompt information into the language model part of the visual language model, and identifying and obtaining the identifiers and parameters of one or more components of the object; Step S130 retrieves the component model of the object based on the component information of the object, including: S320, retrieving one or more component models from a database according to the identifiers of the one or more components; Step S140 adjusts the component model of the object based on the component information of the object to reconstruct the three-dimensional model of the object, including: S330: Adjust parameters of corresponding component models according to the parameters of the one or more components to reconstruct a three-dimensional model of the object.
[0025] In an embodiment of the present disclosure, the image features and prompt information extracted by the visual encoder part of the visual language model are input into the language model part of the visual language model, and the identification (Identity document, ID) and parameters of one or more components contained in the object in the two-dimensional image can be obtained after processing by the language model part. The identification of the component can be a number used to uniquely identify the component, and the number can be composed of numbers, characters, etc. The identification of the component can be used as an index in the database, and based on the ID of the component, the component model pre-stored in the database that matches the ID can be found. The pre-stored component model can have some default parameters. According to the parameters of the identified component, the default parameters of the component model corresponding to the component can be adjusted.
[0026] In the disclosed embodiments, the default parameters of different component models are usually not completely the same. For example, component models can be constructed according to item types, such as component models of cabinets, doors, drawers, fixed panels, movable panels, etc. can be constructed separately. For another example, component models can be constructed according to the composition characteristics of the components. For example, a component model of a cabinet including one grid is constructed. For another example, a component model of a cabinet including two upper grids and one lower grid is constructed.
[0027] In the disclosed embodiment, the posture, size, position, etc. of the component model may also be adjusted based on the parameters of the component partially recognized by the language model, so that multiple component models of an object are combined into a three-dimensional model of the object.
[0028] According to the embodiments of the present disclosure, the identification and parameters of each component of the object are obtained through the language model part of the visual language model, the corresponding component model can be retrieved from the database, and the parameters of the component model can be adjusted to complete the reconstruction of the three-dimensional model. The component model of the object can be found quickly and accurately, thereby improving the accuracy of the model reconstruction result.
[0029] In one embodiment, step S330 adjusts the parameters of the corresponding component model according to the parameters of the one or more components to reconstruct the three-dimensional model of the object, including: The pose parameters and specific parameters of the corresponding component model are adjusted according to the parameters of the one or more components to obtain a three-dimensional model of the object; wherein the pose parameters include component position, component size, and component rotation angle.
[0030] In the embodiment of the present disclosure, the parameters of the component partially identified by the language model may include the pose parameters of the component. For example, the pose parameters of the component include the position of the component, the size of the component, the rotation angle of the component, etc. The size of the component may also be a specific parameter of the component. The component model in the database may not have pose parameters, or a default pose parameter may be set. Using the pose parameters of a certain component identified, the pose parameters of the component model corresponding to the component may be adjusted. For example, based on the position of the component of an object in the original two-dimensional image, the three-dimensional coordinates (x, y, z) of the component may be obtained. For another example, the size of the component of an object identified from the original two-dimensional image may include one or more of the length, width, height, etc. of the component of the object. The size of the component model of the component may be adjusted according to one or more of these sizes. For another example, the rotation angle of the component of an object identified from the original two-dimensional image may be the angle between a line on the component of the object and a reference line or reference plane. For example, the angle between the axis of symmetry of the component and the straight line representing the ground. Based on the identified component rotation angle of the component, the angle between the component model corresponding to the component and the plane representing the ground may be adjusted.
[0031] For example, if the length of a component M1 is 30 cm from the original two-dimensional image, and the length of the component model m1 corresponding to the component M1 retrieved from the database is 5 cm, the length of the component model m1 can be adjusted to 30 cm, and other dimensions can be adjusted based on the recognition results. For another example, if the rotation angle of a component M2 is 90° from the original two-dimensional image, and the rotation angle of the component model m2 corresponding to the component M2 retrieved from the database is 180°, the rotation angle of the component model m2 can be adjusted to 90°.
[0032] In the embodiment of the present disclosure, the default parameters of the component model in the database may include specific parameters of the component model. The specific parameters of the component may be parameters unique to the component, and the specific parameters of different components may be completely different or partially different. For example, a specific parameter example of a cabinet component model is as follows: Figure 4 As shown, the specific parameters of the cabinet component model include N=2, NKA=873, NKB=873, DBXX=1, where N represents the number of cabinet grids, NKA and NKB represent the width of each grid, and DBXX represents the position of the door baffle. Figures 5 to 10 This is an example of changing the default specific parameters of the above component model according to the recognition result. For another example, in the component model of a combined drawer, the specific parameters may include the number of drawers, the width and length of the drawer, etc.
[0033] The default parameters of the component model of the component can be adjusted based on the specific parameters of the component identified by the language model. For example, if the component model of a component in the database includes two grids, the default length of each grid is 40cm, the width is 20cm, and the height is 60cm. It is identified from the two-dimensional image that the cabinet includes 3 grids, and a grid A has a length of 50cm, a width of 24cm, and a height of 80cm. The length of grid A in the component model can be adjusted to 50cm, the width to 24cm, and the height to 80cm. If the parameters of the other two grids B and C identified from the two-dimensional image only include a length of 100cm, but not a width or height, the length of the grid B and grid C can be adjusted to 100cm, and the width can be adjusted to 24cm according to the first grid A, and the height can be adjusted to 80cm according to the first grid A.
[0034] If an object includes multiple parts, the pose parameters and specific parameters of the component models corresponding to the various parts of the object are adjusted according to the recognition results, so as to obtain a three-dimensional model composed of the various parts.
[0035] According to the embodiments of the present disclosure, by adjusting the parameters of the component model retrieved from the database according to the parameters of the various components of the object in the original two-dimensional image, a three-dimensional model that conforms to the shape, posture, position, etc. of the object in the two-dimensional image can be obtained, thereby improving the accuracy of the model reconstruction results.
[0036] Fig.11 : is a flow chart of a 3D model reconstruction method according to another embodiment of the present disclosure, and the method may include one or more features of the above 3D model reconstruction method. In one embodiment, the method further includes: S510, pre-establishing a parameterized component model in a database; wherein the component model includes an identification of the component and its corresponding specific parameters.
[0037] In the disclosed embodiment, each component model may be parameterized to characterize the component model as a mathematical language description. The component model may include an identifier of the parameterized component model in a database and specific parameters corresponding to the parameterized component model. Further, the component model may be characterized in the form of code or data serialization. For example, the code may include code formats such as Python code format and Java code format, and the data serialization format may include data serialization formats such as Json format and YAML format.
[0038] For example, a parametric component model Can be characterized as ,in It is the unique ID of the parametric component model in the database. It is the pose parameter of the component model, which can include the center point position ,size and rotation angle wait. is a set of component model specific parameters. Formula Can be expressed by N A model (e.g., a 3D model) that is assembled from a number of parameterized component models.
[0039] Through the embodiments of the present disclosure, by pre-modeling the components, the components can be represented parameterized, which can improve the efficiency of obtaining the three-dimensional model.
[0040] In one embodiment, the identifier of the component is associated with the word-gram of the component model; wherein the features of the word-gram of the component model are obtained by stitching together the text features of the component name and the image features of the component preview image. In one embodiment, the visual language model can search for the corresponding word-gram based on the identifier of the component, and then search for the component model corresponding to the word-gram. In another embodiment, the visual language model can identify the word-gram from the two-dimensional model, and then search for the identifier or component model of the component corresponding to the word-gram.
[0041] In the disclosed embodiments, the word-grams of the component model can be used to characterize the component in the visual language model, and the word-grams of the component model can have an association with the identification of the component. For example, the component name and component preview image of each component model can be pre-set, and the word-gram features of the component model can be generated based on the text features of the component name and the image features of the component preview image. For example, a multimodal model such as the CLIP (Contrastive Language Image Pre-Training) model can be used to extract text features from the component name, extract image features from the component preview image, and perform feature dimension splicing on the text features and image features to obtain the features of the word-grams of the component model.
[0042] According to the embodiments of the present disclosure, the word-grams of the component model can help the visual language model to better understand the component information, improve the accuracy of the component model extracted from the database, and improve the efficiency of extracting the component model.
[0043] In one embodiment, the visual language model is obtained by fine-tuning a large benchmark model based on training data; wherein the training data includes sample images and their corresponding parameterized component models.
[0044] In the disclosed embodiment, the sample image may include a feature vector of the image, and the parameterized component model may be in the format of a code. The training data may also include prompt information that requires the model to perform a three-dimensional reconstruction task. After the training data is input into the benchmark large model, the parameters of the benchmark large model may be supervised and fine-tuned according to the output result of the benchmark large model. After the output result of the benchmark large model meets the training goal, the training may be stopped to obtain a trained visual language model.
[0045] According to the embodiments of the present disclosure, the visual language model is obtained by fine-tuning the large model using a supervised fine-tuning method, which can improve the accuracy of the output results of the visual language model and further improve the accuracy of the model reconstruction results.
[0046] In one embodiment, the two-dimensional image includes a geometry layer and / or an annotation layer. The geometry layer may include an orthogonal projection for describing the geometric information of the three-dimensional model, and the annotation layer may include dimension annotations, functional symbols, etc. Examples of functional symbols may include surface types, manufacturing instructions, etc.
[0047] According to the embodiments of the present disclosure, by acquiring information of the geometric layer and the annotation layer in the two-dimensional image, the accuracy of component model generation can be improved, thereby improving the accuracy of the model reconstruction result.
[0048] In one embodiment, the two-dimensional image includes a bitmap or a vector image.
[0049] In the disclosed embodiments, the two-dimensional image may have a variety of formats, such as a bitmap composed of pixels, a vector image composed of vector graphic elements. A bitmap may also be called a dot map or a raster map, etc., and may be composed of a plurality of pixels. A vector image may also be called an object-oriented image or a drawing image, etc. The graphic elements in the vector image may have attributes such as color, shape, outline, size, and screen position. The vector image may be drawn by some software, such as AutoCAD, etc.
[0050] According to the embodiments of the present disclosure, two-dimensional images in different formats can be used as input images of a visual language model, which can expand the application scope of model reconstruction scenarios.
[0051] The present disclosure provides a method for reconstructing a three-dimensional model, wherein a schematic diagram of an example of a model network structure is shown in FIG. Fig.12 shown.
[0052] The disclosed embodiment can model the reconstruction problem of the 3D model as a visual question answering problem and implement it by fine-tuning a vision language model.
[0053] In one application scenario, the input of the model can be a 2D image such as a complete CAD drawing (including geometry layers and annotation layers) and prompt questions, or a bitmap in a format such as GIF or JPEG and prompt questions. The output of the model can include a natural language (such as program code) used to describe the 3D model.
[0054] An example of a method for reconstructing a 3D custom cabinet model from a CAD drawing is as follows: 1. Algorithm input The input of the algorithm can be a complete CAD drawing, including the geometry layer and the annotation layer, see Fig.13 and Fig.15 .
[0055] 2. Algorithm output In actual modeling, using a set of pre-modeled parametric component models can speed up the design process, reduce design errors and ensure that the product can be manufactured correctly. For example, modeling the components of a custom cabinet. For example, these component models can include basic cabinets, doors, drawers, fixed panels, movable panels, etc. Each component model can be pre-modeled as a parametric model.
[0056] A custom cabinet can be assembled from a set of pre-modeled component models, for example using the formula In which, each component model Can be characterized as ,in It is the unique ID of the parametric component model in the database. are the pose parameters, including the center point position ,size and rotation angle , is a set of component model specific parameters. If the component model has no parameters, .
[0057] An example of specific parameters for a cabinet component model can be found in Figures 4 to 10 . Figure 4 , the default values of the specific parameters of the cabinet component model may include: the number of cabinet grids N=2, the width of each grid NKA=873 and NKB=873, and the position of the door baffle DBXX=1. Figure 5 This is an example where the number of cabinet compartments is changed to N=1. Figure 6 This is an example where the number of cabinet compartments is changed to N=3. Figure 7 This is an example of changing the cabinet grid width to NKA=1100 and NKB=646. Figure 8 This is an example of changing the cabinet grid width to NKA=646 and NKB=1100. Fig. 9 This is an example of changing the cabinet door baffle position to DBXX=2. Fig.10 This is an example of changing the cabinet door baffle position to DBXX=3.
[0058] The component ID is a randomly generated number and has no actual semantics. In order to help the neural network model better understand the component information, a set of additional special tokens can be used to represent the component. For example, tokens are obtained through the two features of each component: (1) the component name and (2) the component preview image. The component preview image can be rendered using default parameters and a fixed camera perspective. The CLIP model can be used to extract text features from the component name and image features from the component preview image, and these two sets of features are concatenated in the feature dimension to serve as the token of each component.
[0059] Since the Large Language Model (LLM) has good code generation capabilities, the 3D model can be represented in the format of code. For example, using programming languages such as Python as proxy languages, Fig.17 An example of a shape program expressed in Python is shown. bbox_0 to bbox_5 represent the pose parameters of multiple parts of an object, and model_0 to model_5 represent the IDs of these parts and their corresponding specific parameters. Based on the example of the shape program, a preview of the reconstruction result can be run and displayed in the corresponding software tool, see Fig.14 and Fig.16 .
[0060] 3. Algorithm Model The algorithm model used may be a visual language model, such as the Mini-InternVL-1.5-2B model. The visual language model may include a visual encoder such as InternViT-300M and a language model such as InternLM2-1.8B. The InternViT-300M model may be used to extract image features. Fig.18 An example diagram of a complete conversation is provided. An example of the prompt information (prompt) input to the language model may include "Reconstruct cabinet from image. The bounding box reference code is as follows: ...", and the output may include a shape program describing the cabinet expressed in Python. Represents image features. Image features can be used as part of the prompt information and input into the model together.
[0061] Use supervised fine-tuning to fine-tune the Mini-InternVL-1.5-2B model. The training samples of the model can be found in Fig.18 During reasoning, drawings and prompts can be provided, and the algorithm will automatically generate program code that describes an object, such as a cabinet structure.
[0062] Fig.19 1 is a flow chart of a 3D model reconstruction device 1000 according to an embodiment of the present disclosure. The device may include: An extraction module 1010, used to extract visual features of an object to be reconstructed from a two-dimensional image; The recognition module 1020 is used to use the visual language model to identify the component information of the object based on the visual features and the prompt information to reconstruct the three-dimensional model of the object; wherein the prompt information is used to indicate the three-dimensional reconstruction task that needs to be performed by the visual language model; A calling module 1030, configured to call a component model of the object based on the component information of the object; The adjustment module 1040 is configured to adjust the component model of the object based on the component information of the object to reconstruct a three-dimensional model of the object.
[0063] Fig. 20 1 is a flow chart of a 3D model reconstruction device 1100 according to another embodiment of the present disclosure, and the device may include one or more features of the above 3D model reconstruction device. In one implementation, the extraction module 1010 includes: A segmentation submodule 1111 is used to segment the two-dimensional image into multiple image blocks using the visual language model; The visual feature submodule 1112 is used to input the multiple image blocks into the visual encoder and the multi-layer perceptron MLP of the visual language model to obtain the visual features of the object; wherein the visual features include vector representations obtained based on the mapping of the multiple image blocks.
[0064] In one embodiment, the recognition module 1020 is used to input the visual feature and the prompt information into the language model part of the visual language model to obtain the identification and parameters of one or more components of the object; A retrieving module 1030, configured to retrieve one or more component models from a database according to the identification of the one or more components; The adjustment module 1040 is used to adjust the parameters of the corresponding component model according to the parameters of the one or more components to reconstruct the three-dimensional model of the object.
[0065] In one embodiment, the adjustment module 1040 is used to adjust the posture parameters and specific parameters of the corresponding component model according to the parameters of the one or more components to obtain a three-dimensional model of the object; wherein the posture parameters include component position, component size, and component rotation angle.
[0066] In one embodiment, the device further comprises: The modeling module 1130 is used to pre-establish a parameterized component model in the database; wherein the component model includes the identification of the component and its corresponding specific parameters.
[0067] In one embodiment, the identifier of the component is associated with a word element of the component model; wherein the features of the word element of the component model are obtained by combining text features of the component name and image features of the component preview image.
[0068] In one embodiment, the visual language model is obtained by fine-tuning a large benchmark model based on training data; wherein the training data includes sample images and their corresponding parameterized component models.
[0069] In one embodiment, the two-dimensional image includes a geometry layer and / or a annotation layer.
[0070] In one embodiment, the two-dimensional image includes a bitmap or a vector image.
[0071] Fig.21 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Fig.21 As shown, the electronic device includes: a memory 1210 and a processor 1220, and the memory 1210 stores a computer program that can be run on the processor 1220. The number of memories 1210 and processors 1220 can be one or more. The memory 1210 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided by the above method embodiment. The electronic device may also include: a communication interface 1230, which is used to communicate with external devices and perform data exchange transmission.
[0072] If the memory 1210, the processor 1220 and the communication interface 1230 are implemented independently, the memory 1210, the processor 1220 and the communication interface 1230 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.21 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0073] Optionally, in a specific implementation, if the memory 1210, the processor 1220 and the communication interface 1230 are integrated on a chip, the memory 1210, the processor 1220 and the communication interface 1230 can communicate with each other through an internal interface.
[0074] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.
[0075] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).
[0076] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (for example: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (for example: infrared, Bluetooth, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.
[0077] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0078] In the description of the embodiments of the present disclosure, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0079] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0080] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.
[0081] The above description is only an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A three-dimensional model reconstruction method, comprising: Extract visual features of the object to be reconstructed from the two-dimensional image; Using a visual language model to identify component information of the object based on the visual features and the prompt information; wherein the prompt information is used to indicate a three-dimensional reconstruction task that needs to be performed by the visual language model; Retrieving a component model of the object based on the component information of the object; The component model of the object is adjusted based on the component information of the object to reconstruct a three-dimensional model of the object.
2. The method according to claim 1, wherein: Extract visual features of the object to be reconstructed from the two-dimensional image, including: Using the visual language model, segmenting the two-dimensional image into a plurality of image blocks; The multiple image blocks are input into the visual encoder and the multi-layer perceptron MLP of the visual language model to obtain the visual features of the object; wherein the visual features include vector representations obtained based on the mapping of the multiple image blocks.
3. The method according to claim 1, wherein: Using a visual language model to identify component information of the object based on the visual features and the prompt information includes: inputting the visual features and the prompt information into a language model part of the visual language model to identify and obtain identifiers and parameters of one or more components of the object; Retrieving the component model of the object based on the component information of the object, including: retrieving one or more component models from a database according to the identification of the one or more components; Adjusting the component model of the object based on the component information of the object to reconstruct the three-dimensional model of the object includes: adjusting parameters of the corresponding component model according to parameters of the one or more components to reconstruct the three-dimensional model of the object.
4. The method according to claim 3, wherein: Adjusting parameters of corresponding component models according to the parameters of the one or more components to reconstruct a three-dimensional model of the object includes: The pose parameters and specific parameters of the corresponding component model are adjusted according to the parameters of the one or more components to obtain a three-dimensional model of the object; wherein the pose parameters include component position, component size, and component rotation angle.
5. The method according to claim 3, wherein: The method further comprises: A parameterized component model is pre-established in the database; wherein the component model includes the identification of the component and its corresponding specific parameters.
6. The method according to claim 5, wherein: The identification of the component is associated with the word element of the component model; wherein the features of the word element of the component model are obtained by splicing the text features of the component name and the image features of the component preview image.
7. The method according to any one of claims 1 to 6, wherein: The visual language model is obtained by fine-tuning a large benchmark model based on training data; wherein the training data includes sample images and their corresponding parameterized component models.
8. The method according to any one of claims 1 to 6, wherein: The two-dimensional image includes a geometric layer and / or a label layer.
9. The method according to any one of claims 1 to 6, wherein: The two-dimensional image includes a bitmap or a vector image.
10. A three-dimensional model reconstruction device, comprising: An extraction module, used for extracting visual features of an object to be reconstructed from a two-dimensional image; A recognition module, used to use a visual language model to identify component information of the object based on the visual features and prompt information; wherein the prompt information is used to indicate a three-dimensional reconstruction task that needs to be performed by the visual language model; A calling module, used for calling a component model of the object based on the component information of the object; An adjustment module is used to adjust the component model of the object based on the component information of the object to reconstruct a three-dimensional model of the object.
11. The device according to claim 10, wherein: The extraction module comprises: A segmentation submodule, configured to segment the two-dimensional image into a plurality of image blocks using the visual language model; A visual feature submodule is used to input the multiple image blocks into the visual encoder and the multi-layer perceptron MLP of the visual language model to obtain the visual features of the object; wherein the visual features include vector representations obtained based on the mapping of the multiple image blocks.
12. The device according to claim 10, wherein: The recognition module is used to input the visual features and the prompt information into the language model part of the visual language model to obtain the identification and parameters of one or more components of the object; the retrieval module is used to retrieve one or more component models from a database according to the identification of the one or more components; the adjustment module is used to adjust the parameters of the corresponding component model according to the parameters of the one or more components to reconstruct the three-dimensional model of the object.
13. The device according to claim 12, wherein: The adjustment module is used to adjust the posture parameters and specific parameters of the corresponding component model according to the parameters of the one or more components to obtain a three-dimensional model of the object; wherein the posture parameters include component position, component size, and component rotation angle.
14. The device according to claim 12, wherein: The device also includes: The modeling module is used to pre-establish a parameterized component model in a database; wherein the component model includes the identification of the component and its corresponding specific parameters.
15. The device according to claim 14, wherein: The identification of the component is associated with the word element of the component model; wherein the features of the word element of the component model are obtained by splicing the text features of the component name and the image features of the component preview image.
16. The device according to any one of claims 10 to 15, wherein: The visual language model is obtained by fine-tuning a large benchmark model based on training data; wherein the training data includes sample images and their corresponding parameterized component models.
17. The device according to any one of claims 10 to 15, wherein: The two-dimensional image includes a geometric layer and / or a label layer.
18. The device according to any one of claims 10 to 15, wherein: The two-dimensional image includes a bitmap or a vector image.
19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.
21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Three-dimensional reconstruction model training method, three-dimensional reconstruction method and device
CN115147558A
Three-dimensional reconstruction method and device and computer storage medium
CN115587928A
Three-dimensional reconstruction model training method, three-dimensional reconstruction method and device
CN117237538A
Model training method and device, storage medium and electronic equipment
CN117635822A
Drawing processing method and device, equipment and storage medium
CN118115692A
Cited By
Model and scene fusion rendering method and device, equipment and storage medium
CN119516078A
