Image-based model generation method and device, equipment and storage medium

Abstract

The invention provides an image-based model generation method and device, equipment and a storage medium, and the method comprises the steps: extracting feature point information in an initial image, and obtaining the feature information of an entity; the feature information is input into a language model to obtain a target text, and the target text is used for tracing attribute information of an entity in the initial image; adjusting attributes of a preset model based on the target text to obtain an initial model; and adjusting the initial model based on the similarity between the initial models of the initial image to obtain a target model. According to the method, the automatic process of generating the corresponding accurate three-dimensional model by using one image is realized by introducing a language processing technology, so that the modeling efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of modeling technology, and in particular to an image-based model generation method, apparatus, device, and storage medium. Background Art

[0002] Rapid prototyping is designed to achieve efficient 3D modeling and visualization, which is not limited to the furniture field but also has important application potential in a wide range of other fields.

[0003] Currently, the creation of three-dimensional models mostly relies on manual creation and adjustment by designers, resulting in low modeling efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide an image-based model generation method, device, equipment and storage medium to address the above technical problems, aiming to improve the modeling rate.

[0005] In a first aspect, the present application provides an image-based model generation method, comprising:

[0006] Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0007] Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0008] Adjusting the attributes of the preset model based on the target text to obtain an initial model;

[0009] Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

[0010] As a feasible embodiment of the present application, adjusting the initial model based on the similarity between the initial image and the initial model to obtain the target model includes:

[0011] Inputting the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image;

[0012] The initial model is adjusted according to the similarity between the first coding feature and the second coding feature to obtain a target model.

[0013] As a feasible embodiment of the present application, adjusting the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model includes:

[0014] Aligning the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature;

[0015] Determining the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model;

[0016] The target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0017] As a feasible embodiment of the present application, the initial image includes multiple frames of images extracted from the initial video; the method further includes:

[0018] Creating a point cloud model based on the feature points in the multiple frames of initial images; and

[0019] The point cloud model is used as the preset model.

[0020] In one possible embodiment of the present application, the initial image includes multiple frames of images extracted from an initial video; and the method further includes:

[0021] Creating a point cloud model based on the feature points in the multiple frames of initial images; and

[0022] The point cloud model is used as the preset model.

[0023] As a feasible embodiment of the present application, extracting feature point information from the initial image to obtain feature information of at least one entity in the initial image includes:

[0024] Inputting the initial image into the trained target model for processing to obtain feature information of at least one entity in the initial image;

[0025] The target model is trained by the following steps:

[0026] Inputting a sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of partial images of the sample image into a second initial network to obtain a second probability distribution;

[0027] Optimizing parameters of the second initial network according to the first probability distribution and the second probability distribution;

[0028] The parameters of the first initial network are optimized based on the optimization information of the parameters of the second initial network until a preset condition is met, and then the current first initial network is determined as the target model.

[0029] As a feasible embodiment of the present application, the method further includes:

[0030] Determining positional relationships between multiple entities in the initial image;

[0031] Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

[0032] In a second aspect, the present application further provides an image-based model generation method, comprising:

[0033] An extraction module, configured to extract feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0034] a processing module, configured to input the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0035] An adjustment module, configured to adjust the properties of a preset model based on the target text to obtain an initial model;

[0036] The adjustment module adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

[0037] In a third aspect, the present application further provides a computer device, comprising:

[0038] one or more processors;

[0039] Memory; and

[0040] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the method provided above.

[0041] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the method provided above.

[0042] In a fifth aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the generation method provided above.

[0043] The method provided in the present application, after inputting a visual image including the entity to be modeled, obtains the feature information of the entity by extracting the feature point information in the visual image, inputs the feature information into the language model, obtains the target text that can be used to describe the attribute information of the entity, adjusts the attributes of the preset model based on the target text, obtains the initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of the steps of an image-based model generation method provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of the steps for training a target model provided in an embodiment of the present application to extract entity feature information from an image to better achieve subsequent entity modeling effects;

[0047] Figure 3 A schematic diagram of the steps for adjusting the initial model provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a process flow for adjusting an initial model based on similarity provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process flow for creating a point cloud model based on multiple frames of images and adjusting the preset model provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a process flow for creating a point cloud model based on feature point matching provided in an embodiment of the present application;

[0051] Figure 7 A schematic structural diagram of an image-based model generation device provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0054] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of this application, "plurality" means two or more, unless otherwise specifically specified.

[0055] In the description of this application, the word "for example" is used to mean "used as an example, illustration or illustration". Any embodiment described in this application as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0056] To clearly understand the methods provided in the embodiments of the present application, the following first describes the application background of image-based model generation methods. Three-dimensional modeling is widely used in many fields, such as mechanical parts, architectural structures, and home furnishings. Currently proposed modular modeling designs can improve modeling results by individually designing each module within the model. However, these designs rely on manual adjustments by the designer, resulting in less than ideal modeling results.

[0057] Precisely in order to solve the above problems, based on the modular design, this application further provides an image-based model generation method, which aims to analyze the feature information of the entities in the image and convert it into corresponding text descriptions through parsing the input image, so as to accurately convert it into the parameters required for three-dimensional model generation in the modular design, thereby effectively improving the generation efficiency of the three-dimensional model.

[0058] For details, please refer to Figure 1 , Figure 1A schematic flow chart of a method for generating an image-based model according to an embodiment of the present application, specifically comprising steps S110 to S140:

[0059] S110 , extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image.

[0060] In the embodiments of the present application, in order to accurately generate an entity model that matches the entity in the image, it is often necessary to first determine the structural information of the entity contained in the image. Specifically, for ease of understanding, the following description will use furniture modeling as an example. Entity feature information typically includes both structural and attribute feature information. For example, structural feature information typically includes information such as shape and size, while attribute feature information typically includes information such as color and material.

[0061] Since the present application proposes a model generation method based on modular design, the characteristic information of the entity here can generally further refer to the characteristic information of each module component within the entity. For example, taking a table and chair as an example, the table and chair model generally includes a tabletop (seat surface) and table legs (chair legs) components. Therefore, the characteristic information of the entity generally includes the characteristic information of the tabletop and table leg components. Of course, for other types of entities, the various module components contained therein can be specifically determined based on the type of entity. The embodiment of the present application does not limit the module components obtained by dividing the entity.

[0062] Furthermore, in the solution provided in the embodiments of this application, a three-dimensional model of an entity can be accurately restored using only a single image containing the entity. However, this should not be construed as a limitation of the solution of this application. In fact, when the initial image contains multiple images, the solution provided in this application can also more accurately restore the original features of the entity from the multiple images, thereby constructing a more accurate three-dimensional model. Specifically, for ease of understanding, the modeling process of multiple initial images will be described in an embodiment provided later.

[0063] Of course, it should be noted that there are many implementation schemes for extracting feature point information from an image through image processing to obtain feature information of an entity. For example, as a common feasible implementation scheme, the initial image can be processed by introducing a neural network model. That is, in a feasible implementation scheme, extracting feature point information from the initial image to obtain feature information of at least one entity in the initial image includes:

[0064] The initial image is input into the trained target model for processing to obtain feature information of at least one entity in the initial image.

[0065] The trained target model can be a deep network model, a convolutional network model, or other network models trained using machine learning. However, based on research on the effectiveness of subsequent modeling, in one embodiment of this application, a deep model generated through self-supervised learning in the absence of labeled data is proposed to better ensure the effectiveness of subsequent modeling. Specifically, the specific strategy for training the target model is described below.

[0066] See also Figure 2 , Figure 2 A training target model provided in an embodiment of the present application is used to extract entity feature information from an image to better achieve subsequent entity modeling. Specifically, the training target model generally includes steps S210 to S230:

[0067] S210: Inputting a sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of partial images of the sample image into a second initial network to obtain a second probability distribution.

[0068] In order to achieve self-supervised training of the model, a method based on knowledge distillation is proposed in the embodiment of the present application. Specifically, in the embodiment of the present application, a network structure consisting of a first initial network and a second initial network is designed, wherein the first initial network and the second initial network have the same model structure and can both process the input image into a normalized probability distribution in K dimensions.

[0069] On the basis of the foregoing, by generating a set of different views from a given image, such as multiple local views with smaller resolutions, the global view, that is, the original sample image, can be input into the first initial network of the teacher model for processing to obtain a first probability distribution, and then multiple local views with smaller resolutions, that is, local images, can be input into the second initial network of the student model for processing to obtain multiple second probability distributions.

[0070] S220: Optimize parameters of the second initial network according to the first probability distribution and the second probability distribution.

[0071] Unlike conventional strategies that train the model based on the difference between the first and second probability distributions to minimize the difference between the first and second probability distributions, the technical solution provided in this application learns to match these distributions by minimizing the cross-entropy loss of the parameters of the student network and the teacher network. In other words, stochastic gradient descent is used to minimize the equation to learn the parameters of the student model, that is, the parameters of the second initial network. For the teaching network, that is, the first initial network, a different strategy is used to optimize the parameters. Please refer to the subsequent description of step S230 for details.

[0072] S230: Optimize the parameters of the first initial network based on the optimization information of the parameters of the second initial network until a preset condition is met, and then determine the current first initial network as the target model.

[0073] Compared to the strategy of using gradients to synchronously adjust the parameters of the first initial model in related technologies, in the embodiment of the present application, the parameters of the first initial network are optimized at a certain ratio based on the optimization information of the parameters of the second initial network. In other words, the parameters of the optimized first initial network can be understood as the weighted sum of the parameters of the original first initial network and the optimized increments of the parameters of the first initial network. After a certain training, when the preset conditions are met, such as the similarity between the first probability distribution and the second probability distribution is sufficient, the current first initial network can be determined as the trained target model for extracting feature point information from the initial image.

[0074] Specifically, to facilitate understanding of the complete implementation process of the training target model provided above, the following will be explained in detail, as follows:

[0075] Image information from multiple different perspectives of an image is input to the student model and the teacher model respectively. The two network models have the same architecture but different parameters. The output of the teacher model is clustered (centering) by the average value of a batch of training data. Each network outputs a K-dimensional feature and is normalized using Softmax. The cross-entropy loss is then used as the objective function to calculate the similarity between the student model and the teacher model. The stop-gradient (sg) operator is used on the teacher model to block the propagation of the gradient, and only the gradient is passed to the student model so that the student model updates the parameters. The teacher model is updated using the exponential moving average (EMA) of the parameters in the student model. After training is completed, the teacher model that performs better than the student model and can provide higher-quality target features can be regarded as the trained target model for extracting feature point information of the initial image.

[0076] S120: Input the feature information into a language model to obtain a target text.

[0077] After completing the extraction of feature point information in the initial image and obtaining the feature information of the entity through the method provided above, in order to enable the code to better understand the extracted feature information and to more accurately realize the automated modeling process of the model, in an embodiment of the present application, it is considered to combine the natural language processing capabilities of the language model with the three-dimensional modeling of the code, and define the parameters and properties of the three-dimensional model through natural language description, thereby realizing a seamless transition for users from image input to text description to model generation.

[0078] Specifically, in the embodiment of the present application, the language model used can be a generative large language model, that is, it can describe the input information in the form of natural language, wherein the target text obtained can usually be used to describe the attribute information of the entity in the initial image. For example, in a common Xi'an solution, the target text can be descriptive information such as "a rectangular wooden table with four legs and rounded corners on the table top", which usually defines the entity or the size, color, material, shape and other information of multiple components in the entity. Of course, the target text provided above is only a feasible implementation scheme and is not intended to limit the technical solution of the present application. In fact, with the development of language models, the target text obtained by inputting feature information into the language model will be able to describe the feature information of the entity in the image in a more accurate language.

[0079] S130: Adjust the attributes of the preset model based on the target text to obtain an initial model.

[0080] In an embodiment of the present application, after the feature information is processed through the language model as mentioned above and the target text for describing the attribute information of the entity is obtained, by further mapping the text to the code process for the modeling parameters of each component in the modular design, some attribute parameters of the preset model can be adjusted based on the form of automatic adjustment of the code, thereby obtaining an initial model.

[0081] For example, to facilitate understanding, taking furniture as an example, the "rectangle" in the target text can be converted into geometric dimensions (such as length, width, and thickness) and passed to the modular components in the code to complete the adjustment of the properties of the corresponding components. Similarly, "wood" can be mapped to a specific material texture and respond to the modular components passed to the code. The embodiment of the present application does not limit the implementation steps of adjusting the properties of the preset model based on the target text to obtain the initial model.

[0082] Furthermore, in a feasible implementation scheme, the preset model here can be the initial model preset in the modular components, for example, the initial table and chair model or other initial furniture model, but as another feasible implementation scheme of the present application, an implementation scheme is also provided for constructing the initial model through point cloud data when the original image includes multiple images. The specific implementation content will be described in detail later.

[0083] S140: Adjust the initial model based on the similarity between the initial image and the initial model to obtain a target model.

[0084] On the basis of the solution provided above, after completing the adjustment of the properties of the preset model based on the target text, in order to ensure that the generated three-dimensional model is highly matched with the initial image, the solution provided in the embodiment of the present application further considers the similarity between the image and the model to perform appropriate alignment optimization and adjustment on the model to obtain a target model that is more accurately in line with the initial image.

[0085] It should be noted that, considering that the image and the model are contents in different modalities, the solution provided in the embodiment of the present application considers using a multimodal model to map the image and the model to the same dimensional space and then adjust and match the model. Figure 3 , Figure 3 The flowchart of the steps for adjusting the initial model provided in the embodiment of the present application specifically includes steps S310 to S320:

[0086] S310: Input the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image.

[0087] In an embodiment of the present application, by inputting data of different modalities, such as models and images, into a multimodal model with multimodal data processing capabilities, the model and image can be respectively mapped to the first coding feature and the second coding feature in the same dimensional space, so as to describe the corresponding feature information in the initial model and the initial image respectively through the coding features in the same spatial dimension.

[0088] S320: Adjust the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model.

[0089] In the embodiment of the present application, since the initial model and the initial image are mapped to the same dimensional space, by comparing the similarity between the first coding feature and the second coding feature, it can be determined whether the initial model needs to be adjusted, and whether the mismatched parts in the initial model need to be adjusted, thereby obtaining the corresponding target model. Specifically, the similarity can usually be calculated based on distance or cosine similarity, which is not limited in the embodiment of the present application. The higher the similarity, the higher the degree of match between the initial model and the initial image. Conversely, the lower the similarity, the lower the degree of match between the initial model and the initial image, and further adjustments need to be made to the initial model.

[0090] Specifically, based on the similarity between the first coding feature and the second coding feature, the initial model is adjusted to obtain the target model. This can be achieved by aligning the coding features and determining the coding attribute information corresponding to each coding region in the coding features, thereby determining the target region to be adjusted in the initial model to adjust the initial model. For details, please refer to Figure 4 , Figure 4 A flowchart of a process for adjusting an initial model based on similarity provided in an embodiment of the present application, specifically comprising steps S410 to S430:

[0091] S410: Align the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature.

[0092] Based on the idea of multimodal pre-training of CLIP (Contrastive Language-Image Pre-training), in an embodiment of the present application, by performing feature alignment processing on the coding features obtained by model processing of different modalities, the coding features of the target model and the matching content in the initial image can be aligned, so that the coding attribute information corresponding to each coding area in the coding feature can be determined, for example, the shape information corresponding to a certain component, such as a table top, or the size information of the table legs, etc.

[0093] S420: Determine the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model.

[0094] In an embodiment of the present application, after different coding regions in the coding feature correspond to the coding attribute information of different regions in the image (model), the similarity between the first coding feature and the second coding feature can be understood as the similarity between these coding regions. That is, the similarity between these coding regions can be compared with a preset similarity threshold to determine the coding attribute information corresponding to the coding region with a similarity lower than the preset similarity threshold as the target region to be adjusted in the initial model. For example, when the similarity between the coding features of the coding region describing the shape of the desktop in the model and the image is low, it can be considered that the shape of the desktop in the created model does not match the initial image. Therefore, the desktop can be regarded as the target region to be adjusted in the initial model, that is, the desktop region.

[0095] S430: Adjust the target area in the initial model based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0096] In an embodiment of the present application, after the identification of the non-matching target area is completed as described above, the target area in the initial model is further adjusted using the feature information of the corresponding coding area in the second coding feature. For example, the shape of the desktop in the model is adjusted to the feature shape corresponding to the image, and finally a target model that better matches the initial image can be generated, thereby further improving the generation accuracy of the target model.

[0097] The method provided in the present application, after inputting a visual image including the entity to be modeled, obtains the feature information of the entity by extracting the feature point information in the visual image, inputs the feature information into the language model, obtains the target text that can be used to describe the attribute information of the entity, adjusts the attributes of the preset model based on the target text, obtains the initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0098] In addition, in order to more accurately improve the accuracy of the generated model, as another feasible embodiment of the present application, the initial image provided by the embodiment of the present application can also be extracted from the initial scene time frequency to obtain multiple frames of images under a series of different perspectives. In this case, by fusion processing of the multiple frames of images, the characteristic information of each entity can be more accurately analyzed to complete the modeling more accurately. For example, in a feasible implementation scheme, please refer to Figure 5 , Figure 5 A schematic flow chart of a step of creating a point cloud model based on multiple frames of images as a preset model for adjustment is provided in an embodiment of the present application, specifically including steps S510 to S520:

[0099] S510: Create a point cloud model based on the feature points in the multiple frames of initial images.

[0100] In an embodiment of the present application, by analyzing images covering various perspectives of the entity and extracting feature point information, rich multi-perspective information can be extracted. That is, based on matching the feature points extracted from multiple frames of initial images, operations such as deduplication and aggregation of these feature points can be completed, and an initialized sparse point cloud model can be constructed.

[0101] Specifically, in order to further improve the effect of the constructed point cloud model and avoid the problem that the key points extracted when dealing with occlusion and dynamic objects are difficult to meet the requirements, as a further feasible implementation scheme of this application, a method of realizing image pose estimation based on point-by-point voting is proposed to better match and align feature points, thereby creating a more accurate point cloud model. For details, please refer to Figure 6 , Figure 6 A schematic flow chart of steps for creating a point cloud model based on feature point matching provided in an embodiment of the present application, specifically including steps S610 to S630:

[0102] S610: Determine the viewing angle posture corresponding to each initial image based on point-by-point voting.

[0103] In this embodiment, a point-by-point voting method based on geometry guidance is proposed to eliminate the differences in rotation and displacement between frames through normalization. The pose generated in each frame is obtained by cumulative calculation of point-by-point voting, avoiding key point dependence.

[0104] Specifically, the rotational pose of an image is determined by calculating the normal of each point. The rotation vote of each point is determined by the normal and the geometric features of the neighboring points. Ultimately, the rotation votes of all points are aggregated into the final rotational pose R by minimizing the error. R is the rotation matrix of the view pose corresponding to the initial image relative to the reference view pose.

[0105] Similarly, for image translation, point-by-point voting can be generated based on point displacement. The accumulated translation votes for each point yield the overall translation information T. By increasing the visibility weight of each point, robustness under occlusion conditions is further improved. T is the translation matrix of the initial image's view pose relative to the baseline view pose. R and T together determine the initial image's view pose.

[0106] S620: Perform transformation processing on the initial images according to the viewing angle posture corresponding to each of the initial images to obtain a mapping image under the same viewing angle.

[0107] In an embodiment of the present application, after the viewing angle posture of each initial image is memorized based on point-by-point voting, the corresponding initial image is transformed using the viewing angle posture, and all the initial images can be mapped to the same reference posture, that is, the mapping image under the reference view angle, so as to more accurately match the feature points.

[0108] S630: Match and fuse the feature points in the mapped image to generate the point cloud model.

[0109] In an embodiment of the present application, by matching and fusing the feature points in multiple mapped images obtained under the same perspective, duplicate feature points can be effectively filtered out and the feature points can be aggregated, thereby obtaining point cloud data with a certain sparse density. The point cloud model constructed by the point cloud data will further provide guidance for the subsequent model generation process. For example, it can be used as a preset model for the attribute information to be adjusted.

[0110] S520: Using the point cloud model as the preset model.

[0111] In an embodiment of the present application, the point cloud model obtained by the aforementioned multi-frame image processing is used as a preset model for the initial attribute parameters to be adjusted. Compared with directly using the preset initialized model in the program library, the model can better match the three-dimensional feature information of the image entity, thereby improving the accuracy of the subsequent generated model.

[0112] Of course, in addition to the target generation module provided above, when the initial image further includes multiple entities, the positional relationship between these entities can be determined by processing the initial image, thereby further determining the position information of the generated target model and obtaining the target scene. In other words, the image-based model generation method provided by this application also includes:

[0113] Determining positional relationships between multiple entities in the initial image;

[0114] Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

[0115] Specifically, in order to clearly understand the solution provided by the embodiment of the present application, the following will be combined with the above Figures 1 to 6 The content provided provides a complete execution step of an image-based model generation method, which specifically includes the following steps:

[0116] (1) Model-based image analysis and feature extraction, specifically including the following steps:

[0117] Object Detection: Leveraging the feature extraction capabilities of DINO (DETR with Improved deNoising AnchOrboxes), an end-to-end object detector, DINO can identify key objects in an image without manual annotation. DINO can accurately identify specific objects such as furniture and appliances in an image, allowing for separate processing of different types of objects within an image. For example, the system can distinguish and identify "table," "chair," or "lamp" within an image.

[0118] Keypoint Detection: After identifying an object, DINO further identifies its key structural points, such as edges, corners, and connection points, through feature embedding. This process establishes a framework for 3D modeling, ensuring that each part possesses independent geometric characteristics, providing a precise coordinate basis for subsequent 3D reconstruction.

[0119] Shape and Surface Estimation: Through DINO’s feature embedding, the system is able to extract the object’s basic shape, curves, angles, and surface contours. This information not only helps to build the object’s 3D framework but also defines the object’s specific geometric characteristics, making the modeling result more consistent with the true characteristics of the original image.

[0120] (2) Generate text description using a generative language model, which includes the following steps:

[0121] Object Description: A generative language model generates a natural language description based on the structural and material features derived from image analysis, such as "a round wooden table with four legs." In this way, the generative language model transforms structural information into a human-understandable description, laying the foundation for the subsequent 3D modeling step.

[0122] Hints and detailed descriptions: By providing specific hints, the generative language model can generate detailed descriptions, including hierarchical details of each component, such as material, color, size, etc. This allows for a clear description of each object, providing specific attribute parameters for subsequent modeling.

[0123] (3) Parameter mapping and construction of the three-dimensional model, specifically including the following steps:

[0124] Description-to-Parameter Mapping: Natural language descriptions generated by generative language models can be converted into specific 3D model parameters. For example, "round wooden table" can correspond to a circular tabletop and wood texture, while "four legs" specifies the number, position, and thickness of the legs. Through this mapping, the properties of each component are precisely defined.

[0125] Geometry Generation and Composition: The system uses a 3D modeling library (such as Plotly or the Blender API) to gradually generate the 3D shape of each component based on the described parameters, ultimately combining these components into a complete model. This step-by-step generation approach not only ensures accurate representation of each part of the model but also maintains a modular structure, facilitating subsequent adjustments and optimizations.

[0126] (4) Iterative improvement based on optimization, specifically including the following steps:

[0127] Optimized alignment: By applying the image-text alignment features of the Contrastive Language-Image pre-trained model to the optimization of 3D models, the system can identify mismatches in shape, color, or scale in the generated model. The Contrastive Language-Image pre-trained model compares the image with the generated model and outputs areas of inconsistency, which provide feedback to guide further adjustments to the model.

[0128] Dynamic Optimization and Feedback: Based on feedback from the pre-trained language-image model, the system gradually adjusts the model's properties, such as making precise changes to shape, size, and material. Through rounds of feedback and adjustments, the resulting 3D model closely matches the detailed requirements of the original image.

[0129] When the input image includes multiple images, such as multiple images of a video frame, the specific steps of generating the model include:

[0130] First, keyframes are extracted from the video. By segmenting and analyzing the video frames, the system extracts images covering all viewpoints of the target object. These keyframes provide rich multi-view information, enabling the system to reconstruct the complete structure of the 3D model. To automate the extraction process, a deep learning-based keyframe detection algorithm is often used to ensure that each frame contributes unique geometric viewpoint information.

[0131] Based on the keyframes, the system generates key feature points for each frame using DINO or a similar feature extraction model to capture the object's shape, boundaries, and details. During the feature extraction process, self-supervised learning methods can reduce the need for data annotation while also generating consistent features across multiple viewpoints. Next, the system uses these feature points to construct a preliminary 3D point cloud model, creating a sparse 3D point cloud by fusing 2D information from multiple frames into 3D data.

[0132] The generated point cloud model can be optimized using methods such as COLMAP (an open-source computer vision software) or Structure-from-Motion (SfM) to obtain a more complete and dense 3D structure. COLMAP can match and fuse feature points from multiple frames, generating a preliminary 3D structure by triangulating images from multiple perspectives. However, the sparsity of point cloud data is still high, so depth completion technology is needed to further densify the model so that the generated model is closer to the shape and details of the real object.

[0133] Based on the initial point cloud, optimization models (such as CLIP) can be used for post-processing to ensure a more accurate visual model. CLIP uses image-text similarity analysis to provide feedback and optimize the model's shape, texture, and details, ensuring that the resulting 3D model is more consistent with the visual information in the video data. The optimization process, through multiple iterations, gradually adjusts the model's details to ensure the accuracy and consistency of the 3D model.

[0134] In order to understand the effect of the model generated by the model generation method provided in this application, this application also provides an evaluation of the generated model, mainly referring to the relevant indicators of single image reconstruction. The evaluation indicators include:

[0135] Structural Similarity (SSIM): measures the structural similarity between the generated model and the reference image.

[0136] Chamfer Distance: Calculates the point cloud distance between the generated model and the original model to evaluate the geometric error.

[0137] Visual similarity (CLIP similarity): Score the similarity between the input image and the generated model image based on CLIP.

[0138] Material accuracy: Determines how well the material of the generated model matches the material of the real image, ensuring consistency in material and color.

[0139] Among them, in terms of the SSIM indicator, the average structural similarity between the generated model and the reference image reaches 0.85, indicating that the model can better capture the shape and structure of the object.

[0140] In terms of Chamfer Distance, experimental results show that the model's geometric consistency reaches a good level, especially for regular objects such as tables and chairs, with a high degree of geometric restoration.

[0141] In terms of visual similarity, CLIP has a higher similarity score, especially with the help of multi-angle input images, the generated model is closer to the visual effect of the original image.

[0142] In terms of material accuracy, experiments show that the generated material matches the material of the input image by more than 85%, reflecting the accuracy of the model in material generation.

[0143] Overall, this method performs well under the conditions of multi-angle image input and can achieve high-quality 3D reconstruction.

[0144] In addition, in order to verify the contribution of each module to the overall performance, based on ablation experiments, some modules are gradually removed or replaced to observe their impact on the final results:

[0145] Removing CLIP optimization: Without CLIP optimization, the visual similarity score decreases by about 15%, indicating that CLIP optimization plays an important role in ensuring the visual consistency of the generated model.

[0146] Text description generation without the generative language model: After removing the description of the generative language model, the model loses detailed control over materials, shapes, and colors, and the material accuracy of the generated model decreases significantly.

[0147] Remove DINO feature extraction: Directly using the features of the original image, the generated 3D model has a significant decrease in geometric consistency and structural similarity.

[0148] Ablation experiment results show that DINO, generative language model and CLIP module all make important contributions to the accuracy and consistency of the model.

[0149] On the basis of the above-mentioned image-based model generation method, as another embodiment of the present application, an image-based model generation device is also provided. For details, please refer to Figure 7 , Figure 7 A schematic structural diagram of an image-based model generation device provided in an embodiment of the present application, specifically comprising:

[0150] An extraction module 710 is configured to extract feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0151] A processing module 720 is configured to input the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0152] An adjustment module 730 is configured to adjust the attributes of the preset model based on the target text to obtain an initial model;

[0153] The adjustment module 730 adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

[0154] In a feasible embodiment of the present application, the adjustment module 730 is also used to input the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image; and adjust the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model.

[0155] In a feasible embodiment of the present application, the adjustment module 730 is also used to perform feature alignment on the first coding feature and the second coding feature to determine the coding attribute information corresponding to each coding area in the coding feature; the coding attribute information corresponding to the coding area whose similarity is lower than the preset similarity threshold is determined as the target area to be adjusted in the initial model; and the target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0156] In a feasible embodiment of the present application, the adjustment module 730 is further configured to create a point cloud model based on feature points in the multiple frames of initial images; and use the point cloud model as the preset model.

[0157] In a feasible embodiment of the present application, the adjustment module 730 is also used to determine the viewing angle posture corresponding to each initial image based on point-by-point voting; transform the initial image according to the viewing angle posture corresponding to each initial image to obtain a mapping image under the same viewing angle; match and fuse the feature points in the mapping image to generate the point cloud model.

[0158] In a feasible embodiment of the present application, the extraction module 710 is also used to input the initial image into a trained target model for processing to obtain feature information of at least one entity in the initial image; the target model is trained by the extraction module 710 through the following steps: inputting the sample image into the first initial network to obtain a first probability distribution, and inputting multiple local images of the sample image into the second initial network to obtain a second probability distribution; optimizing the parameters of the second initial network according to the first probability distribution and the second probability distribution; optimizing the parameters of the first initial network based on the optimization information of the parameters of the second initial network, until the current first initial network is determined as the target model after the preset conditions are met.

[0159] In a feasible embodiment of the present application, the adjustment module 730 is further used to determine the positional relationship between multiple entities in the initial image; determine the position information of multiple target models based on the positional relationship, and obtain the target scene.

[0160] After inputting a visual image including the entity to be modeled, this application obtains the feature information of the entity by extracting the feature point information in the visual image, and then inputs the feature information into the language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then based on the similarity between the initial image and the initial model, the obtained model is adjusted in detail again, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0161] In some embodiments of the present application, the image-based model generation device can be implemented in the form of a computer program, which can be used in Figure 8 The computer device is run on the computer device shown. The memory of the computer device can store various program modules that constitute the image-based model generation device, such as, Figure 7 The illustrated extraction module 710, processing module 720, and adjustment module 730. The computer program composed of various program modules enables the processor to execute the steps of the particle animation generation method of each embodiment of the present application described in this specification.

[0162] For example, Figure 8 The computer device shown can be Figure 7 The extraction module 710 in the image-based model generation device shown executes step S110. The computer device can execute step S120 through the processing module 720. The computer device can execute steps S130 and S140 through the adjustment module 730. The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, an image-based model generation method is implemented.

[0163] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0164] In some embodiments of the present application, a computer-readable storage medium is provided, storing a computer program. The computer program is loaded by a processor, causing the processor to perform the following steps:

[0165] Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0166] Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0167] Adjusting the attributes of the preset model based on the target text to obtain an initial model;

[0168] Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

[0169] After inputting a visual image including the entity to be modeled, this application obtains the feature information of the entity by extracting the feature point information in the visual image, and then inputs the feature information into the language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then based on the similarity between the initial image and the initial model, the obtained model is adjusted in detail again, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0170] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, information library or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0171] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above is a detailed introduction to the image-based model generation method, device and computer equipment provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image-based model generation method, characterized in that: include: Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image; Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image; Adjusting the attributes of the preset model based on the target text to obtain an initial model; Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

2. The method according to claim 1, characterized in that The adjusting the initial model based on the similarity between the initial image and the initial model to obtain a target model includes: Inputting the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image; The initial model is adjusted according to the similarity between the first coding feature and the second coding feature to obtain a target model.

3. The method according to claim 2, characterized in that The adjusting the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model includes: Aligning the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature; Determining the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model; The target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

4. The method according to claim 1, wherein The initial image includes multiple frames of images extracted from the initial video; the method further includes: Creating a point cloud model based on the feature points in the multiple frames of initial images; and The point cloud model is used as the preset model.

5. The method according to claim 4, characterized in that The step of creating a point cloud model based on the feature points in the multiple frames of initial images includes: Determine the viewpoint pose corresponding to each initial image based on point-by-point voting; transforming the initial images according to the viewing angle and posture corresponding to each of the initial images to obtain a mapping image at the same viewing angle; Matching and fusing the feature points in the mapped image to generate the point cloud model.

6. The method according to claim 1, characterized in that The extracting feature point information from the initial image to obtain feature information of at least one entity in the initial image includes: Inputting the initial image into the trained target model for processing to obtain feature information of at least one entity in the initial image; The target model is trained by the following steps: Inputting a sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of partial images of the sample image into a second initial network to obtain a second probability distribution; Optimizing parameters of the second initial network according to the first probability distribution and the second probability distribution; The parameters of the first initial network are optimized based on the optimization information of the parameters of the second initial network until a preset condition is met, and then the current first initial network is determined as the target model.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determining positional relationships between multiple entities in the initial image; Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

8. An image-based model generation device, characterized in that: include: An extraction module, configured to extract feature point information from an initial image to obtain feature information of at least one entity in the initial image; a processing module, configured to input the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image; An adjustment module, configured to adjust the properties of a preset model based on the target text to obtain an initial model; The adjustment module adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the image-based model generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded by a processor to execute the image-based model generation method according to any one of claims 1 to 7.

Images