Image-based model generation method and device, equipment and storage medium

Abstract

The invention provides an image-based model generation method and device, equipment and a storage medium, and the method comprises the steps: extracting feature point information in an initial image, and obtaining the feature information of an entity; the feature information is input into a language model to obtain a target text, and the target text is used for tracing attribute information of an entity in the initial image; adjusting attributes of a preset model based on the target text to obtain an initial model; and adjusting the initial model based on the similarity between the initial models of the initial image to obtain a target model. According to the method, the automatic process of generating the corresponding accurate three-dimensional model by using one image is realized by introducing a language processing technology, so that the modeling efficiency is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of modeling technology, and in particular to an image-based model generation method, device, equipment and storage medium. Background Art

[0002] Rapid prototyping is designed to achieve efficient 3D modeling and visualization, which is not limited to the furniture field, but also has important application potential in a wide range of other fields.

[0003] Currently, the creation of 3D models mostly relies on manual creation and adjustment by designers, resulting in low modeling efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide an image-based model generation method, device, equipment and storage medium to address the above technical problems, aiming to improve the modeling rate.

[0005] In a first aspect, the present application provides an image-based model generation method, comprising:

[0006] Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0007] Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0008] Adjusting the attributes of the preset model based on the target text to obtain an initial model;

[0009] Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

[0010] As a feasible embodiment of the present application, adjusting the initial model based on the similarity between the initial image and the initial model to obtain the target model includes:

[0011] Inputting the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image;

[0012] According to the similarity between the first coding feature and the second coding feature, the initial model is adjusted to obtain a target model.

[0013] As a feasible embodiment of the present application, adjusting the initial model according to the similarity between the first coding feature and the second coding feature to obtain the target model includes:

[0014] Aligning the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature;

[0015] Determine the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model;

[0016] The target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0017] As a feasible embodiment of the present application, the initial image includes multiple frames of images extracted from the initial video; the method further includes:

[0018] Creating a point cloud model based on the feature points in the multiple frames of initial images; and

[0019] The point cloud model is used as the preset model.

[0020] As a feasible embodiment of the present application, the initial image includes multiple frames of images extracted from the initial video; the method further includes:

[0021] Creating a point cloud model based on the feature points in the multiple frames of initial images; and

[0022] The point cloud model is used as the preset model.

[0023] As a feasible embodiment of the present application, extracting feature point information in the initial image to obtain feature information of at least one entity in the initial image includes:

[0024] Inputting the initial image into the trained target model for processing to obtain feature information of at least one entity in the initial image;

[0025] The target model is trained by the following steps:

[0026] Inputting a sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of partial images of the sample image into a second initial network to obtain a second probability distribution;

[0027] Optimizing parameters of the second initial network according to the first probability distribution and the second probability distribution;

[0028] The parameters of the first initial network are optimized based on the optimization information of the parameters of the second initial network until the current first initial network is determined as the target model after a preset condition is met.

[0029] As a feasible embodiment of the present application, the method further includes:

[0030] Determining positional relationships between a plurality of entities in the initial image;

[0031] Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

[0032] In a second aspect, the present application also provides an image-based model generation method, comprising:

[0033] An extraction module, used to extract feature point information in the initial image to obtain feature information of at least one entity in the initial image;

[0034] A processing module, used for inputting the feature information into a language model to obtain a target text, wherein the target text is used for describing the attribute information of the entity in the initial image;

[0035] An adjustment module, used to adjust the properties of the preset model based on the target text to obtain an initial model;

[0036] The adjustment module adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

[0037] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0038] one or more processors;

[0039] Memory; and

[0040] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method provided above.

[0041] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the method provided above.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the generation method provided above.

[0043] The method provided by the present application, after inputting a visual image including an entity to be modeled, obtains feature information of the entity by extracting feature point information in the visual image, and then inputs the feature information into a language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of the steps of an image-based model generation method provided in an embodiment of the present application;

[0046] Figure 2 A schematic flow chart of the steps of training a target model provided in an embodiment of the present application for extracting entity feature information in an image to better achieve subsequent entity modeling effects;

[0047] Figure 3 A schematic diagram of the steps for adjusting the initial model provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a process flow for adjusting an initial model based on similarity provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a process flow of creating a point cloud model based on multiple frames of images as a preset model for adjustment provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of a process flow for creating a point cloud model based on feature point matching provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of an image-based model generation device provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0054] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0055] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0056] In order to clearly understand the method provided by the embodiment of the present application, the application background of the image-based model generation method is first described below, wherein three-dimensional modeling is relied upon in many fields, such as mechanical parts, building structures, or home furnishings, etc. Some modular modeling designs currently proposed have been able to achieve separate design of each module in the model to improve the modeling effect, but rely on manual adjustment by designers, resulting in less than ideal modeling effects.

[0057] Precisely in order to solve the above-mentioned problems, on the basis of modular design, the present application further provides an image-based model generation method, which aims to analyze the feature information of the entities in the image and convert it into corresponding text descriptions through parsing the input image, so as to accurately convert it into the parameters required for three-dimensional model generation in the modular design, thereby effectively improving the generation efficiency of the three-dimensional model.

[0058] For details, please refer to Figure 1 , Figure 1A schematic flow chart of a method for generating a model based on an image provided in an embodiment of the present application, specifically, comprising steps S110 to S140:

[0059] S110, extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image.

[0060] In the embodiments of the present application, in order to accurately generate an entity model that matches the entity in the image, it is often necessary to first determine the structural information of the entity contained in the image. Specifically, for ease of understanding, the following will take the modeling in the field of furniture as an example for explanation, wherein the characteristic information of the entity generally includes the structural characteristic information and attribute characteristic information of the entity, for example, the structural characteristic information generally includes information such as shape and size, while the attribute characteristic information generally includes information such as color and material.

[0061] Among them, since the present application proposes a model generation method based on modular design, the characteristic information of the entity here can usually further refer to the characteristic information of each module component in the entity. For example, taking tables and chairs as an example, the table and chair model usually includes a tabletop (seat surface) and table legs (chair legs) components. Therefore, the characteristic information of the entity usually includes the characteristic information of the tabletop and table leg components. Of course, for other types of entities, the various module components they contain can be specifically determined based on the type of entity, and the embodiment of the present application does not limit the module components obtained by dividing the entity.

[0062] In addition, in the solution provided in the embodiment of the present application, the three-dimensional model of the entity can be accurately restored only by a single image containing the entity, but this should not be understood as a limitation of the solution of the present application. In fact, when the initial image contains multiple images, the solution provided in the present application can also more accurately restore the original features of the entity from the multiple images, thereby constructing a more accurate three-dimensional model. Specifically, for ease of understanding, the modeling process of multiple initial images will be described in an embodiment provided later.

[0063] Of course, it should be noted that there are many implementation schemes for extracting feature point information in an image by image processing to obtain feature information of an entity. For example, as a common feasible implementation scheme, the initial image can be processed by introducing a neural network model. That is, in a feasible implementation scheme, the feature point information in the initial image is extracted to obtain feature information of at least one entity in the initial image, including:

[0064] The initial image is input into the trained target model for processing to obtain feature information of at least one entity in the initial image.

[0065] Among them, the trained target model here can be a deep network model, a convolutional network model or other network models trained by machine learning. However, based on the research on the effect of subsequent modeling, in one embodiment of the present application, a deep model generated by training based on self-supervised learning in the absence of labeled data is proposed to better ensure the model effect obtained by subsequent modeling. Specifically, the specific strategy for training the target model will be described below.

[0066] See also Figure 2 , Figure 2 A training target model provided in an embodiment of the present application is used to extract entity feature information in an image to better achieve subsequent entity modeling effects. Specifically, the training target model generally includes steps S210 to S230:

[0067] S210, inputting the sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of local images of the sample image into a second initial network to obtain a second probability distribution.

[0068] In order to achieve self-supervised training of the model, a method based on knowledge distillation is proposed in the embodiment of the present application. Specifically, in the embodiment of the present application, a network structure consisting of a first initial network and a second initial network is designed, wherein the first initial network and the second initial network have the same model structure and can process the input image into a normalized probability distribution in K dimensions.

[0069] On the basis of the foregoing, by generating a set of different views from a given image, such as multiple local views with smaller resolutions, the global view, that is, the original sample image, can be input into the first initial network of the teacher model for processing to obtain a first probability distribution, and then multiple local views with smaller resolutions, that is, local images, can be input into the second initial network of the student model for processing to obtain multiple second probability distributions.

[0070] S220: Optimize parameters of the second initial network according to the first probability distribution and the second probability distribution.

[0071] Unlike the conventional strategy of training the model by the difference between the first probability distribution and the second probability distribution to minimize the difference between the first probability distribution and the second probability distribution, in the technical solution provided by the present application, the cross entropy loss of the parameters of the student network and the teacher network is minimized to learn to match these distributions, that is, the student model, that is, the parameters of the second initial network, is learned by using the stochastic gradient descent minimization equation, while for the teaching network, that is, the first initial network, another strategy is used to optimize the parameters. Please refer to the relevant description of the subsequent step S230 for details.

[0072] S230: Optimize the parameters of the first initial network based on the optimization information of the parameters of the second initial network until a preset condition is met, and then determine the current first initial network as the target model.

[0073] Compared with the strategy of using gradients to synchronously adjust the parameters of the first initial model in the related art, in the embodiment of the present application, the parameters of the first initial network are optimized at a certain ratio based on the optimization information of the parameters of the second initial network, that is, the parameters of the optimized first initial network can be understood as the parameters of the original first initial network and the parameters of the first initial network weighted by the optimization increment. After a certain training, when the preset conditions are met, such as the similarity between the first probability distribution and the second probability distribution is sufficient, the current first initial network can be determined as the trained target model for extracting feature point information from the initial image.

[0074] Specifically, in order to facilitate understanding of the complete implementation process of the training target model provided above, the following will be specifically explained as follows:

[0075] Image information of a picture from multiple different perspectives is input to the student model and the teacher model respectively. The two network models have the same architecture but different parameters. The output of the teacher model is clustered (centering) by the average value of a batch of training data. Each network outputs a K-dimensional feature and is normalized using Softmax. Then the cross entropy loss is used as the objective function to calculate the similarity between the student model and the teacher model, and the stop-gradient (sg) operator is used on the teacher model to block the propagation of the gradient, and only the gradient is passed to the student model so that the student model updates the parameters, while the teacher model is updated using the exponential moving average (EMA) of the parameters in the student model. After the training is completed, the teacher model that performs better than the student model and can provide higher quality target features can be regarded as a trained target model for extracting feature point information of the initial image.

[0076] S120: Input the feature information into a language model to obtain a target text.

[0077] After completing the extraction of feature point information in the initial image through the method provided above and obtaining the feature information of the entity, in order to enable the code to better understand the extracted feature information and to more accurately realize the automated modeling process of the model, in an embodiment of the present application, it is considered to combine the natural language processing capability of the language model with the three-dimensional modeling of the code, and define the parameters and properties of the three-dimensional model through natural language description, thereby realizing a seamless transition for users from image input to text description and then to model generation.

[0078] Specifically, in the embodiments of the present application, the language model used can be a generative large language model, that is, it can describe the input information in the form of natural language, wherein the target text obtained can usually be used to describe the attribute information of the entity in the initial image. For example, in a common Xi'an solution, the target text can be a description such as "a rectangular wooden table with four legs and rounded corners on the table top", which usually defines the entity or the size, color, material, shape and other information of multiple components in the entity. Of course, the target text provided above is only a feasible implementation scheme and is not intended to limit the technical solution of the present application. In fact, with the development of language models, the target text obtained by inputting feature information into the language model will be able to describe the feature information of the entity in the image in a more accurate language.

[0079] S130, adjusting the attributes of the preset model based on the target text to obtain an initial model.

[0080] In the embodiment of the present application, after the feature information is processed through the language model as mentioned above to obtain the target text for describing the attribute information of the entity, by further mapping the text to the code process for the modeling parameters of each component in the modular design, some attribute parameters of the preset model can be adjusted based on the form of automatic adjustment of the code, thereby obtaining the initial model.

[0081] For example, to facilitate understanding, taking furniture as an example, the "rectangle" in the target text can be converted into geometric dimensions (such as length, width, thickness) and passed to the modular components in the code to complete the adjustment of the properties of the corresponding components. Similarly, "wood" can be mapped to a specific material texture and respond to the modular components passed to the code. The embodiment of the present application does not limit the implementation steps of adjusting the properties of the preset model based on the target text to obtain the initial model.

[0082] Furthermore, in a feasible implementation scheme, the preset model here can be an initial model preset in the modular components, for example, an initial table and chair model or other initial furniture model. However, as another feasible implementation scheme of the present application, an implementation scheme is also provided for constructing an initial model through point cloud data when the original image includes multiple images. The specific implementation content will be described in detail later.

[0083] S140: Based on the similarity between the initial image and the initial model, adjust the initial model to obtain a target model.

[0084] On the basis of the above-mentioned solution, after the properties of the preset model are adjusted based on the target text, in order to ensure that the generated three-dimensional model is highly matched with the initial image, the solution provided in the embodiment of the present application further considers the similarity between the image and the model to perform appropriate alignment optimization and adjustment on the model to obtain a target model that is more accurately in line with the initial image.

[0085] It should be noted that, considering that the image and the model are contents in different modes, in the solution provided in the embodiment of the present application, it is considered to use a multimodal model to map the image and the model to the same dimensional space and then adjust and match the model. Figure 3 , Figure 3 The schematic diagram of the process flow for adjusting the initial model provided in the embodiment of the present application, specifically, includes steps S310 to S320:

[0086] S310: Input the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image.

[0087] In an embodiment of the present application, by inputting data of different modalities, such as models and images, into a multimodal model with multimodal data processing capabilities, the model and the image can be respectively mapped to the first coding feature and the second coding feature in the same dimensional space, so as to describe the corresponding feature information in the initial model and the initial image respectively through the coding features in the same spatial dimension.

[0088] S320: Adjust the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model.

[0089] In the embodiment of the present application, since the initial model and the initial image are mapped to the same dimensional space, by comparing the similarity between the first coding feature and the second coding feature, it can be determined whether the initial model needs to be adjusted, and the mismatched parts in the initial model need to be adjusted, so as to obtain the corresponding target model. Specifically, the similarity can usually be calculated based on distance or cosine similarity, which is not limited in the embodiment of the present application. The higher the similarity, the higher the matching degree between the initial model and the initial image. Conversely, the lower the similarity, the lower the matching degree between the initial model and the initial image, and further adjustments need to be made to the initial model.

[0090] Specifically, based on the similarity between the first coding feature and the second coding feature, the initial model is adjusted to obtain the target model. This can be done by aligning the coding features and determining the coding attribute information corresponding to each coding region in the coding features, thereby determining the target region to be adjusted in the initial model to adjust the initial model. For details, please refer to Figure 4 , Figure 4 A schematic flow chart of a step of adjusting an initial model based on similarity provided in an embodiment of the present application, specifically, comprising steps S410 to S430:

[0091] S410: Align the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature.

[0092] Based on the idea of ​​multimodal pre-training of CLIP (Contrastive Language-Image Pre-training), in an embodiment of the present application, by performing feature alignment processing on the coding features obtained by model processing of different modalities, the coding features of the target model and the matching content in the initial image can be aligned, so that the coding attribute information corresponding to each coding area in the coding feature can be determined, for example, the shape information corresponding to a certain component, such as a desktop, or the size information of a table leg, etc.

[0093] S420: Determine the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model.

[0094] In an embodiment of the present application, after different coding regions in the coding feature correspond to the coding attribute information of different regions in the image (model), respectively, the similarity between the first coding feature and the second coding feature can be understood as the similarity between these coding regions. That is, the similarity between these coding regions can be compared with a preset similarity threshold to determine the coding attribute information corresponding to the coding region with a similarity lower than the preset similarity threshold as the target region to be adjusted in the initial model. For example, when the similarity between the coding features of the coding region describing the shape of the desktop in the model and the image is low, it can be considered that the shape of the desktop in the created model does not match the initial image. Therefore, the desktop can be regarded as the target region to be adjusted in the initial model, that is, the desktop region.

[0095] S430: Adjust the target area in the initial model based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0096] In an embodiment of the present application, after the identification of the non-matching target area is completed as mentioned above, the target area in the initial model is further adjusted using the feature information of the corresponding coding area in the second coding feature. For example, the shape of the desktop in the model is adjusted to the feature shape corresponding to the image, so that a target model that better matches the initial image can be finally generated, thereby further improving the generation accuracy of the target model.

[0097] The method provided by the present application, after inputting a visual image including an entity to be modeled, obtains feature information of the entity by extracting feature point information in the visual image, and then inputs the feature information into a language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0098] In addition, in order to more accurately improve the accuracy of the generated model, as another feasible embodiment of the present application, the initial image provided in the embodiment of the present application can also be extracted from the initial scene time frequency to obtain multiple frames of images under a series of different perspectives. At this time, through the fusion processing of the multiple frames of images, the characteristic information of each entity can be more accurately analyzed to complete the modeling more accurately. For example, in a feasible implementation scheme, please refer to Figure 5 , Figure 5 A schematic flow chart of a step of creating a point cloud model based on multiple frame images as a preset model for adjustment is provided in an embodiment of the present application, specifically, comprising steps S510 to S520:

[0099] S510: Create a point cloud model according to the feature points in the multiple frames of initial images.

[0100] In an embodiment of the present application, by analyzing images covering various perspectives of the entity and extracting feature point information, rich multi-perspective information can be extracted. That is, based on matching the feature points extracted from multiple frames of initial images, operations such as deduplication and aggregation of these feature points can be completed, and an initialized sparse point cloud model can be constructed.

[0101] Specifically, in order to further improve the effect of the constructed point cloud model and avoid the problem that the key points extracted when dealing with occlusion and dynamic objects are difficult to meet the requirements, as a further feasible implementation scheme of this application, a method of realizing image posture estimation based on point-by-point voting is proposed to better match and align feature points, thereby creating a more accurate point cloud model. For details, please refer to Figure 6 , Figure 6 A schematic diagram of a process flow of creating a point cloud model based on feature point matching provided in an embodiment of the present application, specifically, comprising steps S610 to S630:

[0102] S610, determining the viewing angle posture corresponding to each initial image based on point-by-point voting.

[0103] In the embodiment of the present application, a point-by-point voting method based on geometry guidance is proposed, which eliminates the difference in rotation and displacement between frames through normalization. The posture generated in each frame is obtained by cumulative calculation of point-by-point voting, avoiding key point dependence.

[0104] Specifically, the rotational posture of the image is determined by calculating the normal of the point. The rotation vote of each point is determined by the normal and the geometric features of the neighboring points. Finally, the rotation votes of all points are aggregated into the final rotational posture R by minimizing the error. R is the rotation matrix of the view posture corresponding to the initial image relative to the reference view posture.

[0105] Similarly, for the translation of the image, point-by-point voting can be generated through the displacement of the point, and the translation votes of each point are accumulated to obtain the overall translation information T, where the robustness under occlusion conditions is further improved by increasing the visibility weight of the point. Among them, T is the translation matrix of the view attitude corresponding to the initial image relative to the reference view attitude. R and T jointly determine the view attitude of the initial image.

[0106] S620: transform the initial images according to the viewing angle posture corresponding to each of the initial images to obtain a mapping image under the same viewing angle.

[0107] In an embodiment of the present application, after the viewing angle posture of each initial image is memorized based on point-by-point voting, the corresponding initial image is transformed using the viewing angle posture, so that all the initial images can be mapped to the same reference posture, that is, the mapping image under the reference view angle, so as to more accurately match the feature points.

[0108] S630: Match and fuse the feature points in the mapping image to generate the point cloud model.

[0109] In an embodiment of the present application, by matching and fusing feature points in multiple mapped images obtained under the same perspective, duplicate feature points can be effectively filtered out and feature points can be clustered, thereby obtaining point cloud data with a certain sparse density. The point cloud model constructed by the point cloud data will further provide guidance for the subsequent model generation process, for example, it can be used as a preset model for attribute information to be adjusted.

[0110] S520: Using the point cloud model as the preset model.

[0111] In an embodiment of the present application, the point cloud model obtained by the aforementioned multi-frame image processing is used as a preset model for the initial attribute parameters to be adjusted. Compared with directly using the preset initialized model in the program library, the model can better match the three-dimensional feature information of the image entity, thereby improving the accuracy of the subsequent generated model.

[0112] Of course, in addition to the target generation module provided above, when the initial image further includes multiple entities, the positional relationship between these entities can be determined by processing the initial image, thereby further determining the position information of the generated target model and obtaining the target scene. That is to say, the image-based model generation method provided in the present application also includes:

[0113] Determining positional relationships between a plurality of entities in the initial image;

[0114] Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

[0115] Specifically, in order to clearly understand the solution provided by the embodiment of the present application, the following will be combined with the aforementioned Figure 1 to Figure 6 The content provided provides a complete execution step of an image-based model generation method, which specifically includes the following steps:

[0116] (1) Model-based image analysis and feature extraction, specifically including the following steps:

[0117] Object detection: Relying on the feature extraction capability of DINO (DETR with Improved deNoising AnchOr boxes, an end-to-end object detector), the main objects in the image can be identified without manual annotation. Through DINO, specific objects such as furniture and equipment in the image can be accurately identified, so that different types of objects in the image can be processed separately. For example, the system can distinguish and identify "table", "chair" or "lamp" in the image.

[0118] Key point detection: After identifying the object, DINO further determines the key structural points of the object, such as edges, corners, and connection points, through feature embedding. This process is equivalent to establishing a framework for 3D modeling, so that each part has independent geometric characteristics, providing an accurate coordinate basis for subsequent 3D reconstruction.

[0119] Shape and surface estimation: Through DINO’s feature embedding, the system is able to extract the basic shape, curves, angles, and surface contours of the object. This information not only helps to build the 3D framework of the object, but also defines the specific geometric features of the object, making the modeling result more consistent with the real characteristics of the original image.

[0120] (2) Generate text description using a generative language model, which includes the following steps:

[0121] Object description: The generative language model generates a natural language description based on the structural and material features derived from the image analysis, such as "a round wooden table with four legs." In this way, the generative language model converts structural information into a human-understandable description, laying the foundation for the next 3D modeling step.

[0122] Hints and detailed descriptions: By providing specific hints, the generative language model can generate detailed descriptions, including hierarchical details of each component, such as material, color, size, etc. In this way, the description of each object can be clearly presented, providing specific attribute parameters for subsequent modeling.

[0123] (3) 3D model parameter mapping and construction, specifically including the following steps:

[0124] Description-to-parameter mapping: The natural language description generated by the generative language model can be converted into specific 3D model parameters. For example, "round wooden table" can correspond to a round tabletop and wood texture, and "four legs" can specify the number, position, and thickness of the legs. Through this mapping, the properties of each component are precisely defined.

[0125] Geometry generation and combination: The system uses a 3D modeling library (such as Plotly or Blender API) to gradually generate the 3D shape of each component according to the described parameters, and finally combines these components into an overall model. This step-by-step generation method not only allows each part of the model to be accurately expressed, but also ensures the modular structure of the model, which is convenient for subsequent adjustment and optimization.

[0126] (4) Iterative improvement based on optimization, specifically including the following steps:

[0127] Optimize alignment: By applying the image-text alignment feature of the contrastive language-image pre-trained model to the optimization of the 3D model, the system is able to determine where the generated model does not match in shape, color, or scale. The contrastive language-image pre-trained model compares the image with the generated model and outputs inconsistent areas, and feedback from these areas guides further adjustments to the model.

[0128] Dynamic optimization and feedback: The system gradually adjusts the model's properties based on the feedback from the pre-trained language-image model, such as making precise changes in shape, size, and material. Through rounds of feedback and adjustments, the resulting 3D model can highly meet the detail requirements of the original image.

[0129] When the input image includes multiple images, such as multiple images of a video frame, the specific steps of generating the model include:

[0130] First, key frames are extracted from the video. By segmenting and analyzing the video, the system can extract images covering all perspectives of the target object. These key frames provide rich multi-perspective information, allowing the system to reconstruct the complete structure of the 3D model. In order to automate the extraction process, a key frame detection algorithm based on deep learning is usually used to ensure that each frame can contribute unique geometric perspective information.

[0131] Based on the key frames, the system generates key feature points for each frame of the image through DINO or a similar feature extraction model to obtain the shape, boundary, and details of the object. In the process of feature extraction, self-supervised learning methods can reduce the need for data annotation and generate consistent features under multi-view conditions. Next, the system uses these feature points to build a preliminary 3D point cloud model, creating a sparse 3D point cloud by fusing 2D information from multiple frames into 3D data.

[0132] The generated point cloud model can be optimized through methods such as COLMAP (an open source computer vision software) or Structure-from-Motion (SfM) to obtain a more complete and denser 3D structure. COLMAP can match and fuse the feature points of multiple frames, and generate a preliminary 3D structure by triangulating images from multiple perspectives. However, the sparsity of point cloud data is still high, so it is necessary to further densify the model through depth completion technology so that the generated model is closer to the shape and details of the real object.

[0133] Based on the preliminary point cloud, an optimization model (such as CLIP) can be used for post-correction to ensure that the visual effect of the model is more accurate. CLIP can provide feedback optimization on the shape, texture and details of the model through image-text similarity analysis, so that the final generated 3D model is more consistent with the visual information in the video data. The optimization process gradually adjusts the details of the model through multiple iterations to ensure the accuracy and consistency of the 3D model.

[0134] In order to understand the effect of the model obtained by the model generation method provided in this application, this application also provides an evaluation effect of the generated model, mainly referring to the relevant indicators of single image reconstruction. The evaluation indicators include:

[0135] Structural Similarity (SSIM): measures the structural similarity between the generated model and the reference image.

[0136] Geometric consistency (Chamfer Distance): Calculate the point cloud distance between the generated model and the original model to evaluate the geometric error.

[0137] Visual Similarity (CLIP Similarity): Score the similarity between the input image and the generated model image based on CLIP.

[0138] Material accuracy: Determines the degree of match between the material of the generated model and the real image material to ensure the consistency of material and color.

[0139] Among them, in terms of the SSIM indicator, the structural similarity between the generated model and the reference image reaches an average of 0.85, indicating that the model can capture the shape and structure of the object well.

[0140] In terms of Chamfer Distance, the experimental results show that the geometric consistency of the model has reached a good level, especially for regular objects such as tables and chairs, with a high degree of geometric restoration.

[0141] In terms of visual similarity, CLIP has a higher similarity score, especially with the help of multi-angle input images, the generated model is closer to the visual effect of the original image.

[0142] In terms of material accuracy, experiments show that the generated material matches the material of the input image by more than 85%, reflecting the accuracy of the model in material generation.

[0143] In general, this method performs well under the condition of multi-angle image input and can achieve high-quality 3D reconstruction.

[0144] In addition, in order to verify the contribution of each module to the overall performance, based on the ablation experiment, some modules are gradually removed or replaced to observe their impact on the final results:

[0145] Removing CLIP optimization: Without CLIP optimization, the visual similarity score decreases by about 15%, indicating that CLIP optimization plays an important role in ensuring the visual consistency of the generated model.

[0146] Text description generation without the generative language model: After removing the description of the generative language model, the model loses detailed control over materials, shapes, and colors, and the material accuracy of the generated model decreases significantly.

[0147] Removing DINO feature extraction: Directly using the features of the original image, the generated 3D model has a significant decrease in geometric consistency and structural similarity.

[0148] The ablation experiment results show that DINO, generative language model and CLIP modules all make important contributions to the accuracy and consistency of the model.

[0149] On the basis of the image-based model generation method provided above, as another embodiment of the present application, an image-based model generation device is also provided. For details, please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an image-based model generation device provided in an embodiment of the present application, specifically, includes:

[0150] An extraction module 710 is used to extract feature point information in an initial image to obtain feature information of at least one entity in the initial image;

[0151] A processing module 720 is used to input the feature information into a language model to obtain a target text, where the target text is used to describe the attribute information of the entity in the initial image;

[0152] An adjustment module 730, configured to adjust the properties of a preset model based on the target text to obtain an initial model;

[0153] The adjustment module 730 adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

[0154] In a feasible embodiment of the present application, the adjustment module 730 is also used to input the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image; and adjust the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model.

[0155] In a feasible embodiment of the present application, the adjustment module 730 is also used to align the first coding feature and the second coding feature to determine the coding attribute information corresponding to each coding area in the coding feature; the coding attribute information corresponding to the coding area whose similarity is lower than a preset similarity threshold is determined as the target area to be adjusted in the initial model; and the target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

[0156] In a feasible embodiment of the present application, the adjustment module 730 is further used to create a point cloud model according to the feature points in the multiple frames of initial images; and to use the point cloud model as the preset model.

[0157] In a feasible embodiment of the present application, the adjustment module 730 is also used to determine the viewing angle posture corresponding to each initial image based on point-by-point voting; transform the initial image according to the viewing angle posture corresponding to each initial image to obtain a mapping image under the same viewing angle; match and fuse the feature points in the mapping image to generate the point cloud model.

[0158] In a feasible embodiment of the present application, the extraction module 710 is also used to input the initial image into a trained target model for processing to obtain feature information of at least one entity of the initial image; the target model is trained by the extraction module 710 through the following steps: inputting the sample image into the first initial network to obtain a first probability distribution, and inputting multiple local images of the sample image into the second initial network to obtain a second probability distribution; optimizing the parameters of the second initial network according to the first probability distribution and the second probability distribution; optimizing the parameters of the first initial network based on the optimization information of the parameters of the second initial network, until the current first initial network is determined as the target model after the preset conditions are met.

[0159] In a feasible embodiment of the present application, the adjustment module 730 is further used to determine the positional relationship between multiple entities in the initial image; determine the position information of multiple target models based on the positional relationship to obtain the target scene.

[0160] After inputting a visual image including an entity to be modeled, the present application obtains feature information of the entity by extracting feature point information in the visual image, and then inputs the feature information into a language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0161] In some embodiments of the present application, the image-based model generation device can be implemented in the form of a computer program. The computer program can be used in Figure 8 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the image-based model generation device, such as, Figure 7 The illustrated extraction module 710, processing module 720 and adjustment module 730. The computer program composed of various program modules enables the processor to execute the steps of the particle animation generation method of each embodiment of the present application described in this specification.

[0162] For example, Figure 8 The computer device shown can be Figure 7 The extraction module 710 in the image-based model generation device shown executes step S110. The computer device can execute step S120 through the processing module 720. The computer device can execute steps S130 and S140 through the adjustment module 730. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external computer device through a network connection. When the computer program is executed by the processor, an image-based model generation method is implemented.

[0163] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In some embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. The computer program is loaded by a processor, so that the processor performs the following steps:

[0165] Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image;

[0166] Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image;

[0167] Adjusting the attributes of the preset model based on the target text to obtain an initial model;

[0168] Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

[0169] After inputting a visual image including an entity to be modeled, the present application obtains feature information of the entity by extracting feature point information in the visual image, and then inputs the feature information into a language model to obtain a target text that can be used to describe the attribute information of the entity, so as to adjust the attributes of the preset model based on the target text to obtain an initial model that matches the attribute information described in the target text, and then adjusts the details of the obtained model again based on the similarity between the initial image and the initial model, thereby realizing an automated process of generating a corresponding accurate three-dimensional model from an image, thereby effectively improving the modeling efficiency.

[0170] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, information library or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0171] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above is a detailed introduction to an image-based model generation method, device and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating a model based on an image, characterized in that: include: Extracting feature point information from an initial image to obtain feature information of at least one entity in the initial image; Inputting the feature information into a language model to obtain a target text, wherein the target text is used to describe attribute information of an entity in the initial image; Adjusting the attributes of the preset model based on the target text to obtain an initial model; Based on the similarity between the initial image and the initial model, the initial model is adjusted to obtain a target model.

2. The method according to claim 1, characterized in that The step of adjusting the initial model based on the similarity between the initial image and the initial model to obtain a target model comprises: Inputting the initial model and the initial image into a multimodal model to obtain a first coding feature of the initial model and a second coding feature of the initial image; According to the similarity between the first coding feature and the second coding feature, the initial model is adjusted to obtain a target model.

3. The method according to claim 2, characterized in that The step of adjusting the initial model according to the similarity between the first coding feature and the second coding feature to obtain a target model includes: Aligning the first coding feature with the second coding feature to determine coding attribute information corresponding to each coding region in the coding feature; Determine the coding attribute information corresponding to the coding region whose similarity is lower than a preset similarity threshold as the target region to be adjusted in the initial model; The target area in the initial model is adjusted based on the feature information of the corresponding coding area in the second coding feature to obtain a target model.

4. The method according to claim 1, characterized in that: The initial image includes multiple frames of images extracted from the initial video; the method further includes: Creating a point cloud model based on the feature points in the multiple frames of initial images; and The point cloud model is used as the preset model.

5. The method according to claim 4, characterized in that The step of creating a point cloud model according to the feature points in the multiple frames of initial images includes: Determine the viewpoint posture corresponding to each initial image based on point-by-point voting; Transforming the initial images according to the viewing angle posture corresponding to each of the initial images to obtain a mapping image at the same viewing angle; Matching and fusing the feature points in the mapping image to generate the point cloud model.

6. The method according to claim 1, characterized in that The step of extracting feature point information from the initial image to obtain feature information of at least one entity in the initial image includes: Inputting the initial image into the trained target model for processing to obtain feature information of at least one entity in the initial image; The target model is trained by the following steps: Inputting a sample image into a first initial network to obtain a first probability distribution, and inputting a plurality of partial images of the sample image into a second initial network to obtain a second probability distribution; Optimizing parameters of the second initial network according to the first probability distribution and the second probability distribution; The parameters of the first initial network are optimized based on the optimization information of the parameters of the second initial network until the current first initial network is determined as the target model after a preset condition is met.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determining positional relationships between a plurality of entities in the initial image; Based on the positional relationship, position information of the plurality of target models is determined to obtain a target scene.

8. An image-based model generation device, characterized in that: include: An extraction module, used to extract feature point information in the initial image to obtain feature information of at least one entity in the initial image; A processing module, used for inputting the feature information into a language model to obtain a target text, wherein the target text is used for describing the attribute information of the entity in the initial image; An adjustment module, used to adjust the properties of the preset model based on the target text to obtain an initial model; The adjustment module adjusts the initial model based on the similarity between the initial image and the initial model to obtain a target model.

9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the image-based model generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded by a processor to execute the image-based model generation method according to any one of claims 1 to 7.

Images