A three-dimensional modeling method, device, electronic device and storage medium

By fitting the symbol distance function and discrete voxel grid, the problems of robustness and low accuracy of three-dimensional modeling in the prior art are solved, and a higher quality three-dimensional model construction is achieved.

CN118334265BActive Publication Date: 2024-11-05广州极点三维信息科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410269550.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-11
Publication Date
2024-11-05
Estimated Expiration
2044-03-11

AI Technical Summary

Technical Problem

The existing implicit three-dimensional modeling methods based on deep learning lack robustness and average accuracy, making it difficult to effectively build high-quality three-dimensional models.

Method used

By obtaining multiple images of the target object, fitting the symbol distance function, discrete the three-dimensional space of the voxel mesh into a cube mesh, and bringing the vertex space coordinates of each cube mesh into the symbol distance function, obtaining the relative vertex to be built The spatial location of the mold object is combined into a three-dimensional model.

Benefits of technology

It improves the accuracy and robustness of three-dimensional modeling, and can learn the characteristics of the target object from the two-dimensional image more accurately and generate high-quality three-dimensional models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118334265B_ABST
    Figure CN118334265B_ABST
Patent Text Reader

Abstract

This application discloses a three-dimensional modeling method, apparatus, electronic device, and storage medium, relating to the field of three-dimensional modeling technology. The method includes: acquiring multiple images of a target object; fitting a signed distance function to the surface of the target object based on the multiple images; discretizing a voxel mesh into multiple cubic meshes in three-dimensional space; inputting the spatial coordinates of each vertex in each cubic mesh into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; and combining the vertices on the surface of the target object to form a three-dimensional model of the target object. This application learns the features of the target object from two-dimensional images to a three-dimensional object from multiple images of the target object using a signed distance function, learns a signed distance function that approximates the surface of the target object, and then extracts each vertex on the surface of the target object based on the signed distance function. A three-dimensional model is then built from each vertex on the surface of the target object, improving the accuracy and robustness of three-dimensional modeling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional modeling method, device, electronic device and storage medium. Background Art

[0002] Currently, deep learning technology and neural network models have been widely used in three-dimensional modeling. Neural network models can learn the features of three-dimensional objects from a large amount of two-dimensional image data and realize automated three-dimensional modeling. However, the existing implicit modeling methods based on deep learning lack robustness and have average accuracy. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a three-dimensional modeling method, device, electronic device and storage medium to improve the robustness and accuracy of three-dimensional modeling.

[0004] To achieve the above purpose, one aspect of an embodiment of the present application provides a three-dimensional modeling method, the method comprising:

[0005] Acquire multiple images of the target object;

[0006] Fitting a signed distance function of the surface of the target object according to the plurality of images;

[0007] Discretize the voxel grid three-dimensional space into multiple cube grids;

[0008] Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled;

[0009] The vertices on the surface of the target object are combined into a three-dimensional model of the target object.

[0010] In some embodiments, acquiring multiple images of the target object includes:

[0011] A plurality of images are obtained by photographing the target object from multiple angles by a camera.

[0012] In some embodiments, fitting the signed distance function of the surface of the target object according to the plurality of images comprises:

[0013] Acquire the posture parameters of the camera according to the multiple images; wherein the multiple images are taken by the camera;

[0014] constructing a neural radiation field rendering model according to the posture parameters;

[0015] The neural radiance field rendering model is trained to fit the signed distance function in the neural radiance field rendering model.

[0016] In some embodiments, the acquiring the posture parameters of the camera according to the plurality of images includes:

[0017] Performing camera pose estimation on the multiple images to obtain the pose of the camera;

[0018] The posture is:

[0019] T w2c =[R|t]

[0020] Among them, T w2c is the posture, R is the rotation matrix, and t is the position matrix;

[0021] Setting camera intrinsic parameters, converting the rotation matrix into a rotation matrix equation expressed in the Rodriguez formula, and setting the position matrix; wherein the camera intrinsic parameters, the rotation matrix equation, and the position matrix serve as the posture parameters;

[0022] The camera intrinsic parameter is represented by K, where f x 、f y are the focal length of the camera, c x =w / 2, c y =h / 2, w and h are the width and height of the resolution of the plurality of images respectively;

[0023] The rotation matrix equation is expressed as

[0024]

[0025] Where I represents the identity matrix and α represents the rotation angle;

[0026]

[0027] The position matrix is ​​denoted as t:

[0028]

[0029] Among them, φ0, φ1, φ2, t x ,t y ,t z are learnable parameters.

[0030] In some embodiments, constructing a neural radiation field rendering model according to the posture parameters includes:

[0031] Constructing a first rendering equation to render pixel values ​​of all pixels in the plurality of images;

[0032] The first rendering equation is:

[0033]

[0034] Wherein, C(o, v) represents the first rendering equation, o represents the center point of the camera, v represents a ray pointing from point o to a pixel point on the imaging plane, the ray is recorded as p(t), p(t)=o+tv, t≥0; w(t) is the weight of the spatial point on the ray in space;

[0035] Converting the first rendering equation into a second rendering equation;

[0036] The second rendering equation is:

[0037]

[0038] in, represents the second rendering equation, n represents dividing the ray into n sample lines in [0, t], each sample line is recorded as p(t i ), c i It is the pixel value predicted by the neural radiation field rendering model after the noise data of the sample line is removed by the neural network model;

[0039]

[0040]

[0041]

[0042] Among them, s represents the learnable parameter, x represents the function independent variable, β i The f(·) in is the signed distance function, and the value of the signed distance function is the output result of the neural radiation field rendering model.

[0043] In some embodiments, the training the neural radiation field rendering model to fit the signed distance function in the neural radiation field rendering model includes:

[0044] Set the pixel coordinates in the multiple images to:

[0045] p pixel =(u,v)

[0046] Among them, p pixel represents the pixel coordinates; v is the ray, and v is expressed as:

[0047]

[0048] represents the rotation matrix equation;

[0049] Training the learnable parameters in the rotation matrix equation according to a total loss function;

[0050] The total loss function is:

[0051] L=L color +L reg

[0052] Wherein, L represents the total loss function, L color represents the color loss function, L reg represents the constraint loss function;

[0053]

[0054] m is the total number of images; C k represents the pixel value, k represents the pixel number;

[0055]

[0056] is the gradient of the signed distance function.

[0057] In some embodiments, the step of bringing the spatial coordinates of each vertex in each of the cubic meshes into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled includes:

[0058] Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled;

[0059] The cubic mesh in which all the vertices are on the surface of the object to be modeled is divided into 8 small cubic meshes of the same size as the current cubic mesh, and then the spatial coordinates of each vertex in each cubic mesh are input into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled, until the cubic mesh is reduced to a preset maximum subdivision resolution to obtain the current spatial position of each vertex relative to the object to be modeled.

[0060] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a three-dimensional modeling device, the device comprising:

[0061] An image acquisition unit, used to acquire multiple images of a target object;

[0062] A function fitting unit, used for fitting a signed distance function of the surface of the target object according to the plurality of images;

[0063] A spatial discretization unit, used for discretizing the voxel grid three-dimensional space into multiple cubic grids;

[0064] A vertex position determination unit, used for bringing the spatial coordinates of each vertex in each of the cubic meshes into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled;

[0065] The vertex modeling unit is used to combine the vertices on the surface of the target object into a three-dimensional model of the target object.

[0066] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0067] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0068] The embodiments of the present application include at least the following beneficial effects:

[0069] The present application acquires multiple images of the target object; fits the signed distance function of the target object surface according to the multiple images; discretizes the voxel grid three-dimensional space into multiple cubic grids; brings the spatial coordinates of each vertex in each cubic grid into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; and combines the vertices on the surface of the target object into a three-dimensional model of the target object. The present application learns the characteristics of the target object from a two-dimensional image to a three-dimensional object from multiple images of the target object through the signed distance function, learns the signed distance function that approximates the surface of the target object, and then extracts each vertex on the surface of the target object according to the signed distance function, and establishes a three-dimensional model from each vertex on the surface of the target object, thereby improving the accuracy and robustness of three-dimensional modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a process of a three-dimensional modeling method provided in an embodiment of the present application;

[0071] Figure 2 An example flowchart of rendering pixel values ​​for pixel points provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of the structure of a three-dimensional modeling device provided in an embodiment of the present application;

[0073] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0075] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0076] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0078] Before describing the embodiments of the present application in detail, some related technologies involved in the embodiments of the present application are first described as follows:

[0079] Users of home design software need to present design concepts to customers in an intuitive 3D model format. A large number of 3D models provide more choices, allowing designers to better present design concepts and improve customers' understanding and acceptance of design solutions.

[0080] At present, the three-dimensional models of furniture in the home furnishing industry mainly have the following modeling methods:

[0081] Manual modeling: This is the traditional 3D modeling method, where designers use professional 3D modeling software (such as Blender, 3ds Max, Maya, SolidWorks) to manually create the 3D model of furniture. This requires designers to have strong modeling skills, is suitable for situations that require a high degree of customization and unique design, and has low modeling efficiency.

[0082] Scanning and point cloud processing: 3D scanning technology can collect 3D point cloud data of existing furniture through laser scanning or camera. Designers can then use point cloud processing software (such as Geomagic, CloudCompare) to convert these point cloud data into 3D models and perform subsequent editing and optimization. This method requires specific scanning equipment, which limits its application scenarios.

[0083] Deep learning and neural networks: Deep learning technology has been applied in furniture 3D modeling. The neural network model can learn the characteristics of the 3D model from a large amount of 2D image data to realize the automated 3D modeling process. This method is particularly suitable for processing complex furniture shapes and structures.

[0084] At present, deep learning has demonstrated its superior characteristics in automated 3D modeling, especially the 3D reconstruction technology based on neural rendering, which automatically reconstructs high-quality 3D models through multi-view consistency constraints, and has become a competitive direction in the field of furniture 3D modeling.

[0085] However, the related technologies have the following problems:

[0086] 1. Traditional handmade furniture modeling requires professional modelers, which is time-consuming and inefficient;

[0087] 2. The method of obtaining furniture models through scanning equipment requires specific equipment and is difficult to promote and apply on a large scale;

[0088] 3. The implicit furniture modeling method based on deep learning lacks robustness and has average accuracy.

[0089] Therefore, the embodiments of the present application provide a three-dimensional modeling method, device, electronic device and storage medium. The scheme obtains multiple images of the target object; fits the signed distance function of the target object surface according to the multiple images; discretizes the voxel grid three-dimensional space into multiple cubic grids; brings the spatial coordinates of each vertex in each cubic grid into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; and combines the vertices on the surface of the target object into a three-dimensional model of the target object. The present application learns the characteristics of the target object from a two-dimensional image to a three-dimensional object from multiple images of the target object through the signed distance function, learns the signed distance function that approximates the surface of the target object, and then extracts each vertex on the surface of the target object according to the signed distance function, and establishes a three-dimensional model from each vertex on the surface of the target object, thereby improving the accuracy and robustness of three-dimensional modeling.

[0090] The embodiment of the present application provides a three-dimensional modeling method, which relates to the field of three-dimensional modeling technology. The three-dimensional modeling method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a knowledge extraction method, etc., but is not limited to the above forms.

[0091] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0092] Reference Figure 1 The present application embodiment provides a three-dimensional modeling method, which may include but is not limited to S100 to S140, as follows:

[0093] S100: Acquire multiple images of the target object.

[0094] Specifically, the target object of the embodiment of the present application is an object that needs three-dimensional modeling, which can be any three-dimensional object, such as a car, electronic product or furniture, etc. Optionally, the embodiment of the present application is illustrated using furniture as an example, and the implementation methods for modeling other types of target objects can be referred to accordingly, and examples are not given here one by one.

[0095] The multiple images of this embodiment may include images of the target object at multiple angles, such as multiple images of a full view or multiple images of a certain side at multiple angles.

[0096] Furthermore, S100 may include: acquiring a plurality of images obtained by photographing the target object at multiple angles using a camera.

[0097] Specifically, the camera in the embodiment of the present application refers to all devices with a shooting function, such as a mobile phone, a digital camera or a camera.

[0098] S110: Fitting a signed distance function of the surface of the target object according to the multiple images.

[0099] Specifically, the Sign Distance Function (SDF) determines the distance from a point to the boundary of a finite region in space and defines the sign of the distance: the point is positive inside the boundary of the region, negative outside, and 0 when it is on the boundary.

[0100] The signed distance function of this embodiment can use the surface of the target object (such as the surface of furniture) as the region boundary, and then determine the distance from each point in the space to the surface of the target object according to the signed distance function.

[0101] Furthermore, S110 may include S111 to S113:

[0102] S111: Acquire the posture parameters of the camera according to the multiple images; wherein the multiple images are obtained by taking the camera.

[0103] Specifically, this embodiment may use the colmap algorithm to perform preliminary camera pose estimation on multiple images (eg, furniture images).

[0104] Furthermore, S111 may include:

[0105] Performing camera pose estimation on the multiple images to obtain the pose of the camera;

[0106] The posture is:

[0107] T w2c =[R|t]

[0108] Among them, T w2c is the posture, R is the rotation matrix, and t is the position matrix;

[0109] Setting camera intrinsic parameters, converting the rotation matrix into a rotation matrix equation expressed in the Rodriguez formula, and setting the position matrix; wherein the camera intrinsic parameters, the rotation matrix equation, and the position matrix serve as the posture parameters;

[0110] The camera intrinsic parameter is represented by K, where f x 、f y are the focal length of the camera, c x =w / 2, c y =h / 2, w and h are the width and height of the resolution of the plurality of images respectively;

[0111] The rotation matrix equation is expressed as

[0112]

[0113] Where I represents the identity matrix and α represents the rotation angle;

[0114]

[0115] The position matrix is ​​denoted as t:

[0116]

[0117] Among them, φ0, φ1, φ2, t x ,t y ,t z are learnable parameters.

[0118] S112: Constructing a neural radiation field rendering model according to the posture parameters.

[0119] Specifically, in the neural radiation field rendering model, the pixel value of each pixel in multiple images can be represented by the rendering equation.

[0120] Furthermore, S112 may include:

[0121] Constructing a first rendering equation to render pixel values ​​of all pixels in the plurality of images;

[0122] The first rendering equation is:

[0123]

[0124] Wherein, C(o, v) represents the first rendering equation, o represents the center point of the camera, v represents a ray pointing from point o to a pixel point on the imaging plane, the ray is recorded as p(t), p(t)=o+tv, t≥0; w(t) is the weight of the spatial point on the ray in space;

[0125] Since the first rendering equation is an integral expression, in order to facilitate computer calculation, in this embodiment, the first rendering equation can be converted into a second rendering equation: the first rendering equation is converted into the second rendering equation;

[0126] The second rendering equation is:

[0127]

[0128] in, represents the second rendering equation, n represents dividing the ray into n sample lines in [0, t], each sample line is recorded as p(t i ), c i It is the pixel value predicted by the neural radiation field rendering model after the noise data of the sample line is removed by the neural network model;

[0129]

[0130]

[0131]

[0132] Among them, s represents the learnable parameter, x represents the function independent variable, β i The f(·) in is the signed distance function, and the value of the signed distance function is the output result of the neural radiation field rendering model.

[0133] Next, the process of removing noise data using the neural network model is described: Since this embodiment needs to shoot images from multiple angles, changes in background and lighting and camera shake may occur during the shooting process. Therefore, this embodiment can use a neural network model to encode environmental information, that is, to remove noise data. Figure 2 , this embodiment uses a multi-layer perceptron (MLP2) to encode environmental information, Figure 2MLP1 in the figure is a neural radiation field rendering model. By removing noise data and fine-tuning the camera internal and external parameters through neural radiation field rendering model learning, blurry and strange images are removed, and the influence of different backgrounds on the 3D reconstruction of the target object (furniture) is improved, thereby improving the reconstruction accuracy.

[0134] S113: Training the neural radiation field rendering model to fit the signed distance function in the neural radiation field rendering model.

[0135] Specifically, each learnable parameter can be initialized according to the estimated value of the colmap algorithm in S111 and optimized along with the training process of the neural radiation field rendering model.

[0136] Furthermore, S113 may include:

[0137] Set the pixel coordinates in the multiple images to:

[0138] p pixel =(u,v)

[0139] Among them, p pixel represents the pixel coordinates; v is the ray, and v is expressed as:

[0140]

[0141] represents the rotation matrix equation;

[0142] The learnable parameters in the rotation matrix equation are trained according to the total loss function; specifically, each learnable parameter is end-to-end supervised training is performed through the total loss function;

[0143] The total loss function is:

[0144] L=L color +L reg

[0145] Wherein, L represents the total loss function, L color represents the color loss function, L reg represents the constraint loss function;

[0146]

[0147] m is the total number of images; C k represents the pixel value, k represents the pixel number;

[0148]

[0149] is the gradient of the signed distance function. The physical meaning of the constrained loss function is to constrain the gradient of the signed distance function to tend to 1.

[0150] The trained neural radiation field rendering model fits the signed distance function of the target object (the target object in this embodiment is furniture), and the three-dimensional model of the furniture can be extracted using the signed distance function.

[0151] This embodiment designs a signed distance function suitable for three-dimensional reconstruction based on deep learning technology, realizes unbiased estimation of the signed distance function and adaptive reduction of weights of multiple surfaces of the three-dimensional model.

[0152] S120: discretize the voxel grid three-dimensional space into a plurality of cubic grids.

[0153] Specifically, this embodiment can initialize the voxel grid three-dimensional space V, where the x, y, and z axes are all normalized to [0, 1]. Then the voxel grid three-dimensional space V is discretized, such as the x, y, and z axes are discretized into s parts respectively, to obtain S 3 cube mesh and set the maximum subdivision resolution S max .

[0154] S130: Substitute the spatial coordinates of each vertex in each of the cubic meshes into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled.

[0155] Specifically, the 8 vertex space coordinates V of each discretized cube mesh are i =(v x , v y , v z ), substitute the signed distance function, if the vertex f(V i ) has values ​​greater than 0 and less than 0 at the same time, it means that the cube grid is on the surface of the three-dimensional model (i.e., furniture). The relationship between the value and position of the signed distance function is expressed as:

[0156]

[0157] Further, S130 may include S131-S132:

[0158] S131: Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled;

[0159] S132: Divide the cubic mesh in which all the vertices are on the surface of the object to be modeled into 8 small cubic meshes of the same size as the current cubic mesh, and then return to the step of substituting the spatial coordinates of each vertex in each cubic mesh into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled, until the cubic mesh is reduced to a preset maximum subdivision resolution to obtain the current spatial position of each vertex relative to the object to be modeled.

[0160] Specifically, the cube grid on the furniture surface is further divided, that is, one cube grid is subdivided into 9 small cube grids of the same size, and then step S131 is repeatedly performed until the small cube grid is divided and reduced to the maximum subdivision resolution S max This embodiment sets a maximum resolution limit for the cube grid, and efficiently extracts the three-dimensional model of the furniture by continuously subdividing the cube grid.

[0161] S140: Combining the vertices on the surface of the target object into a three-dimensional model of the target object.

[0162] Specifically, the set of all f(V i )=0 is the 3D model of the target object (furniture).

[0163] The embodiment of the present application can utilize multiple images of the target object to achieve end-to-end fine-tuning of camera internal and external parameters, and realize three-dimensional model reconstruction of the target object (furniture) while eliminating environmental disturbance noise data. It is more robust than existing solutions and can achieve higher-precision three-dimensional reconstruction.

[0164] Reference Figure 3 The embodiment of the present application further provides a three-dimensional model building device, which can implement the above-mentioned three-dimensional model building method, and the device includes:

[0165] An image acquisition unit, used to acquire multiple images of a target object;

[0166] A function fitting unit, used for fitting a signed distance function of the surface of the target object according to the plurality of images;

[0167] A spatial discretization unit, used for discretizing the voxel grid three-dimensional space into multiple cubic grids;

[0168] A vertex position determination unit, used for bringing the spatial coordinates of each vertex in each of the cubic meshes into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled;

[0169] The vertex modeling unit is used to combine the vertices on the surface of the target object into a three-dimensional model of the target object.

[0170] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0171] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned knowledge extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0172] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0173] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0174] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0175] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the knowledge extraction method of the embodiment of this application;

[0176] Input / output interface 403, used to implement information input and output;

[0177] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0178] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0179] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0180] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned knowledge extraction method is implemented.

[0181] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0182] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0184] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0185] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0187] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0188] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0193] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A three-dimensional modeling method, characterized in that: The method comprises: Acquire multiple images of the target object; The acquiring of multiple images of the target object comprises: acquiring multiple images of the target object obtained by photographing the target object at multiple angles by a camera; Fitting a signed distance function of the surface of the target object according to the plurality of images; The step of fitting the signed distance function of the surface of the target object according to the plurality of images comprises: Acquire the posture parameters of the camera according to the multiple images; wherein the multiple images are taken by the camera; The step of obtaining the posture parameters of the camera according to the plurality of images comprises: Performing camera pose estimation on the multiple images to obtain the pose of the camera; The posture is: T w2c =[R|t] Among them, T w2c is the posture, R is the rotation matrix, and t is the position matrix; Setting camera intrinsic parameters, converting the rotation matrix into a rotation matrix equation expressed in the Rodriguez formula, and setting the position matrix; wherein the camera intrinsic parameters, the rotation matrix equation, and the position matrix serve as the posture parameters; The camera intrinsic parameter is represented by K, where f x 、f y are the focal length of the camera, c x =w / 2, c y =h / 2, w and h are the width and height of the resolution of the plurality of images respectively; The rotation matrix equation is expressed as Where I represents the identity matrix and α represents the rotation angle; The position matrix is ​​denoted as t: Among them, φ0, φ1, φ, t x ,t y ,t z is a learnable parameter; constructing a neural radiation field rendering model according to the posture parameters; The step of constructing a neural radiation field rendering model according to the posture parameters includes: Constructing a first rendering equation to render pixel values ​​of all pixels in the plurality of images; The first rendering equation is: Wherein, C(o, v) represents the first rendering equation, o represents the center point of the camera, v represents a ray pointing from point o to a pixel point on the imaging plane, the ray is recorded as p(t), p(t)=o+tv, t≥0; w(t) is the weight of the spatial point on the ray in space; Converting the first rendering equation into a second rendering equation; The second rendering equation is: in, represents the second rendering equation, n represents dividing the ray into n sample lines in [0, t], each sample line is recorded as p(t i ), c i It is the pixel value predicted by the neural radiation field rendering model after the noise data of the sample line is removed by the neural network model; Among them, s represents the learnable parameter, x represents the function independent variable, β i f(·) in which is the signed distance function, and the value of the signed distance function is the output result of the neural radiation field rendering model; the camera internal and external parameters are fine-tuned by learning the neural radiation field rendering model to remove blurred and singular images, so as to remove the influence of different backgrounds on the three-dimensional reconstruction of the target object; Training the neural radiation field rendering model to fit the signed distance function in the neural radiation field rendering model; Discretize the voxel grid three-dimensional space into multiple cube grids; Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; The vertices on the surface of the target object are combined into a three-dimensional model of the target object.

2. A three-dimensional modeling method according to claim 1, characterized in that: The training of the neural radiation field rendering model to fit the signed distance function in the neural radiation field rendering model includes: Set the pixel coordinates in the multiple images to: p pixel =(u,v) Among them, p pixel represents the pixel coordinates; v is the ray, and v is expressed as: represents the rotation matrix equation; Training the learnable parameters in the rotation matrix equation according to a total loss function; The total loss function is: L=L colo r+L reg Wherein, L represents the total loss function, L color represents the color loss function, L reg represents the constraint loss function; m is the total number of images; C k represents the pixel value, k represents the pixel number; is the gradient of the signed distance function.

3. A three-dimensional modeling method according to any one of claims 1 to 2, characterized in that: Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled includes: Substituting the spatial coordinates of each vertex in each of the cubic grids into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; The cubic mesh in which all the vertices are on the surface of the object to be modeled is divided into 8 small cubic meshes of the same size as the current cubic mesh, and then the spatial coordinates of each vertex in each cubic mesh are input into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled, until the cubic mesh is reduced to a preset maximum subdivision resolution to obtain the current spatial position of each vertex relative to the object to be modeled.

4. A three-dimensional modeling device, characterized in that: The device comprises: An image acquisition unit, used to acquire multiple images of a target object; The acquiring of multiple images of the target object comprises: acquiring multiple images of the target object obtained by photographing the target object at multiple angles by a camera; A function fitting unit, used for fitting a signed distance function of the surface of the target object according to the plurality of images; The step of fitting the signed distance function of the surface of the target object according to the plurality of images comprises: Acquire the posture parameters of the camera according to the multiple images; wherein the multiple images are taken by the camera; The step of obtaining the posture parameters of the camera according to the plurality of images comprises: Performing camera pose estimation on the multiple images to obtain the pose of the camera; The posture is: T w2c =[R|t] Among them, T w2c is the posture, R is the rotation matrix, and t is the position matrix; Setting camera intrinsic parameters, converting the rotation matrix into a rotation matrix equation expressed in the Rodriguez formula, and setting the position matrix; wherein the camera intrinsic parameters, the rotation matrix equation, and the position matrix serve as the posture parameters; The camera intrinsic parameter is represented by K, where f x 、f y are the focal length of the camera, c x =w / 2, c y =h / 2, w and h are the width and height of the resolution of the plurality of images respectively; The rotation matrix equation is expressed as Where I represents the identity matrix and α represents the rotation angle; The position matrix is ​​denoted as t: Among them, φ0, φ1, φ, t x ,t y ,t z is a learnable parameter; constructing a neural radiation field rendering model according to the posture parameters; The step of constructing a neural radiation field rendering model according to the posture parameters includes: Constructing a first rendering equation to render pixel values ​​of all pixels in the plurality of images; The first rendering equation is: Wherein, C(o, v) represents the first rendering equation, o represents the center point of the camera, v represents a ray pointing from point o to a pixel point on the imaging plane, the ray is recorded as p(t), p(t)=o+tv, t≥0; w(t) is the weight of the spatial point on the ray in space; Converting the first rendering equation into a second rendering equation; The second rendering equation is: in, represents the second rendering equation, n represents dividing the ray into n sample lines in [0, t], each sample line is recorded as p(t i ), c i It is the pixel value predicted by the neural radiation field rendering model after the noise data of the sample line is removed by the neural network model; Among them, s represents the learnable parameter, x represents the function independent variable, β i f(·) in which is the signed distance function, and the value of the signed distance function is the output result of the neural radiation field rendering model; the camera internal and external parameters are fine-tuned by learning the neural radiation field rendering model to remove blurred and singular images, so as to remove the influence of different backgrounds on the three-dimensional reconstruction of the target object; Training the neural radiation field rendering model to fit the signed distance function in the neural radiation field rendering model; A spatial discretization unit, used for discretizing the voxel grid three-dimensional space into multiple cubic grids; A vertex position determination unit, used for bringing the spatial coordinates of each vertex in each of the cubic meshes into the signed distance function to obtain the spatial position of each vertex relative to the object to be modeled; The vertex modeling unit is used to combine the vertices on the surface of the target object into a three-dimensional model of the target object.

5. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.