Object reconstruction method and device based on neural network, storage medium and electronic equipment
Through multimodal data fusion and dual neural network training, the problem of traditional object reconstruction methods' poor handling of sparse point clouds is solved, and high-fidelity, dynamically adjusted object reconstruction is achieved, which is suitable for high-precision scenes.
Patent Information
- Application Number
- CN202510791784.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-09-12
AI Technical Summary
Traditional object reconstruction methods do not handle sparse point clouds and occlusion well, resulting in holes, noise or distortion in the reconstruction results, a separation between geometry and appearance optimization, difficulty in expressing high-frequency details, and a lack of dynamic adjustment capabilities.
Multimodal data fusion and dual neural network joint training are adopted, RGB multi-layer perceptron and SDF multi-layer perceptron neural networks are used for feature extraction and constraint, combined with 3D Gaussian model optimization, to achieve collaborative optimization of geometry and appearance and implicit field representation.
It achieves high-fidelity reconstruction from sparse input, adapts to sparse data, solves the problem of separation between geometry and appearance, has high expressive power and differentiable rendering-driven optimization, and is suitable for scenes with high precision requirements.
Smart Images
Figure CN120635362A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of object modeling, and in particular to a method, device, storage medium and electronic device for object reconstruction based on a neural network. Background Art
[0002] Object reconstruction technology refers to the process of converting real-world objects into digital three-dimensional models through multi-field technical means such as computer vision, graphics, and deep learning.
[0003] Traditional object reconstruction methods have the following problems:
[0004] 1. Traditional object reconstruction methods (such as multi-view stereo matching and structured light scanning) typically require dense input data (large numbers of high-precision images or complete scan point clouds). If the input is a sparse point cloud (such as a preliminary point cloud generated by structure-from-motion) or there is occlusion, the reconstruction results are prone to holes, noise, or distortion.
[0005] 2. Most traditional workflows require a separate process: first rebuilding the geometry (mesh / voxels), then adding color through texture mapping. These two independent optimizations can lead to inconsistencies between geometry and appearance (e.g., texture misalignment and seam issues).
[0006] 3. Voxel-based (TSDF) or explicit grid methods are limited by resolution and have difficulty expressing high-frequency details; while direct reconstruction of point clouds lacks the ability to represent continuous surfaces and requires post-processing smoothing.
[0007] 4. The mesh models generated by traditional methods are difficult to dynamically adjust details, and the optimization process relies on manual parameter adjustment (such as mesh simplification and smoothing), lacking data-driven adaptive capabilities. Summary of the Invention
[0008] The embodiments of the present application provide a neural network-based object reconstruction method, device, storage medium, and electronic device to solve the technical problems existing in the prior art.
[0009] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.
[0010] According to a first aspect of an embodiment of the present application, a method for object reconstruction based on a neural network is provided, comprising:
[0011] Obtain multi-view images and sparse 3D point clouds of objects;
[0012] Using an RGB multi-layer perceptron neural network to extract features from the multi-view image to obtain color values of the object surface;
[0013] Using an SDF multi-layer perceptron neural network to extract features from the sparse 3D point cloud and obtain a signed distance value;
[0014] Performing appearance constraints on the RGB multilayer perceptron neural network based on the color value;
[0015] Performing geometric constraints on the SDF multilayer perceptron neural network based on the matching distance value;
[0016] Jointly train the RGB multi-layer perceptron neural network with appearance constraints and the SDF multi-layer perceptron neural network with geometric constraints;
[0017] Based on the sparse 3D point cloud, the mesh model is extracted using the jointly trained SDF multi-layer perceptron neural network;
[0018] The grid model is converted into a 3D Gaussian model, and the real image of the object is used as a supervision signal to continuously optimize the 3D Gaussian model.
[0019] In some embodiments of the present application, based on the aforementioned solution, acquiring multi-view images of an object includes:
[0020] An RGB camera is used to capture multi-view images of the target object from multiple perspectives.
[0021] In some embodiments of the present application, based on the above solution, obtaining a sparse 3D point cloud of an object includes:
[0022] The sparse three-dimensional coordinate points on the surface of an object are obtained through structured light, laser scanning or depth camera to form a sparse 3D point cloud of the object.
[0023] In some embodiments of the present application, based on the aforementioned solution, performing appearance constraints on the RGB multi-layer perceptron neural network based on the color value includes:
[0024] The color values are aggregated into a volume rendering result through voxel integration, and compared with the real multi-view image, and the color pixel-level error is calculated to obtain the color loss.
[0025] In some embodiments of the present application, based on the aforementioned solution, geometrically constraining the SDF multilayer perceptron neural network based on the matching distance value includes:
[0026] Using the true SDF constraint, the error between the predicted SDF and the true SDF is calculated to obtain the signed distance loss.
[0027] In some embodiments of the present application, based on the aforementioned solution, the mesh model is extracted based on the sparse 3D point cloud using a jointly trained SDF multi-layer perceptron neural network, including:
[0028] Input the sparse 3D point cloud into the jointly trained SDF multi-layer perceptron neural network to obtain the complete SDF field of the object surface;
[0029] The Marching Cubes algorithm is used to extract the isosurface of the SDF field and generate a mesh model of the object surface.
[0030] In some embodiments of the present application, based on the aforementioned solution, the method of using the real image of the object as a supervisory signal and continuously optimizing the 3D Gaussian model includes:
[0031] Perform ray casting on the target view in the real image of the object and calculate the intersection of the ray and the 3D Gaussian model;
[0032] Predict the color and opacity at the intersection according to the Gaussian distribution and render the new view image;
[0033] Calculate the pixel-level error between the rendered image and the real image of the object, and backpropagate to optimize the Gaussian parameters.
[0034] According to a second aspect of an embodiment of the present application, a neural network-based object reconstruction device is provided, comprising:
[0035] An acquisition unit, configured to acquire multi-view images and sparse 3D point clouds of an object;
[0036] A first extraction unit is used to extract features from the multi-view image using an RGB multi-layer perceptron neural network to obtain a color value of the object surface;
[0037] A second extraction unit is used to extract features from the sparse 3D point cloud using an SDF multi-layer perceptron neural network to obtain a signed distance value;
[0038] A first constraint unit is used to perform appearance constraints on the RGB multi-layer perceptron neural network based on the color value;
[0039] A second constraint unit is used to geometrically constrain the SDF multi-layer perceptron neural network based on the matching distance value;
[0040] A training unit, for jointly training the RGB multi-layer perceptron neural network after appearance constraints and the SDF multi-layer perceptron neural network after geometric constraints;
[0041] The third extraction unit is used to extract a mesh model based on the sparse 3D point cloud using the jointly trained SDF multi-layer perceptron neural network;
[0042] A conversion unit, configured to convert the grid model into a 3D Gaussian model;
[0043] The optimization unit is used to continuously optimize the 3D Gaussian model by using the real image of the object as a supervision signal.
[0044] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method according to the first aspect.
[0045] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including: a memory and a processor;
[0046] The memory is used to store computer instructions;
[0047] The processor is configured to call the computer instructions stored in the memory so that the electronic device executes the method according to the first aspect.
[0048] The technical solution of this application achieves a breakthrough from sparse / incomplete input to high-fidelity reconstruction through multimodal data fusion + dual neural network joint training + implicit field representation + 3D Gaussian refinement. Its advantages are:
[0049] 1. Data efficiency (adapting to sparse input);
[0050] 2. Geometry-appearance collaborative optimization (solving the fragmentation problem);
[0051] 3. High expressiveness of implicit fields (arbitrary resolution surfaces);
[0052] 4. Differentiable rendering driver optimization (to achieve automatic iteration and upgrade of results).
[0053] 5. Suitable for scenarios with high precision requirements and limited input data (such as archaeological fragment restoration, monocular RGB-D sparse scanning reconstruction, and VR / AR content generation).
[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:
[0056] Figure 1 A schematic flow chart of a neural network-based object reconstruction method according to an embodiment of the present application is shown;
[0057] Figure 2 A block diagram of an object reconstruction device based on a neural network according to an embodiment of the present application is shown;
[0058] Figure 3 A block diagram of an electronic device according to an embodiment of the present application is shown;
[0059] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0060] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.
[0061] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0062] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0063] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0064] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.
[0065] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0066] The following will describe some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.
[0067] See also Figure 1 , shows a flow chart of a neural network-based object reconstruction method according to an embodiment of the present application.
[0068] like Figure 1 As shown, a neural network-based object reconstruction method is presented, which specifically includes steps S100 to S800.
[0069] refer to Figure 1 ,Step S100, obtain multi-view images and sparse 3D point clouds of the object.
[0070] In some feasible embodiments, based on the above solution, acquiring multi-view images of an object includes:
[0071] An RGB camera is used to capture multi-view images of the target object from multiple perspectives.
[0072] It is understandable that the acquired multi-view images can provide appearance information (such as color, texture, etc.). In some feasible embodiments, based on the above scheme, a sparse 3D point cloud of an object is acquired, including:
[0073] The sparse three-dimensional coordinate points on the surface of an object are obtained through structured light, laser scanning or depth camera to form a sparse 3D point cloud of the object.
[0074] Continue to refer Figure 1 In step S200 , a RGB multi-layer perceptron neural network is used to perform feature extraction on the multi-view image to obtain the color value of the object surface.
[0075] It is understood that the obtained color values are used to capture appearance details.
[0076] Continue to refer Figure 1 , step S300, using the SDF multi-layer perceptron neural network to extract features from the sparse 3D point cloud and obtain a signed distance value.
[0077] It can be understood that the obtained signed distance value is used to characterize the geometric shape.
[0078] Continue to refer Figure 1 , step S400, performing appearance constraints on the RGB multi-layer perceptron neural network based on the color value.
[0079] It can be understood that performing appearance constraints can optimize appearance consistency.
[0080] In some feasible embodiments, based on the aforementioned solution, performing appearance constraints on the RGB multilayer perceptron neural network based on the color value includes:
[0081] The color values are aggregated into a volume rendering result through voxel integration, and compared with the real multi-view image, and the color pixel-level error is calculated to obtain the color loss.
[0082] Continue to refer Figure 1 , step S500, geometrically constraining the SDF multi-layer perceptron neural network based on the matching distance value.
[0083] In some feasible embodiments, based on the above solution, the geometrically constraining the SDF multilayer perceptron neural network based on the matching distance value includes:
[0084] Using the true SDF constraint, the error between the predicted SDF and the true SDF is calculated to obtain the signed distance loss.
[0085] It can be understood that the actual SDF constraint is the precise geometric prior or implicit surface constraint of a known object.
[0086] Continue to refer Figure 1 In step S600, the RGB multi-layer perceptron neural network subjected to appearance constraints and the SDF multi-layer perceptron neural network subjected to geometric constraints are jointly trained.
[0087] It is understandable that during the joint training process, the parameters of the RGB multi-layer perceptron neural network and the SDF multi-layer perceptron neural network are optimized simultaneously, so that the geometric prediction and the appearance prediction are spatially aligned to form a unified implicit representation.
[0088] Continue to refer Figure 1 , step S700, based on the sparse 3D point cloud, a grid model is extracted using the jointly trained SDF multi-layer perceptron neural network.
[0089] In some feasible embodiments, based on the above solution, the mesh model is extracted based on the sparse 3D point cloud using a jointly trained SDF multi-layer perceptron neural network, including:
[0090] Input the sparse 3D point cloud into the jointly trained SDF multi-layer perceptron neural network to obtain the complete SDF field of the object surface;
[0091] The Marching Cubes algorithm is used to extract the isosurface of the SDF field and generate a mesh model of the object surface.
[0092] Continue to refer Figure 1 , step S800, converting the grid model into a 3D Gaussian model, and using the real image of the object as a supervision signal to continuously optimize the 3D Gaussian model.
[0093] In some feasible embodiments, based on the above solution, using the real image of the object as a supervisory signal and continuously optimizing the 3D Gaussian model includes:
[0094] Perform ray casting on the target view in the real image of the object and calculate the intersection of the ray and the 3D Gaussian model;
[0095] Predict the color and opacity at the intersection according to the Gaussian distribution and render the new view image;
[0096] Calculate the pixel-level error between the rendered image and the real image of the object, and backpropagate to optimize the Gaussian parameters.
[0097] The following describes an embodiment of the device of the present application, which can be used to implement a neural network-based object reconstruction method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0098] Reference Figure 2 As shown, according to one embodiment of the present application, a neural network-based object reconstruction device 200 includes:
[0099] An acquisition unit 201 is configured to acquire a multi-view image and a sparse 3D point cloud of an object;
[0100] The first extraction unit 202 is configured to perform feature extraction on the multi-view image using an RGB multi-layer perceptron neural network to obtain a color value of the object surface;
[0101] The second extraction unit 203 is used to extract features from the sparse 3D point cloud using an SDF multi-layer perceptron neural network to obtain a signed distance value;
[0102] A first constraint unit 204 is configured to perform appearance constraints on the RGB multilayer perceptron neural network based on the color value;
[0103] A second constraint unit 205 is configured to geometrically constrain the SDF multilayer perceptron neural network based on the matching distance value;
[0104] A training unit 206 is used to jointly train the RGB multi-layer perceptron neural network after performing appearance constraints and the SDF multi-layer perceptron neural network after performing geometric constraints;
[0105] The third extraction unit 207 is configured to extract a mesh model based on the sparse 3D point cloud using the jointly trained SDF multi-layer perceptron neural network;
[0106] A conversion unit 208, configured to convert the mesh model into a 3D Gaussian model;
[0107] The optimization unit 209 is configured to use the real image of the object as a supervisory signal to continuously optimize the 3D Gaussian model.
[0108] like Figure 3 As shown, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, the steps of the above-mentioned neural network-based object reconstruction method are implemented.
[0109] Since the electronic device introduced in this embodiment is a device used to implement a neural network-based object reconstruction device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is no longer introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application falls within the scope of protection to be protected by this application.
[0110] During the specific implementation process, when the computer program 311 is executed by the processor, any implementation method of the embodiments corresponding to the first aspect can be implemented.
[0111] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.
[0112] It should be noted that Figure 4 The computer system 400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0113] like Figure 4As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 into the random access memory (RAM) 403, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 403. The CPU 401, ROM 402 and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0114] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read therefrom can be installed into the storage section 408 as needed.
[0115] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the various functions defined in the system of the present application are executed.
[0116] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.
[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0118] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.
[0119] As another aspect, the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the neural network-based object reconstruction method described in the above embodiment.
[0120] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the neural network-based object reconstruction method described in the above embodiments.
[0121] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0122] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.
[0123] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in this application. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of this application is limited only by the appended claims.
Claims
1. A method for object reconstruction based on a neural network, characterized in that: include: Obtain multi-view images and sparse 3D point clouds of objects; Using an RGB multi-layer perceptron neural network to extract features from the multi-view image to obtain color values of the object surface; Using an SDF multi-layer perceptron neural network to extract features from the sparse 3D point cloud and obtain a signed distance value; Performing appearance constraints on the RGB multilayer perceptron neural network based on the color value; Performing geometric constraints on the SDF multilayer perceptron neural network based on the matching distance value; Jointly train the RGB multi-layer perceptron neural network with appearance constraints and the SDF multi-layer perceptron neural network with geometric constraints; Based on the sparse 3D point cloud, the mesh model is extracted using the jointly trained SDF multi-layer perceptron neural network; The grid model is converted into a 3D Gaussian model, and the real image of the object is used as a supervision signal to continuously optimize the 3D Gaussian model.
2. The method according to claim 1, characterized in that Acquire multi-view images of an object, including: An RGB camera is used to capture multi-view images of the target object from multiple perspectives.
3. The method according to claim 1, characterized in that Get a sparse 3D point cloud of an object, including: The sparse three-dimensional coordinate points on the surface of an object are obtained through structured light, laser scanning or depth camera to form a sparse 3D point cloud of the object.
4. The method according to claim 1, wherein The performing appearance constraint on the RGB multi-layer perceptron neural network based on the color value includes: The color values are aggregated into a volume rendering result through voxel integration, and compared with the real multi-view image, and the color pixel-level error is calculated to obtain the color loss.
5. The method according to claim 1, wherein The geometrically constraining the SDF multi-layer perceptron neural network based on the matching distance value includes: Using the true SDF constraint, the error between the predicted SDF and the true SDF is calculated to obtain the signed distance loss.
6. The method according to claim 1, characterized in that The mesh model is extracted based on the sparse 3D point cloud using the jointly trained SDF multi-layer perceptron neural network, including: Input the sparse 3D point cloud into the jointly trained SDF multi-layer perceptron neural network to obtain the complete SDF field of the object surface; The Marching Cubes algorithm is used to extract the isosurface of the SDF field and generate a mesh model of the object surface.
7. The method according to claim 1, characterized in that The method of using the real image of the object as a supervisory signal and continuously optimizing the 3D Gaussian model includes: Perform ray casting on the target view in the real image of the object and calculate the intersection of the ray and the 3D Gaussian model; Predict the color and opacity at the intersection according to the Gaussian distribution and render the new view image; Calculate the pixel-level error between the rendered image and the real image of the object, and backpropagate to optimize the Gaussian parameters.
8. An object reconstruction device based on a neural network, characterized in that: include: An acquisition unit, configured to acquire multi-view images and sparse 3D point clouds of an object; A first extraction unit is used to extract features from the multi-view image using an RGB multi-layer perceptron neural network to obtain a color value of the object surface; A second extraction unit is used to extract features from the sparse 3D point cloud using an SDF multi-layer perceptron neural network to obtain a signed distance value; A first constraint unit is used to perform appearance constraints on the RGB multi-layer perceptron neural network based on the color value; A second constraint unit is used to geometrically constrain the SDF multi-layer perceptron neural network based on the matching distance value; A training unit, for jointly training the RGB multi-layer perceptron neural network after appearance constraints and the SDF multi-layer perceptron neural network after geometric constraints; The third extraction unit is used to extract a mesh model based on the sparse 3D point cloud using the jointly trained SDF multi-layer perceptron neural network; A conversion unit, configured to convert the grid model into a 3D Gaussian model; The optimization unit is used to continuously optimize the 3D Gaussian model by using the real image of the object as a supervision signal.
9. A computer-readable storage medium, characterized in that The storage medium stores computer instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.
10. An electronic device, characterized in that: include: memory and processor; The memory is used to store computer instructions; The processor is configured to call the computer instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.