Object three-dimensional reconstruction method and device based on deep learning

Abstract

The application discloses a kind of object three-dimensional reconstruction method and device based on deep learning, it is related to three-dimensional reconstruction technical field, wherein, the reconstruction method includes: using camera imaging equipment to collect target object video of target object, and based on target object video, determine target image set, based on target image set, determine target camera parameter and initial camera pose, and correct initial camera pose, obtain target camera pose, input target image set, target camera parameter and target camera pose to preset neural radiance field model, output target radiance field information, based on target radiance field information, construct the object three-dimensional model of target object.The application solves the technical problem that the accuracy of three-dimensional reconstruction of object in related art is low.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and more specifically, to a method and apparatus for 3D object reconstruction based on deep learning. Background Technology

[0002] 3D reconstruction technology reconstructs a 3D model of an object by acquiring its geometric and texture information. During the 3D reconstruction process, it is necessary to simultaneously acquire information such as the object's external shape and surface texture to support subsequent visualization, analysis, and applications. This technology has broad application prospects in fields such as industrial manufacturing, cultural relic preservation, virtual reality, and medical imaging.

[0003] While LiDAR (Light Detection and Ranging) technology can acquire high-precision 3D point cloud information of objects, it is expensive and time-consuming to acquire data. Furthermore, while using depth camera-based 3D phenotypic analysis methods for 3D model reconstruction can acquire 3D models of objects quickly and at low cost, the data obtained by depth cameras has low resolution, and its 3D representational capabilities are severely degraded due to distortion and noise, making it difficult to accurately reconstruct objects with complex structures.

[0004] In related technologies, reconstruction techniques based on multi-view color images are generally employed to reconstruct point cloud models using low-cost equipment. Examples include SFM (Structure from Motion), MVS (Multi-view Stereo, generating dense point clouds), and voxel sculpting algorithms. These algorithms can extract 3D information of objects from color images captured from multiple perspectives, thereby reconstructing point cloud models. However, to achieve automatic extraction of the object's 3D model and accelerate reconstruction, these algorithms require a solid-color background and stable lighting conditions during data acquisition. They also rely on a checkerboard pattern to provide sufficient feature points for the algorithm. Furthermore, to convert the algorithm-derived model into accurate reconstructions in metric units, a calibrator is needed. This makes it difficult to achieve fast and high-precision 3D reconstruction of a single object against any background.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method and apparatus for three-dimensional reconstruction of objects based on deep learning, in order to at least solve the technical problem of low accuracy in three-dimensional reconstruction of objects in related technologies.

[0007] According to one aspect of the present invention, a method for three-dimensional reconstruction of an object is provided, comprising: acquiring a target object video using a camera imaging device, and determining a target image set based on the target object video, wherein the target image set includes a preset number of target images, each of the target images containing the target object; determining target camera parameters and an initial camera pose based on the target image set, and correcting the initial camera pose to obtain a target camera pose; inputting the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model, and outputting target radiation field information, wherein the preset neural radiation field model includes at least a preset density network structure and a preset color network structure, the preset density network structure being used to extract volume density information of each ray sampling point, and the preset color network structure being used to extract color information of each ray sampling point; the target radiation field information including at least the volume density information and the color information; and constructing a three-dimensional model of the target object based on the target radiation field information.

[0008] Further, the step of acquiring a target object video using a camera imaging device and determining a target image set based on the target object video includes: capturing the target object using the camera imaging device to obtain the target object video, wherein the target object video is a multi-view video, and the multi-view video includes: multiple single-view videos; downsampling each of the single-view videos based on a preset time interval to obtain a first image set for each of the single-view videos; for each of the first image sets, removing blurred images from the first image set to obtain a second image set; sampling each of the second image sets to obtain a third image set, and adding each third image from each of the third image sets to the target image set.

[0009] Further, the step of removing blurry images from the first image set to obtain the second image set includes: for each first image in the first image set, converting the first image into a grayscale image based on the pixel values ​​of each channel on the first image; determining the Laplacian operator for each grayscale image, and determining the variance value of each grayscale image based on the Laplacian operator; determining the average variance value based on all the variance values, and determining a preset variance threshold based on the average variance value; identifying grayscale images with variance values ​​less than the preset variance threshold as blurry images, and removing the blurry images from the first image set to obtain the second image set.

[0010] Furthermore, the initial camera pose is a camera pose determined on a virtual coordinate system. The step of correcting the initial camera pose to obtain the target camera pose includes: determining a rotation matrix and a scaling factor, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling factor is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; and correcting the initial camera pose based on the rotation matrix and the scaling factor to determine the target camera pose.

[0011] Further, the step of determining the rotation matrix includes: obtaining the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fitting all the camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; determining a direction vector based on the direction axis of the real coordinate system, and determining a rotation vector based on the plane normal vector and the direction vector; determining the cosine value between the plane normal vector and the direction vector, and adjusting the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold if the cosine value is less than the preset cosine threshold; constructing an antisymmetric matrix based on the rotation vector, and determining the rotation matrix based on the antisymmetric matrix, the identity matrix, the magnitude of the rotation vector, and the cosine value.

[0012] Further, the step of determining the scaling ratio includes: determining a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera is shooting the target object, extracted from the target image; the second radius is the distance between the camera and the target object in the real coordinate system, determined based on the camera imaging device; and determining the scaling ratio based on the first radius and the second radius.

[0013] Further, before inputting the target image set, the target camera parameters, and the target camera pose into the preset neural radiation field model and outputting the target radiation field information, the method further includes: constructing a camera pose optimizer, the preset density network structure, and the preset color network structure, wherein the camera pose optimizer is used to optimize the input target camera pose; constructing an initial neural radiation field model based on the camera pose optimizer, the preset density network structure, and the preset color network structure; acquiring a historical image set and labeling each historical image in the historical image set to obtain labeling information, wherein the labeling information includes at least: the camera parameters and camera pose corresponding to the historical image; representing all the historical images associated with the labeling information as a training data set; and training the initial neural radiation field model using the training data set to obtain the preset neural radiation field model.

[0014] Further, the step of inputting the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model and outputting target radiation field information includes: optimizing the target camera pose to obtain an optimized target camera pose; generating ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points sampled from the rays formed by the pixels emitted by the camera onto each of the target images; encoding the position of the ray sampling points to obtain a position code, and encoding the ray direction of the ray corresponding to the ray sampling point to obtain a direction code; determining the volume density information and multidimensional feature vector of the ray sampling points using the preset density network structure based on the position code; determining the color information of the ray sampling points using the preset color network structure based on the direction code and the multidimensional feature vector; and constructing the target radiation field information based on the volume density information and color information of each ray sampling point.

[0015] Furthermore, before constructing the target radiation field information based on the volume density information and color information of each of the ray sampling points, the method further includes: determining an initial image exposure rate, and constructing a two-dimensional image based on the initial image exposure rate, the volume density information, and the color information; and optimizing the initial image exposure rate, the volume density information, and the color information based on the two-dimensional image and the target image to obtain the target image exposure rate, the optimized volume density information, and the optimized color information.

[0016] According to another aspect of the present invention, a three-dimensional object reconstruction apparatus is also provided, comprising: a determining unit, configured to acquire a target object video using a camera imaging device, and determine a target image set based on the target object video, wherein the target image set includes: a preset number of target images, each of the target images containing the target object; a correcting unit, configured to determine target camera parameters and an initial camera pose based on the target image set, and correct the initial camera pose to obtain a target camera pose; an output unit, configured to input the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model, and output target radiation field information, wherein the preset neural radiation field model includes at least: a preset density network structure and a preset color network structure, the preset density network structure being used to extract volume density information of each ray sampling point, and the preset color network structure being used to extract color information of each ray sampling point; the target radiation field information includes at least: the volume density information and the color information; and a constructing unit, configured to construct a three-dimensional object model of the target object based on the target radiation field information.

[0017] Further, the first determining unit includes: a first shooting module, configured to use the camera imaging device to shoot the target object to obtain a video of the target object, wherein the target object video is a multi-view video, and the multi-view video includes: multiple single-view videos; a first sampling module, configured to downsample each of the single-view videos based on a preset time interval to obtain a first image set for each of the single-view videos; a first elimination module, configured to eliminate blurred images in each of the first image sets to obtain a second image set; and a second sampling module, configured to sample each of the second image sets to obtain a third image set, and add each third image in each of the third image sets to the target image set.

[0018] Further, the first elimination module includes: a first conversion submodule, configured to convert each first image in the first image set into a grayscale image based on the pixel values ​​of each channel in the first image; a first determination submodule, configured to determine the Laplacian operator for each grayscale image and determine the variance value of each grayscale image based on the Laplacian operator; a second determination submodule, configured to determine the average variance value based on all the variance values ​​and determine a preset variance threshold based on the average variance value; and a third determination submodule, configured to determine the grayscale images indicated by the variance values ​​less than the preset variance threshold as the blurred images and eliminate the blurred images from the first image set to obtain the second image set.

[0019] Further, the correction unit includes: a first determining module, used to determine a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; and a second determining module, used to correct the initial camera pose based on the rotation matrix and the scaling ratio, and determine the target camera pose.

[0020] Further, the first determining module includes: a first acquiring submodule, used to acquire the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fit all the camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; a fourth determining submodule, used to determine a direction vector based on the direction axis of the real coordinate system, and to determine a rotation vector based on the plane normal vector and the direction vector; a fifth determining submodule, used to determine the cosine value between the plane normal vector and the direction vector, and to adjust the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold if the cosine value is less than a preset cosine threshold; and a sixth determining submodule, used to construct an antisymmetric matrix based on the rotation vector, and to determine the rotation matrix based on the antisymmetric matrix, the identity matrix, the magnitude of the rotation vector, and the cosine value.

[0021] Furthermore, the first determining module further includes: a seventh determining submodule, used to determine a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera is shooting the target object, extracted from the target image; the second radius is the distance between the camera and the target object in the real coordinate system, determined based on the camera imaging device; and an eighth determining submodule, used to determine the scaling ratio based on the first radius and the second radius.

[0022] Further, the object 3D reconstruction device includes: a first construction module, used to construct a camera pose optimizer, a preset density network structure, and a preset color network structure before inputting the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model and outputting target radiation field information, wherein the camera pose optimizer is used to optimize the input target camera pose; a second construction module, used to construct an initial neural radiation field model based on the camera pose optimizer, the preset density network structure, and the preset color network structure; a first processing module, used to acquire a historical image set and annotate each historical image in the historical image set to obtain annotation information, wherein the annotation information includes at least: the camera parameters and camera pose corresponding to the historical image; a first representation module, used to represent all the historical images associated with the annotation information as a training data set; and a first training module, used to train the initial neural radiation field model using the training data set to obtain the preset neural radiation field model.

[0023] Further, the output unit includes: a first optimization module, used to optimize the target camera pose to obtain an optimized target camera pose; a first generation module, used to generate ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points sampled from a ray formed by pixels emitted by the camera to each target image; a first encoding module, used to encode the position of the ray sampling points to obtain a position code, and to encode the ray direction of the ray corresponding to the ray sampling points to obtain a direction code; a third determination module, used to determine the volume density information and multidimensional feature vector of the ray sampling points based on the position code and using the preset density network structure; a fourth determination module, used to determine the color information of the ray sampling points based on the direction code and the multidimensional feature vector and using the preset color network structure; and a third construction module, used to construct the target radiation field information based on the volume density information and color information of each ray sampling point.

[0024] Furthermore, the object 3D reconstruction device further includes: a fifth determining module, used to determine an initial image exposure rate before constructing the target radiation field information based on the volume density information and color information of each of the ray sampling points, and to construct a two-dimensional image based on the initial image exposure rate, the volume density information, and the color information; and a second optimizing module, used to optimize the initial image exposure rate, the volume density information, and the color information based on the two-dimensional image and the target image to obtain the target image exposure rate, the optimized volume density information, and the optimized color information.

[0025] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the object three-dimensional reconstruction method described in any one of the above embodiments.

[0026] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described object three-dimensional reconstruction methods.

[0027] In this invention, a camera imaging device can be used to acquire video of the target object, and based on the video, a target image set is determined. Based on the target image set, target camera parameters and initial camera pose are determined, and the initial camera pose is corrected to obtain the target camera pose. The target image set, target camera parameters, and target camera pose are input into a preset neural radiation field model, and target radiation field information is output. Based on the target radiation field information, a three-dimensional model of the target object is constructed, thereby solving the technical problem of low accuracy in three-dimensional reconstruction of objects in related technologies.

[0028] In this invention, a preset number of target images containing the target object can be obtained from the target object video captured by the camera imaging device. Then, based on the target images, the target camera parameters and initial camera pose can be determined. The initial camera pose can be corrected according to the direction and scale relationship between the coordinate systems to obtain the target camera pose. After that, the preset number of target images, target camera parameters, and target camera pose are input into a preset neural radiation field model, which outputs volume density information and color information. Based on the output volume density information and color information, a three-dimensional model of the target object can be constructed without segmenting the image background. This achieves high throughput and high precision three-dimensional object reconstruction, improves the reconstruction efficiency and quality of the target object, and thus achieves the technical effect of accurately and efficiently reconstructing the three-dimensional model of the object under any background. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0030] Figure 1 This is a flowchart of an optional three-dimensional object reconstruction method according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of an optional camera imaging system according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of an optional SFM process according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of an optional camera pose correction according to an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of an optional extraction of the region of interest according to an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of an optional density network structure according to an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of an optional color network structure according to an embodiment of the present invention;

[0037] Figure 8 This is a schematic diagram of the optimization process of an optional neural radiation field model according to an embodiment of the present invention;

[0038] Figure 9 This is a schematic diagram of an optional ray sampling point selection according to an embodiment of the present invention;

[0039] Figure 10 This is a schematic diagram of an optional hash encoding according to an embodiment of the present invention;

[0040] Figure 11 This is a schematic diagram of an optional overall framework for object 3D reconstruction according to an embodiment of the present invention;

[0041] Figure 12 This is a schematic diagram of an optional three-dimensional object reconstruction process according to an embodiment of the present invention;

[0042] Figure 13 This is a schematic diagram of an optional three-dimensional object reconstruction device according to an embodiment of the present invention;

[0043] Figure 14 This is a hardware structure block diagram of an electronic device (or mobile device) for a three-dimensional reconstruction method of an object according to an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] It should be noted that all related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) collected and involved in this invention are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0047] In this invention, because related technologies use a monochrome background to segment objects from their surroundings, current 3D reconstruction methods struggle to achieve fast and high-precision 3D reconstruction of single objects against arbitrary backgrounds. Furthermore, 3D reconstruction methods based on Nerf (Neural Radiance Fields, a computer vision technique) have limitations in representing geometric details. Therefore, this invention proposes a deep learning-based 3D object reconstruction method. This method employs a shallow multilayer perceptron to approximate the density and color fields, improves the Nerf algorithm by introducing a novel ray sampling strategy, and applies hash encoding to map input points to a high-dimensional space. Simultaneously, an optimizer is constructed for camera pose, enabling Nerf to recover stronger spatial geometric details, enhancing its geometric accuracy, significantly improving network training speed, and increasing the overall efficiency of the reconstruction process.

[0048] The present invention will now be described in detail with reference to various embodiments.

[0049] Example 1

[0050] According to an embodiment of the present invention, an embodiment of a method for three-dimensional reconstruction of an object is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0051] Figure 1 This is a flowchart of an optional three-dimensional object reconstruction method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0052] Step S101: Use a camera imaging device to acquire a target object video, and determine a target image set based on the target object video. The target image set includes a preset number of target images, and each target image contains the target object.

[0053] Optionally, a target object video (i.e., a video containing the target object) can be acquired using a camera imaging device (such as an image acquisition device in a "camera-to-object" mode). Based on the target object video, frames can be extracted from the video, and the extracted images can be preprocessed to obtain a preset number (e.g., 90) of target images.

[0054] In this embodiment, the target object can be located in any background, and a camera imaging device can be used to directly capture the target object at its location to obtain a video of the target object.

[0055] Figure 2This is a schematic diagram of an optional camera imaging system according to an embodiment of the present invention, such as... Figure 2 As shown, the camera imaging system includes: a controller, limit switches, a hexagonal base, a DC motor, a driven wheel, a metal ring, a camera bracket, a camera, and a counterweight. The camera's shooting trajectory is a spatial circle, with the plane of the circle parallel to the ground. A short dashed circle represents the upper imaging trajectory, and a dotted dashed circle represents the lower imaging trajectory. The radius of the metal ring is 75cm. The camera bracket base is made of aluminum with a grooved structure, allowing for free adjustment of the extension length of the camera bracket base, thereby adjusting the radius of the camera bracket from the center of the metal ring, up to a maximum adjustment of 30cm (meaning the radius of the spatial circle is also variable, ranging from 75cm to 105cm). During image acquisition for each object, the object is placed at the center of the metal ring. Powered by the DC motor, the metal ring with the camera bracket rotates, and the camera circles the object for one revolution, automatically stopping after video recording. Each shooting process takes approximately 15 seconds. Any identical RGB (Red Green Blend, an additive color model) imaging device can be used.

[0056] In one example, two mobile terminals can be used as imaging devices to shoot dual-view video, with a video resolution of 1440×2560.

[0057] In some alternative embodiments, imaging devices can be used to capture three-view or more-view videos.

[0058] Step S102: Based on the target image set, determine the target camera parameters and initial camera pose, and correct the initial camera pose to obtain the target camera pose.

[0059] Optionally, based on the target image set, the target camera parameters (i.e., camera intrinsic parameters) and the initial camera pose (i.e., camera extrinsic parameters) can be obtained. The target camera parameters describe the inherent properties of the camera itself, including parameters such as focal length, pixel pitch, and camera lens distortion. The camera pose describes the position and orientation of the camera relative to the world coordinate system.

[0060] In this embodiment, SFM can be used to estimate the target camera parameters and initial camera pose from an unordered set of images. For example, COLMAP software (an open-source 3D reconstruction software that combines SFM and MVS technologies) uses an incremental approach for SFM. First, it selects a pair of images with the most geometrically consistent matching from all images. For the selected image pair, COLMAP uses a five-point algorithm or other methods to estimate the camera parameters and initial camera pose. Then, it registers additional images based on shared keypoints (i.e., feature points) (i.e., estimates the camera pose of the remaining images). After estimating the camera parameters and initial camera pose, COLMAP globally optimizes the camera parameters, initial camera pose, 3D structure, and image keypoints through bundle adjustment, resulting in more accurate target camera parameters and initial camera pose.

[0061] Figure 3 This is a schematic diagram of an optional SFM process according to an embodiment of the present invention, such as... Figure 3 As shown, firstly, feature points are extracted from the input target image. Then, feature matching is performed between multiple images to extract feature point pairs. The camera pose (i.e., camera parameters and camera pose) is estimated using the feature point pairs obtained from feature matching. The camera parameters and camera pose include: image size (e.g., 1440×2560), horizontal field of view (e.g., 1.13), vertical field of view (e.g., 1.66), camera lens focal length (e.g., 1169), and camera type (e.g., PINHOLE). Then, sparse reconstruction is performed based on the camera parameters, camera pose, and the matched feature point pairs to obtain the 3D point cloud of the scene.

[0062] In this embodiment, since the obtained initial camera pose has a different global coordinate system than the real world, the target camera pose can be obtained by correcting the initial camera pose.

[0063] Figure 4 This is an optional camera pose correction schematic diagram according to an embodiment of the present invention, such as... Figure 4 As shown, the normal vector of the trajectory plane captured by the target camera is parallel to the coordinate axis (i.e., the z-axis). Figure 4 As shown on the right), the trajectory plane normal vector obtained from the estimated initial camera position has a certain angle with the coordinate axes (e.g., Figure 4 As shown on the left), it can be seen that the initial camera pose deviates from the target camera pose. By correcting the initial camera pose, the target camera pose can be obtained.

[0064] Step S103: Input the target image set, target camera parameters, and target camera pose into the preset neural radiation field model and output the target radiation field information. The preset neural radiation field model includes at least: a preset density network structure and a preset color network structure. The preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point. The target radiation field information includes at least: volume density information and color information.

[0065] Optionally, the preset neural radiation field model includes at least a preset density network structure and a preset color network structure. The target image set, target camera parameters, and target camera pose are input into the preset neural radiation field model (such as an improved Nerf model). Based on the target camera pose, a ray emitted from the camera to the image pixels is constructed, and sampling points are sampled on the ray. The sampling points are input into the preset density network structure to extract the volume density information of each ray sampling point. Then, the output vector of the preset density network structure and the observation direction of the ray sampling points are input into the preset color network structure to extract the color information of each ray sampling point.

[0066] In this embodiment, the preset neural radiation field model implicitly represents the 3D scene as a 3D density field and a 5D light field (i.e., radiation field) approximated by a neural network. The radiation field describes the volume density of each sampling point in the scene and the color of each sampling point in each viewing direction, and can be represented as a function. Among them, F θ Let represent a multilayer perceptron network, θ represent the network parameter combination, x represent the position coordinates of the sampling point in the input network, d represent the observation direction of the ray sampling point in the input network, and c represent the color of the sampling point output by the network. This represents the volume density of the sampling points output by the network.

[0067] Step S104: Based on the target radiation field information, construct a three-dimensional model of the target object.

[0068] Optionally, the preset neural radiation field model does not directly store the explicit geometric data of the 3D scene (such as voxels, point clouds, or mesh models), but provides an implicit continuous representation of the 3D scene. Based on the volume density and color information predicted by the preset neural radiation field model, an algorithm (such as the Marching Cubes algorithm) can be used to construct a 3D model of the target object.

[0069] In this embodiment, the Marching Cubes algorithm requires a predefined region of interest (the area around the object captured by the camera, i.e., the foreground region) and a density threshold (used to determine whether a voxel belongs to the surface of the region of interest). By adjusting the density threshold, surfaces with different density levels can be extracted, thereby achieving automated extraction of the 3D mesh model (i.e., automatically generating the 3D model of the target object).

[0070] Figure 5 This is a schematic diagram of an optional extraction of the region of interest according to an embodiment of the present invention, such as... Figure 5 As shown, it includes the Area of ​​Interest (i.e., region of interest), the Upper imaging orbit, and the Lower imaging orbit. The camera's shooting orbit separates the foreground region from the background region, and the foreground region is the region of interest. The region of interest can be accurately defined based on the camera's shooting orbit, enabling the automated extraction of the 3D mesh model.

[0071] In summary, a target object video can be acquired using a camera imaging device, and image sampling can be performed on the target object video to obtain a target image set. Based on the target image set, the target camera parameters and initial camera pose can be determined, and the initial camera pose can be corrected to obtain the target camera pose. Then, the target image set, target camera parameters, and target camera pose are input into a preset neural radiation field model, which outputs target radiation field information. Based on the target radiation field information, a 3D model of the target object is constructed. This not only completes the 3D reconstruction of the target object under any background but also effectively improves the efficiency of the 3D reconstruction of the target object, thereby solving the technical problem of low accuracy in 3D reconstruction of objects in related technologies.

[0072] To accurately obtain the target image set, in the object 3D reconstruction method provided in Embodiment 1 of this application, a camera imaging device is used to capture the target object to obtain a target object video. The target object video is a multi-view video, which includes: multiple single-view videos; based on a preset time interval, each single-view video is downsampled to obtain a first image set for each single-view video; for each first image set, blurred images in the first image set are removed to obtain a second image set; each second image set is sampled to obtain a third image set, and each third image in each third image set is added to the target image set.

[0073] In this embodiment, the relationship between the number of images and the reconstruction time is non-linear. If the number of images exceeds a certain threshold, the reconstruction time will increase significantly, and insufficient image quantity or poor image quality will affect the quality of the 3D reconstruction of the object. Therefore, it is necessary to determine the optimal balance between reconstruction quality and reconstruction time (i.e., the optimal number of images extracted, i.e., the preset number).

[0074] In this embodiment, by excluding blurry frames and limiting the number of images to approximately 90, high-quality 3D reconstruction can be achieved in a shorter time. Therefore, an intelligent frame extraction algorithm is proposed to remove blurry frames from multi-view videos and extract a certain number of images from each view, totaling 90 images, for use in subsequent reconstruction processes.

[0075] In this embodiment, the camera imaging device can downsample multiple perspectives (such as two single-view videos) captured by the camera imaging device at a preset time interval (such as 0.2s) to obtain a first image set for each single-view video (such as obtaining 50 images from each single-view video, for a total of 100 images). By removing the blurry frames in each first image set, a second image set can be obtained. Random sampling can be performed on each second image set to obtain a third image set. Each third image in each third image set is added to the target image set to ensure that a preset number of target images (such as 90 images) can be obtained.

[0076] In this embodiment, if the number of blurred frames removed from the first image set is greater than a preset removal threshold (e.g., 10), the quality of the collected data is determined to be unqualified, and the video needs to be collected again.

[0077] To accurately obtain the second image set, in the object 3D reconstruction method provided in Embodiment 1 of this application, for each first image in the first image set, the first image is converted into a grayscale image based on the pixel values ​​of each channel on the first image; the Laplacian operator for each grayscale image is determined, and the variance value of each grayscale image is determined based on the Laplacian operator; the average variance value is determined based on all variance values, and a preset variance threshold is determined based on the average variance value; grayscale images with variance values ​​less than the preset variance threshold are identified as blurred images, and the blurred images are removed from the first image set to obtain the second image set.

[0078] Optionally, the image sharpness can be quantitatively evaluated using a method based on the variance of the Laplacian operator. For each first image in the first image set, the pixel value of the grayscale image can be obtained by weighting the pixel values ​​of each channel (i.e., red (R), green (G), and blue (L)) in the first image (e.g., Y = 0.299R + 0.587G + 0.114B, where Y represents the pixel value of the grayscale image).

[0079] In this embodiment, the Laplace operator is a second-order differential operator, which can be expressed as:

[0080]

[0081] in, This represents the second-order partial derivative of the grayscale image in the x-direction. This represents the second-order partial derivative of the grayscale image in the y-direction. The discrete approximation of the Laplacian operator can be implemented using a convolution kernel; the 8-neighborhood Laplacian operator (L8) can be represented as a 3×3 convolution kernel.

[0082]

[0083] By multiplying the convolution kernel by the corresponding pixel in the grayscale image and summing the products, a new value can be generated for each pixel in the grayscale image (i.e., an approximation of the second derivative of the grayscale image at that point). Based on this pixel value, the variance (Var(L)) of the grayscale image can be calculated.

[0084]

[0085] Where i represents the i-th pixel in the image, N represents the total number of pixels in the image, and L i Let represent the pixel value of the i-th pixel, and μ represent the average value of all pixel values ​​in the image.

[0086] In this embodiment, variance is used as a measure of sharpness. The larger the variance, the higher the image sharpness; the smaller the variance, the lower the image sharpness. The average variance of all grayscale images can be calculated based on the calculated variance value. If the average variance of a grayscale image is less than a preset variance threshold (pre-set based on the average variance value, such as less than 20% of the average variance value), the grayscale image can be identified as a blurred image and removed from the first image set to obtain the second image set.

[0087] In order to accurately determine the pose of the target camera, in the object 3D reconstruction method provided in Embodiment 1 of this application, a rotation matrix and a scaling ratio are determined. The rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system. Based on the rotation matrix and the scaling ratio, the initial camera pose is corrected to determine the pose of the target camera.

[0088] Optionally, camera extrinsic parameters describe the camera's position and orientation relative to the world coordinate system. These parameters can be represented by a rotation matrix R and a translation vector t. The translation vector t describes the camera's position in the world coordinate system, while the rotation matrix R describes the camera's viewing angle, determining the angle from which the camera observes the scene.

[0089] In this embodiment, due to differences in shooting position, camera angle and camera configuration, the camera pose obtained based on SFM technology has a different global coordinate system (i.e., virtual coordinate system) than the real world, resulting in the 3D model reconstructed by Nerf having a different size and pose than the real world.

[0090] In this embodiment, the transformation relationship between the virtual coordinate system and the real coordinate system (i.e., the directional relationship between the directions of the two coordinate systems) can be found by using the rotation matrix R. vtr The camera's extrinsic parameters are corrected by using the scaling relationship between the two coordinate systems (represented by the scaling factor k) and the scale relationship between the two coordinate systems. This can be expressed as:

[0091] R = R vtr ·R virtual (4);

[0092] t = k·t virtual (5);

[0093] Among them, R virtual t represents the virtual rotation matrix recovered by the SFM technique. virtual R represents the virtual translation vector recovered by the SFM technique, R represents the corrected rotation matrix, and t represents the corrected translation vector.

[0094] To improve the accuracy of determining the rotation matrix, in the object 3D reconstruction method provided in Embodiment 1 of this application, the camera position coordinates of the camera in the camera imaging device when shooting the target object are obtained, and all camera position coordinates are fitted to obtain the plane normal vector, wherein the camera position coordinates are coordinates in a virtual coordinate system; based on the direction axis of the real coordinate system, the direction vector is determined, and based on the plane normal vector and the direction vector, the rotation vector is determined; the cosine value between the plane normal vector and the direction vector is determined, and if the cosine value is less than a preset cosine threshold, the plane normal vector is adjusted until the cosine value is greater than or equal to the preset cosine threshold; based on the rotation vector, an antisymmetric matrix is ​​constructed, and based on the antisymmetric matrix, the identity matrix, the magnitude of the rotation vector, and the cosine value, the rotation matrix is ​​determined.

[0095] Optionally, in order to determine the orientation relationship between the 3D virtual coordinate system and the real coordinate system, it is necessary to find a suitable reference to calculate the rotation matrix (i.e., R). vtr The normal vector of the trajectory plane can be used as a reference for the direction of the coordinate system.

[0096] In this embodiment, the camera's shooting trajectory is parallel to the ground, while the normal vector of the trajectory plane is parallel to the z-axis (i.e., the direction axis) of the real coordinate system. The camera position coordinates when shooting the target object can be obtained from the camera imaging device. By fitting all camera position coordinates, the plane normal vector 'a' is obtained (that is, the reconstructed camera position is considered a discrete representation of the camera's shooting trajectory, and the trajectory plane normal vector is obtained by plane fitting). The direction vector (i.e., the direction vector of the z-axis) can be determined based on the direction axis of the real coordinate system. ).

[0097] In this embodiment, the plane normal vector a and the direction vector b can be normalized to obtain a unit vector. Based on the plane normal vector and the direction vector (i.e., the plane normal vector...) and direction vector Cross product calculation) to determine the rotation vector (i.e., the rotation axis V):

[0098]

[0099] In this embodiment, the plane normal vector can be calculated. and direction vector The dot product of the vectors yields the plane normal vector. and direction vector The cosine of the angle between them:

[0100]

[0101] In this embodiment, the plane normal vector can be adjusted (e.g., in the plane normal vector) when the cosine value is less than a preset cosine threshold (e.g., -1 + 1e-10, i.e., a negative value close to -1). Add a small random perturbation (i.e., a very small positive number) until the cosine value is greater than or equal to the preset cosine threshold.

[0102] In this embodiment, the magnitude of the rotation vector is calculated and stored in the variable s (i.e., s = ||v||). An antisymmetric matrix (used to describe the cross product operation of the rotation vector) can be constructed from the components of the rotation vector.

[0103]

[0104] Where v1, v2, and v3 represent the components of the rotation vector on the coordinate axes x, y, and z, respectively.

[0105] The rotation matrix (R) can be determined based on the antisymmetric matrix (kmat), the identity matrix (I), the magnitude of the rotation vector (s = ||v||), and the cosine value (c). vtr ):

[0106]

[0107] Where 1e-10 represents 10 to the power of negative 10, which is 0.0000000001.

[0108] To improve the accuracy of determining the first radius and the second radius, in the object 3D reconstruction method provided in Embodiment 1 of this application, the first radius and the second radius are determined. The first radius is the distance between the camera and the target object in the virtual coordinate system when the camera is shooting the target object, extracted from the target image. The second radius is the distance between the camera and the target object in the real coordinate system, determined based on the camera imaging device. The scaling ratio is determined based on the first radius and the second radius.

[0109] Optionally, in order to obtain the scale relationship between the three-dimensional virtual coordinate system and the real coordinate system, it is necessary to find a corresponding reference to calculate the scaling ratio (i.e., k). The radius of the trajectory can be used as a reference for the coordinate system scale.

[0110] In this embodiment, the distance between the camera and the target object in the virtual coordinate system when the camera is shooting the target object can be determined based on the first radius (i.e., the distance between the camera and the target object in the virtual coordinate system extracted from the target image). virtual The second radius (i.e., the distance between the camera and the target object in the real coordinate system, determined by the camera imaging device) and the second radius (i.e., the distance between the camera and the target object in the real coordinate system, determined by the camera imaging device). real To calculate the scaling ratio:

[0111]

[0112] To accurately obtain the preset neural radiation field model, the object 3D reconstruction method provided in Embodiment 1 of this application constructs a camera pose optimizer, a preset density network structure, and a preset color network structure. The camera pose optimizer is used to optimize the input target camera pose. Based on the camera pose optimizer, the preset density network structure, and the preset color network structure, an initial neural radiation field model is constructed. A historical image set is collected, and each historical image in the historical image set is labeled to obtain labeling information. The labeling information includes at least: the camera parameters and camera pose corresponding to the historical image. All historical images associated with the labeling information are represented as a training data set. The initial neural radiation field model is trained using the training data set to obtain the preset neural radiation field model.

[0113] Optionally, the accuracy of camera pose is crucial for defining the relationship between the image capture viewpoint and the global world coordinate system (i.e., the real coordinate system). It can ensure the accurate transformation from real coordinates to camera coordinates, providing a foundation for model training and rendering. However, the camera pose recovered and corrected based on SFM technology deviates from the real value. This deviation will lead to errors in the light projection path, which will negatively affect the model's estimation of scene geometry and color distribution. Therefore, accurate correction of camera pose is required.

[0114] In this embodiment, a camera pose optimizer can be constructed. The optimizer has two trainable parameters: the camera position offset and the rotation offset. These two parameters can be integrated into the forward propagation process of the network, and the camera position offset and rotation offset parameters can be iteratively updated through the backpropagation algorithm based on the gradient information of the external parameters of the loss function. In order to suppress overfitting, an L2 regularization term can be added to the optimizer parameters based on the loss function.

[0115] In this embodiment, in order to improve the speed of the 3D reconstruction process, a preset density network structure (such as a two-layer MLP network (Multilayer Perceptron, i.e., artificial neural network)) and a preset color network structure (such as a three-layer MLP network) can be constructed to predict density information and color information respectively, and an initial neural radiation field model can be constructed based on the preset density network structure and the preset color network structure.

[0116] Figure 6 This is a schematic diagram of an optional density network structure according to an embodiment of the present invention, such as... Figure 6 As shown, the vector includes an input vector (i.e., γ(x) in 32 dimensions), two intermediate hidden layers, and an output vector. Here, γ(x) represents the positional encoding, the vector dimension of the intermediate hidden layers is 64, the output vector is σ (i.e., volume density information), and a 15-dimensional geometric feature vector. Solid arrows represent ReLU (Rectified Linear Unit) activation layers, dashed arrows represent sigmoid (Logistic Sigmoid Function) activation layers, and + indicates vector concatenation.

[0117] Figure 7 This is a schematic diagram of an optional color network structure according to an embodiment of the present invention, such as... Figure 7As shown, it includes an input vector (i.e., a 16-dimensional γ(d) and a 15-dimensional geometric feature vector), three intermediate hidden layers, and an output vector. Here, γ(d) represents the ray direction encoding, the vector dimension of the intermediate hidden layers is 64, the output vector is RGB (i.e., color information), solid arrows represent ReLU activation layers, dotted dashed arrows represent ELU (Exponential Linear Unit, i.e., an activation function) activation layers, and + indicates vector concatenation.

[0118] In this embodiment, a set of historical images can be collected and each historical image in the set can be labeled to obtain the annotation information such as camera parameters and camera pose corresponding to the historical images. All historical images with associated annotation information can be represented as a training data set. The initial neural radiation field model can be trained using this training data set until the model converges to obtain the trained preset neural radiation field model (such as the Nerf model).

[0119] Figure 8 This is a schematic diagram illustrating the optimization process of an optional neural radiation field model according to an embodiment of the present invention, such as... Figure 8 As shown, firstly, the target camera parameters (i.e., camera intrinsics) and initial camera pose (i.e., camera extrinsics) can be determined based on the input image. The initial camera pose can be corrected to obtain the target camera pose. Then, the target camera pose can be optimized using an optimizer to obtain the optimized target camera pose. Based on this target camera pose, ray sampling is performed on the rays emitted by the camera to obtain the position of each sampling point. The positions of the sampling points are then hashed to obtain position codes, and the ray directions are encoded to obtain the view vector. The position codes can be input into an MLP network (i.e., a density network), which is used to fit a 3D density field (i.e., the density field in space). Each sampling point is mapped to a value representing density, and the volume density information (i.e., σ) corresponding to each point can be obtained. Then, the volume density information and the view vector are concatenated together and input into another MLP network (i.e., color network). This network is used to fit the 5D light field (i.e., the direction and intensity of light propagation in three-dimensional space). It can obtain the color information (i.e., RGB) of each point in the viewing direction. Then, through volume rendering, the initial exposure rate is set and the exposure rate is adjusted to synthesize the image. The network is optimized by backpropagation (i.e., L) using the difference between the synthesized view and the real view and the color value as the loss function to obtain the optimized neural radiation field model.

[0120] To accurately obtain the volume density and color information of ray sampling points, the object 3D reconstruction method provided in Embodiment 1 of this application optimizes the target camera pose to obtain an optimized target camera pose. Based on the optimized target camera pose, ray sampling points are generated, wherein the ray sampling points are points sampled from the rays formed by the pixels emitted by the camera onto each target image. The positions of the ray sampling points are encoded to obtain position codes, and the ray directions of the corresponding rays are encoded to obtain direction codes. Based on the position codes, a preset density network structure is used to determine the volume density information and multi-dimensional feature vectors of the ray sampling points. Based on the direction codes and multi-dimensional feature vectors, a preset color network structure is used to determine the color information of the ray sampling points. Based on the volume density and color information of each ray sampling point, target radiation field information is constructed.

[0121] Optionally, a camera pose optimizer can be used to optimize the target camera pose to obtain an optimized target camera pose. Based on the optimized target camera pose, ray sampling points (i.e., points sampled on the ray formed by the pixels emitted by the camera to each target image) are generated.

[0122] In this embodiment, the selection of ray sampling points can affect the time and quality of 3D object reconstruction. Based on the camera imaging position, the scene can be divided into a foreground region and a background region. The foreground region is the area where the target object is located (i.e., the area where the camera surrounds the target object), and it is also a volume-dense region in space. The quality of the neural radiation field in the foreground region (i.e., the accuracy of the preset neural radiation field model in representing the scene) determines the reconstruction quality. The quality of the neural radiation field in the background region has no impact on the reconstruction quality. However, since there are a large number of blank areas between the foreground and background, sampling points falling in the blank areas not only have no effect on the optimization process, but also increase the computation and memory overhead, and prolong the reconstruction time of the target object.

[0123] In this embodiment, a fixed camera track can be used to distinguish between the foreground and background regions. A new sampling strategy generates more sampling points in the foreground region, allowing the model to focus more on the foreground region and skip blank areas during the optimization process, thereby improving reconstruction quality and reducing reconstruction time.

[0124] Figure 9 This is a schematic diagram illustrating an optional ray sampling point selection according to an embodiment of the present invention, as shown below. Figure 9As shown, ray sampling points can be selected in areas such as Empty (blank area), Foreground (foreground area), and Background (background area). Black dots on the ray represent high volume density points, while white dots represent low volume density points. By collecting more ray sampling points in the foreground area, the network can focus more on the foreground region.

[0125] In this embodiment, since shallow MLPs tend to learn low-frequency functions, they are not good at reconstructing high-frequency details of color and geometry, and cannot effectively represent the hierarchical structure of objects and organs. Multi-resolution hashing technology in instant-ngp (Instant Neural Graphics Primitives, a type of graphics primitive based on neural networks) can be used for encoding to obtain position codes, and the ray direction of the ray corresponding to the ray sampling point is encoded to obtain direction codes (such as using spherical harmonic functions to encode the gaze direction, mapping the gaze direction into a fixed-dimensional vector).

[0126] In this embodiment, multi-resolution hashing combines the advantages of spatial hashing and multi-level coding, allowing the network to capture both coarse and fine features of a scene at different resolution levels. The hash table can include L resolution levels, each containing T feature vectors, with each feature vector having a dimension of F. For different resolution levels, the hash table is graded from coarse to fine, and the grid size of different levels increases at a fixed ratio. To avoid increasing the output vector dimension and computational burden by simply adding more hash table layers, thus prolonging reconstruction time, the hash table can be optimized. In the 8th and 9th levels, the hash table resolution can not increase proportionally (e.g., the resolution of the 9th level is four times that of the 8th level). In the last level of the hash table, the space can be divided into 2048 hash voxels instead of the conventional 512. Without changing the output dimension of the hash coding, the coding capability at the microscale is enhanced, thereby improving the reconstruction capability at the scale of objects and organs.

[0127] For example, the hash table parameters are shown in Table 1:

[0128] Table 1

[0129]

[0130] Figure 10 This is a schematic diagram of an optional hash encoding according to an embodiment of the present invention, such as... Figure 10As shown, the space can be divided into a grid (i.e., a hash grid). L=0 and L=1 represent two levels, indicating different resolution levels. By finding voxels (i.e., a grid cell in three-dimensional space) of ray sampling points at different resolution levels, a hash function can be used to calculate the hash value of the voxel where the ray sampling point is located. The hash value is then used to look up the corresponding feature vector in the hash table (i.e., lookup). Linear interpolation is performed on the found feature vector to generate a feature vector corresponding to the input coordinates. The feature vectors obtained from different resolution levels can be connected to form the final feature vector (e.g., a d×L dimensional feature vector with a resolution of L levels and a data width of d), thus completing the position encoding of the ray sampling point.

[0131] In this embodiment, the position encoding of the ray sampling point can be input into a preset density network structure to predict the volume density information and multi-dimensional feature vector (such as a 15-dimensional geometric feature vector) of the ray sampling point. The direction encoding and the multi-dimensional feature vector predicted by the preset density network structure can be concatenated and input into a preset color network structure to predict the color information of the ray sampling point.

[0132] In order to obtain the exposure rate of the target image, the optimized volume density information, and the optimized color information, the object three-dimensional reconstruction method provided in Embodiment 1 of this application determines the initial image exposure rate, and constructs a two-dimensional image based on the initial image exposure rate, volume density information, and color information; based on the two-dimensional image and the target image, the initial image exposure rate, volume density information, and color information are optimized to obtain the target image exposure rate, the optimized volume density information, and the optimized color information.

[0133] Optionally, since the lighting conditions of a scene are uneven in a natural environment, images from different perspectives may have different exposure levels. RGB image-based 3D reconstruction algorithms are extremely sensitive to lighting conditions. Therefore, traditional 3D reconstruction methods set up artificial backgrounds to ensure the stability of the light source.

[0134] In this embodiment, to achieve 3D reconstruction against any background, the exposure rate can be integrated into the forward propagation process. The exposure rate of each image is incorporated as a learnable parameter into the model training process (e.g., an initial exposure rate can be set). The backpropagation algorithm can be used to adjust the exposure rate of each image in real time based on the gradient information calculated from the loss function. The learned exposure rate can be used to adjust color values ​​during image synthesis.

[0135]

[0136] RGBtarget =s i ·RGB (12);

[0137] Among them, e i Indicates exposure rate, s i This represents the scaling factor for pixel values; RGB represents image color.

[0138] In this embodiment, to prevent overfitting, L2 regularization (such as Δe) can be used. i The optimization process of exposure rate is supervised by λ), where λ is the regularization strength parameter, used to control the degree of influence of the regularization term, Δe. i This indicates a change in exposure rate. Based on the exposure adjustment step, the exposure rate under the synthesized new perspective image can be controlled to achieve a relighting effect. It can also significantly improve the accuracy of 3D reconstruction under uneven lighting backgrounds and enhance the robustness of the algorithm to complex and variable lighting conditions.

[0139] Figure 11 This is a schematic diagram of an optional overall framework for three-dimensional reconstruction of an object according to an embodiment of the present invention, such as... Figure 11 As shown, firstly, video of the target object is acquired and keyframes are extracted. Then, SFM is used to recover the camera pose and perform global camera pose calibration. After that, an improved Nerf network is used to reconstruct the neural radiation field of the target object, and a Marching Cubes network is used to extract the object mesh model from the neural radiation field to complete the 3D reconstruction of the target object.

[0140] Figure 12 This is a schematic diagram of an optional three-dimensional object reconstruction process according to an embodiment of the present invention, such as... Figure 12 As shown, the acquired video is first preprocessed to obtain a set of target images. The initial camera pose (i.e., camera extrinsic parameters) and target camera parameters (i.e., camera intrinsic parameters) are estimated using SFM technology. Then, the initial camera pose is corrected to obtain the target camera pose, which is input into the Nerf model. Based on the output of the model, new perspective images can be synthesized, and the mesh model can be extracted to achieve 3D reconstruction of the target object.

[0141] In an alternative embodiment, the following can be employed: Figure 12 The illustrated object 3D reconstruction process executes the object 3D reconstruction method.

[0142] In this embodiment of the invention, a target object video is acquired using a camera imaging device, and the target object video is downsampled to obtain a first image set for each single-view video. For each first image set, each image in the first image set is converted into a grayscale image, and the variance value of each grayscale image is calculated according to the Laplacian operator. A second image set is obtained based on the variance value. Then, each second image set is sampled to obtain a target image set. Based on the target image set, the target camera parameters and initial camera pose are determined, and the initial camera pose is corrected to obtain the target camera pose. Then, the target image set, target camera parameters, and target camera pose are input into a preset neural radiation field model, and the volume density information and color information of each ray sampling point are output. Based on the volume density information and color information, a three-dimensional model of the target object is constructed to complete the reconstruction of the three-dimensional model of the target object. This improves the training speed of the model, eliminates the need for background segmentation, and achieves the reconstruction of the three-dimensional object in a short time, thereby effectively improving the reconstruction efficiency and quality of the target object.

[0143] The following is a detailed description with reference to another embodiment.

[0144] Example 2

[0145] The object three-dimensional reconstruction device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0146] Figure 13 This is a schematic diagram of an optional three-dimensional object reconstruction device according to an embodiment of the present invention, such as... Figure 13 As shown, the object 3D reconstruction device may include: a determination unit 130, a correction unit 131, an output unit 132, and a construction unit 133.

[0147] The determining unit 130 is used to acquire a target object video using a camera imaging device and determine a target image set based on the target object video. The target image set includes a preset number of target images, each of which contains a target object.

[0148] The correction unit 131 is used to determine the target camera parameters and the initial camera pose based on the target image set, and to correct the initial camera pose to obtain the target camera pose.

[0149] The output unit 132 is used to input the target image set, target camera parameters and target camera pose into the preset neural radiation field model and output the target radiation field information. The preset neural radiation field model includes at least: a preset density network structure and a preset color network structure. The preset density network structure is used to extract the volume density information of each ray sampling point and the preset color network structure is used to extract the color information of each ray sampling point. The target radiation field information includes at least: volume density information and color information.

[0150] Construction unit 133 is used to construct a three-dimensional model of the target object based on the target radiation field information.

[0151] The aforementioned object 3D reconstruction device can, through the determining unit 130 using a camera imaging device to acquire target object video, and based on the target object video, determine a target image set; through the correction unit 131 using the target image set, determine the target camera parameters and initial camera pose, and correct the initial camera pose to obtain the target camera pose; through the output unit 132, input the target image set, target camera parameters, and target camera pose into a preset neural radiation field model, and output target radiation field information; and through the construction unit 133, construct a 3D model of the target object based on the target radiation field information. In this embodiment of the invention, by using a camera imaging device to acquire target... The system collects target object videos and performs image preprocessing to obtain a target image set. Then, based on the target image set, it determines the target camera parameters and initial camera pose, corrects the initial camera pose, and obtains the target camera pose. Subsequently, it inputs the target image set, target camera parameters, and target camera pose into a preset neural radiation field model, which outputs volume density information and color information. Based on the output volume density information and color information, it constructs a 3D model of the target object to achieve rapid and high-quality reconstruction of the target object, thereby solving the technical problem of low efficiency and accuracy in 3D reconstruction of objects with arbitrary backgrounds in related technologies.

[0152] Optionally, the first determining unit includes: a first shooting module, configured to capture a target object using a camera imaging device to obtain a target object video, wherein the target object video is a multi-view video, and the multi-view video includes: multiple single-view videos; a first sampling module, configured to downsample each single-view video based on a preset time interval to obtain a first image set for each single-view video; a first elimination module, configured to eliminate blurred images from each first image set to obtain a second image set; and a second sampling module, configured to sample each second image set to obtain a third image set, and add each third image from each third image set to the target image set.

[0153] Optionally, the first elimination module includes: a first conversion submodule, configured to convert each first image in the first image set into a grayscale image based on the pixel values ​​of each channel in the first image; a first determination submodule, configured to determine the Laplacian operator for each grayscale image and determine the variance value of each grayscale image based on the Laplacian operator; a second determination submodule, configured to determine the average variance value based on all variance values ​​and determine a preset variance threshold based on the average variance value; and a third determination submodule, configured to determine grayscale images with variance values ​​less than the preset variance threshold as blurred images and eliminate the blurred images from the first image set to obtain a second image set.

[0154] Optionally, the correction unit includes: a first determining module, used to determine the rotation matrix and the scaling ratio, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; and a second determining module, used to correct the initial camera pose based on the rotation matrix and the scaling ratio, and determine the target camera pose.

[0155] Optionally, the first determining module includes: a first acquiring submodule, used to acquire the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fit all camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in a virtual coordinate system; a fourth determining submodule, used to determine a direction vector based on the direction axis of the real coordinate system, and to determine a rotation vector based on the plane normal vector and the direction vector; a fifth determining submodule, used to determine the cosine value between the plane normal vector and the direction vector, and to adjust the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold if the cosine value is less than a preset cosine threshold; and a sixth determining submodule, used to construct an antisymmetric matrix based on the rotation vector, and to determine the rotation matrix based on the antisymmetric matrix, the identity matrix, the magnitude of the rotation vector, and the cosine value.

[0156] Optionally, the first determining module further includes: a seventh determining submodule, used to determine a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in a virtual coordinate system when the camera is shooting the target object, extracted from the target image; the second radius is the distance between the camera and the target object in a real coordinate system determined based on the camera imaging device; and an eighth determining submodule, used to determine the scaling ratio based on the first radius and the second radius.

[0157] Optionally, the object 3D reconstruction device includes: a first construction module, used to construct a camera pose optimizer, a preset density network structure, and a preset color network structure before inputting a target image set, target camera parameters, and target camera pose into a preset neural radiation field model and outputting target radiation field information, wherein the camera pose optimizer is used to optimize the input target camera pose; a second construction module, used to construct an initial neural radiation field model based on the camera pose optimizer, the preset density network structure, and the preset color network structure; a first processing module, used to acquire a historical image set and annotate each historical image in the historical image set to obtain annotation information, wherein the annotation information includes at least: the camera parameters and camera pose corresponding to the historical image; a first representation module, used to represent all historical images associated with the annotation information as a training data set; and a first training module, used to train the initial neural radiation field model using the training data set to obtain the preset neural radiation field model.

[0158] Optionally, the output unit includes: a first optimization module for optimizing the target camera pose to obtain an optimized target camera pose; a first generation module for generating ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points sampled from the rays formed by the pixels emitted by the camera to each target image; a first encoding module for encoding the position of the ray sampling points to obtain a position code, and encoding the ray direction of the ray corresponding to the ray sampling points to obtain a direction code; a third determination module for determining the volume density information and multi-dimensional feature vector of the ray sampling points based on the position code and using a preset density network structure; a fourth determination module for determining the color information of the ray sampling points based on the direction code and the multi-dimensional feature vector and using a preset color network structure; and a third construction module for constructing target radiation field information based on the volume density information and color information of each ray sampling point.

[0159] Optionally, the object 3D reconstruction device further includes: a fifth determining module, used to determine the initial image exposure rate before constructing the target radiation field information based on the volume density information and color information of each ray sampling point, and to construct a two-dimensional image based on the initial image exposure rate, volume density information and color information; and a second optimizing module, used to optimize the initial image exposure rate, volume density information and color information based on the two-dimensional image and the target image to obtain the target image exposure rate, optimized volume density information and optimized color information.

[0160] The aforementioned object 3D reconstruction device may also include a processor and a memory. The aforementioned determining unit 130, correction unit 131, output unit 132, construction unit 133, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0161] The processor described above contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the 3D model of the target object can be constructed by adjusting the kernel parameters.

[0162] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0163] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the object three-dimensional reconstruction method described above.

[0164] When a computer program product is executed on a data processing device, it is suitable to execute an initialization program with the following method steps: acquiring target object video using a camera imaging device, determining a target image set based on the target object video, determining target camera parameters and initial camera pose based on the target image set, correcting the initial camera pose to obtain the target camera pose, inputting the target image set, target camera parameters, and target camera pose into a preset neural radiation field model, outputting target radiation field information, and constructing a three-dimensional model of the target object based on the target radiation field information.

[0165] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the above-described object three-dimensional reconstruction method.

[0166] Figure 14 This is a hardware structure block diagram of an electronic device (or mobile device) for a three-dimensional object reconstruction method according to an embodiment of the present invention. Figure 14 As shown, an electronic device may include one or more processors (e.g., Figure 14The processors 1402a, 1402b, ..., 1402n, etc., may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), and a memory 1404 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 14 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 14 The more or fewer components shown, or having the same Figure 14 The different configurations shown.

[0167] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0168] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0169] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0174] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for three-dimensional reconstruction of an object, characterized in that, include: A target object video is acquired using a camera imaging device, and a target image set is determined based on the target object video. The target image set includes a preset number of target images, and each target image contains the target object. Based on the target image set, the target camera parameters and initial camera pose are determined, and the initial camera pose is corrected to obtain the target camera pose, wherein the initial camera pose is used to describe the position and orientation of the camera relative to the world coordinate system; The target image set, the target camera parameters, and the target camera pose are input into a preset neural radiation field model, and the target radiation field information is output. The preset neural radiation field model includes at least: a preset density network structure and a preset color network structure. The preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point. The target radiation field information includes at least: the volume density information and the color information. Based on the target radiation field information, a three-dimensional model of the target object is constructed; The initial camera pose is a camera pose determined on a virtual coordinate system. The step of correcting the initial camera pose to obtain the target camera pose includes: Determine the rotation matrix and scaling ratio, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; Based on the rotation matrix and the scaling ratio, the initial camera pose is corrected to determine the target camera pose.

2. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The steps of acquiring target object video using a camera imaging device and determining a target image set based on the target object video include: The target object is captured by the camera imaging device to obtain a video of the target object, wherein the target object video is a multi-view video, and the multi-view video includes: multiple single-view videos; Based on a preset time interval, each of the single-view videos is downsampled to obtain a first image set for each of the single-view videos; For each of the first image sets, blurry images are removed from the first image set to obtain the second image set; Each of the second image sets is sampled to obtain a third image set, and each third image in each of the third image sets is added to the target image set.

3. The method for three-dimensional reconstruction of an object according to claim 2, characterized in that, The step of removing blurry images from the first image set to obtain the second image set includes: For each first image in the first image set, the first image is converted into a grayscale image based on the pixel values ​​of each channel on the first image; Determine the Laplacian operator for each of the grayscale images, and determine the variance value of each of the grayscale images based on the Laplacian operator; Based on all the variance values, determine the average variance value, and based on the average variance value, determine the preset variance threshold. The grayscale image whose variance value is less than the preset variance threshold is identified as the blurred image, and the blurred image is removed from the first image set to obtain the second image set.

4. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The steps to determine the rotation matrix include: The camera position coordinates of the camera in the camera imaging device when shooting the target object are obtained, and all the camera position coordinates are fitted to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; Based on the direction axis of the real coordinate system, determine the direction vector, and based on the plane normal vector and the direction vector, determine the rotation vector; Determine the cosine value between the plane normal vector and the direction vector, and adjust the plane normal vector if the cosine value is less than a preset cosine threshold until the cosine value is greater than or equal to the preset cosine threshold. Based on the rotation vector, an antisymmetric matrix is ​​constructed, and the rotation matrix is ​​determined based on the antisymmetric matrix, the identity matrix, the magnitude of the rotation vector, and the cosine value.

5. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that, The steps to determine the scaling ratio include: A first radius and a second radius are determined, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera is capturing the target object, extracted from the target image; and the second radius is the distance between the camera and the target object in the real coordinate system, determined based on the camera imaging device. The scaling ratio is determined based on the first radius and the second radius.

6. The method of claim 1, wherein, Before inputting the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model and outputting the target radiation field information, the method further includes: Construct a camera pose optimizer, the preset density network structure, and the preset color network structure, wherein the camera pose optimizer is used to optimize the input target camera pose; Based on the camera pose optimizer, the preset density network structure, and the preset color network structure, an initial neural radiation field model is constructed. A set of historical images is collected, and each historical image in the set is labeled to obtain labeling information, wherein the labeling information includes at least: the camera parameters and camera pose corresponding to the historical image; All the historical images associated with the annotation information are represented as a training data set; The initial neural radiation field model is trained using the training dataset to obtain the preset neural radiation field model.

7. The method of claim 1, wherein, The steps of inputting the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model and outputting target radiation field information include: The target camera pose is optimized to obtain the optimized target camera pose. Based on the optimized target camera pose, ray sampling points are generated, wherein the ray sampling points are points obtained by sampling on the ray formed by the pixels emitted by the camera to each of the target images; The position of the ray sampling point is encoded to obtain a position code, and the ray direction of the ray corresponding to the ray sampling point is encoded to obtain a direction code; Based on the location encoding, the volume density information and multidimensional feature vector of the ray sampling point are determined using the preset density network structure. Based on the directional encoding and the multidimensional feature vector, the color information of the ray sampling point is determined using the preset color network structure. The target radiation field information is constructed based on the volume density information and color information of each of the ray sampling points.

8. The method of claim 7, wherein, Before constructing the target radiation field information based on the volume density information and color information of each of the ray sampling points, the method further includes: Determine the initial image exposure rate, and construct a two-dimensional image based on the initial image exposure rate, the volume density information, and the color information; Based on the two-dimensional image and the target image, the initial image exposure rate, the volume density information, and the color information are optimized to obtain the target image exposure rate, the optimized volume density information, and the optimized color information.

9. An apparatus for three-dimensional reconstruction of an object, characterized by include: A determining unit is configured to acquire a target object video using a camera imaging device, and determine a target image set based on the target object video, wherein the target image set includes: a preset number of target images, and each target image contains the target object; The correction unit is used to determine the target camera parameters and the initial camera pose based on the target image set, and to correct the initial camera pose to obtain the target camera pose, wherein the initial camera pose is used to describe the position and orientation of the camera relative to the world coordinate system; An output unit is used to input the target image set, the target camera parameters, and the target camera pose into a preset neural radiation field model, and output target radiation field information. The preset neural radiation field model includes at least: a preset density network structure and a preset color network structure. The preset density network structure is used to extract volume density information for each ray sampling point, and the preset color network structure is used to extract color information for each ray sampling point. The target radiation field information includes at least: the volume density information and the color information. A construction unit is used to construct a three-dimensional model of the target object based on the target radiation field information; The initial camera pose is a camera pose determined on a virtual coordinate system. The correction unit includes: a first determining module, used to determine a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; and a second determining module, used to correct the initial camera pose based on the rotation matrix and the scaling ratio to determine the target camera pose.

Images