Object three-dimensional reconstruction method and device based on deep learning

Abstract

The invention discloses an object three-dimensional reconstruction method and device based on deep learning, and relates to the technical field of three-dimensional reconstruction, and the reconstruction method comprises the steps: collecting a target object video of a target object through a camera imaging device, determining a target image set based on the target object video, and obtaining a target image set based on the target image set; the method comprises the steps of obtaining a target image set, determining target camera parameters and an initial camera pose, correcting the initial camera pose to obtain a target camera pose, inputting the target image set, the target camera parameters and the target camera pose to a preset neural radiation field model, outputting target radiation field information, and constructing an object three-dimensional model of a target object based on the target radiation field information. According to the invention, the technical problem of low accuracy of three-dimensional reconstruction of an object in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a method and device for three-dimensional reconstruction of an object based on deep learning. Background Art

[0002] 3D reconstruction technology reconstructs the 3D model of an object by collecting geometric and texture information on the surface of the object. In the process of 3D reconstruction of an object, it is necessary to simultaneously obtain information such as the external shape and surface texture of the object to support subsequent visualization, analysis and application. This technology has broad application prospects in industrial manufacturing, cultural relics protection, virtual reality, medical imaging and other fields.

[0003] Although LiDAR (Light Detection and Ranging, a remote sensing technology) technology can obtain high-precision three-dimensional point cloud information of objects, it is expensive and takes a long time to collect data. In addition, although the three-dimensional phenotyping method based on depth cameras can be used to reconstruct three-dimensional models of objects at low cost and quickly, the data obtained by the depth camera has low resolution, and the 3D expression ability is severely degraded due to distortion and noise, making it difficult to accurately reconstruct objects with complex structures.

[0004] In the related art, reconstruction technology based on multi-view color images is generally used to reconstruct point cloud models with low-cost equipment, such as SFM (structure from motion), MVS (multi-viewstereo, generating dense point clouds), voxel carving algorithms, etc., which can extract the three-dimensional information of objects from color images taken from multiple perspectives, and then reconstruct point cloud models. However, in order to realize the automatic extraction of the three-dimensional model of the object and speed up the reconstruction rate, the above-mentioned algorithms need to use a pure color background and stable lighting conditions during data collection, and need to rely on a checkerboard to provide enough feature points for the algorithm to use. In addition, in order to further convert the model obtained by the algorithm into an accurate reconstruction in metric units, the assistance of a calibrator is also required, which makes it difficult to achieve fast and high-precision three-dimensional reconstruction of a single object under any background.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a method and device for three-dimensional reconstruction of an object based on deep learning, so as to at least solve the technical problem of low accuracy in three-dimensional reconstruction of an object in the related art.

[0007] According to one aspect of an embodiment of the present invention, a method for three-dimensional reconstruction of an object is provided, comprising: using a camera imaging device to capture a target object video of a target object, and based on the target object video, determining a target image set, wherein the target image set includes: a preset number of target images, each of the target images includes the target object; based on the target image set, determining target camera parameters and an initial camera pose, and correcting the initial camera pose to obtain a target camera pose; inputting the target image set, the target camera parameters and the target camera pose into a preset neural radiation field model, and outputting target radiation field information, wherein the preset neural radiation field model at least includes: a preset density network structure and a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information at least includes: the volume density information and the color information; based on the target radiation field information, constructing a three-dimensional object model of the target object.

[0008] Furthermore, the step of acquiring a target object video of a target object by an acquisition camera imaging device, and determining a target image set based on the target object video, includes: photographing the target object with the camera imaging device to obtain the target object video, wherein the target object video is a multi-perspective video, and the multi-perspective video includes: a plurality of single-perspective videos; based on a preset time interval, downsampling each of the single-perspective videos to obtain a first image set of each of the single-perspective videos; for each of the first image sets, eliminating blurred images in the first image set to obtain a second image set; sampling each of the second image sets to obtain a third image set, and adding each third image in each of the third image sets to the target image set.

[0009] Furthermore, the step of eliminating blurred images in the first image set to obtain the second image set includes: for each first image in the first image set, based on the pixel values ​​of each channel on the first image, converting the first image into a grayscale image; determining the Laplace operator of each grayscale image, and based on the Laplace operator, determining the variance value of each grayscale image; determining the average variance value based on all the variance values, and determining a preset variance threshold based on the average variance value; determining the grayscale image indicated by the variance value less than the preset variance threshold as the blurred image, and eliminating the blurred image from the first image set to obtain the second image set.

[0010] Furthermore, the initial camera pose is a camera pose determined on a virtual coordinate system, and the step of correcting the initial camera pose to obtain a target camera pose includes: determining a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the direction relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; based on the rotation matrix and the scaling ratio, the initial camera pose is corrected to determine the target camera pose.

[0011] Furthermore, the step of determining the rotation matrix includes: obtaining the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fitting all the camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; determining a direction vector based on the direction axis of the real coordinate system, and determining a rotation vector based on the plane normal vector and the direction vector; determining a cosine value between the plane normal vector and the direction vector, and when the cosine value is less than a preset cosine threshold, adjusting the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold; constructing an antisymmetric matrix based on the rotation vector, and determining the rotation matrix based on the antisymmetric matrix, the unit matrix, the modulus of the rotation vector and the cosine value.

[0012] Furthermore, the step of determining the scaling ratio includes: determining a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera is photographing the target object, which is extracted from the target image; the second radius is the distance between the camera and the target object in the real coordinate system determined based on the camera imaging device; and the scaling ratio is determined based on the first radius and the second radius.

[0013] Furthermore, before the target image set, the target camera parameters and the target camera pose are input into the preset neural radiation field model and the target radiation field information is output, it also includes: constructing a camera pose optimizer, the preset density network structure and the preset color network structure, wherein the camera pose optimizer is used to optimize the input target camera pose; constructing an initial neural radiation field model based on the camera pose optimizer, the preset density network structure and the preset color network structure; collecting a historical image set, and annotating each historical image in the historical image set to obtain annotation information, wherein the annotation information at least includes: camera parameters and camera pose corresponding to the historical image; representing all the historical images associated with the annotation information as a training data set; using the training data set to train the initial neural radiation field model to obtain the preset neural radiation field model.

[0014] Furthermore, the target image set, the target camera parameters and the target camera pose are input into a preset neural radiation field model, and the step of outputting target radiation field information includes: optimizing the target camera pose to obtain the optimized target camera pose; generating ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points obtained by sampling on rays composed of pixel points emitted by the camera to each of the target images; encoding the positions of the ray sampling points to obtain position codes, and encoding the ray directions of the rays corresponding to the ray sampling points to obtain direction codes; based on the position codes, using the preset density network structure to determine the volume density information and the multidimensional feature vector of the ray sampling points; based on the direction codes and the multidimensional feature vectors, using the preset color network structure to determine the color information of the ray sampling points; and constructing the target radiation field information based on the volume density information and the color information of each of the ray sampling points.

[0015] Furthermore, before constructing the target radiation field information based on the volume density information and the color information of each of the ray sampling points, it also includes: determining an initial image exposure rate, and constructing a two-dimensional image based on the initial image exposure rate, the volume density information and the color information; based on the two-dimensional image and the target image, optimizing the initial image exposure rate, the volume density information and the color information to obtain the target image exposure rate, the optimized volume density information and the optimized color information.

[0016] According to another aspect of an embodiment of the present invention, a three-dimensional reconstruction device for an object is also provided, including: a determination unit, used to use a camera imaging device to collect a target object video of a target object, and based on the target object video, determine a target image set, wherein the target image set includes: a preset number of target images, each of the target images contains the target object; a correction unit, used to determine target camera parameters and an initial camera pose based on the target image set, and correct the initial camera pose to obtain the target camera pose; an output unit, used to input the target image set, the target camera parameters and the target camera pose into a preset neural radiation field model, and output target radiation field information, wherein the preset neural radiation field model at least includes: a preset density network structure and a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information includes at least: the volume density information and the color information; a construction unit, used to construct a three-dimensional object model of the target object based on the target radiation field information.

[0017] Furthermore, the first determination unit includes: a first shooting module, used to use the camera imaging device to shoot the target object to obtain the target object video, wherein the target object video is a multi-perspective video, and the multi-perspective video includes: multiple single-perspective videos; a first sampling module, used to downsample each of the single-perspective videos based on a preset time interval to obtain a first image set of each of the single-perspective videos; a first elimination module, used to eliminate blurred images in the first image set for each of the first image sets to obtain a second image set; a second sampling module, used to sample each of the second image sets to obtain a third image set, and add each third image in each of the third image sets to the target image set.

[0018] Furthermore, the first elimination module includes: a first conversion submodule, used for converting each first image in the first image set into a grayscale image based on the pixel values ​​of each channel on the first image; a first determination submodule, used for determining the Laplace operator of each grayscale image, and determining the variance value of each grayscale image based on the Laplace operator; a second determination submodule, used for determining the average variance value based on all the variance values, and determining a preset variance threshold based on the average variance value; a third determination submodule, used for determining the grayscale image indicated by the variance value less than the preset variance threshold as the blurred image, and eliminating the blurred image from the first image set to obtain the second image set.

[0019] Furthermore, the correction unit includes: a first determination module, used to determine a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the direction relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; a second determination module, used to correct the initial camera pose based on the rotation matrix and the scaling ratio to determine the target camera pose.

[0020] Furthermore, the first determination module includes: a first acquisition submodule, used to acquire the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fit all the camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; a fourth determination submodule, used to determine the direction vector based on the direction axis of the real coordinate system, and determine the rotation vector based on the plane normal vector and the direction vector; a fifth determination submodule, used to determine the cosine value between the plane normal vector and the direction vector, and when the cosine value is less than a preset cosine threshold, adjust the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold; a sixth determination submodule, used to construct an antisymmetric matrix based on the rotation vector, and determine the rotation matrix based on the antisymmetric matrix, the unit matrix, the modulus of the rotation vector and the cosine value.

[0021] Furthermore, the first determination module also includes: a seventh determination submodule, used to determine a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera shoots the target object, which is extracted from the target image; the second radius is the distance between the camera and the target object in the real coordinate system determined based on the camera imaging device; and an eighth determination submodule, used to determine the scaling ratio based on the first radius and the second radius.

[0022] Furthermore, the object three-dimensional reconstruction device includes: a first construction module, used to construct a camera pose optimizer, the preset density network structure and the preset color network structure before inputting the target image set, the target camera parameters and the target camera pose into the preset neural radiation field model and outputting the target radiation field information, wherein the camera pose optimizer is used to optimize the input target camera pose; a second construction module, used to construct an initial neural radiation field model based on the camera pose optimizer, the preset density network structure and the preset color network structure; a first processing module, used to collect a historical image set, and annotate each historical image in the historical image set to obtain annotation information, wherein the annotation information at least includes: camera parameters and camera pose corresponding to the historical image; a first representation module, used to represent all the historical images associated with the annotation information as a training data set; a first training module, used to train the initial neural radiation field model using the training data set to obtain the preset neural radiation field model.

[0023] Furthermore, the output unit includes: a first optimization module, used to optimize the target camera pose to obtain the optimized target camera pose; a first generation module, used to generate ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points sampled on rays composed of pixel points emitted by the camera to each of the target images; a first encoding module, used to encode the positions of the ray sampling points to obtain position codes, and to encode the ray directions of the rays corresponding to the ray sampling points to obtain direction codes; a third determination module, used to determine the volume density information and the multidimensional feature vector of the ray sampling points based on the position codes and the preset density network structure; a fourth determination module, used to determine the color information of the ray sampling points based on the direction codes and the multidimensional feature vectors and the preset color network structure; a third construction module, used to construct the target radiation field information based on the volume density information and the color information of each of the ray sampling points.

[0024] Furthermore, the three-dimensional reconstruction device of the object also includes: a fifth determination module, used to determine the initial image exposure rate before constructing the target radiation field information based on the volume density information and the color information of each of the ray sampling points, and construct a two-dimensional image based on the initial image exposure rate, the volume density information and the color information; a second optimization module, used to optimize the initial image exposure rate, the volume density information and the color information based on the two-dimensional image and the target image to obtain the target image exposure rate, the optimized volume density information and the optimized color information.

[0025] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional reconstruction method of an object described in any one of the above is implemented.

[0026] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for three-dimensional reconstruction of an object.

[0027] In the present invention, a camera imaging device can be used to capture a target object video of a target object, and based on the target object video, a target image set is determined, based on the target image set, target camera parameters and an initial camera pose are determined, and the initial camera pose is corrected to obtain the target camera pose, the target image set, target camera parameters and target camera pose are input into a preset neural radiation field model, target radiation field information is output, and based on the target radiation field information, a three-dimensional model of the target object is constructed, thereby solving the technical problem of low accuracy in three-dimensional reconstruction of objects in related technologies.

[0028] In the present invention, a preset number of target images containing the target object can be obtained based on the target object video of the target object captured by the camera imaging device, and then based on the target image, the target camera parameters and the initial camera pose can be determined, and the initial camera pose can be corrected according to the direction and scale relationship between the coordinate systems to obtain the target camera pose, and then the preset number of target images, target camera parameters and target camera pose are input into the preset neural radiation field model, and volume density information and color information are output. According to the output volume density information and color information, a three-dimensional model of the target object can be constructed without segmenting the image background, thereby achieving high-throughput and high-precision three-dimensional object reconstruction, improving the reconstruction efficiency and quality of the target object, thereby achieving the technical effect of accurately and efficiently reconstructing the three-dimensional model of the object under any background. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 is a flow chart of an optional method for three-dimensional reconstruction of an object according to an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of an optional camera imaging system according to an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of an optional SFM process according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of an optional camera posture correction according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of an optional extraction of a region of interest according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram of an optional density network structure according to an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of an optional color network structure according to an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram of an optional optimization process of a neural radiation field model according to an embodiment of the present invention;

[0038] Fig. 9 is a schematic diagram of an optional ray sampling point selection according to an embodiment of the present invention;

[0039] Fig.10 is an optional hash coding schematic diagram according to an embodiment of the present invention;

[0040] Fig.11 is a schematic diagram of an overall framework of an optional three-dimensional reconstruction of an object according to an embodiment of the present invention;

[0041] Fig.12 is a schematic diagram of an optional object 3D reconstruction process according to an embodiment of the present invention;

[0042] Fig.13 is a schematic diagram of an optional device for three-dimensional reconstruction of an object according to an embodiment of the present invention;

[0043] Fig.14 It is a hardware structure block diagram of an electronic device (or mobile device) used for a method for three-dimensional reconstruction of an object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set between the system and the relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0047] In the present invention, since the objects are separated from the surrounding environment by setting a monochrome background in the related art, it is difficult for the current 3D reconstruction method to achieve fast and high-precision 3D reconstruction of a single object under any background. The 3D reconstruction method based on Nerf (Neural Radiance Fields, a computer vision technology) has limitations in representing geometric details. Therefore, the present invention proposes a 3D reconstruction method of an object based on deep learning, which adopts a shallow multi-layer perceptron to approximate the density field and the color field, improves the Nerf algorithm, introduces a new ray sampling strategy, and applies hash coding to map the input points to a high-dimensional space, and constructs an optimizer for the camera pose, so that Nerf has a stronger ability to restore spatial geometric details, enhances the geometric accuracy of Nerf, significantly improves the training speed of the network, and enhances the overall efficiency of the reconstruction process.

[0048] The present invention is described in detail below in conjunction with various embodiments.

[0049] Embodiment 1

[0050] According to an embodiment of the present invention, an embodiment of a method for three-dimensional reconstruction of an object is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] Figure 1 is a flowchart of an optional method for three-dimensional reconstruction of an object according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0052] Step S101 , using a camera imaging device to capture a target object video of a target object, and determining a target image set based on the target object video, wherein the target image set includes: a preset number of target images, each target image containing a target object.

[0053] Optionally, a target object video (i.e., a video containing the target object) of the target object can be captured based on a camera imaging device (such as an image capture device in a "camera to object" mode), and based on the target object video, the video can be framed and the framed images can be preprocessed to obtain a preset number (such as 90) of target images.

[0054] In this embodiment, the target object may be located in any background, and a camera imaging device may be used to directly shoot at the location of the target object to obtain a video of the target object.

[0055] Figure 2is a schematic diagram of an optional camera imaging system according to an embodiment of the present invention, such as Figure 2 As shown, the camera imaging system includes: a controller, a limit switch, a hexagonal base, a DC motor, a driven wheel, a metal ring, a camera bracket, a camera and a counterweight. Among them, the shooting trajectory of the camera is a space circle, the plane where the space circle is located is parallel to the ground, the short dashed circle represents the upper imaging trajectory (i.e., Upper imaging trajectory), the dotted circle represents the lower imaging trajectory (i.e., Lower imaging trajectory), the radius of the metal ring is 75cm, the camera bracket base is made of aluminum material with a slide structure, and the extension length of the camera bracket base can be freely adjusted, thereby adjusting the radius of the camera bracket from the center of the metal ring, which can be adjusted to a maximum of 30cm (i.e., the radius of the space circle is also variable, ranging from 75cm to 105cm). In the image acquisition process of each object, the object is placed at the center of the metal ring, and the DC motor can provide power to drive the metal ring with the camera bracket installed to rotate. The camera circles the object for one cycle and automatically stops after completing the video shooting. Each shooting process takes about 15s, and the camera can choose any RGB (Red Green Blue, i.e., an additive color model) imaging device.

[0056] In one example, two mobile phone terminals may be used as imaging devices to shoot dual-viewing video, and the resolution of the video may be 1440×2560.

[0057] In some optional embodiments, an imaging device may be used to capture videos with three or more perspectives.

[0058] Step S102, based on the target image set, determine the target camera parameters and the initial camera pose, and correct the initial camera pose to obtain the target camera pose.

[0059] Optionally, based on the target image set, the target camera parameters (i.e., camera intrinsic parameters) and the initial camera pose (i.e., camera extrinsic parameters) can be obtained. The target camera parameters describe the inherent properties of the camera itself, including parameters such as focal length, pixel spacing, and camera lens distortion. The camera pose describes the position and orientation of the camera relative to the world coordinate system.

[0060] In this embodiment, SFM can be used to estimate the target camera parameters and initial camera pose from an unordered image set. For example, COLMAP software (an open source 3D reconstruction software that combines SFM technology and MVS technology) uses an incremental method to perform SFM. First, a pair of images with the most geometrically consistent matches can be selected from all images. For the selected image pair, COLMAP uses a five-point algorithm or other methods to estimate the camera parameters and the initial camera pose, and then registers additional images based on shared key points (i.e., feature points) (i.e., estimates the camera pose of the remaining images). After estimating the camera parameters and the initial camera pose, COLMAP uses bundle adjustment to globally optimize the camera parameters and the initial camera pose, the 3D structure, and the image key points, so that more accurate target camera parameters and initial camera pose can be obtained.

[0061] Figure 3 is a schematic diagram of an optional SFM process according to an embodiment of the present invention, such as Figure 3 As shown in the figure, first, feature points are extracted from the input target image, and then feature matching is performed between multiple images to extract feature point pairs. The camera posture (i.e., camera parameters and camera pose) is estimated through the feature point pairs obtained by feature matching. The camera parameters and camera pose include: image size (such as 1440×2560), horizontal field of view (such as 1.13), vertical field of view (such as 1.66), camera lens focal length (such as 1169), camera type (such as PINHOLE), and then sparse reconstruction is performed based on the camera parameters, camera pose and matched feature point pairs to obtain a three-dimensional point cloud of the scene.

[0062] In this embodiment, since the obtained initial camera pose has a global coordinate system different from that of the real world, the target camera pose can be obtained by correcting the initial camera pose.

[0063] Figure 4 is a schematic diagram of an optional camera posture correction according to an embodiment of the present invention, such as Figure 4 As shown, the normal vector of the trajectory plane captured by the target camera is parallel to the coordinate axis (i.e., the z-axis) (e.g. Figure 4 The estimated initial camera position results in a trajectory plane normal vector that has a certain angle with the coordinate axis (as shown in the right side of Figure 4 As shown on the left side of ), it can be seen that the initial camera pose deviates from the target camera pose. By correcting the initial camera pose, the target camera pose can be obtained.

[0064] Step S103, input the target image set, target camera parameters and target camera posture into a preset neural radiation field model, and output the target radiation field information, wherein the preset neural radiation field model at least includes: a preset density network structure, a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information at least includes: volume density information and color information.

[0065] Optionally, the preset neural radiation field model includes at least a preset density network structure and a preset color network structure. The target image set, target camera parameters and target camera pose are input into the preset neural radiation field model (such as an improved Nerf model). According to the target camera pose, a ray emitted from the camera to the image pixel point is constructed, and sampling points are sampled on the ray. The sampling point is input into the preset density network structure, and the volume density information of each ray sampling point can be extracted. Then, the output vector of the preset density network structure and the observation direction of the ray sampling point are input into the preset color network structure, and the color information of each ray sampling point can be extracted.

[0066] In this embodiment, the preset neural radiation field model implicitly represents the three-dimensional scene as a 3D density field and a 5D light field (i.e., radiation field) approximated by a neural network. The radiation field describes the volume density of each sampling point in the scene and the color of each sampling point in each viewing direction, which can be expressed as a function Among them, F θ represents a multilayer perceptron network, θ represents the parameter combination of the network, x represents the position coordinates of the sampling point in the input network, d represents the observation direction of the ray sampling point in the input network, and c represents the color of the sampling point output by the network. Represents the volume density of sample points output by the network.

[0067] Step S104: constructing a three-dimensional model of the target object based on the target radiation field information.

[0068] Optionally, the preset neural radiation field model does not directly store explicit geometric data of the three-dimensional scene (such as voxels, point clouds or mesh models), but is an implicit continuous representation of the three-dimensional scene. A three-dimensional model of the target object can be constructed using an algorithm (such as the Marching Cubes algorithm) based on the volume density and color information predicted by the preset neural radiation field model.

[0069] In this embodiment, the Marching Cubes algorithm requires a predefined region of interest (the area captured by the camera around the object, i.e., the foreground area) and a density threshold (used to determine whether a voxel belongs to the surface of the region of interest). By adjusting the density threshold, surfaces of different density levels can be extracted to achieve automatic extraction of three-dimensional mesh models (i.e., automatic generation of a three-dimensional model of the target object).

[0070] Figure 5 is a schematic diagram of an optional extraction of a region of interest according to an embodiment of the present invention, such as Figure 5 As shown in the figure, it includes Area of ​​interest, Upper imaging orbit and Lower imaging orbit. The camera shooting orbit separates the foreground area from the background area. The foreground area is the area of ​​interest. The area of ​​interest can be accurately defined according to the camera shooting orbit to realize the automatic extraction of the three-dimensional mesh model.

[0071] In summary, a camera imaging device can be used to capture a target object video of the target object, and image sampling can be performed on the target object video to obtain a target image set. According to the target image set, the target camera parameters and the initial camera pose can be determined, and the initial camera pose can be corrected to obtain the target camera pose. Then, the target image set, the target camera parameters and the target camera pose are input into a preset neural radiation field model, and the target radiation field information is output. According to the target radiation field information, a three-dimensional model of the target object is constructed, which not only completes the three-dimensional reconstruction of the target object under any background, but also effectively improves the efficiency of the three-dimensional reconstruction of the target object, thereby solving the technical problem of low accuracy of three-dimensional reconstruction of objects in related technologies.

[0072] In order to accurately obtain a target image set, in the object three-dimensional reconstruction method provided in Example 1 of the present application, a camera imaging device is used to shoot the target object to obtain a target object video, wherein the target object video is a multi-perspective video, and the multi-perspective video includes: multiple single-perspective videos; based on a preset time interval, each single-perspective video is downsampled to obtain a first image set of each single-perspective video; for each first image set, blurred images in the first image set are eliminated to obtain a second image set; each second image set is sampled to obtain a third image set, and each third image in each third image set is added to the target image set.

[0073] In this embodiment, the relationship between the number of images and the reconstruction time is nonlinear. If the number of images exceeds a certain threshold, the reconstruction time will increase significantly, and insufficient number of images or poor image quality will affect the quality of the 3D reconstruction of the object. Therefore, it is necessary to determine the optimal balance between reconstruction quality and reconstruction time (i.e., the optimal number of extracted images, i.e., the preset number).

[0074] In this embodiment, high-quality 3D reconstruction can be achieved in a shorter time by excluding blurry frames and limiting the number of images to about 90. Therefore, an intelligent frame extraction algorithm is proposed to remove blurry frames from multi-view videos and extract a certain number of images from each view, totaling 90, for use in the subsequent reconstruction process.

[0075] In this embodiment, the multi-view (such as two single-view videos) captured by the camera imaging device can be down-sampled according to a preset time interval (such as 0.2s) to obtain a first image set of each single-view video (such as 50 images are sampled for each single-view video, for a total of 100 images). By eliminating blurred frames in each first image set, a second image set can be obtained, and each second image set can be randomly sampled to obtain a third image set, and each third image in each third image set can be added to the target image set to ensure that a preset number of target images (such as 90 images) can be obtained.

[0076] In this embodiment, if the number of blurred frames to be eliminated in the first image set is greater than a preset elimination threshold (eg, 10), it is determined that the quality of the acquired data is unqualified and the video needs to be re-acquired.

[0077] In order to accurately obtain the second image set, in the object three-dimensional reconstruction method provided in Example 1 of the present application, for each first image in the first image set, the first image is converted into a grayscale image based on the pixel values ​​of each channel on the first image; the Laplace operator of each grayscale image is determined, and the variance value of each grayscale image is determined based on the Laplace operator; the average variance value is determined based on all variance values, and the preset variance threshold is determined based on the average variance value; the grayscale image indicated by the variance value less than the preset variance threshold is determined as a blurred image, and the blurred image is eliminated from the first image set to obtain the second image set.

[0078] Optionally, a Laplace operator variance-based method can be used to quantitatively evaluate the clarity of the image. For each first image in the first image set, a weighted calculation can be performed based on the pixel values ​​of each channel (i.e., red (R), green (G), and blue (L)) in the first image to obtain the pixel value of the grayscale image (such as Y = 0.299R + 0.587G + 0.114B, where Y represents the pixel value of the grayscale image).

[0079] In this embodiment, the Laplace operator is a second-order differential operator and can be expressed as:

[0080]

[0081] in, represents the second-order partial derivative of the grayscale image in the x direction, Represents the second-order partial derivative of the grayscale image in the y direction. The discrete approximation of the Laplacian operator can be realized by the convolution kernel. The 8-neighborhood Laplacian operator (i.e. L8) can be expressed as a 3×3 convolution kernel:

[0082]

[0083] The convolution kernel can be multiplied with the pixel at the corresponding position in the grayscale image, and the products can be summed to generate a new value for each pixel in the grayscale image (i.e., the approximate value of the second-order derivative of the grayscale image at that point). Based on the pixel value, the variance of the grayscale image (i.e., Var(L)) can be calculated:

[0084]

[0085] Among them, i represents the i-th pixel in the image, N represents the total number of pixels in the image, and L i represents the pixel value of the i-th pixel, and μ represents the average value of all pixel values ​​in the image.

[0086] In this embodiment, variance is used as a measure of clarity. The larger the variance, the higher the image clarity, and the smaller the variance, the lower the image clarity. The average variance value of all grayscale images can be calculated based on the calculated variance value. If the average variance value of the grayscale image is less than a preset variance threshold (pre-set based on the average variance value, such as a value less than 20% of the average variance value), the grayscale image can be determined as a blurred image, and the blurred image can be removed from the first image set to obtain a second image set.

[0087] In order to accurately determine the target camera pose, in the object three-dimensional reconstruction method provided in Example 1 of the present application, a rotation matrix and a scaling ratio are determined, wherein the rotation matrix is ​​used to represent the direction relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; based on the rotation matrix and the scaling ratio, the initial camera pose is corrected to determine the target camera pose.

[0088] Optionally, the camera extrinsics describe the position and orientation of the camera relative to the world coordinate system, which can be represented by a rotation matrix R and a translation vector t. The translation vector t describes the position of the camera in the world coordinate system, and the rotation matrix R describes the viewing angle of the camera, which determines the angle from which the camera observes the scene.

[0089] In this embodiment, due to the differences in shooting positions, camera angles and camera configurations, the camera pose obtained based on the SFM technology has a global coordinate system (i.e., a virtual coordinate system) that is different from the real world, resulting in the three-dimensional model reconstructed by Nerf having a size and posture that is different from the real world.

[0090] In this embodiment, the transformation relationship between the virtual coordinate system and the real coordinate system (that is, the direction relationship between the directions of the two coordinate systems (using the rotation matrix R vtr The calibration of the camera extrinsic parameters is completed by using the scale relationship between the two coordinate systems (expressed by the scaling ratio k), which can be expressed as:

[0091] R=R vtr ·R virtual (4);

[0092] t=k·t virtual (5);

[0093] Among them, R virtual represents the virtual rotation matrix recovered by the SFM technique, t virtual represents the virtual translation vector restored by the SFM technique, R represents the corrected rotation matrix, and t represents the corrected translation vector.

[0094] In order to improve the accuracy of determining the rotation matrix, in the object three-dimensional reconstruction method provided in Example 1 of the present application, the camera position coordinates of the camera in the camera imaging device when shooting the target object are obtained, and all the camera position coordinates are fitted to obtain the plane normal vector, wherein the camera position coordinates are the coordinates in the virtual coordinate system; based on the direction axis of the real coordinate system, the direction vector is determined, and based on the plane normal vector and the direction vector, the rotation vector is determined; the cosine value between the plane normal vector and the direction vector is determined, and when the cosine value is less than a preset cosine threshold, the plane normal vector is adjusted until the cosine value is greater than or equal to the preset cosine threshold; based on the rotation vector, an antisymmetric matrix is ​​constructed, and based on the antisymmetric matrix, the unit matrix, the modulus length of the rotation vector and the cosine value, the rotation matrix is ​​determined.

[0095] Optionally, in order to determine the orientation relationship between the three-dimensional virtual coordinate system and the real coordinate system, it is necessary to find an appropriate reference to calculate the rotation matrix (ie, R vtr ), the normal vector of the trajectory plane can be used as a reference for the direction of the coordinate system.

[0096] In this embodiment, the photographic trajectory of the camera is parallel to the ground, and the normal vector of the trajectory plane is parallel to the z-axis (i.e., the direction axis) of the real coordinate system. The camera position coordinates of the camera in the camera imaging device when photographing the target object can be obtained, and all camera position coordinates are fitted to obtain the plane normal vector a (i.e., the reconstructed camera position is regarded as a discrete representation of the camera photographic trajectory, and the plane fitting is performed to obtain the trajectory plane normal vector). The direction vector (i.e., the direction vector of the z-axis) can be determined according to the direction axis of the real coordinate system. ).

[0097] In this embodiment, the plane normal vector a and the direction vector b can be normalized to obtain a unit vector. and direction vector Cross product calculation), determine the rotation vector (i.e. the rotation axis V):

[0098]

[0099] In this embodiment, the plane normal vector can be calculated and direction vector The dot product of and direction vector The cosine of the angle between:

[0100]

[0101] In this embodiment, the plane normal vector can be adjusted when the cosine value is less than a preset cosine threshold (such as -1+1e-10, that is, a negative value close to -1). Add a small random disturbance, that is, a very small positive number) until the cosine value is greater than or equal to the preset cosine threshold.

[0102] In this embodiment, the modulus of the rotation vector is calculated and saved in the variable s (ie, s=||v||), and an antisymmetric matrix (used to describe the cross product operation of the rotation vector) can be constructed through the components of the rotation vector:

[0103]

[0104] Among them, v1, v2, and v3 represent the components of the rotation vector on the coordinate axes x, y, and z, respectively.

[0105] The rotation matrix (ie, R) can be determined based on the antisymmetric matrix (ie, kmat), the unit matrix (ie, I), the modulus of the rotation vector (ie, s = ||v||), and the cosine value (ie, c). vtr ):

[0106]

[0107] Among them, 1e-10 means 10 to the negative 10th power, which is 0.0000000001.

[0108] In order to improve the accuracy of determining the first radius and the second radius, in the object three-dimensional reconstruction method provided in Example 1 of the present application, the first radius and the second radius are determined, wherein the first radius is the distance between the camera and the target object in a virtual coordinate system when the camera shoots the target object extracted from the target image; the second radius is the distance between the camera and the target object in a real coordinate system determined based on the camera imaging device; and the scaling ratio is determined based on the first radius and the second radius.

[0109] Optionally, in order to obtain the scale relationship between the three-dimensional virtual coordinate system and the real coordinate system, it is necessary to find a corresponding reference to calculate the scaling ratio (ie, k), and the radius of the trajectory can be used as a reference for the scale of the coordinate system.

[0110] In this embodiment, the first radius (i.e., the distance between the camera and the target object in the virtual coordinate system when the camera shoots the target object extracted from the target image) can be used. virtual )), and the second radius (i.e., the distance between the camera and the target object in the real coordinate system determined by the camera imaging device (i.e., long real )) to calculate the scaling ratio:

[0111]

[0112] In order to accurately obtain the preset neural radiation field model, in the object three-dimensional reconstruction method provided in Example 1 of the present application, a camera pose optimizer, a preset density network structure and a preset color network structure are constructed, wherein the camera pose optimizer is used to optimize the input target camera pose; based on the camera pose optimizer, the preset density network structure and the preset color network structure, an initial neural radiation field model is constructed; a historical image set is collected, and each historical image in the historical image set is annotated to obtain annotation information, wherein the annotation information at least includes: camera parameters and camera pose corresponding to the historical image; all historical images associated with the annotation information are represented as a training data set; the initial neural radiation field model is trained using the training data set to obtain the preset neural radiation field model.

[0113] Optionally, the accuracy of the camera pose is crucial to defining the relationship between the image capture perspective and the global world coordinate system (i.e., the real coordinate system), which can ensure the accurate conversion of real coordinates to camera coordinates and provide a basis for model training and model rendering. However, the camera pose restored and corrected based on the SFM technology deviates from the true value, which will cause errors in the ray casting path, and then negatively affect the model's estimation of the scene geometry and color distribution. Therefore, the camera pose needs to be accurately corrected.

[0114] In this embodiment, a camera pose optimizer can be constructed. The optimizer has two trainable parameters: the position offset and the rotation offset of the camera. These two parameters can be integrated into the forward propagation process of the network, and the position offset parameters and the rotation offset parameters of the camera can be iteratively updated according to the gradient information of the loss function to the external parameters through the back propagation algorithm. In order to suppress overfitting, an L2 regularization term can be added to the optimizer parameters based on the loss function.

[0115] In this embodiment, in order to improve the speed of the three-dimensional reconstruction process, a preset density network structure (such as a two-layer MLP network (Multilayer Perceptron, i.e., artificial neural network)) and a preset color network structure (such as a three-layer MLP network) can be constructed to predict density information and color information respectively, and an initial neural radiation field model can be constructed based on the preset density network structure and the preset color network structure.

[0116] Figure 6 is a schematic diagram of an optional density network structure according to an embodiment of the present invention, such as Figure 6 As shown in the figure, it includes an input vector (i.e., 32-dimensional γ(x)), two intermediate hidden layers, and an output vector. Among them, γ(x) represents position encoding, the vector dimension of the intermediate hidden layer is 64, the output vector is σ (i.e., volume density information) and a 15-dimensional geometric feature vector, the solid arrow represents the ReLU (Rectified Linear Unit, i.e., an activation function) activation layer, the dashed arrow represents the sigmoid (Logistic Sigmoid Function, i.e., a mathematical function) activation layer, and + represents vector concatenation.

[0117] Figure 7 is a schematic diagram of an optional color network structure according to an embodiment of the present invention, such as Figure 7As shown in the figure, it includes an input vector (i.e., 16-dimensional γ(d) and a 15-dimensional geometric feature vector), three intermediate hidden layers, and an output vector. Among them, γ(d) represents the ray direction encoding, the vector dimension of the intermediate hidden layer is 64, the output vector is RGB (i.e., color information), the solid arrow represents the ReLU activation layer, the dotted arrow represents the ELU (Exponential Linear Unit, i.e., an activation function) activation layer, and + represents vector concatenation.

[0118] In this embodiment, by collecting a historical image set and annotating each historical image in the historical image set, the annotation information such as camera parameters and camera pose corresponding to the historical image can be obtained, and all historical images with associated annotation information can be represented as a training data set. The initial neural radiation field model is trained with the training data set until the model converges to obtain a preset neural radiation field model (such as a Nerf model) after training.

[0119] Figure 8 is a schematic diagram of an optional optimization process of a neural radiation field model according to an embodiment of the present invention, such as Figure 8 As shown, first, the target camera parameters (i.e., camera intrinsic parameters) and the initial camera pose (i.e., camera extrinsic parameters) can be determined according to the input image, and the initial camera pose can be corrected to obtain the target camera pose, and then the target camera pose can be optimized according to the optimizer to obtain the optimized target camera pose, and according to the target camera pose, the rays emitted by the camera are sampled to obtain the position of each sampling point, and then the positions of the sampling points are hashed to obtain the position code, and the ray direction is encoded to obtain the viewing angle vector, and the position code can be input into an MLP network (i.e., density network), which is used to fit the 3D density field (i.e., the density field in space). Each sampling point is mapped to a value representing density), and the volume density information corresponding to each point (i.e., σ) can be obtained. The volume density information and the viewing angle vector are then concatenated together and input into another MLP network (i.e., the color network), which is used to fit the 5D light field (i.e., the propagation direction and intensity of light in three-dimensional space). The color information of each point in the viewing direction (i.e., RGB) can be obtained, and then the image is synthesized by setting the initial exposure rate and adjusting the exposure rate through volume rendering. The network is optimized by back-propagating the difference between the synthetic view and the real view and the color value as the loss function (i.e., L) to obtain the optimized neural radiation field model.

[0120] In order to accurately obtain the volume density information and color information of the ray sampling points, in the object three-dimensional reconstruction method provided in Example 1 of the present application, the target camera pose is optimized to obtain the optimized target camera pose; based on the optimized target camera pose, ray sampling points are generated, wherein the ray sampling points are points obtained by sampling on the rays composed of the pixel points emitted by the camera to each target image; the positions of the ray sampling points are encoded to obtain the position codes, and the ray directions of the rays corresponding to the ray sampling points are encoded to obtain the direction codes; based on the position codes, a preset density network structure is used to determine the volume density information and the multidimensional feature vectors of the ray sampling points; based on the direction codes and the multidimensional feature vectors, a preset color network structure is used to determine the color information of the ray sampling points; based on the volume density information and the color information of each ray sampling point, the target radiation field information is constructed.

[0121] Optionally, a camera pose optimizer is used to optimize the target camera pose to obtain an optimized target camera pose, and ray sampling points (i.e., points sampled on rays formed by pixels emitted by the camera to each target image) are generated according to the optimized target camera pose.

[0122] In this embodiment, the selection of ray sampling points can affect the time and quality of three-dimensional reconstruction of the object. According to the camera imaging position, the scene can be divided into a foreground area and a background area. The foreground area is the area where the target object is located (that is, the area where the camera shoots around the target object), and it is also a volume dense area in space. The quality of the neural radiation field in the foreground area (that is, the representation accuracy of the scene by the preset neural radiation field model) determines the reconstruction quality, and the quality of the neural radiation field in the background area has no effect on the reconstruction quality. However, due to the large number of blank areas between the foreground and the background, the sampling points falling in the blank areas not only have no effect on the optimization process, but also increase the calculation and memory overhead, and prolong the reconstruction time of the target object.

[0123] In this embodiment, a fixed camera track can be used to distinguish the foreground area from the background area, and more sampling points are generated in the foreground area through a new sampling strategy, so that the model pays more attention to the foreground area and skips the blank area during the optimization process, so as to improve the reconstruction quality and reduce the reconstruction time.

[0124] Fig. 9 is a schematic diagram of an optional ray sampling point selection according to an embodiment of the present invention, such as Fig. 9As shown in the figure, ray sampling points can be selected in the Empty (blank area), Foreground (foreground area), Background (background area), etc. Among them, the black dots on the ray represent high volume density sampling points (i.e. High volume density points), and the white dots on the ray represent low volume density sampling points (i.e. Low volume density points). By collecting more ray sampling points in the foreground area, the network can pay more attention to the foreground area.

[0125] In this embodiment, since the shallow MLP tends to learn low-frequency functions, resulting in poor reconstruction effect on high-frequency details of color and geometric shapes, and cannot effectively express the hierarchical structure of the object's organs, the multi-resolution hashing technology in instant-ngp (Instant Neural Graphics Primitives, a neural network-based graphics primitive) can be used for encoding to obtain position encoding, and the ray direction of the ray corresponding to the ray sampling point is encoded to obtain direction encoding (such as using spherical harmonics to encode the line of sight direction and map the line of sight direction into a vector of fixed dimension).

[0126] In this embodiment, multi-resolution hash coding combines the advantages of spatial hashing and multi-level coding, allowing the network to capture the coarse and fine features of the scene at different resolution levels. The hash table may include L resolution levels, each layer contains T feature vectors, and the dimension of each feature vector is F. For different resolution levels, the hash table is graded from coarse to fine, and the grid sizes of different levels increase at a fixed ratio. In order to avoid the expansion of the output vector dimension and the increase of the computational burden by simply increasing the number of hash table layers, thereby extending the reconstruction time, the hash table can be optimized. At the 8th and 9th levels, the resolution of the hash table may not increase in equal proportion (such as the resolution of the 9th level is four times that of the 8th level). At the last level of the hash table, the space can be divided into 2048 hash voxels instead of the conventional 512. Without changing the output dimension of the hash code, the encoding capability at the microscopic scale is enhanced, thereby improving the reconstruction capability of the object organ scale.

[0127] Exemplarily, the hash table parameters are shown in Table 1:

[0128] Table 1

[0129]

[0130] Fig.10 is an optional hash coding schematic diagram according to an embodiment of the present invention, such as Fig.10As shown, the space can be divided into a grid (i.e., Hash Grid), L=0 and L=1 represent two levels, respectively, representing different resolution levels. By finding the voxel of the ray sampling point at different resolution levels (i.e., a grid unit in the three-dimensional space), the hash function can be used to calculate the hash value of the voxel where the ray sampling point is located, and the hash value is used to find the corresponding feature vector (i.e., Lookup) in the hash table (i.e., Hash Table). The found feature vector is linearly interpolated (i.e., Linear Interpolation) to generate a feature vector corresponding to the input coordinates, and the feature vectors obtained from different resolution levels can be connected to form a final feature vector (e.g., the resolution is L layers, the data width is d, that is, a feature vector with d×L dimensions) to complete the position encoding of the ray sampling point.

[0131] In this embodiment, the position code of the ray sampling point can be input into the preset density network structure to predict the volume density information and multi-dimensional feature vector (such as a 15-dimensional geometric feature vector) of the ray sampling point, and the direction code and the multi-dimensional feature vector predicted by the preset density network structure can be spliced ​​and input into the preset color network structure to predict the color information of the ray sampling point.

[0132] In order to obtain the target image exposure rate, optimized volume density information and optimized color information, in the three-dimensional reconstruction method of an object provided in Example 1 of the present application, the initial image exposure rate is determined, and a two-dimensional image is constructed based on the initial image exposure rate, volume density information and color information; based on the two-dimensional image and the target image, the initial image exposure rate, volume density information and color information are optimized to obtain the target image exposure rate, optimized volume density information and optimized color information.

[0133] Alternatively, since the lighting conditions of the scene in a natural environment are uneven, images from different perspectives may have different exposure levels. The 3D reconstruction algorithm based on RGB images is extremely sensitive to lighting conditions. Therefore, the traditional 3D reconstruction method sets an artificial background to ensure the stability of the light source.

[0134] In this embodiment, in order to achieve 3D reconstruction in any background, the exposure rate can be integrated into the forward propagation process, and the exposure rate of each image can be included in the model training process as a learnable parameter (for example, an initial exposure rate can be set), and the back propagation algorithm can be used to adjust the exposure rate of each image in real time according to the gradient information calculated by the loss function. The learned exposure rate can be used to adjust the color value during the image synthesis process:

[0135]

[0136] RGBtarget =s i RGB (12);

[0137] Among them, e i Represents exposure rate, s i Represents the scaling factor of the pixel value, and RGB represents the image color.

[0138] In this embodiment, in order to prevent overfitting, L2 regularization (such as Δe i ·λ) to supervise the optimization process of exposure rate, where λ is the regularization strength parameter, which is used to control the influence of the regularization term, Δe i It represents the change of exposure rate. According to the exposure adjustment link, the exposure rate of the synthesized new perspective image can be controlled to achieve the re-lighting effect. It can also significantly improve the accuracy of 3D reconstruction under uneven lighting background and enhance the robustness of the algorithm to complex and variable lighting conditions.

[0139] Fig.11 FIG. 1 is a schematic diagram of an overall framework of an optional three-dimensional reconstruction of an object according to an embodiment of the present invention. Fig.11 As shown in the figure, firstly, the target object video is collected and key frames are extracted. Then, the SFM is used to restore the camera posture and the camera posture is globally calibrated. Then, the improved Nerf network is used to reconstruct the neural radiation field of the target object, and the Marching Cubes network is used to extract the object mesh model from the neural radiation field to complete the three-dimensional reconstruction of the target object.

[0140] Fig.12 is a schematic diagram of an optional object 3D reconstruction process according to an embodiment of the present invention, such as Fig.12 As shown in the figure, the captured video is first preprocessed to obtain a set of target images, and the initial camera pose (i.e., camera extrinsic parameters) and target camera parameters (i.e., camera intrinsic parameters) are estimated according to the SFM technology. Then, the initial camera pose is corrected to obtain the target camera pose, which is input into the Nerf model. According to the output of the model, a new perspective image can be synthesized and a mesh model can be extracted to realize three-dimensional reconstruction of the target object.

[0141] In an optional embodiment, the following may be used: Fig.12 The object three-dimensional reconstruction process shown executes the object three-dimensional reconstruction method.

[0142] In an embodiment of the present invention, a target object video of a target object is captured by using a camera imaging device, and the target object video is downsampled to obtain a first image set of each single-view video. For each first image set, each image in each first image set is converted into a grayscale image, and the variance value of each grayscale image is calculated according to the Laplace operator, and a second image set is obtained according to the variance value. Then, each second image set is sampled to obtain a target image set. According to the target image set, target camera parameters and an initial camera pose are determined, and the initial camera pose is corrected to obtain a target camera pose. Then, the target image set, the target camera parameters and the target camera pose are input into a preset neural radiation field model, and the volume density information and color information of each ray sampling point are output. According to the volume density information and the color information, a three-dimensional object model of the target object is constructed to complete the reconstruction of the three-dimensional model of the target object, thereby improving the training speed of the model, and realizing the reconstruction of the three-dimensional object in a relatively short time without background segmentation, thereby effectively improving the reconstruction efficiency and quality of the target object.

[0143] The following is a detailed description in conjunction with another embodiment.

[0144] Embodiment 2

[0145] The three-dimensional reconstruction device of an object provided in this embodiment includes multiple implementation units, and each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1.

[0146] Fig.13 is a schematic diagram of an optional device for three-dimensional reconstruction of an object according to an embodiment of the present invention, such as Fig.13 As shown, the object three-dimensional reconstruction device may include: a determination unit 130, a correction unit 131, an output unit 132, and a construction unit 133.

[0147] The determination unit 130 is used to capture a target object video of the target object using a camera imaging device, and determine a target image set based on the target object video, wherein the target image set includes: a preset number of target images, each target image including the target object;

[0148] The correction unit 131 is used to determine the target camera parameters and the initial camera pose based on the target image set, and correct the initial camera pose to obtain the target camera pose;

[0149] The output unit 132 is used to input the target image set, the target camera parameters and the target camera posture into a preset neural radiation field model, and output the target radiation field information, wherein the preset neural radiation field model at least includes: a preset density network structure and a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information at least includes: volume density information and color information;

[0150] The construction unit 133 is used to construct a three-dimensional object model of the target object based on the target radiation field information.

[0151] The above-mentioned object three-dimensional reconstruction device can use the camera imaging device to collect the target object video of the target object through the determination unit 130, and determine the target image set based on the target object video, and determine the target camera parameters and the initial camera pose based on the target image set through the correction unit 131, and correct the initial camera pose to obtain the target camera pose, and input the target image set, the target camera parameters and the target camera pose into the preset neural radiation field model through the output unit 132, and output the target radiation field information, and construct the object three-dimensional model of the target object based on the target radiation field information through the construction unit 133. In an embodiment of the present invention, by using the camera imaging device to collect the target A target object video of the object is obtained, and image preprocessing is performed based on the collected target object video to obtain a target image set, and then according to the target image set, target camera parameters and initial camera pose are determined, and the initial camera pose is corrected to obtain the target camera pose, and then the target image set, target camera parameters and target camera pose are input into a preset neural radiation field model, and volume density information and color information are output. According to the output volume density information and color information, a three-dimensional model of the target object is constructed to complete the rapid and high-quality reconstruction of the target object, thereby solving the technical problem of low efficiency and accuracy in performing three-dimensional reconstruction of objects under any background in the related technology.

[0152] Optionally, the first determination unit includes: a first shooting module, used to use a camera imaging device to shoot the target object to obtain a target object video, wherein the target object video is a multi-perspective video, and the multi-perspective video includes: multiple single-perspective videos; a first sampling module, used to downsample each single-perspective video based on a preset time interval to obtain a first image set of each single-perspective video; a first elimination module, used to eliminate blurred images in the first image set for each first image set to obtain a second image set; a second sampling module, used to sample each second image set to obtain a third image set, and add each third image in each third image set to the target image set.

[0153] Optionally, the first elimination module includes: a first conversion submodule, used to convert each first image in the first image set into a grayscale image based on the pixel values ​​of each channel on the first image; a first determination submodule, used to determine the Laplace operator of each grayscale image, and determine the variance value of each grayscale image based on the Laplace operator; a second determination submodule, used to determine the average variance value based on all variance values, and determine a preset variance threshold based on the average variance value; a third determination submodule, used to determine the grayscale image indicated by the variance value less than the preset variance threshold as a blurred image, and eliminate the blurred image from the first image set to obtain a second image set.

[0154] Optionally, the correction unit includes: a first determination module, used to determine a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the directional relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; a second determination module, used to correct the initial camera pose based on the rotation matrix and the scaling ratio to determine the target camera pose.

[0155] Optionally, the first determination module includes: a first acquisition submodule, used to acquire the camera position coordinates of the camera in the camera imaging device when shooting the target object, and fit all camera position coordinates to obtain the plane normal vector, wherein the camera position coordinates are the coordinates in the virtual coordinate system; a fourth determination submodule, used to determine the direction vector based on the direction axis of the real coordinate system, and determine the rotation vector based on the plane normal vector and the direction vector; a fifth determination submodule, used to determine the cosine value between the plane normal vector and the direction vector, and when the cosine value is less than a preset cosine threshold, adjust the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold; a sixth determination submodule, used to construct an antisymmetric matrix based on the rotation vector, and determine the rotation matrix based on the antisymmetric matrix, the unit matrix, the modulus of the rotation vector and the cosine value.

[0156] Optionally, the first determination module also includes: a seventh determination submodule, used to determine a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in a virtual coordinate system when the camera is shooting the target object, which is extracted from the target image; the second radius is the distance between the camera and the target object in a real coordinate system determined based on the camera imaging device; and an eighth determination submodule, used to determine the scaling ratio based on the first radius and the second radius.

[0157] Optionally, the object three-dimensional reconstruction device includes: a first construction module, which is used to construct a camera pose optimizer, a preset density network structure and a preset color network structure before inputting the target image set, target camera parameters and target camera pose into the preset neural radiation field model and outputting the target radiation field information, wherein the camera pose optimizer is used to optimize the input target camera pose; a second construction module, which is used to construct an initial neural radiation field model based on the camera pose optimizer, the preset density network structure and the preset color network structure; a first processing module, which is used to collect a historical image set and annotate each historical image in the historical image set to obtain annotation information, wherein the annotation information at least includes: camera parameters and camera pose corresponding to the historical image; a first characterization module, which is used to characterize all historical images with associated annotation information as a training data set; a first training module, which is used to train the initial neural radiation field model using the training data set to obtain a preset neural radiation field model.

[0158] Optionally, the output unit includes: a first optimization module, used to optimize the target camera pose to obtain an optimized target camera pose; a first generation module, used to generate ray sampling points based on the optimized target camera pose, wherein the ray sampling points are points obtained by sampling on rays composed of pixel points emitted by the camera to each target image; a first encoding module, used to encode the positions of the ray sampling points to obtain position codes, and to encode the ray directions of the rays corresponding to the ray sampling points to obtain direction codes; a third determination module, used to determine the volume density information and multi-dimensional feature vectors of the ray sampling points based on the position codes and using a preset density network structure; a fourth determination module, used to determine the color information of the ray sampling points based on the direction codes and the multi-dimensional feature vectors and using a preset color network structure; a third construction module, used to construct target radiation field information based on the volume density information and color information of each ray sampling point.

[0159] Optionally, the three-dimensional reconstruction device of the object also includes: a fifth determination module, which is used to determine the initial image exposure rate before constructing the target radiation field information based on the volume density information and color information of each ray sampling point, and construct a two-dimensional image based on the initial image exposure rate, volume density information and color information; a second optimization module, which is used to optimize the initial image exposure rate, volume density information and color information based on the two-dimensional image and the target image, and obtain the target image exposure rate, optimized volume density information and optimized color information.

[0160] The above-mentioned object three-dimensional reconstruction device may also include a processor and a memory. The above-mentioned determination unit 130, correction unit 131, output unit 132, construction unit 133, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0161] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels may be provided, and the three-dimensional model of the target object is constructed by adjusting kernel parameters.

[0162] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0163] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above-mentioned methods for three-dimensional reconstruction of an object is implemented.

[0164] When the computer program product is executed on a data processing device, it is suitable for executing an initialization program having the following method steps: using a camera imaging device to collect a target object video of a target object, and based on the target object video, determining a target image set, based on the target image set, determining target camera parameters and an initial camera pose, and correcting the initial camera pose to obtain a target camera pose, inputting the target image set, target camera parameters and target camera pose into a preset neural radiation field model, outputting target radiation field information, and constructing a three-dimensional object model of the target object based on the target radiation field information.

[0165] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned method for three-dimensional reconstruction of an object.

[0166] Fig.14 1 is a hardware structure block diagram of an electronic device (or mobile device) for a method for three-dimensional reconstruction of an object according to an embodiment of the present invention. Fig.14 As shown, the electronic device may include one or more processors (e.g., Fig.14The processor 1402a, processor 1402b, ..., processor 1402n, etc., which may include but are not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1404 for storing data. In addition, it may also include: a display, an input / output interface (I / 0 interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / 0 interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.14 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.14 More or fewer components as shown, or with Fig.14 Different configurations shown.

[0167] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0168] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0169] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0170] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0172] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0173] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0174] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for three-dimensional reconstruction of an object, characterized in that: include: A target object video of a target object is captured by a camera imaging device, and a target image set is determined based on the target object video, wherein the target image set includes: a preset number of target images, each of which contains the target object; Based on the target image set, determine target camera parameters and an initial camera pose, and correct the initial camera pose to obtain a target camera pose; The target image set, the target camera parameters and the target camera posture are input into a preset neural radiation field model, and target radiation field information is output, wherein the preset neural radiation field model at least includes: a preset density network structure and a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information at least includes: the volume density information and the color information; Based on the target radiation field information, a three-dimensional object model of the target object is constructed.

2. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that: The step of using a camera imaging device to capture a target object video of the target object, and determining a target image set based on the target object video, comprises: The target object is photographed by the camera imaging device to obtain a video of the target object, wherein the video of the target object is a multi-view video, and the multi-view video includes: a plurality of single-view videos; Based on a preset time interval, downsampling each of the single-view videos to obtain a first image set of each of the single-view videos; For each of the first image sets, remove blurred images in the first image set to obtain a second image set; Each of the second image sets is sampled to obtain a third image set, and each third image in each of the third image sets is added to the target image set.

3. The method for three-dimensional reconstruction of an object according to claim 2, characterized in that: The step of removing blurred images from the first image set to obtain a second image set includes: For each first image in the first image set, convert the first image into a grayscale image based on pixel values ​​of each channel of the first image; Determine a Laplacian operator for each of the grayscale images, and determine a variance value of each of the grayscale images based on the Laplacian operator; Based on all the variance values, determining an average variance value, and based on the average variance value, determining a preset variance threshold; The grayscale image indicated by the variance value less than the preset variance threshold is determined as the blurred image, and the blurred image is removed from the first image set to obtain the second image set.

4. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that: The initial camera pose is a camera pose determined on a virtual coordinate system. The step of correcting the initial camera pose to obtain a target camera pose includes: Determine a rotation matrix and a scaling ratio, wherein the rotation matrix is ​​used to represent the direction relationship between the virtual coordinate system and the real coordinate system, and the scaling ratio is used to represent the scale relationship between the virtual coordinate system and the real coordinate system; Based on the rotation matrix and the scaling ratio, the initial camera pose is corrected to determine the target camera pose.

5. The method for three-dimensional reconstruction of an object according to claim 4, characterized in that: The steps to determine the rotation matrix include: Obtaining the camera position coordinates of the camera in the camera imaging device when photographing the target object, and fitting all the camera position coordinates to obtain a plane normal vector, wherein the camera position coordinates are coordinates in the virtual coordinate system; Determine a direction vector based on the direction axis of the real coordinate system, and determine a rotation vector based on the plane normal vector and the direction vector; Determine a cosine value between the plane normal vector and the direction vector, and if the cosine value is less than a preset cosine threshold, adjust the plane normal vector until the cosine value is greater than or equal to the preset cosine threshold; An antisymmetric matrix is ​​constructed based on the rotation vector, and the rotation matrix is ​​determined based on the antisymmetric matrix, a unit matrix, a modulus of the rotation vector, and the cosine value.

6. The method for three-dimensional reconstruction of an object according to claim 4, characterized in that: The steps to determine the scaling ratio include: Determine a first radius and a second radius, wherein the first radius is the distance between the camera and the target object in the virtual coordinate system when the camera photographs the target object, extracted from the target image; and the second radius is the distance between the camera and the target object in the real coordinate system, determined based on the camera imaging device; The scaling ratio is determined based on the first radius and the second radius.

7. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that: Before inputting the target image set, the target camera parameters and the target camera position into a preset neural radiation field model and outputting target radiation field information, the method further includes: Constructing a camera pose optimizer, the preset density network structure and the preset color network structure, wherein the camera pose optimizer is used to optimize the input target camera pose; Based on the camera pose optimizer, the preset density network structure and the preset color network structure, construct an initial neural radiation field model; Collecting a historical image set, and annotating each historical image in the historical image set to obtain annotation information, wherein the annotation information at least includes: camera parameters and camera pose corresponding to the historical image; Characterizing all of the historical images associated with the annotation information as a training data set; The initial neural radiation field model is trained using the training data set to obtain the preset neural radiation field model.

8. The method for three-dimensional reconstruction of an object according to claim 1, characterized in that: The step of inputting the target image set, the target camera parameters and the target camera posture into a preset neural radiation field model and outputting target radiation field information comprises: Optimizing the target camera posture to obtain an optimized target camera posture; Based on the optimized target camera pose, generating ray sampling points, wherein the ray sampling points are points obtained by sampling on rays formed by pixels emitted by the camera to each of the target images; Encoding the position of the ray sampling point to obtain a position code, and encoding the ray direction of the ray corresponding to the ray sampling point to obtain a direction code; Based on the position coding, the volume density information and the multi-dimensional feature vector of the ray sampling point are determined by using the preset density network structure; Based on the direction code and the multi-dimensional feature vector, the preset color network structure is adopted to determine the color information of the ray sampling point; The target radiation field information is constructed based on the volume density information and the color information of each ray sampling point.

9. The method for three-dimensional reconstruction of an object according to claim 8, characterized in that: Before constructing the target radiation field information based on the volume density information and the color information of each ray sampling point, the method further includes: Determining an initial image exposure rate, and constructing a two-dimensional image based on the initial image exposure rate, the volume density information, and the color information; Based on the two-dimensional image and the target image, the initial image exposure rate, the volume density information and the color information are optimized to obtain the target image exposure rate, the optimized volume density information and the optimized color information.

10. A device for three-dimensional reconstruction of an object, characterized in that: include: a determination unit, configured to capture a target object video of the target object using a camera imaging device, and determine a target image set based on the target object video, wherein the target image set includes: a preset number of target images, each of which contains the target object; A correction unit, used for determining target camera parameters and an initial camera pose based on the target image set, and correcting the initial camera pose to obtain a target camera pose; An output unit, used for inputting the target image set, the target camera parameters and the target camera posture into a preset neural radiation field model, and outputting target radiation field information, wherein the preset neural radiation field model at least includes: a preset density network structure and a preset color network structure, the preset density network structure is used to extract the volume density information of each ray sampling point, and the preset color network structure is used to extract the color information of each ray sampling point; the target radiation field information at least includes: the volume density information and the color information; A construction unit is used to construct a three-dimensional object model of the target object based on the target radiation field information.

Images