Implicit field reconstruction method and system based on view angle matching and epipolar geometric constraint

Abstract

The invention discloses an implicit field reconstruction method, system and terminal based on view angle matching and epipolar geometric constraint, and the method comprises the steps: obtaining an initial image group, calculating a luminosity error between the initial image group and a multi-view-angle rendering value of a target object, and training a three-dimensional implicit field; extracting and combining sampling point features to optimize a three-dimensional implicit field, outputting a target feature body and inputting the target feature body into a geometric network, outputting a directed distance field value, and extracting a geometric result of a target object from the directed distance field value; and obtaining normal information according to the geometric result, inputting the normal information into the texture network, and outputting a rendering value of the camera ray. According to the method, the image set with the inaccurate pose is used as input, the three-dimensional implicit field is reconstructed, the optimized camera pose and the three-dimensional implicit field are subjected to joint optimization, light optimization and matching light consistency are achieved through camera light penetrating through key points in the input image, the pose optimization result can be effectively improved, and the accuracy of the pose optimization is improved. And the three-dimensional implicit field and the camera pose can be further optimized by using a point cloud feature enhancement mode.

Description

Technical Field

[0001] The present invention relates to the field of scene rendering technology, and in particular to an implicit field reconstruction method, system, terminal and computer-readable storage medium based on view matching and epipolar geometry constraints. Background Art

[0002] Camera rays are rays emitted from the camera position and pass through the corresponding image plane pixels to determine which parts of the scene should be displayed in the image. In the neural implicit field rendering method, the sampling points on the camera ray play a crucial role.

[0003] However, the reconstruction of the neural implicit field depends on an accurate pose input, and the algorithms designed for inaccurate poses find it difficult to effectively optimize both the camera pose and the neural implicit field expression at the same time. The resulting new perspective rendering results lack details, and the three-dimensional mesh extracted from the density field is usually overly smooth and cannot reflect the geometric details of the target object well.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an implicit field reconstruction method, system, terminal and computer-readable storage medium based on view matching and epipolar geometry constraints, aiming to solve the problem in the prior art that it is difficult to effectively optimize the camera pose and neural implicit field expression at the same time, resulting in blurred rendering results and inability to accurately express the geometric details of the object.

[0006] To achieve the above object, the present invention provides an implicit field reconstruction method based on view matching and epipolar geometry constraints, the implicit field reconstruction method based on view matching and epipolar geometry constraints comprising the following steps: Acquire an initial image group of the target object, and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; Extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature volume; Inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; The normal information of the target object at each sampling point is obtained according to the geometric result, all the normal information is input into a texture network, and the rendering value of the matching ray among the multiple camera rays is output.

[0007] Optionally, the implicit field reconstruction method based on view matching and epipolar geometry constraints, wherein the initial image group of the target object is obtained, and an initial three-dimensional implicit field is constructed, and the photometric error between the initial image group and the multi-view rendering value of the target object is input into the initial three-dimensional implicit field for training to obtain the three-dimensional implicit field, specifically includes: Acquire an initial image group of the target object captured by a target camera, wherein the initial image group is generated by photographing the target object from multiple viewing angles by the target camera; Construct an initial three-dimensional implicit field, and calculate the photometric error between the initial image group and the multi-view rendering value of the target object: ; ; ; ; in, Indicates sampling point The photometric error, represents the sampling point, represents the image index in the initial image group, Represents the pixels of the image, Represents the image and target object at pixel points The color error at Indicates The image is at pixel The RGB value at Indicates that the target object is at the pixel The The rendering value of the camera perspective corresponding to the image, Indicates The camera pose of each image, Indicates The rotation matrix of the target camera corresponding to the image, Indicates The displacement vector of the target camera corresponding to the image, represents the three-dimensional rotation group, Represents three-dimensional rational number space; The photometric error is input into the initial three-dimensional implicit field for training to obtain a three-dimensional implicit field.

[0008] Optionally, in the implicit field reconstruction method based on view matching and epipolar geometry constraints, the camera rays include: key rays and auxiliary rays; The extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting the target feature body, further includes: Acquire a plurality of the sampling points through which the plurality of key rays pass, and acquire color information and feature information of each of the sampling points; Extract multiple color information and multiple feature information with consistency, select corresponding sampling points, match the key rays corresponding to the selected multiple sampling points, and obtain matching rays: ; ; in, represents the matching ray, represents the camera center of the target camera, represents the number of rays, Represents matching rays The normalized direction.

[0009] Optionally, the implicit field reconstruction method based on view matching and epipolar geometry constraints, wherein the extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, and inputting the initial image group into the three-dimensional implicit field, and outputting the target feature volume, specifically includes: Extracting key features of key sampling points generated by the matching ray passing through the initial image group, and extracting auxiliary features of multiple auxiliary sampling points generated by multiple auxiliary rays passing through the initial image group: ; in, Indicates sampling point The characteristics of represents the sampling point, represents the progressive feature mask, represents the initial feature body; The key feature is enhanced by using all the auxiliary features to obtain a target key feature, and the target key feature is input into the three-dimensional implicit field to output a target feature body.

[0010] Optionally, the implicit field reconstruction method based on view matching and epipolar geometry constraints, wherein the step of enhancing the key feature by using the auxiliary feature to obtain the target key feature, and inputting the target key feature into the three-dimensional implicit field to output the target feature body, specifically includes: The key feature is combined with all the auxiliary features to obtain the target key feature: ; in, Indicates key sampling points The key characteristics of the target, Indicates key sampling points The number of auxiliary rays, represents a mathematical function that converts an arbitrary set of real numbers into real numbers representing a probability distribution. represents the key sampling point, represents the auxiliary sampling point, Indicates key sampling points The key features of Indicates auxiliary sampling point Auxiliary features at The target key feature is input into the three-dimensional implicit field, the camera pose in the three-dimensional implicit field is optimized according to the target key feature, and the target key feature is output.

[0011] Optionally, the implicit field reconstruction method based on view matching and epipolar geometry constraints, wherein the target feature body is input into a geometric network, a signed distance field value and an output feature are output, and a geometric result of the target object is obtained according to the signed distance field value and the output feature, specifically includes: Inputting the target key features of the target feature body into the constructed geometric network, and outputting the signed distance field value and the output features; According to the output feature, a geometric result of the target object in three-dimensional space is extracted from the signed distance field value: ; in, Indicates key sampling points The signed distance field value of Indicates key sampling points The output features of Represents a geometric network.

[0012] Optionally, the implicit field reconstruction method based on view matching and epipolar geometry constraints, wherein the normal information of the target object at each sampling point is obtained according to the geometric result, all the normal information is input into a texture network, and the rendering value of the matching ray among the multiple camera rays is output, specifically includes: Calculating the normal of the target object at the sampling point according to the gradient value of the geometric result, and obtaining the normal information at the sampling point according to the normal; The normal information is input into the constructed texture network, and the RGB value of the sampling point is output: ; in, Indicates sampling point RGB values, represents the texture network, Indicates sampling point The output features of Represents matching rays The normalized direction of Indicates sampling point Normal information of The color of the matching ray is rendered using the opacity density and RGB values ​​of all sampling points to obtain the rendering value of the matching ray: ; in, Represents matching rays The rendering value of Indicates sampling point The transmittance at Indicates sampling point opacity.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides an implicit field reconstruction system based on view matching and epipolar geometry constraints, wherein the implicit field reconstruction system based on view matching and epipolar geometry constraints comprises: An implicit field optimization module is used to obtain an initial image group of a target object and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; A feature volume optimization module, used for extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature volume; A geometric feature collection module, used for inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; A rendering module is used to obtain normal information of the target object at each sampling point according to the geometric result, input all the normal information into a texture network, and output rendering values ​​of matching rays among the multiple camera rays.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an implicit field reconstruction program based on perspective matching and epipolar geometry constraints stored in the memory and executable on the processor, wherein the implicit field reconstruction program based on perspective matching and epipolar geometry constraints implements the steps of the implicit field reconstruction method based on perspective matching and epipolar geometry constraints as described above when executed by the processor.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an implicit field reconstruction program based on perspective matching and epipolar geometry constraints, and when the implicit field reconstruction program based on perspective matching and epipolar geometry constraints is executed by a processor, the steps of the implicit field reconstruction method based on perspective matching and epipolar geometry constraints as described above are implemented.

[0016] In the present invention, an initial image group of a target object is obtained, and an initial three-dimensional implicit field is constructed. The photometric error between the initial image group and the multi-view rendering value of the target object is input into the initial three-dimensional implicit field for training to obtain a three-dimensional implicit field; sampling point features of multiple camera rays are extracted, target key features are generated according to the multiple sampling point features, the target key features are input into the three-dimensional implicit field, and the initial image group is input into the three-dimensional implicit field to output a target feature body; the target feature body is input into a geometric network, and a signed distance field value and an output feature are output. According to the signed distance field value and the output feature, a geometric result of the target object is obtained; according to the geometric result, the normal information of the target object at each sampling point is obtained, all the normal information is input into a texture network, and the rendering value of the matching rays among the multiple camera rays is output. The present invention uses a set of images with inaccurate poses as input to reconstruct a three-dimensional implicit field, and uses the optimized camera pose to jointly optimize the three-dimensional implicit field. By using camera rays passing through key points in the input image, light optimization and matching light consistency are achieved, which can effectively improve the results of pose optimization. The three-dimensional implicit field and camera pose can be further optimized by enhancing point cloud features. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the implicit field reconstruction method based on view matching and epipolar geometry constraints of the present invention; Figure 2 It is a flow chart of a preferred embodiment of the implicit field reconstruction method based on view matching and epipolar geometry constraints of the present invention; Figure 3 It is a schematic diagram of feature enhancement of a preferred embodiment of the implicit field reconstruction method based on view matching and epipolar geometry constraints of the present invention; Figure 4It is a structural diagram of a preferred embodiment of the implicit field reconstruction system based on view matching and epipolar geometry constraints of the present invention; Figure 5 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] The implicit field reconstruction method based on view matching and epipolar geometry constraints described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the implicit field reconstruction method based on view matching and epipolar geometry constraints includes the following steps: Step S10, obtaining an initial image group of the target object, and constructing an initial three-dimensional implicit field, inputting the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtaining a three-dimensional implicit field.

[0020] The camera rays include key rays and auxiliary rays. The initial image group is a group of images with uncalibrated poses, which are used as input to obtain the three-dimensional target object, and the purpose of scene reconstruction based on point cloud enhancement is to design a neural network that reconstructs the three-dimensional expression of the target object from a group of images captured from multiple perspectives, that is, to achieve optimized training of the three-dimensional implicit field.

[0021] Specifically, an initial image group of the target object captured by a target camera is obtained, wherein the initial image group is generated by the target camera photographing the target object from multiple perspectives; an initial three-dimensional implicit field is constructed, and a photometric error between the initial image group and a multi-perspective rendering value of the target object is calculated: ; ; ; ; in, Indicates sampling point The photometric error, represents the sampling point, represents the image index in the initial image group, Represents the pixels of the image, Represents the image and target object at pixel points The color error at Indicates The image is at pixel The RGB value at Indicates that the target object is at the pixel The The rendering value of the camera perspective corresponding to the image, Indicates The camera pose of each image, Indicates The rotation matrix of the target camera corresponding to the image, Indicates The displacement vector of the target camera corresponding to the image, represents the three-dimensional rotation group, Representing a three-dimensional rational number space; inputting the photometric error into the initial three-dimensional implicit field for training to obtain a three-dimensional implicit field.

[0022] Each image in the initial image set is associated with a camera pose that is assumed to be known or estimated using an algorithm. The initial 3D implicit field is trained by minimizing the photometric error between the input image and the multi-view rendering of the target object, thereby obtaining an optimized 3D implicit field. By optimizing the 3D implicit field, the target object is optimized (i.e., a 3D feature volume that can accurately represent the target object is constructed). By directly optimizing the target, the original opacity or density is changed to a signed distance field, which is itself a way of expressing 3D geometric shapes. The geometric details of the triangular mesh extracted from the reconstructed 3D implicit field are finer. By modeling the correspondence between the signed distance field and the density in NeRF (Neural Radiance Fields), the density can be optimized by optimizing the signed distance field, thereby enhancing the rendering result.

[0023] Furthermore, a plurality of sampling points through which the plurality of key rays pass are obtained, and color information and feature information of each sampling point are obtained; a plurality of color information and a plurality of feature information having consistency are extracted, and corresponding sampling points are selected, and the key rays corresponding to the selected plurality of sampling points are matched to obtain matching rays: ; ; in, represents the matching ray, represents the camera center of the target camera, represents the number of rays, Represents matching rays The normalized direction.

[0024] Among them, Figure 2As shown, after extracting key points from the input image and matching them using a pre-trained network, the initial 3D implicit field is trained to optimize the 3D feature volume, which encodes the geometric and photometric information about the target object and can be queried through camera rays to obtain the corresponding color information and feature information for new view synthesis and 3D reconstruction.

[0025] Here, each pixel is actually associated with a specific 3D ray from the object / scene through the center of the pixel and towards the center of the camera: ; The color and opacity of the points that can be sampled along the ray The integration is performed to generate the rendering color of the ray. Since the camera pose may be noisy, the reconstructed radiation field or implicit field may not produce a clear rendering with details. In the case of large noise, some methods may not even produce results, so Beyond a neural rendering network that implicitly maps to opacity density (or density) and color, we enhance sample point features by leveraging contextual feature relationships between sample points by matching rays in different images and using multi-view stereo geometry priors to learn implicit fields.

[0026] Step S20, extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting the target feature body.

[0027] Among them, in this embodiment, SuperPoint (Self-Supervised Interest Point Detection and Description, a feature point detection and description method based on deep learning) is used to detect the key points on each input image, and SuperGlue (Learning Feature Matching with Graph Neural Networks, a feature matching method based on graph neural networks) is used to perform point-to-point matching between image pairs to extract the key points. These steps can obtain a set of sparse ray matching between image pairs, which is used as the input of the three-dimensional implicit field, thereby obtaining the target feature body.

[0028] Among them, the optimization of the feature volume is affected by the photometric loss of rendering along the camera ray passing through the key point (i.e., the key ray). In the key ray feature enhancement module, this optimization can enhance the features of the key sampling point by combining features from auxiliary rays (i.e., rays passing through auxiliary sampling points near the key sampling point in the image).

[0029] Specifically, the key features of the key sampling points generated by the matching ray passing through the initial image group are extracted, and the auxiliary features of the multiple auxiliary sampling points generated by the multiple auxiliary rays passing through the initial image group are extracted: ; in, Indicates sampling point The characteristics of represents the sampling point, represents the progressive feature mask, Representing an initial feature body; utilizing all the auxiliary features to enhance the key feature to obtain a target key feature, and inputting the target key feature into the three-dimensional implicit field to output a target feature body.

[0030] The camera rays proposed in this embodiment include key rays and auxiliary rays. Both rays are emitted from the camera through the pixel center and are associated with the camera view. The feature body can obtain the corresponding features from the sampling points on the rays obtained from the camera center passing through the pixels of the input image. The key rays Through the key sampling points detected in the image, these key sampling points usually correspond to surface points with rich texture and geometric features; auxiliary rays It is the point around the key sampling point. When learning the features of the sampling points on the key ray in the optimization framework of this embodiment, context or local structure information is provided for point cloud enhancement; the sampling point information on the key ray and the auxiliary ray is used to enhance the sampling point features of the key ray, thereby better optimizing the three-dimensional representation of the target object.

[0031] Furthermore, multiple key rays need to be matched, that is, by matching the ray consistency enhancement module, constraints are imposed on the matched key rays in the form of color consistency, and the potential unmatched rays can be determined by comparing the feature similarity of the key ray integrals in the feature body. When two key rays in the input image are correctly matched and the corresponding camera posture is accurate, the two rays intersect at the object surface captured by the corresponding pixels in the three-dimensional space. Therefore, the color values ​​of the two correctly matched key rays in the corresponding three-dimensional implicit field should be consistent; in addition, when the three-dimensional implicit field optimization of the target object converges, the opacity of any sampling point on the key ray in the three-dimensional space is maximized when the sampling point is located on the object surface, and the color value of the object surface position accounts for the maximum contribution of the corresponding color value integral result, so the color integral values ​​of the two key rays are consistent and correspond to the color value of the object surface; similarly to the integral of the color value, when the ray matching relationship is correct and the camera posture is correct, the features between the matched key rays should be consistent, so the feature similarity between the two matched key rays is used to interpolate the color values ​​of the two key rays. In this case, the features between the two matching key rays and the color and opacity of the sampling points can be optimized simultaneously. This method of jointly optimizing the matching rays can adapt the matching accuracy by utilizing the similarity of the ray features.

[0032] in, It is a progressive feature mask used to filter fine-grained features in the early iterations of coarse-to-fine training. In order to consider the noise in the camera pose, the present invention parameterizes the camera's transformation matrix and jointly optimizes it with the implicit field parameters in the three-dimensional feature volume to obtain the target feature volume.

[0033] Furthermore, the key feature is fused with all the auxiliary features to obtain the target key feature: ; in, Indicates key sampling points The key characteristics of the target, Indicates key sampling points The number of auxiliary rays, represents a mathematical function that converts an arbitrary set of real numbers into real numbers representing a probability distribution. represents the key sampling point, represents the auxiliary sampling point, Indicates key sampling points The key features of Indicates auxiliary sampling point input the target key features into the three-dimensional implicit field, optimize the camera pose in the three-dimensional implicit field according to the target key features, and output the target key features.

[0034] Among them, due to the noise in the camera pose, the key sampling points on the input image, that is, the three-dimensional surface points collected by the corresponding pixel points in the image, may be inconsistent with the two matching key rays. In three-dimensional space, it is not possible to intersect accurately. In this case, auxiliary rays from around the key ray The local neighboring points sampled on The characteristics can greatly facilitate the More stable optimization of related features. Since the camera pose is unreliable, the points on the surface of the object may not completely intersect with the surface points obtained by the ray corresponding to the key points on the input image. Therefore, in this embodiment, The context information of the upsampled surface points is used to optimize the camera pose, so that the geometric information of the surface points and the corresponding key points can be better learned, and this context information can be used The adjacent rays sampled are obtained, that is, the key sampling points are fused The target key features and surrounding auxiliary sampling points Auxiliary features at the position generate key sampling points The key features of the target. Among them, the auxiliary ray Features remain unchanged, that is . In the key ray consistency module, the neural field and camera pose are jointly optimized by combining contextual information for point cloud enhancement and by using the matching ray enhancement module to enforce geometric and photometric consistency. The sampling points of the key ray are enhanced by using the features of the sampling points in the auxiliary ray, resulting in high-quality rendering and reconstruction with fine geometric details. The key ray enhancement module enriches the features of the key ray using the contextual information of the auxiliary ray and enhances the point cloud of the sampling points on the key ray to improve the robustness to noisy poses. The matching ray enhancement module learns cross-view information between matching rays to maintain the coherence of ray matching and eliminate ambiguity between input images.

[0035] Among them, Figure 3 As shown, in the key ray feature enhancement module, the features of the sampling points on the key ray are used to perform point cloud enhancement on the features of the sampling points on the key ray. In the case of inaccurate pose, the spatial context can be used to obtain the object surface feature information that correctly corresponds to the key ray. Since the input image is obtained through perspective projection, all rays in the same image should converge at a point in the same camera space.

[0036] Step S30: input the target feature body into a geometric network, output a signed distance field value and an output feature, and obtain a geometric result of the target object according to the signed distance field value and the output feature.

[0037] Once the initial 3D implicit field and the camera pose are jointly optimized, the expression of the 3D implicit field is realized, that is, the target feature volume can be used for new view synthesis or multi-view 3D reconstruction. The color prediction of the new view synthesis is performed by the texture network. The geometric information of the target object (i.e., the geometric result) is completed by the geometric network Come get.

[0038] Specifically, the target key features of the target feature body are input into the constructed geometric network, and the signed distance field value and the output features are output; according to the output features, the geometric results of the target object in the three-dimensional space are extracted from the signed distance field value: ; in, Indicates key sampling points The signed distance field value of Indicates key sampling points The output features of Represents a geometric network.

[0039] After the target feature body is input into the geometric network, the target key features at the sampling points along the matching ray passing through the target feature body are obtained. The geometric network generates a signed distance field value according to the target key features, and then obtains an opacity from the signed distance field value. To render three-dimensional objects and extract three-dimensional reconstruction information, that is, to obtain the geometric results of the target object in three-dimensional space through the geometric network; in this process, the key ray feature enhancement module is used to enhance the features along the key rays through the features of auxiliary ray sampling, and point cloud enhancement is performed, thereby improving the robustness of this optimization process.

[0040] Step S40: obtaining normal information of the target object at each sampling point according to the geometric result, inputting all the normal information into a texture network, and outputting rendering values ​​of matching rays among the multiple camera rays.

[0041] Specifically, according to the gradient value of the geometric result, the normal of the target object at the sampling point is calculated, and the normal information at the sampling point is obtained according to the normal; the normal information is input into the constructed texture network, and the RGB value of the sampling point is output: ; in, Indicates sampling point RGB values, represents the texture network, Indicates sampling point The output features of Represents matching rays The normalized direction of Indicates sampling point Normal information of the matching ray; using the opacity density and RGB values ​​of all sampling points, the color of the matching ray is rendered to obtain the rendering value of the matching ray: ; in, Represents matching rays The rendering value of Indicates sampling point The transmittance at Indicates sampling point opacity.

[0042] Among them, the function Indicates that along the corresponding ray The integrated transmittance, opaque density It is calculated using the signed distance field value, the normal vector, and the ray direction.

[0043] Among them, it is calculated by using the gradient of the signed distance field (that is, the gradient value of the geometric result) Normal information at can be used to enhance the geometric network Training, thereby achieving iterative training, in each iteration, sampling a pair of matching key rays and , along with the corresponding auxiliary rays, the geometry network, texture network and feature volume optimization are jointly trained end-to-end.

[0044] Among them, the texture network receives the output features and ray directions from the geometric network The normal at point is used as input to predict the point Color In addition, a matching ray consistency enhancement module is designed in this embodiment. By considering the color matching between rays and learning to maintain the consistency between ray matching features, the rendering quality of the optimization results and the final neural implicit field is improved by matching ray features. The matching ray consistency enhancement module can effectively reduce the impact of mismatched rays by eliminating confusion between camera ray matching.

[0045] The present invention uses a set of images with inaccurate poses as input to reconstruct a three-dimensional implicit field, and uses the optimized camera pose to jointly optimize the three-dimensional implicit field. By using camera rays passing through key points in the input image, light optimization and matching light consistency are achieved, which can effectively improve the results of pose optimization. The three-dimensional implicit field and camera pose can be further optimized by enhancing point cloud features.

[0046] Furthermore, if Figure 4 As shown, based on the above-mentioned implicit field reconstruction method based on view matching and epipolar geometry constraints, the present invention also provides an implicit field reconstruction system based on view matching and epipolar geometry constraints, wherein the implicit field reconstruction system based on view matching and epipolar geometry constraints includes: An implicit field optimization module 51 is used to obtain an initial image group of a target object and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; A feature volume optimization module 52 is used to extract sampling point features of multiple camera rays, generate target key features according to the multiple sampling point features, input the target key features into the three-dimensional implicit field, input the initial image group into the three-dimensional implicit field, and output the target feature volume; A geometric feature collection module 53 is used to input the target feature body into a geometric network, output a signed distance field value and an output feature, and obtain a geometric result of the target object according to the signed distance field value and the output feature; The rendering module 54 is used to obtain the normal information of the target object at each sampling point according to the geometric result, input all the normal information into the texture network, and output the rendering value of the matching ray among the multiple camera rays.

[0047] Furthermore, if Figure 5 As shown, based on the above-mentioned implicit field reconstruction method and system based on view matching and epipolar geometry constraints, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0048] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an implicit field reconstruction program 40 based on perspective matching and epipolar geometric constraints is stored on the memory 20, and the implicit field reconstruction program 40 based on perspective matching and epipolar geometric constraints can be executed by the processor 10, thereby realizing the implicit field reconstruction method based on perspective matching and epipolar geometric constraints in the present application.

[0049] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the implicit field reconstruction method based on view matching and epipolar geometry constraints.

[0050] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0051] In one embodiment, when the processor 10 executes the implicit field reconstruction program 40 based on view matching and epipolar geometry constraints in the memory 20, the following steps are implemented: Acquire an initial image group of the target object, and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; Extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature volume; Inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; The normal information of the target object at each sampling point is obtained according to the geometric result, all the normal information is input into a texture network, and the rendering value of the matching ray among the multiple camera rays is output.

[0052] The step of obtaining an initial image group of the target object and constructing an initial three-dimensional implicit field, inputting the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtaining the three-dimensional implicit field specifically includes: Acquire an initial image group of the target object captured by a target camera, wherein the initial image group is generated by photographing the target object from multiple viewing angles by the target camera; Construct an initial three-dimensional implicit field, and calculate the photometric error between the initial image group and the multi-view rendering value of the target object: ; ; ; ; in, Indicates sampling point The photometric error, represents the sampling point, represents the image index in the initial image group, Represents the pixels of the image, Represents the image and target object at pixel points The color error at Indicates The image is at pixel The RGB value at Indicates that the target object is at the pixel The The rendering value of the camera perspective corresponding to the image, Indicates The camera pose of each image, Indicates The rotation matrix of the target camera corresponding to the image, Indicates The displacement vector of the target camera corresponding to the image, represents the three-dimensional rotation group, Represents three-dimensional rational number space; The photometric error is input into the initial three-dimensional implicit field for training to obtain a three-dimensional implicit field.

[0053] Wherein, the camera rays include: key rays and auxiliary rays; The extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting the target feature body, further includes: Acquire a plurality of the sampling points through which the plurality of key rays pass, and acquire color information and feature information of each of the sampling points; Extract multiple color information and multiple feature information with consistency, select corresponding sampling points, match the key rays corresponding to the selected multiple sampling points, and obtain matching rays: ; ; in, represents the matching ray, represents the camera center of the target camera, represents the number of rays, Represents matching rays The normalized direction.

[0054] The step of extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting the target feature body specifically includes: Extracting key features of key sampling points generated by the matching ray passing through the initial image group, and extracting auxiliary features of multiple auxiliary sampling points generated by multiple auxiliary rays passing through the initial image group: ; in, Indicates sampling point The characteristics of represents the sampling point, represents the progressive feature mask, represents the initial feature body; The key feature is enhanced by using all the auxiliary features to obtain a target key feature, and the target key feature is input into the three-dimensional implicit field to output a target feature body.

[0055] The step of enhancing the key feature by using the auxiliary feature to obtain the target key feature, and inputting the target key feature into the three-dimensional implicit field to output the target feature body specifically includes: The key feature is combined with all the auxiliary features to obtain the target key feature: ; in, Indicates key sampling points The key characteristics of the target, Indicates key sampling points The number of auxiliary rays, represents a mathematical function that converts an arbitrary set of real numbers into real numbers representing a probability distribution. represents the key sampling point, represents the auxiliary sampling point, Indicates key sampling points The key features of Indicates auxiliary sampling point Auxiliary features at The target key feature is input into the three-dimensional implicit field, the camera pose in the three-dimensional implicit field is optimized according to the target key feature, and the target key feature is output.

[0056] The step of inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature specifically includes: Inputting the target key features of the target feature body into the constructed geometric network, and outputting the signed distance field value and the output features; According to the output feature, a geometric result of the target object in three-dimensional space is extracted from the signed distance field value: ; in, Indicates key sampling points The signed distance field value of Indicates key sampling points The output features of Represents a geometric network.

[0057] The step of obtaining normal information of the target object at each sampling point according to the geometric result, inputting all the normal information into a texture network, and outputting rendering values ​​of matching rays among the multiple camera rays specifically includes: Calculating the normal of the target object at the sampling point according to the gradient value of the geometric result, and obtaining the normal information at the sampling point according to the normal; The normal information is input into the constructed texture network, and the RGB value of the sampling point is output: ; in, Indicates sampling point RGB values, represents the texture network, Indicates sampling point The output features of Represents matching rays The normalized direction of Indicates sampling point Normal information of The color of the matching ray is rendered using the opacity density and RGB values ​​of all sampling points to obtain the rendering value of the matching ray: ; in, Represents matching rays The rendering value of Indicates sampling point The transmittance at Indicates sampling point opacity.

[0058] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an implicit field reconstruction program based on perspective matching and epipolar geometry constraints, and when the implicit field reconstruction program based on perspective matching and epipolar geometry constraints is executed by a processor, the steps of the implicit field reconstruction method based on perspective matching and epipolar geometry constraints as described above are implemented.

[0059] In summary, the present invention provides an implicit field reconstruction method based on view matching and epipolar geometry constraints and related equipment, the method comprising: obtaining an initial image group of a target object, and constructing an initial three-dimensional implicit field, inputting the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtaining a three-dimensional implicit field; extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, and inputting the initial image group into the three-dimensional implicit field, and outputting a target feature body; inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; obtaining normal information of the target object at each sampling point according to the geometric result, inputting all the normal information into a texture network, and outputting rendering values ​​of matching rays among the multiple camera rays. The present invention uses a set of images with inaccurate poses as input to reconstruct a three-dimensional implicit field, and uses the optimized camera pose to jointly optimize the three-dimensional implicit field. By using camera rays passing through key points in the input image, light optimization and matching light consistency are achieved, which can effectively improve the results of pose optimization. The three-dimensional implicit field and camera pose can be further optimized by enhancing point cloud features.

[0060] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0061] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0062] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An implicit field reconstruction method based on view matching and epipolar geometry constraints, characterized in that: The implicit field reconstruction method based on view matching and epipolar geometry constraints includes: Acquire an initial image group of the target object, and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; Extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature volume; Inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; The normal information of the target object at each sampling point is obtained according to the geometric result, all the normal information is input into a texture network, and the rendering value of the matching ray among the multiple camera rays is output.

2. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 1, characterized in that: The step of obtaining an initial image group of the target object and constructing an initial three-dimensional implicit field, inputting the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtaining the three-dimensional implicit field specifically includes: Acquire an initial image group of the target object captured by a target camera, wherein the initial image group is generated by photographing the target object from multiple viewing angles by the target camera; Construct an initial three-dimensional implicit field, and calculate the photometric error between the initial image group and the multi-view rendering value of the target object: ； ； ； ； in, Indicates sampling point The photometric error, represents the sampling point, represents the image index in the initial image group, Represents the pixels of the image, Represents the image and target object at pixel points The color error at Indicates The image is at pixel The RGB value at Indicates that the target object is at the pixel The The rendering value of the camera perspective corresponding to the image, Indicates The camera pose of each image, Indicates The rotation matrix of the target camera corresponding to the image, Indicates The displacement vector of the target camera corresponding to the image, represents the three-dimensional rotation group, Represents three-dimensional rational number space; The photometric error is input into the initial three-dimensional implicit field for training to obtain a three-dimensional implicit field.

3. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 1, characterized in that: The camera rays include: key rays and auxiliary rays; The extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting the target feature body, further includes: Acquire a plurality of the sampling points through which the plurality of key rays pass, and acquire color information and feature information of each of the sampling points; Extract multiple color information and multiple feature information with consistency, select corresponding sampling points, match the key rays corresponding to the selected multiple sampling points, and obtain matching rays: ； ； in, represents the matching ray, represents the camera center of the target camera, represents the number of rays, Represents matching rays The normalized direction.

4. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 3, characterized in that: The step of extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature body specifically includes: Extracting key features of key sampling points generated by the matching ray passing through the initial image group, and extracting auxiliary features of multiple auxiliary sampling points generated by multiple auxiliary rays passing through the initial image group: ； in, Indicates sampling point The characteristics of represents the sampling point, represents the progressive feature mask, represents the initial feature body; The key feature is enhanced by using all the auxiliary features to obtain a target key feature, and the target key feature is input into the three-dimensional implicit field to output a target feature body.

5. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 4, characterized in that: The method of enhancing the key feature by using the auxiliary feature to obtain the target key feature, and inputting the target key feature into the three-dimensional implicit field to output the target feature body specifically includes: The key feature is combined with all the auxiliary features to obtain the target key feature: ； in, Indicates key sampling points The key characteristics of the target, Indicates key sampling points The number of auxiliary rays, represents a mathematical function that converts an arbitrary set of real numbers into real numbers representing a probability distribution, represents the key sampling point, represents the auxiliary sampling point, Indicates key sampling points The key features of Indicates auxiliary sampling point Auxiliary features at The target key feature is input into the three-dimensional implicit field, the camera pose in the three-dimensional implicit field is optimized according to the target key feature, and the target key feature is output.

6. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 5, characterized in that: The step of inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature specifically includes: Inputting the target key features of the target feature body into the constructed geometric network, and outputting the signed distance field value and the output features; According to the output feature, a geometric result of the target object in three-dimensional space is extracted from the signed distance field value: ； in, Indicates key sampling points The signed distance field value of Indicates key sampling points The output features of Represents a geometric network.

7. The implicit field reconstruction method based on view matching and epipolar geometry constraints according to claim 2, characterized in that: The step of obtaining normal information of the target object at each sampling point according to the geometric result, inputting all the normal information into a texture network, and outputting rendering values ​​of matching rays among the multiple camera rays specifically includes: Calculating the normal of the target object at the sampling point according to the gradient value of the geometric result, and obtaining the normal information at the sampling point according to the normal; The normal information is input into the constructed texture network, and the RGB value of the sampling point is output: ； in, Indicates sampling point RGB values, represents the texture network, Indicates sampling point The output features of Represents matching rays The normalized direction of Indicates sampling point Normal information of The color of the matching ray is rendered using the opacity density and RGB values ​​of all sampling points to obtain the rendering value of the matching ray: ； in, Represents matching rays The rendering value of Indicates sampling point The transmittance at Indicates sampling point opacity.

8. An implicit field reconstruction system based on view matching and epipolar geometry constraints, characterized in that: The implicit field reconstruction system based on view matching and epipolar geometry constraints includes: An implicit field optimization module is used to obtain an initial image group of a target object and construct an initial three-dimensional implicit field, input the photometric error between the initial image group and the multi-view rendering value of the target object into the initial three-dimensional implicit field for training, and obtain a three-dimensional implicit field; A feature volume optimization module, used for extracting sampling point features of multiple camera rays, generating target key features according to the multiple sampling point features, inputting the target key features into the three-dimensional implicit field, inputting the initial image group into the three-dimensional implicit field, and outputting a target feature volume; A geometric feature collection module, used for inputting the target feature body into a geometric network, outputting a signed distance field value and an output feature, and obtaining a geometric result of the target object according to the signed distance field value and the output feature; A rendering module is used to obtain normal information of the target object at each sampling point according to the geometric result, input all the normal information into a texture network, and output rendering values ​​of matching rays among the multiple camera rays.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an implicit field reconstruction program based on perspective matching and epipolar geometry constraints stored in the memory and executable on the processor. When the implicit field reconstruction program based on perspective matching and epipolar geometry constraints is executed by the processor, the steps of the implicit field reconstruction method based on perspective matching and epipolar geometry constraints as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implicit field reconstruction program based on perspective matching and epipolar geometry constraints. When the implicit field reconstruction program based on perspective matching and epipolar geometry constraints is executed by a processor, the steps of the implicit field reconstruction method based on perspective matching and epipolar geometry constraints as described in any one of claims 1 to 7 are implemented.

Images