Surface reconstruction method, device and equipment based on unsupervised end-to-end rarefaction and storage medium

Abstract

The invention discloses a surface reconstruction method and device based on unsupervised end-to-end rarefaction, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a target image obtained by a camera through shooting from a plurality of visual angles; determining a single-channel matching body and a multi-channel feature body corresponding to voxels of different scales in each target image; based on the single-channel matching body and the multi-channel feature body, determining surface areas corresponding to voxels of different scales and first color information; and based on the surface areas corresponding to different scales and the first color information, carrying out sparsification on voxels of the target images so as to realize unsupervised end-to-end object surface reconstruction. According to the method, by constructing the unsupervised end-to-end and object surface area sparse surface reconstruction method, memory occupation and calculated amount overhead are reduced, unsupervised end-to-end surface reconstruction is realized, and better calculation efficiency and reconstruction performance are achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a surface reconstruction method, device, equipment and storage medium based on unsupervised end-to-end sparsification. Background Art

[0002] Reconstructing the surface of an object from multi-view images is a key technology with important practical value, but existing technologies have many defects in performance and efficiency. For example, implicit neural surface reconstruction methods combine implicit representation with differentiable rendering techniques (such as volume rendering or surface rendering) to achieve end-to-end reconstruction of surfaces directly from multi-view images. However, this type of method requires retraining a model for each scene separately, which is time-consuming and difficult to apply directly. At the same time, this type of method relies on dense perspective input and cannot reconstruct accurate surfaces in sparse perspective conditions, which further limits its practicality. Summary of the invention

[0003] The main purpose of this application is to provide a surface reconstruction method, device, equipment and storage medium based on unsupervised end-to-end sparsification, aiming to reduce memory usage and computational overhead, and realize unsupervised end-to-end surface reconstruction, thereby having better computational efficiency and reconstruction performance.

[0004] To achieve the above objectives, the present application proposes a surface reconstruction method based on unsupervised end-to-end sparsification, the method comprising:

[0005] Obtain target images captured by the camera from several viewing angles;

[0006] Determine single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images;

[0007] Determining surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume;

[0008] Based on the surface areas corresponding to different scales and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

[0009] In one embodiment, determining the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images includes:

[0010] Extracting multi-scale features in each of the target images through an image feature extraction network;

[0011] Get the pixel positions corresponding to voxels of different scales;

[0012] Based on the pixel position and the multi-scale feature, determining fusion weights and multi-view features corresponding to voxels of different scales;

[0013] Based on the fusion weights and the multi-view features, single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images are determined.

[0014] In one embodiment, determining the fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features includes:

[0015] Based on the pixel position, upsampling the multi-scale feature;

[0016] Calculating the upsampled multi-scale features to obtain multi-view features;

[0017] The multi-view features are input into the integration network to obtain fusion weights corresponding to voxels of different scales.

[0018] In one embodiment, determining the surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume includes:

[0019] Get target sampling points on several rays;

[0020] Determining surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume;

[0021] Obtaining second color information obtained by projecting a reference viewing angle and viewing angle weights corresponding to each viewing angle;

[0022] Based on the second color information and each of the viewing angle weights, first color information corresponding to voxels of different scales is calculated.

[0023] In one embodiment, determining the surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume includes:

[0024] interpolating each of the target sampling points with the single-channel matching body to obtain an interpolation result;

[0025] Based on the interpolation result, determining the weight distribution of the target ray corresponding to each of the target sampling points;

[0026] Based on the weight distribution, a surface area of ​​the target ray is determined.

[0027] In one embodiment, the thinning of voxels of each of the target images based on the surface areas corresponding to different scales and the first color information to achieve unsupervised end-to-end object surface reconstruction includes:

[0028] Based on the surface positions in the surface area corresponding to the voxels of different scales, surface maps corresponding to different scales are constructed;

[0029] Projecting each of the voxels onto each of the surface images, and calculating the voxel distance between each of the voxels and the surface area;

[0030] Based on the voxel distance and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

[0031] In one embodiment, the thinning of the voxels of each of the target images based on the voxel distance and the first color information to achieve unsupervised end-to-end object surface reconstruction includes:

[0032] Based on the voxel distance, determining the amount of surface area within which each voxel falls;

[0033] If the number of surface areas corresponding to the voxel meets the preset area number range, the voxel is used as a target voxel, and the first color information corresponding to the target voxel is determined;

[0034] Based on the target voxel and the first color information, a multi-scale feature volume is obtained, and volume rendering is performed on the multi-scale feature volume to achieve unsupervised end-to-end object surface reconstruction.

[0035] In addition, to achieve the above objectives, the present application also proposes a surface reconstruction device based on unsupervised end-to-end sparseness, and the surface reconstruction device based on unsupervised end-to-end sparseness includes:

[0036] An acquisition module is used to acquire target images captured by a camera from several viewing angles;

[0037] A first determination module is used to determine the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images;

[0038] A second determination module is used to determine the surface areas corresponding to voxels of different scales and the first color information based on the single-channel matching volume and the multi-channel feature volume;

[0039] The reconstruction module is used to thin out the voxels of each of the target images based on the surface areas corresponding to different scales and the first color information, so as to achieve unsupervised end-to-end object surface reconstruction.

[0040] In addition, to achieve the above-mentioned objectives, the present application also proposes a surface reconstruction device based on unsupervised end-to-end sparsification, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0043] The present application provides a surface reconstruction method, apparatus, device and storage medium based on unsupervised end-to-end sparseness. The surface reconstruction method based on unsupervised end-to-end sparseness obtains a target image captured by a camera from several perspectives, and then determines a single-channel matching body and a multi-channel feature body corresponding to voxels of different scales in each of the target images, thereby determining the surface area and the first color information corresponding to the voxels of different scales based on the single-channel matching body and the multi-channel feature body, and then thinning the voxels of each of the target images based on the surface area and the first color information corresponding to the different scales, so as to achieve unsupervised end-to-end reconstruction of the surface of the object, thereby reducing memory usage and computational overhead, and achieving unsupervised end-to-end surface reconstruction, so as to have better computational efficiency and reconstruction performance, such as achieving reconstruction of a scene surface with rich details at a faster speed and with lower memory usage directly through a single model inference from a multi-perspective image or even a sparse image input. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1A schematic diagram of a flow chart provided for the first embodiment of the surface reconstruction method based on unsupervised end-to-end sparsification of the present application;

[0047] Figure 2 A schematic diagram of a flow chart provided for the second embodiment of the surface reconstruction method based on unsupervised end-to-end sparsification of the present application;

[0048] Figure 3 Three different examples of regional thinning methods for surface reconstruction methods based on unsupervised end-to-end thinning provided in this application and schematic diagrams of surface sampling;

[0049] Figure 4 A brief flowchart of the surface reconstruction method based on unsupervised end-to-end sparsification provided in this application;

[0050] Figure 5 This is a schematic diagram of the module structure of a surface reconstruction device based on unsupervised end-to-end sparsification according to an embodiment of the present application;

[0051] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the surface reconstruction method based on unsupervised end-to-end sparsification in an embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a big data service platform, a surface reconstruction system based on unsupervised end-to-end sparseness, etc. The following takes the surface reconstruction system based on unsupervised end-to-end sparseness as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides a surface reconstruction method based on unsupervised end-to-end sparseness, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the surface reconstruction method based on unsupervised end-to-end sparsification of the present application is provided.

[0057] In this embodiment, the surface reconstruction method based on unsupervised end-to-end sparsification includes steps S11 to S14:

[0058] Step S11, obtaining target images captured by a camera from several viewing angles;

[0059] It should be noted that the shooting angle and shooting parameters of the camera can be set according to actual conditions and are not limited here. In addition, the target image refers to the image of an object or other thing captured by the camera from several perspectives of the target scene, wherein the target scene refers to the scene that currently needs to be reconstructed. For example, if a table is to be reconstructed, the target scene is the surrounding scene around the table, and then multi-perspective shooting is performed around the table (i.e., the target scene) to achieve surface reconstruction for the current scene.

[0060] It should be further explained that the surface reconstruction method based on unsupervised end-to-end sparsification is to reconstruct the scene surface for the target scene, that is, to process the images obtained by shooting the target scene from multiple perspectives, including unsupervised, end-to-end (that is, directly through a model inference) and voxel sparsification, and finally obtain a high-fidelity surface reconstruction model, so as to achieve reconstruction of the scene surface with rich details at a faster speed and lower memory usage directly through a model inference from multi-perspective images or even sparse image inputs. Among them, the target scene contains the subject object to be reconstructed, and the main object refers to the object whose surface needs to be reconstructed in the target image. There is no restriction on the method of determining the main object, which can be set according to the actual situation, such as an object that occupies more than 80% of the main part of the picture is the main object, and it can also be set according to the user.

[0061] Step S12, determining single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images;

[0062] It should be noted that the scale may refer to different resolutions or granularity levels of data, or may refer to the size of voxels or the resolution of voxel grids, etc. The voxel refers to a volume pixel, which is the basic unit of a three-dimensional voxel grid, similar to a pixel in a two-dimensional image, where each voxel represents a discrete point in a three-dimensional space and includes attributes such as color, density, and normal.

[0063] It should be further explained that the single-channel matching volume refers to a voxel grid, in which each voxel stores a single type of data, such as an SDF value (indicating the signed distance from the voxel to the nearest surface), which focuses on providing geometric information rather than the color or other attributes of the surface. In addition, the multi-channel feature volume refers to a voxel grid, in which each voxel stores multiple types of data, such as color, normal, density, etc., which can provide rich information about each point in 3D space.

[0064] Specifically, multi-scale features in each of the target images are extracted through an image feature extraction network, and then pixel positions corresponding to voxels of different scales are obtained, so that based on the pixel positions and the multi-scale features, fusion weights and multi-view features corresponding to voxels of different scales are determined, and then based on the fusion weights and the multi-view features, single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images are determined.

[0065] Step S13, determining surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume;

[0066] It should be noted that the surface area refers to the surface area where the voxel is located in the 3D space, and the first color information refers to the color corresponding to the voxel in the 3D space, including color information corresponding to different viewing angles.

[0067] Specifically, target sampling points on a number of rays are obtained, and then based on each of the target sampling points and the single-channel matching volume, surface areas corresponding to voxels of different scales are determined, thereby obtaining second color information obtained by projecting a reference perspective and a perspective weight corresponding to each perspective, and then based on the second color information and each of the perspective weights, first color information corresponding to voxels of different scales is calculated.

[0068] Step S14, based on the surface areas corresponding to different scales and the first color information, thinning the voxels of each of the target images to achieve unsupervised end-to-end object surface reconstruction.

[0069] It should be noted that for existing surface reconstruction methods, for example, multi-view stereo matching is a classic algorithm for reconstructing scenes from multi-view images. It first predicts a depth map for each view, then filters the depth map to obtain the precise depth and projects it back into the three-dimensional space to obtain a point cloud, and finally restores the surface from the point cloud. It can be seen that the reconstruction process is multi-stage and non-end-to-end. Different stages cannot be jointly optimized, and it is inevitable that cumulative errors will be introduced to affect performance. In addition, the multi-stage process is too cumbersome and time-consuming. Therefore, in order to achieve end-to-end joint optimization, avoid cumulative errors, and reduce time consumption, the traditional dense voxel grid is improved by using multi-scale sparse voxel grids, and end-to-end sparseness is achieved using matching fields and regional thinning methods. Since the improved multi-scale sparse voxel grid no longer occupies a large memory and computational overhead and each scale can be jointly optimized end-to-end, the model has better computational efficiency and reconstruction performance, which can aggregate higher frequency features with less memory and computational overhead, and reconstruct better surface meshes with faster training and testing speeds. Among them, end-to-end refers to a complete process or system, which runs directly from the starting point (end) to the end point (end) without any intermediate breakpoints or parts that require intervention from other systems.

[0070] Specifically, based on the surface positions in the surface area corresponding to voxels of different scales, surface maps corresponding to different scales are constructed, and then each voxel is projected into each surface map, and the voxel distance between each voxel and the surface area is calculated, so as to thin the voxels of each target image based on the voxel distance and the first color information to achieve unsupervised end-to-end reconstruction of the object surface.

[0071] This embodiment obtains the target images captured by the camera from several perspectives, and then determines the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images, and then determines the surface areas and the first color information corresponding to the voxels of different scales based on the single-channel matching volume and the multi-channel feature volume, and then thins the voxels of each of the target images based on the surface areas corresponding to the different scales and the first color information, so as to achieve unsupervised end-to-end reconstruction of the object surface, thereby reducing memory usage and computational overhead, and achieving unsupervised end-to-end surface reconstruction, so as to have better computational efficiency and reconstruction performance, such as achieving reconstruction of a scene surface with rich details at a faster speed and with lower memory usage directly through a single model inference from multi-perspective images or even sparse image inputs.

[0072] Based on this, the embodiment of the present application provides a surface reconstruction method based on unsupervised end-to-end sparseness, referring to Figure 2 , Figure 2A flow chart of the second embodiment of the surface reconstruction method based on unsupervised end-to-end sparsification of the present application is provided.

[0073] In a feasible implementation manner, determining the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images includes:

[0074] Step S21, extracting multi-scale features in each of the target images through an image feature extraction network;

[0075] It should be noted that the image feature extraction network refers to a deep learning architecture for automatically extracting important visual features from input images, including textures, edges, corners, color distribution, etc. In addition, the multi-scale features refer to features extracted at different resolutions or size levels (i.e., scales), thereby allowing the network to capture both local details and global structures of the image, which helps to process objects and scenes of different sizes.

[0076] Specifically, in one embodiment, a set of target images taken from N different perspectives is obtained, and the target images can be expressed as:

[0077]

[0078] Among them, I i refers to the i-th target image, R refers to the set of real numbers, C refers to the number of image channels, H refers to the image height, W refers to the image width, and N refers to the total N shooting angles.

[0079] Furthermore, a multi-view weight-sharing feature extraction network (i.e., image feature extraction network extraction) is used to extract multi-scale features, which can be expressed as:

[0080]

[0081] in, It refers to the image features of the jth scale of the i-th view, R refers to the set of real numbers, i refers to the view index subscript, j refers to the scale index subscript, H refers to the height, W refers to the width, N refers to the total N shooting view angles, and L refers to the total L scales.

[0082] Step S22, obtaining pixel positions corresponding to voxels of different scales;

[0083] It should be noted that the pixel position refers to the coordinate position of a pixel (voxel) in an image.

[0084] Specifically, in one embodiment, for a Volume V (i.e., a voxel body) having U voxels, firstly, according to the intrinsic parameters of the camera and external reference Project each voxel v = (x, y, z) to the pixel position of the corresponding viewing angle to obtain the pixel positions corresponding to voxels of different scales, as shown in the following formula:

[0085]

[0086] Among them, π([x,y,z] T )=[x / z,y / z] T ,q i refers to the pixel position of voxel v in the i-th viewing angle image, K i refers to the internal parameter of the i-th camera, refers to the transpose of the rotation matrix in the extrinsic parameters of the i-th camera, v refers to the voxel, t i Refers to the translation matrix in the extrinsic parameters of the i-th camera.

[0087] Step S23, determining fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features;

[0088] It should be noted that the fusion weight refers to the weight used to weight the contribution of different features in the feature fusion process. These weights can be fixed or obtained through learning. They are used to determine the proportion of each feature in feature fusion. There is no restriction here. By intelligently determining the fusion weight, it is possible to avoid equal treatment of all features, thereby reducing unnecessary calculations and optimizing resource usage.

[0089] It should be further explained that the multi-view features refer to features extracted from images captured from different viewpoints. In 3D reconstruction, these features help to understand the scene from different directions and provide comprehensive information about the scene geometry and appearance.

[0090] Specifically, based on the pixel position, the multi-scale feature is upsampled, and then the upsampled multi-scale feature is calculated to obtain a multi-view feature, which is then input into an integrated network to obtain fusion weights corresponding to voxels of different scales.

[0091] Step S24: determining single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images based on the fusion weights and the multi-view features.

[0092] Specifically, in one embodiment, the weighted mean and variance of the fusion weight and the multi-view feature are calculated and concatenated together as the feature [Mean(v), Var(v)] of each voxel, as follows:

[0093]

[0094] Wherein, Mean(v) refers to the weighted mean of voxel v, w i refers to the fusion weight of the i-th perspective, f i refers to the features extracted from the i-th view, Var(v) refers to the variance obtained for voxel v, and N refers to the number of shooting viewpoints, i.e., a total of N shooting viewpoints.

[0095] Furthermore, a 3D convolutional network is used to regularize the above fused features to obtain the final single-channel matching volume V m and multi-channel feature volume V f .

[0096] This embodiment extracts multi-scale features in each of the target images through an image feature extraction network, and then obtains the pixel positions corresponding to voxels of different scales, thereby determining the fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features, and then determining the single-channel matching volume and multi-channel feature volume corresponding to voxels of different scales in each of the target images based on the fusion weights and the multi-view features, thereby achieving comprehensive multi-scale feature extraction, which helps to better understand the image content, including local details and global structures, to improve the accuracy of voxel positioning. At the same time, by determining the multi-view features, the model can integrate information from different perspectives, which helps to build a more complete and consistent 3D scene understanding, thereby building an efficient voxel grid, improving the quality of 3D surface reconstruction, and providing a more refined and accurate data basis for subsequent surface reconstruction. In addition, the multi-scale and multi-view fusion strategy improves the robustness to occlusion, noise and illumination changes, makes the reconstruction process more stable, and realizes automated and end-to-end processing.

[0097] In a feasible implementation manner, determining the fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features includes:

[0098] Step S31, upsampling the multi-scale feature based on the pixel position;

[0099] It should be noted that upsampling is a data preprocessing technique used to increase the spatial resolution of data, involving interpolation of low-resolution data to produce a higher-resolution output. In 3D data, upsampling is used to increase the density of voxel data, and the upsampling methods include nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, etc.

[0100] Specifically, in one embodiment, according to the pixel position, bilinear interpolation (ie, upsampling) is used to interpolate corresponding multi-scale features from all image planes, and the multi-scale features can be expressed as:

[0101]

[0102] Step S32, calculating the up-sampled multi-scale features to obtain multi-view features;

[0103] Specifically, each of the multi-scale features is added at the scale dimension corresponding to the multi-scale feature to obtain a multi-view feature, and the multi-view feature can be expressed as:

[0104]

[0105] Step S33: input the multi-view features into the integration network to obtain fusion weights corresponding to voxels of different scales.

[0106] It should be noted that for traditional surface reconstruction methods, such as generalized neural surface reconstruction methods, which use feature extraction networks to extract image priors from input multi-view images based on implicit neural surface reconstruction methods to directly generate reconstructed scene surfaces, this type of method does not need to be retrained for each scene and can also handle sparse view situations. However, existing methods of this type have poor performance and large memory and computational overhead. Existing methods rely on display representations such as dense voxel grids or view cones to aggregate multi-view features, which have large memory and computational overhead and cannot improve resolution due to memory limitations, thereby limiting reconstruction accuracy. Existing sparsification schemes to reduce storage either rely on real 3D labels for supervision or require multi-stage training (first training a coarse model and then initializing a fine model based on the coarse model), which will also introduce cumulative errors. Therefore, in order to solve the above problems, an integrated network and fusion weights are introduced to construct a surface reconstruction method based on unsupervised end-to-end sparsification.

[0107] It should be further explained that the integrated network refers to a deep learning architecture used to process and fuse multi-scale features, multi-view features, etc. to generate richer and more accurate data representations, thereby integrating the advantages of different features and improving the performance of the model, such as in 3D reconstruction, target recognition or scene understanding.

[0108] Specifically, the multi-view features are input into an integrated network to obtain the fusion weights corresponding to voxels of different scales in each view.

[0109] This embodiment upsamples the multi-scale features based on the pixel positions, and then calculates the upsampled multi-scale features to obtain multi-view features, thereby inputting the multi-view features into an integrated network to obtain fusion weights corresponding to voxels of different scales, and then improving feature resolution through upsampling, so that voxels at higher scales can obtain more detailed feature representations, retain or enhance image detail information, and integrate different levels of information from coarse to fine, which can improve the alignment between features of different scales, ensure the consistency of feature mapping in space, and help to fully understand the scene, thereby improving the accuracy of the final reconstruction result, reducing errors or noise that may be introduced by a single perspective, and thus improving the robustness of the overall algorithm to adapt to different application scenarios.

[0110] In a feasible implementation manner, determining the surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume includes:

[0111] Step S41, obtaining target sampling points on a number of rays;

[0112] It should be noted that the ray refers to a straight line extending in a specific direction from an observation point (usually the position of a camera or a virtual observer). In volume rendering and 3D reconstruction, rays are used to extract information from voxel data. The target sampling point refers to a specific point selected on the ray, which is used to sample attributes from voxel data, such as color, density or SDF value. The selection of the target sampling point can be set according to actual conditions, such as uniform sampling or importance-based sampling, etc., and is not limited here.

[0113] Specifically, in one embodiment, for any ray, in order to extract the surface position of a ray r = (o, d), M can be uniformly sampled from the nearest and farthest boundary regions on the ray. c Points The target sampling point on the ray is obtained by this method.

[0114] Step S42, determining surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume;

[0115] Specifically, each of the target sampling points is interpolated with the single-channel matching body to obtain an interpolation result, and then based on the interpolation result, the weight distribution of the target ray corresponding to each of the target sampling points is determined, and then based on the weight distribution, the surface area of ​​the target ray is determined.

[0116] Step S43, obtaining second color information obtained by projecting the reference viewing angle and viewing angle weights corresponding to each viewing angle;

[0117] It should be noted that the reference perspective refers to the perspective used as the main reference in the multi-perspective image, thereby providing a reference view in the 3D reconstruction process for alignment or comparison with images or data from other perspectives. In addition, the second color information refers to the color information of the surface point projected from the reference perspective. In multi-perspective reconstruction, the second color information can be used to guide the color estimation and fusion of other perspectives.

[0118] It should be further explained that the view weight refers to the weight value assigned to different views, which is used to balance the contribution of each view in the feature fusion or color synthesis process. The weight can be determined based on the reliability, importance or consistency of the view with the reference view.

[0119] Specifically, the second color information obtained by projecting the reference perspective is obtained, and the perspective weight of each perspective is obtained through a Softmax activation function, where the perspective weight is input by taking a feature interpolated from the perspective and a line of sight relationship between the perspective and the reference perspective.

[0120] Step S44: Calculate and obtain first color information corresponding to voxels of different scales based on the second color information and each of the viewing angle weights.

[0121] Specifically, since the feature volume is sparse and the traditional interpolation operation cannot perform fast and differentiable interpolation from the sparse feature volume, a sparse trilinear interpolation algorithm is designed to achieve fast and efficient interpolation. In order to predict the color of each 3D point in different directions, this method adopts the Blending strategy.

[0122]

[0123] Among them, c is the color of each 3D point in different directions, is the color projected from the reference view, is the view weight for each view.

[0124] Furthermore, similar to volume rendering, alpha fusion is used to calculate the target sampling point {p(t k )=o+t k d|k=1,…,M} to get the final color, as shown in the following formula:

[0125]

[0126] Where α is an unbiased and occlusion-known result from the SDF value, calculated as follows:

[0127]

[0128] in, is the color of the predicted ray, T k is the cumulative opacity, c k refers to the color of the kth sampling point on the ray, k refers to the index subscript of the sampling point, and M refers to the total number of sampling points, that is, there are a total of M sampling points on each ray. refers to a network that predicts a signed distance function (SDF).

[0129] This embodiment obtains target sampling points on a number of rays, and then determines the surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume, thereby obtaining the second color information obtained by the reference perspective projection and the perspective weight corresponding to each perspective, and then calculates the first color information corresponding to voxels of different scales based on the second color information and each of the perspective weights, thereby facilitating the capture of microscopic and macroscopic details of the object surface, ensuring the consistency of color and surface when observed from different perspectives, achieving a more natural and accurate color fusion effect, and thus improving the accuracy of the reconstructed object surface and achieving a smooth transition and a more realistic visual effect in the final rendering. At the same time, the integration of multi-perspective features can reduce errors or noise that may be introduced due to a single perspective, improve the robustness of the algorithm, and optimize the use of computing resources.

[0130] In a feasible implementation manner, determining the surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume includes:

[0131] Step S51, interpolating each of the target sampling points with the single-channel matching body to obtain an interpolation result;

[0132] It should be noted that in order to achieve surface area positioning with less memory and faster time, the surface area positioning needs to have the following characteristics: First, it is necessary to use limited memory consumption to encode the geometric information of the entire scene. Existing methods such as GeoNeRF and C2F2NeuS borrow the core idea of ​​MVS to build a frustum for each perspective. This method requires a lot of memory and calculation, and is less practical when there are more perspectives; second, it is necessary to quickly locate the surface area with a small computational cost, which makes multi-stage training and the use of additional networks impractical; third, it needs to be able to perceive and handle occlusion and it needs to change with the perspective. Because the input perspective is limited, those surface areas that are not visible or less visible to the input perspective are of little use and have poor solvability.

[0133] It should be further explained that the interpolation result refers to the value obtained by interpolating each of the target sampling points and the single-channel matching body, representing the attribute (such as color, density, depth, etc.) of a certain position in the 3D space.

[0134] Specifically, each of the target sampling points is moved from the matching volume V m The value corresponding to the interpolation in is obtained to obtain the interpolation result.

[0135] Step S52, determining the weight distribution of the target rays corresponding to each of the target sampling points based on the interpolation result;

[0136] It should be noted that the target ray refers to the target ray corresponding to the target sampling point, and the weight distribution refers to the relative importance or influence of each sampling point on the target ray.

[0137] Specifically, for any target ray, according to its corresponding interpolation result, a Softmax activation function is used to obtain the weight distribution on the ray, which can be expressed as:

[0138] Step S53: determining the surface area of ​​the target ray based on the weight distribution.

[0139] It should be noted that the preset area range refers to an area range near the surface position of the target sampling point.

[0140] Specifically, for any target ray, the surface position p(t s )=o+t s d, where t s The calculation formula is as follows:

[0141]

[0142] Among them, p(t s ) refers to the coordinates of the sth sampling point, d refers to the ray direction, o refers to the ray center, t s It refers to the distance from the sth sampling point to the center of the ray, M c Refers to the number of sampling points, that is, a total of M c sampling points, γ k refers to the weight of the kth sampling point, t k It refers to the distance from the kth sampling point to the center of the ray.

[0143] Furthermore, the surface area is defined as a region range near the surface position:

[0144] sr=[t s -∈,t s +∈]

[0145] Here, ∈ is an adjustable hyperparameter. In one embodiment, ∈ is set to be gradually tightened as the scale increases.

[0146] This embodiment interpolates each of the target sampling points with the single-channel matching body to obtain an interpolation result, and then determines the weight distribution of the target ray corresponding to each of the target sampling points based on the interpolation result, thereby determining the surface area of ​​the target ray based on the weight distribution, which helps to determine the exact contour of the object in surface reconstruction, improves the precision and accuracy of surface area positioning, and achieves efficient weight calculation, which helps to identify and isolate key parts of the object surface, enhances the quality and efficiency of capturing surface details, reduces errors caused by noise or inaccurate data, and improves the robustness of the reconstruction process.

[0147] In a feasible implementation, the thinning of the voxels of each of the target images based on the surface areas corresponding to different scales and the first color information to achieve unsupervised end-to-end object surface reconstruction includes:

[0148] Step S61, constructing surface maps corresponding to different scales based on surface positions in the surface area corresponding to voxels of different scales;

[0149] It should be noted that the surface map is a data structure used to represent the distribution of the surface of an object in 3D space. It can be a two-dimensional image, in which each pixel corresponds to a point in 3D space and contains specific information from the point to the nearest surface, such as distance, normal or color. It can also be a slice or a group of slices in a voxel grid, and the slices together define one or more cross-sections of the object surface.

[0150] Specifically, for a reference image I0 with a resolution of H×W, the surface position is extracted for each ray emitted from the reference image, thereby constructing a surface map, which can be expressed as: E0∈R 3×H×W .

[0151] Step S62, projecting each of the voxels into each of the surface images, and calculating the voxel distance between each of the voxels and the surface area;

[0152] It should be noted that the voxel distance refers to the straight-line distance between a voxel and the nearest point on a surface map in 3D space, and is used to determine whether the voxel is part of the surface area of ​​a main object, or to calculate the visual contribution of the voxel to the surface.

[0153] Specifically, in order to cut off those voxels that are far from the surface, first Projected to the pixel positions of all surface maps, and then bilinear interpolation is used to interpolate the corresponding visible surface points from the surface map corresponding to each view After obtaining the surface point corresponding to each voxel, we can determine whether each voxel is in the surface area according to the distance between the voxel and the corresponding surface point. The calculation formula is as follows:

[0154]

[0155] Among them, float is the operation of converting Boolean values ​​to floating-point values. It refers to whether the voxel v is within the surface area of ​​the jth scale at the i-th shooting angle. It refers to the surface position of voxel v at the jth scale of the i-th shooting angle, where v refers to a voxel, ∈ j It refers to the threshold of the j-th scale (the width of the surface area at the j-th scale).

[0156] It can be understood that these surface points are projected back to the pixel coordinate system of the source view according to the formula, and bilinear interpolation is used to interpolate from the source view image to obtain a projection image with a more predictable surface position from the source view. Theoretically, the colors projected from these surface points to each view should remain consistent. Therefore, an effective supervision signal can be generated by using the difference between the image interpolated from the source view and the true reference image. At the same time, in order to enhance robustness, this patent combines pixel-level color loss and block similarity loss SSIM to construct a constraint signal:

[0157]

[0158] Among them, L wl refers to the image projection loss, K refers to the total number of viewpoints used to calculate the image projection loss (a manually designed hyperparameter), and W Li refers to the image projection loss of the i-th viewpoint.

[0159] At this point, the model can unsupervisedly optimize the matching field to efficiently and quickly locate surface areas.

[0160] Furthermore, in order to further improve the efficiency of the model, before using volume rendering to obtain the final color for each ray, a surface sampling strategy is used to efficiently sample more sampling points for the surface area. Therefore, it is convenient to use these surface areas to directly assist the sampling of each ray, so that more sampling points fall on more valuable areas near the surface. Specifically, for the surface interval from low resolution scale to high resolution scale, by uniformly sampling with decreasing number of sampling points in these intervals, more sampling points will eventually fall near the surface area. Figure 3 In addition, when interpolating features from a multi-scale feature volume, features of certain sampling points outside the surface area are filled with 0, thereby eliminating the need to use additional networks and computational effort to implement importance sampling as in existing methods.

[0161] Step S63: based on the voxel distance and the first color information, thinning the voxels of each of the target images to achieve unsupervised end-to-end object surface reconstruction.

[0162] Specifically, based on the voxel distance, the number of surface areas that each voxel falls into is determined, and if the number of surface areas corresponding to the voxel meets the preset area number range, the voxel is used as the target voxel, and the first color information corresponding to the target voxel is determined, so as to obtain a multi-scale feature volume based on the target voxel and the first color information, and volume rendering is performed on the multi-scale feature volume to achieve unsupervised end-to-end object surface reconstruction.

[0163] This embodiment constructs surface maps corresponding to different scales based on the surface positions in the surface area corresponding to voxels of different scales, and then projects each voxel into each surface map, calculates the voxel distance between each voxel and the surface area, and then thins out the voxels of each target image based on the voxel distance and the first color information to achieve unsupervised end-to-end reconstruction of the surface of the object, and then constructs surface maps at different scales to capture the details of the surface of the object from rough to fine, and helps to accurately locate the position of the voxel relative to the surface, thereby achieving unsupervised end-to-end surface reconstruction, reducing dependence on labeled data, reducing the cost and complexity of data preparation, and at the same time, surface area thinning can reduce unnecessary calculations, improve the efficiency of the algorithm and memory usage efficiency. The end-to-end method directly generates reconstruction results from input data, simplifies the entire processing flow, and can maintain high fidelity of the reconstruction results even during the thinning process to ensure visual effects.

[0164] In a feasible implementation, the thinning of the voxels of each of the target images based on the voxel distance and the first color information to achieve unsupervised end-to-end object surface reconstruction includes:

[0165] Step S71, determining the number of surface areas where each voxel falls based on the voxel distance;

[0166] It should be noted that the number of surface areas refers to the number of surface areas determined by voxel distance calculation in 3D space.

[0167] Step S72, if the number of surface regions corresponding to the voxel meets the preset region number range, the voxel is used as a target voxel, and the first color information corresponding to the target voxel is determined;

[0168] It should be noted that the preset region number range refers to the preset region number range that meets the target requirements. If there are more than 2, there is no limitation here and it can be set according to the actual situation. The target voxel refers to the voxel selected as the target of surface reconstruction based on the judgment of the number of surface regions and the preset region number range. The voxel is considered to be a key part of surface reconstruction and will be used in subsequent feature extraction and rendering steps.

[0169] Specifically, because the surface area that is only visible to a few viewpoints or not visible at all is of little value to the reconstructed model, in order to ensure consistency between viewpoints and avoid the influence of occluded areas, in one embodiment, only voxels that fall into the surface area visible to at least two viewpoints are retained, as shown below:

[0170] Vox j+1 ={v|sum(H j (v))≥2}

[0171] Among them, sum is the sum operation, Vox j+1 refers to the voxel at the j+1th scale, v refers to the voxel, and H j (v) refers to whether voxel v is within the surface area at the jth scale (if it is, the value is 1, if not, the value is 0).

[0172] Furthermore, the voxel is used as a target voxel, and first color information corresponding to the target voxel is determined.

[0173] In addition, each target voxel may be split to aggregate higher-frequency information for the next stage.

[0174] Step S73: obtaining a multi-scale feature volume based on the target voxel and the first color information, and performing volume rendering on the multi-scale feature volume to achieve unsupervised end-to-end object surface reconstruction.

[0175] It should be noted that the multi-scale feature volume is a data structure that stores information about the 3D space at different scales, including geometric features, colors, textures, normals, etc., and each scale provides different levels of detail information.

[0176] Specifically, by repeating the sparseness strategy in step S72 for each scale, the multi-scale feature volume is finally obtained, which can be expressed as:

[0177]

[0178] Understandably, this multi-scale strategy is conducive to the model to reconstruct a globally smooth and sufficiently detailed surface, and can also use higher-resolution voxels with lower memory usage. At the same time, because the surface area range used in this method is gradually narrowed, its interval at low-resolution scale is wider, which can ensure that the model can still extract reliable features from low-resolution scale feature bodies when there are errors in subsequent surface positioning.

[0179] This embodiment determines the number of surface areas that each voxel falls into based on the voxel distance, and then if the number of surface areas corresponding to the voxel meets the preset area number range, the voxel is used as the target voxel, and the first color information corresponding to the target voxel is determined, so as to obtain a multi-scale feature body based on the target voxel and the first color information, and the multi-scale feature body is volume rendered to achieve unsupervised end-to-end object surface reconstruction, thereby more accurately identifying the surface area of ​​the object, and at the same time, by volume rendering the multi-scale feature body, a high-quality 3D surface view can be generated, and the computational efficiency is maintained, thereby speeding up the 3D surface reconstruction. In addition, the entire process from voxel distance determination to volume rendering is realized end-to-end and automatically, reducing manual intervention, improving the consistency and repeatability of processing, and having high adaptability to different scenes, that is, adapting to various 3D data and surface complexity, having good versatility, and being able to maintain high fidelity of the reconstruction results even in an efficient computing process to ensure visual effects.

[0180] For example, to help understand the implementation process of the surface reconstruction method based on unsupervised end-to-end sparsification, please refer to Figure 4 , Figure 4 A brief flowchart example diagram is provided for the surface reconstruction method based on unsupervised end-to-end sparsification in this application.

[0181] Specifically, F img is the image feature extraction network, V i is the feature body of the i-th scale, is the 3D regularized network of the i-th scale, is the matching body of the i-th scale, is the feature body of the i-th scale, SDF MLP refers to the multi-layer perceptron for predicting signed distance function, and Blending MLP refers to the multi-layer perceptron for predicting color.

[0182] It should be noted that the examples in the figure are only used to understand the present application and do not constitute a limitation on the surface reconstruction method based on unsupervised end-to-end sparsification of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0183] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0184] This application also provides a surface reconstruction device based on unsupervised end-to-end sparsification, please refer to Figure 5 , the surface reconstruction device based on unsupervised end-to-end sparsification includes:

[0185] An acquisition module 51 is used to acquire target images captured by a camera from several viewing angles;

[0186] A first determination module 52 is used to determine the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images;

[0187] A second determination module 53, configured to determine surface areas corresponding to voxels of different scales and first color information based on the single-channel matching volume and the multi-channel feature volume;

[0188] The reconstruction module 54 is used to thin out the voxels of each of the target images based on the surface areas corresponding to different scales and the first color information, so as to achieve unsupervised end-to-end object surface reconstruction.

[0189] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0190] Extracting multi-scale features in each of the target images through an image feature extraction network;

[0191] Get the pixel positions corresponding to voxels of different scales;

[0192] Based on the pixel position and the multi-scale feature, determining fusion weights and multi-view features corresponding to voxels of different scales;

[0193] Based on the fusion weights and the multi-view features, single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images are determined.

[0194] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0195] Based on the pixel position, upsampling the multi-scale feature;

[0196] Calculating the upsampled multi-scale features to obtain multi-view features;

[0197] The multi-view features are input into the integration network to obtain fusion weights corresponding to voxels of different scales.

[0198] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0199] Get target sampling points on several rays;

[0200] Determining surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume;

[0201] Obtaining second color information obtained by projecting a reference viewing angle and viewing angle weights corresponding to each viewing angle;

[0202] Based on the second color information and each of the viewing angle weights, first color information corresponding to voxels of different scales is calculated.

[0203] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0204] interpolating each of the target sampling points with the single-channel matching body to obtain an interpolation result;

[0205] Based on the interpolation result, determining the weight distribution of the target ray corresponding to each of the target sampling points;

[0206] Based on the weight distribution, a surface area of ​​the target ray is determined.

[0207] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0208] Based on the surface positions in the surface area corresponding to the voxels of different scales, surface maps corresponding to different scales are constructed;

[0209] Projecting each of the voxels onto each of the surface images, and calculating the voxel distance between each of the voxels and the surface area;

[0210] Based on the voxel distance and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

[0211] The surface reconstruction device based on unsupervised end-to-end sparsification is also used for:

[0212] Based on the voxel distance, determining the amount of surface area within which each voxel falls;

[0213] If the number of surface areas corresponding to the voxel meets the preset area number range, the voxel is used as a target voxel, and the first color information corresponding to the target voxel is determined;

[0214] Based on the target voxel and the first color information, a multi-scale feature volume is obtained, and volume rendering is performed on the multi-scale feature volume to achieve unsupervised end-to-end object surface reconstruction.

[0215] The surface reconstruction device based on unsupervised end-to-end sparseness provided by the present application adopts the surface reconstruction method based on unsupervised end-to-end sparseness in the above-mentioned embodiment, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the surface reconstruction device based on unsupervised end-to-end sparseness provided by the present application are the same as the beneficial effects of the surface reconstruction method based on unsupervised end-to-end sparseness provided by the above-mentioned embodiment, and the other technical features of the surface reconstruction device based on unsupervised end-to-end sparseness are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0216] The present application provides a surface reconstruction device based on unsupervised end-to-end sparsification, and the surface reconstruction device based on unsupervised end-to-end sparsification includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the surface reconstruction method based on unsupervised end-to-end sparsification in the above-mentioned embodiment one.

[0217] Reference below Figure 6 , which shows a schematic diagram of the structure of a surface reconstruction device based on unsupervised end-to-end sparseness suitable for implementing the embodiment of the present application. The surface reconstruction device based on unsupervised end-to-end sparseness in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The surface reconstruction device based on unsupervised end-to-end sparsification shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0218] like Figure 6As shown, the surface reconstruction device based on unsupervised end-to-end sparseness may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the surface reconstruction device based on unsupervised end-to-end sparseness are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the surface reconstruction device based on unsupervised end-to-end sparseness to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a surface reconstruction device based on unsupervised end-to-end sparseness with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0219] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0220] The surface reconstruction device based on unsupervised end-to-end sparseness provided by the present application adopts the surface reconstruction method based on unsupervised end-to-end sparseness in the above-mentioned embodiment, which can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the surface reconstruction device based on unsupervised end-to-end sparseness provided by the present application are the same as the beneficial effects of the surface reconstruction method based on unsupervised end-to-end sparseness provided by the above-mentioned embodiment, and the other technical features in the surface reconstruction device based on unsupervised end-to-end sparseness are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0221] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0222] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0223] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the surface reconstruction method based on unsupervised end-to-end sparsification in the above-mentioned embodiment.

[0224] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0225] The computer-readable storage medium may be included in the surface reconstruction device based on unsupervised end-to-end sparsification; or may exist independently without being assembled into the surface reconstruction device based on unsupervised end-to-end sparsification.

[0226] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a surface reconstruction device based on unsupervised end-to-end sparseness, the surface reconstruction device based on unsupervised end-to-end sparseness:

[0227] Obtain target images captured by the camera from several viewing angles;

[0228] Determine single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images;

[0229] Determining surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume;

[0230] Based on the surface areas corresponding to different scales and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

[0231] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0233] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0234] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned surface reconstruction method based on unsupervised end-to-end sparseness, and can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the surface reconstruction method based on unsupervised end-to-end sparseness provided by the above-mentioned embodiment, and will not be repeated here.

[0235] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0236] The computer program product provided by the present application can solve the technical problems in the background technology. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the surface reconstruction method based on unsupervised end-to-end sparsification provided by the above embodiment, which will not be repeated here.

[0237] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A surface reconstruction method based on unsupervised end-to-end sparsification, characterized in that: include: Obtain target images captured by the camera from several viewing angles; Determine single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images; Determining surface areas and first color information corresponding to voxels of different scales based on the single-channel matching volume and the multi-channel feature volume; Based on the surface areas corresponding to different scales and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

2. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 1, characterized in that: The determining of the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images comprises: Extracting multi-scale features in each of the target images through an image feature extraction network; Get the pixel positions corresponding to voxels of different scales; Based on the pixel position and the multi-scale feature, determining fusion weights and multi-view features corresponding to voxels of different scales; Based on the fusion weights and the multi-view features, single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images are determined.

3. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 2, characterized in that: The determining, based on the pixel position and the multi-scale feature, fusion weights and multi-view features corresponding to voxels of different scales includes: Based on the pixel position, upsampling the multi-scale feature; Calculating the upsampled multi-scale features to obtain multi-view features; The multi-view features are input into the integration network to obtain fusion weights corresponding to voxels of different scales.

4. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 1, characterized in that: The determining, based on the single-channel matching volume and the multi-channel feature volume, surface areas corresponding to voxels of different scales and first color information includes: Get target sampling points on several rays; Determining surface areas corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume; Obtaining second color information obtained by projecting a reference viewing angle and viewing angle weights corresponding to each viewing angle; Based on the second color information and each of the viewing angle weights, first color information corresponding to voxels of different scales is calculated.

5. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 4, characterized in that: The determining, based on each of the target sampling points and the single-channel matching volume, surface areas corresponding to voxels of different scales includes: interpolating each of the target sampling points with the single-channel matching body to obtain an interpolation result; Based on the interpolation result, determining the weight distribution of the target ray corresponding to each of the target sampling points; Based on the weight distribution, a surface area of ​​the target ray is determined.

6. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 1, characterized in that: The voxels of each target image are thinned based on the surface areas corresponding to different scales and the first color information to achieve unsupervised end-to-end object surface reconstruction, including: Based on the surface positions in the surface area corresponding to the voxels of different scales, surface maps corresponding to different scales are constructed; Projecting each of the voxels onto each of the surface images, and calculating the voxel distance between each of the voxels and the surface area; Based on the voxel distance and the first color information, the voxels of each target image are thinned to achieve unsupervised end-to-end object surface reconstruction.

7. The surface reconstruction method based on unsupervised end-to-end sparsification according to claim 6, characterized in that: The thinning of the voxels of each of the target images based on the voxel distance and the first color information to achieve unsupervised end-to-end object surface reconstruction includes: Based on the voxel distance, determining the amount of surface area within which each voxel falls; If the number of surface areas corresponding to the voxel meets the preset area number range, the voxel is used as a target voxel, and the first color information corresponding to the target voxel is determined; Based on the target voxel and the first color information, a multi-scale feature volume is obtained, and volume rendering is performed on the multi-scale feature volume to achieve unsupervised end-to-end object surface reconstruction.

8. A surface reconstruction device based on unsupervised end-to-end sparsification, characterized in that: include: An acquisition module is used to acquire target images captured by a camera from several viewing angles; A first determination module is used to determine the single-channel matching volume and the multi-channel feature volume corresponding to the voxels of different scales in each of the target images; A second determination module is used to determine the surface areas corresponding to voxels of different scales and the first color information based on the single-channel matching volume and the multi-channel feature volume; The reconstruction module is used to thin out the voxels of each of the target images based on the surface areas corresponding to different scales and the first color information, so as to achieve unsupervised end-to-end object surface reconstruction.

9. A surface reconstruction device based on unsupervised end-to-end sparsification, characterized in that: The surface reconstruction device based on unsupervised end-to-end sparsification includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described in any one of claims 1 to 7 are implemented.

Images