Surface reconstruction method and device based on unsupervised end-to-end sparsification, equipment and storage medium

Abstract

The application discloses a surface reconstruction method and device based on unsupervised end-to-end sparsification, equipment and a storage medium, relates to the technical field of image processing, and the method comprises the steps of obtaining target images photographed by a camera from several view angles; determining single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image; determining surface regions and first color information corresponding to voxels of different scales based on the single-channel matching bodies and the multi-channel feature bodies; and sparsifying voxels of each target image based on the surface regions and the first color information corresponding to voxels of different scales to realize unsupervised end-to-end object surface reconstruction. The application constructs an unsupervised end-to-end surface reconstruction method with sparsification of object surface regions, thereby reducing memory occupation and computational complexity, and realizing unsupervised end-to-end surface reconstruction, so that better calculation efficiency and reconstruction performance are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a surface reconstruction method and device based on unsupervised end-to-end sparsification, equipment and storage medium. BACKGROUND

[0002] Reconstructing the surface of an object from multi-view images is a key technology with important practical value, and the existing technology has many defects in performance and efficiency, such as: implicit neural surface reconstruction method combines implicit representation and differentiable rendering technology (such as volume rendering or surface rendering) to realize end-to-end reconstruction of the surface directly from multi-view images. However, this kind of method needs to retrain a model for each scene, which has a large time overhead and is difficult to apply directly. At the same time, this kind of method depends on the input of dense views, and cannot reconstruct an accurate surface in the case of sparse views, which further limits its practicality. SUMMARY

[0003] The main purpose of the present application is to provide a surface reconstruction method and device based on unsupervised end-to-end sparsification, equipment and storage medium, which aims to reduce memory occupation and computational overhead, and realizes unsupervised end-to-end surface reconstruction, thereby having better computational efficiency and reconstruction performance.

[0004] To achieve the above purpose, the present application provides a surface reconstruction method based on unsupervised end-to-end sparsification, which comprises:

[0005] Obtaining target images obtained by a camera from several view angles;

[0006] Determining a single-channel matching volume and a multi-channel feature volume corresponding to voxels of different scales in each target image;

[0007] Based on the single-channel matching volume and the multi-channel feature volume, determining the surface area and the first color information corresponding to the voxels of different scales;

[0008] Based on the surface area and the first color information corresponding to different scales, sparsifying the voxels of each target image to realize unsupervised end-to-end object surface reconstruction.

[0009] In an embodiment, the method further comprises:

[0010] Extracting multi-scale features in each target image through an image feature extraction network;

[0011] Obtaining pixel positions corresponding to voxels of different scales;

[0012] determine a fusion weight and a multi-view feature corresponding to a voxel of different scales based on the pixel position and the multi-scale feature;

[0013] determine a single-channel matching volume and a multi-channel feature volume corresponding to a voxel of different scales in each target image based on the fusion weight and the multi-view feature.

[0014] In an embodiment, the determining the fusion weight and the multi-view feature corresponding to a voxel of different scales based on the pixel position and the multi-scale feature comprises:

[0015] perform up-sampling on the multi-scale feature based on the pixel position;

[0016] perform calculation on the up-sampled multi-scale feature to obtain a multi-view feature;

[0017] input the multi-view feature into a unified network to obtain a fusion weight corresponding to a voxel of different scales.

[0018] In an embodiment, the determining a surface area and a first color information corresponding to a voxel of different scales based on the single-channel matching volume and the multi-channel feature volume comprises:

[0019] obtain target sampling points on a plurality of rays;

[0020] determine a surface area corresponding to a voxel of different scales based on each target sampling point and the single-channel matching volume;

[0021] obtain a second color information obtained by projection from a reference view and a view weight corresponding to each view;

[0022] calculate a first color information corresponding to a voxel of different scales based on the second color information and each view weight.

[0023] In an embodiment, the determining a surface area corresponding to a voxel of different scales based on each target sampling point and the single-channel matching volume comprises:

[0024] perform interpolation on each target sampling point and the single-channel matching volume to obtain an interpolation result;

[0025] determine a weight distribution of a target ray corresponding to each target sampling point based on the interpolation result;

[0026] determine a surface area of the target ray based on the weight distribution.

[0027] In an embodiment, the surface regions corresponding to the voxels of different scales are determined based on the single-channel matching bodies and the multi-channel feature bodies, and the first color information is determined based on the single-channel matching bodies and the multi-channel feature bodies.

[0028] The surface positions in the surface regions corresponding to the voxels of different scales are determined based on the surface positions in the surface regions corresponding to the voxels of different scales.

[0029] The voxel distances between each of the voxels and the surface regions are calculated by projecting each of the voxels into each of the surface graphs.

[0030] The voxels of each of the target images are sparsified based on the voxel distances and the first color information, so as to realize the unsupervised end-to-end object surface reconstruction.

[0031] In an embodiment, the voxels of each of the target images are sparsified based on the voxel distances and the first color information, so as to realize the unsupervised end-to-end object surface reconstruction, including:

[0032] The number of surface regions into which each of the voxels falls is determined based on the voxel distances.

[0033] If the number of surface regions corresponding to the voxels meets a preset region number range, the voxels are taken as target voxels, and the first color information corresponding to the target voxels is determined.

[0034] The multi-scale feature bodies are obtained based on the target voxels and the first color information, and the multi-scale feature bodies are rendered, so as to realize the unsupervised end-to-end object surface reconstruction.

[0035] In addition, to achieve the above object, the present application further provides a surface reconstruction device based on unsupervised end-to-end sparsification, which comprises:

[0036] An acquisition module is configured to acquire target images obtained by a camera from a plurality of view angles.

[0037] A first determination module is configured to determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each of the target images.

[0038] A second determination module is configured to determine surface regions corresponding to the voxels of different scales and first color information based on the single-channel matching bodies and the multi-channel feature bodies.

[0039] A reconstruction module is configured to sparsify the voxels of each of the target images based on the surface regions corresponding to the voxels of different scales and the first color information, so as to realize the unsupervised end-to-end object surface reconstruction.

[0040] In addition, to achieve the above object, the present application also provides a surface reconstruction device based on unsupervised end-to-end sparsification, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0041] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0042] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0043] The present application provides a surface reconstruction method, device, equipment and storage medium based on unsupervised end-to-end sparsification, which acquires target images taken by a camera from several viewing angles, and then determines single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image, so as to determine surface regions and first color information corresponding to voxels of different scales based on the single-channel matching bodies and the multi-channel feature bodies, and then sparsify voxels of each target image based on the surface regions and the first color information corresponding to different scales, so as to realize unsupervised end-to-end object surface reconstruction, thereby reducing memory occupation and computational overhead, and realizing unsupervised end-to-end surface reconstruction, thereby having better computational efficiency and reconstruction performance, such as reconstructing a detailed scene surface at a faster speed and with lower memory occupation through one model inference from multi-view images or even sparse image input. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0046] Figure 1A flowchart provided by the surface reconstruction method based on unsupervised end-to-end sparsification according to an embodiment of the present application is shown in FIG. 1.

[0047] Figure 2 A flowchart provided by the surface reconstruction method based on unsupervised end-to-end sparsification according to an embodiment of the present application is shown in FIG. 1.

[0048] Figure 3 Three different instances of the area sparsification method of the surface reconstruction method based on unsupervised end-to-end sparsification provided by the present application are shown in FIGS. 2-4, and a surface sampling diagram is shown in FIG. 5.

[0049] Figure 4 A brief flowchart of the surface reconstruction method based on unsupervised end-to-end sparsification provided by the present application is shown in FIG. 6.

[0050] Figure 5 A module structure diagram of the surface reconstruction device based on unsupervised end-to-end sparsification according to an embodiment of the present application is shown in FIG. 7.

[0051] Figure 6 A device structure diagram of the hardware running environment involved in the surface reconstruction method based on unsupervised end-to-end sparsification according to an embodiment of the present application is shown in FIG. 8.

[0052] The object, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0054] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a big data service platform, a surface reconstruction system based on unsupervised end-to-end sparsification, etc. The present embodiment and the following embodiments will be described below taking the surface reconstruction system based on unsupervised end-to-end sparsification as an example.

[0056] Based on this, the present embodiment provides a surface reconstruction method based on unsupervised end-to-end sparsification, which is described in detail below with reference to Figure 1 , Figure 1 A flowchart provided by the surface reconstruction method based on unsupervised end-to-end sparsification according to an embodiment of the present application is shown in FIG. 1.

[0057] In this embodiment, the surface reconstruction method based on unsupervised end-to-end sparsification includes steps S11-S14:

[0058] Step S11, obtaining target images captured by a camera from multiple perspectives;

[0059] It should be noted that the shooting perspective and shooting parameters of the camera can be set according to actual conditions, which are not limited herein. In addition, the target image refers to an image of an object or other things captured by the camera from multiple perspectives of a target scene, wherein the target scene refers to a scene that needs to be reconstructed at present, such as reconstructing a table, the target scene is the surrounding scene of the table, and then multiple perspective shooting is performed around the table (i.e. the target scene) to realize surface reconstruction of the current scene.

[0060] It should be further noted that the surface reconstruction method based on unsupervised end-to-end sparsification is for scene surface reconstruction of a target scene, that is, the images captured from multiple perspectives of the target scene are processed, including unsupervised, end-to-end (i.e. directly through one model inference) and voxel sparsification, to finally obtain a high-fidelity surface reconstruction model, so as to realize reconstruction of a scene surface with rich details from multiple perspective images or even sparse image input through one model inference at a faster speed and with lower memory occupation. The target scene contains a subject object that needs to be reconstructed, wherein the subject object refers to an object that needs to be reconstructed in the target image, and the determination method of the subject object is not limited herein and can be set according to actual conditions, such as an object occupying more than 80% of the main part of the picture being a subject object, or being set by a user.

[0061] Step S12, determining single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image;

[0062] It should be noted that the scale can refer to different resolutions or granularity levels of data, or the size of voxels or the resolution of voxel grids, etc. The voxel refers to a volume pixel, which is a basic unit of a three-dimensional voxel grid, similar to a pixel in a two-dimensional image, wherein each voxel represents a discrete point in a three-dimensional space and contains attributes such as color, density, normal, etc.

[0063] It should be further noted that the single-channel matching body refers to a voxel grid, wherein each voxel stores a single type of data, such as an SDF value (representing the signed distance of the voxel to the nearest surface), which focuses on providing geometric information rather than the color or other attributes of the surface. In addition, the multi-channel feature body refers to a voxel grid, wherein each voxel stores multiple types of data, such as color, normal, density, etc., which can provide rich information about each point in 3D space.

[0064] Specifically, multi-scale features in each of the target images are extracted by an image feature extraction network, and then pixel positions corresponding to voxels of different scales are obtained, so as to determine fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features, and then determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each of the target images based on the fusion weights and the multi-view features.

[0065] In step S13, surface regions and first color information corresponding to voxels of different scales are determined based on the single-channel matching bodies and the multi-channel feature bodies.

[0066] It should be noted that the surface region refers to a surface region in 3D space where the voxel is located, and the first color information refers to the color corresponding to the voxel in 3D space, including color information corresponding to different viewing angles.

[0067] Specifically, target sampling points on a plurality of rays are obtained, and then surface regions corresponding to voxels of different scales are determined based on each of the target sampling points and the single-channel matching bodies, so as to obtain second color information obtained by projecting a reference viewing angle and viewing angle weights corresponding to each viewing angle, and then first color information corresponding to voxels of different scales is calculated based on the second color information and each of the viewing angle weights.

[0068] In step S14, the voxels of each of the target images are sparsified based on the surface regions and the first color information corresponding to different scales, so as to realize unsupervised end-to-end object surface reconstruction.

[0069] It should be noted that, for the existing surface reconstruction method, for example, multi-view stereo matching is a kind of classic algorithm for realizing the reconstruction of a scene from multi-view pictures, which first predicts a depth map for each view, then filters the depth map to obtain accurate depth and projects it back to the three-dimensional space to obtain a point cloud, and finally recovers the surface from the point cloud. It can be seen that the reconstruction process is multi-stage and non-end-to-end, which cannot be jointly optimized between different stages, and it is difficult to avoid introducing cumulative errors to affect performance, and the multi-stage process is too cumbersome and time-consuming. Therefore, in order to realize end-to-end joint optimization and avoid cumulative errors and reduce time consumption, the traditional dense voxel grid is improved by using a multi-scale sparse voxel grid, and the end-to-end sparsification is realized by using a matching field and a regional sparsification method. Since the improved multi-scale sparse voxel grid no longer occupies a large memory and computational overhead, and each scale can be end-to-end jointly optimized, the model has better computational efficiency and reconstruction performance, which aggregates to higher frequency features with less memory and computational overhead, and reconstructs a better surface grid with faster training and testing speed. Wherein, end-to-end refers to a complete process or system that directly runs from the starting point (end) to the ending point (end) without intermediate breakpoints or the need for other system intervention.

[0070] Specifically, based on the surface positions in the surface regions corresponding to the voxels of different scales, surface maps corresponding to different scales are constructed, and then each voxel is projected into each surface map, the voxel distance of each voxel and the surface region is calculated, and the voxels of each target image are sparsified based on the voxel distance and the first color information, to realize unsupervised end-to-end object surface reconstruction.

[0071] The embodiment obtains target images captured by a camera from several views, and then determines single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image, and then determines surface regions and first color information corresponding to voxels of different scales based on the single-channel matching bodies and the multi-channel feature bodies, and then sparsifies the voxels of each target image based on the surface regions and the first color information corresponding to voxels of different scales, to realize unsupervised end-to-end object surface reconstruction, thereby reducing memory occupation and computational overhead, and realizing unsupervised end-to-end surface reconstruction, thereby having better computational efficiency and reconstruction performance, such as reconstructing a detailed scene surface directly from multi-view images or even sparse image input through one model inference at a faster speed and with lower memory occupation.

[0072] Based on this, the embodiment of the present application provides a surface reconstruction method based on unsupervised end-to-end sparsification, which refers to Figure 2 , Figure 2A flowchart diagram provided for the second embodiment of the surface reconstruction method based on unsupervised end-to-end sparsification of the present application.

[0073] In an implementable embodiment, the determining of the single-channel matching body and the multi-channel feature body corresponding to the voxels of different scales in each of the target images comprises:

[0074] Step S21, extracting multi-scale features in each of the target images through an image feature extraction network;

[0075] It should be noted that the image feature extraction network refers to a deep learning architecture for automatically extracting important visual features from input images, including texture, edge, corner, color distribution, etc. In addition, the multi-scale features refer to features extracted at different resolution or size levels (i.e. scales), allowing the network to capture both local details and global structures of the image, which helps to process objects and scenes of different sizes.

[0076] Specifically, in an embodiment, a set of target images taken from N different perspectives is obtained, which can be represented as:

[0077]

[0078] where I i represents the i-th target image, R represents a set of real numbers, C represents the number of image channels, H represents the image height, W represents the image width, and N represents the total number of N shooting perspectives.

[0079] Further, a multi-perspective weight shared feature extraction network (i.e. image feature extraction network extraction) is used to extract multi-scale features, which can be represented as:

[0080]

[0081] where, represents the image feature of the i-th perspective and the j-th scale, R represents a set of real numbers, i represents the perspective index subscript, j represents the scale index subscript, H represents the height, W represents the width, N represents the total number of N shooting perspectives, and L represents the total number of L scales.

[0082] Step S22, obtaining the pixel position corresponding to the voxels of different scales;

[0083] It should be noted that the pixel position refers to the coordinate position of the pixel (voxel) in the image.

[0084] Specifically, in an embodiment, for a Volume V (i.e. voxel body) with U voxels, the intrinsic parameters and extrinsic parameters Projecting each voxel v = (x, y, z) to the pixel position of the corresponding view, so as to obtain the pixel position corresponding to the voxel of different scales, as shown in the following formula:

[0085]

[0086] wherein, π([x, y, z] T ) = [x / z, y / z] T , q i refers to the pixel position of the voxel v on the i-th view image, K i refers to the intrinsic parameter of the i-th camera, refers to the transpose of the rotation matrix in the i-th camera extrinsic parameter, v refers to the voxel, t i refers to the translation matrix in the i-th camera extrinsic parameter.

[0087] Step S23, based on the pixel position and the multi-scale feature, determining the fusion weight corresponding to the voxel of different scales and the multi-view feature;

[0088] It should be noted that the fusion weight refers to the contribution degree of weighting different features in the feature fusion process, and these weights can be fixed or obtained through learning, which is used to determine the proportion of each feature in the feature fusion, which is not limited herein. By intelligently determining the fusion weight, it can be avoided to equally process all features, so as to possibly reduce unnecessary calculation and optimize resource use.

[0089] It should be further noted that the multi-view feature refers to the feature extracted from the image captured from different views, which helps to understand the scene from different directions and provides comprehensive information about the geometry and appearance of the scene in 3D reconstruction.

[0090] Specifically, based on the pixel position, the multi-scale feature is up-sampled, and then the up-sampled multi-scale feature is calculated to obtain the multi-view feature, so as to input the multi-view feature into the unified network to obtain the fusion weight corresponding to the voxel of different scales.

[0091] Step S24, based on the fusion weight and the multi-view feature, determining the single-channel matching body and the multi-channel feature body corresponding to the voxel of different scales in each target image.

[0092] Specifically, in an embodiment, the fusion weight and the multi-view feature are weighted mean and variance, and are spliced together as the feature [Mean(v), Var(v)] of each voxel, as follows:

[0093]

[0094] wherein, Mean(v) refers to the weighted mean obtained for the voxel v, w i refers to the fusion weight of the i-th view, f i refers to the feature extracted from the i-th view, Var(v) refers to the variance obtained for the voxel v, and N refers to the number of shooting views, i.e., a total of N shooting views.

[0095] Further, using a 3D convolutional network to regularize the fused features, the final single-channel matching body V m and multi-channel feature body V f

[0096] The embodiment extracts multi-scale features in each of the target images through an image feature extraction network, and then obtains pixel positions corresponding to voxels of different scales, so as to determine fusion weights and multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features, and then determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each of the target images based on the fusion weights and the multi-view features, so as to comprehensively extract multi-scale features, better understand image content including local details and global structure, improve the accuracy of voxel positioning, and make the model integrate information from different views by determining multi-view features, which helps to build a more complete and consistent 3D scene understanding, so as to build an efficient voxel grid, improve the quality of 3D surface reconstruction, and provide a more fine and accurate data basis for subsequent surface reconstruction. In addition, the multi-scale and multi-view fusion strategy improves the robustness to occlusion, noise and illumination changes, makes the reconstruction process more stable, and realizes automatic and end-to-end processing.

[0097] In a possible implementation, the determination of the fusion weights and the multi-view features corresponding to voxels of different scales based on the pixel positions and the multi-scale features includes:

[0098] Step S31: upsampling the multi-scale features based on the pixel positions;

[0099] It should be noted that the upsampling is a data preprocessing technique for increasing the spatial resolution of data, which involves interpolating low-resolution data to produce higher-resolution output. In 3D data, upsampling is used to increase the density of voxel data, and the upsampling method includes nearest neighbor interpolation, linear interpolation, bilinear interpolation, bicubic interpolation, etc.

[0100] Specifically, in an embodiment, according to the pixel positions, the corresponding multi-scale features are interpolated from all image planes by bilinear interpolation (i.e., upsampling), and the multi-scale features can be represented as:

[0101] ​

[0102] Step S32, the multi-scale features are calculated to obtain multi-view features;

[0103] Specifically, the multi-scale features are added in the scale dimension corresponding to the multi-scale features to obtain multi-view features, which can be represented as:

[0104]

[0105] Step S33, the multi-view features are input into the integration network to obtain the fusion weight corresponding to the voxel of different scales.

[0106] It should be noted that for traditional surface reconstruction methods, such as the generalized neural surface reconstruction method, the feature extraction network is used to extract image priors from the input multi-view images to directly generate the reconstructed scene surface based on the implicit neural surface reconstruction method. Such methods do not need to be retrained for each scene and can also handle sparse view cases, but the existing methods have poor performance and large memory and computational complexity. The existing methods rely on explicit representations such as dense voxel grids or view frustums to aggregate multi-view features, which have large memory and computational complexity overheads and cannot improve the resolution due to memory limitations, thereby limiting the reconstruction accuracy. The existing sparse storage scheme either relies on real three-dimensional labels for supervision or requires multi-stage training (first train a coarse model and then initialize a fine model based on the coarse model), which also introduces cumulative errors. Therefore, in order to solve the above problems, the integration network and the fusion weight are introduced to construct an unsupervised end-to-end sparse surface reconstruction method.

[0107] It should be further noted that the integration network refers to a deep learning architecture for processing and integrating multi-scale features, multi-view features, etc. to generate more rich and accurate data representations, thereby integrating the advantages of different features and improving the performance of the model, such as in 3D reconstruction, target recognition or scene understanding.

[0108] Specifically, the multi-view features are input into an integration network to obtain the fusion weight corresponding to the voxel of different scales in each view

[0109] The embodiment obtains the multi-view feature by upsampling the multi-scale feature based on the pixel position, and then calculates the multi-scale feature after upsampling to obtain a multi-view feature, so as to input the multi-view feature into a unified network to obtain a fusion weight corresponding to a voxel of different scales, and then improve the feature resolution by upsampling, so that the voxel at a higher scale can obtain more detailed feature representation, retain or enhance the detail information of the image, and integrate different levels of information from coarse to fine, which can improve the alignment degree between different scale features, ensure the consistency of feature mapping in space, help to comprehensively understand the scene, thereby improving the accuracy of the final reconstruction result, reducing the error or noise that may be introduced by a single view, and then improving the robustness of the overall algorithm, and adapting to different application scenarios.

[0110] In an implementable embodiment, the determining of the surface area and the first color information corresponding to the voxels of different scales based on the single-channel matching body and the multi-channel feature body comprises:

[0111] Step S41, obtaining target sampling points on a plurality of rays;

[0112] It should be noted that the ray refers to a straight line extending from an observation point (usually the position of a camera or a virtual observer) in a specific direction. In volume rendering and 3D reconstruction, rays are used to extract information from voxel data. The target sampling point refers to a specific point selected on the ray, which is used to sample attributes such as color, density or SDF value from voxel data. The selection of the target sampling point can be set according to the actual situation, such as uniform sampling or importance-based sampling, which is not limited here.

[0113] Specifically, in an embodiment, for any ray, in order to extract the surface position of a ray r=(o,d), the target sampling points on the ray can be obtained by uniformly sampling M c points from the nearest and farthest boundary regions on the ray.

[0114] Step S42, determining the surface area corresponding to the voxels of different scales based on each target sampling point and the single-channel matching body;

[0115] Specifically, each target sampling point is interpolated with the single-channel matching body to obtain an interpolation result, and then the weight distribution of the target ray corresponding to each target sampling point is determined based on the interpolation result, so as to determine the surface area of the target ray based on the weight distribution.

[0116] Step S43, obtaining second color information obtained by projecting a reference view and view weights corresponding to each view;

[0117] It should be noted that the reference view refers to the view in the multi-view images that serves as the primary reference, thereby providing a baseline view for alignment or comparison with images or data of other views in the 3D reconstruction process. In addition, the second color information refers to the color information of the surface points projected from the reference view, which can be used to guide color estimation and fusion of other views in multi-view reconstruction.

[0118] Further, it should be noted that the view weight refers to the weight value assigned to different views, which is used to balance the contribution of each view in the feature fusion or color synthesis process. The weight can be determined based on the reliability, importance or consistency with the reference view of the view.

[0119] Specifically, the second color information projected from the reference view is obtained, and the view weight of each view is obtained through the Softmax activation function, which takes the features interpolated from the view and the view and reference view line relationship as input.

[0120] In step S44, the first color information corresponding to the voxels of different scales is calculated based on the second color information and the view weights of each view.

[0121] Specifically, since the feature volume is sparse, traditional interpolation operation cannot quickly and differentiably interpolate from sparse feature volume, so a sparse trilinear interpolation algorithm is designed to realize fast and efficient interpolation. In order to predict the color of each three-dimensional point in different directions, the method adopts Blending strategy,

[0122]

[0123] where c is the color of each three-dimensional point in different directions, is the color projected from the reference view, is the view weight of each view.

[0124] Further, similar to volume rendering, alpha blending is used to fuse the final color according to the results of target sampling points {p(t k )=o+t k d|k=1,…,M} on each ray, as shown in the following formula:

[0125]

[0126] where α is the unbiased and occlusion known result obtained from the SDF value, and the calculation method is as follows:

[0127]

[0128] where refers to the color of the predicted ray, T k refers to the accumulated opacity, c k refers to the color of the k-th sampling point on the ray, k refers to the index of the sampling point, and M refers to the total number of sampling points, that is, a total of M sampling points on each ray, refers to a network that predicts a signed distance function (SDF).

[0129] The embodiment obtains target sampling points on a plurality of rays, and then determines the surface regions corresponding to voxels of different scales based on the target sampling points and the single-channel matcher, to obtain second color information obtained by projection from a reference view angle and view angle weights corresponding to each view angle, and then calculate first color information corresponding to voxels of different scales based on the second color information and the view angle weights, thereby helping to capture microscopic and macroscopic details of the surface of the object, ensuring consistency of color and surface when observed from different view angles, achieving a more natural and accurate color fusion effect, and improving the accuracy of the reconstructed object surface and achieving smooth transition and more realistic visual effects in the final rendering. Meanwhile, the integration of multi-view features can reduce errors or noise that may be introduced due to a single view angle, improve the robustness of the algorithm, and optimize the use of computing resources.

[0130] In a feasible implementation, the determination of the surface regions corresponding to voxels of different scales based on the target sampling points and the single-channel matcher comprises:

[0131] In step S51, each target sampling point is interpolated with the single-channel matcher to obtain an interpolation result.

[0132] It should be noted that, in order to position the surface region with small memory and fast time, the surface region positioning needs to have the following characteristics: first, the limited memory consumption needs to be used to encode the geometric information of the entire scene. The existing method such as GeoNeRF and C2F2NeuS borrows the core idea of MVS to construct a view cone for each view angle, which requires a large amount of memory and computation, and is less practical when there are many view angles. Second, it needs to quickly locate the surface region with small computational cost, which makes multi-stage training and the use of additional networks impractical. Third, it needs to be able to perceive and handle occlusions and needs to be view-dependent. Because the input view angles are limited, those surface regions that are not visible or less visible to the input view angles are of little use and have poor solvability.

[0133] It should be further noted that the interpolation result refers to a value obtained by interpolating each target sampling point with the single-channel matcher, which represents the attributes (such as color, density, depth, etc.) of a certain position in the 3D space.

[0134] Specifically, each of the target sampling points is interpolated from the matching body V m to obtain an interpolation result.

[0135] Step S52, determining a weight distribution of the target ray corresponding to each of the target sampling points based on the interpolation result.

[0136] It should be noted that the target ray refers to the target ray corresponding to the target sampling point, and the weight distribution refers to the relative importance or influence of each sampling point on the target ray.

[0137] Specifically, for any target ray, the weight distribution on the ray is obtained by passing through a Softmax activation function according to the corresponding interpolation result, and the weight distribution can be expressed as:

[0138] Step S53, determining a surface region of the target ray based on the weight distribution.

[0139] It should be noted that the preset region range refers to a region range near the surface position of the target sampling point.

[0140] Specifically, for any target ray, the surface position p(t s ) of this target ray can be obtained by the weight distribution as follows: s d, wherein t s The calculation formula is as follows:

[0141]

[0142] wherein p(t s ) is the coordinate of the s-th sampling point, d is the ray direction, o is the ray center, t s is the distance from the s-th sampling point to the ray center, M c is the number of sampling points, i.e., a total of M c sampling points, γ k is the weight of the k-th sampling point, and t k is the distance from the k-th sampling point to the ray center.

[0143] Further, the surface region is defined as a region range near the surface position:

[0144] sr=[t s -∈,t s +∈]

[0145] wherein ∈ is an adjustable hyperparameter, and in an embodiment, ∈ is set to gradually tighten as the scale increases.

[0146] This embodiment interpolates each of the target sampling points with the single-channel matching body to obtain the interpolation result. Based on the interpolation result, the weight distribution of the target ray corresponding to each target sampling point is determined. Based on the weight distribution, the surface area of ​​the target ray is determined, which helps to determine the exact contour of the object in surface reconstruction, improves the accuracy and precision of surface area positioning, and achieves efficient weight calculation. This helps to identify and isolate key parts of the object surface, enhances the quality and efficiency of surface detail capture, reduces errors caused by noise or inaccurate data, and improves the robustness of the reconstruction process.

[0147] In one feasible implementation, the step of sparsifying the voxels of each target image based on surface regions corresponding to different scales and first color information to achieve unsupervised end-to-end object surface reconstruction includes:

[0148] Step S61: Based on the surface positions in the surface regions corresponding to voxels of different scales, construct surface maps corresponding to different scales.

[0149] It should be noted that the surface map is a data structure used to represent the distribution of object surfaces in 3D space. It can be a two-dimensional image, where each pixel corresponds to a point in 3D space and contains specific information about that point to the nearest surface, such as distance, normal, or color. It can also be a slice or a group of slices in a voxel mesh, which together define one or more sections of the object surface.

[0150] Specifically, for a reference image I0 with a resolution of H×W, the surface position is extracted by the rays emitted from each pixel in this reference image, thereby constructing a surface map, which can be represented as: E0∈R 3×H×W .

[0151] Step S62: Project each voxel onto each surface map and calculate the voxel distance between each voxel and the surface region.

[0152] It should be noted that the voxel distance refers to the straight-line distance between a voxel and the nearest point on the surface map in 3D space. It is used to determine whether a voxel belongs to the surface area of ​​the main object, or to calculate the visual contribution of a voxel to the surface.

[0153] Specifically, in order to cut off those voxels that are far from the surface, the voxels are first... The pixel locations are projected onto all surface maps, and then bilinear interpolation is used to interpolate the corresponding visible surface points from the surface map corresponding to each viewpoint. Once the surface point corresponding to each voxel is obtained, the distance between the voxel and the corresponding surface point can be used to determine whether each voxel is within the surface region. The calculation formula is as follows:

[0154]

[0155] where float is the operation of converting boolean value to float value, denotes whether the voxel v is inside the surface region of the j-th scale of the i-th view, denotes the surface position of the voxel v in the j-th scale of the i-th view, v denotes a voxel, ∈ j denotes the threshold of the j-th scale (the width of the surface region of the j-th scale).

[0156] It can be understood that these surface points are projected back to the pixel coordinate system of the source view according to the formula, and the projection image obtained from the source view is obtained by using bilinear interpolation from the source view image In theory, the color projected from these surface points to each view should remain consistent. Therefore, the difference between the image interpolated from the source view and the real reference image can be used to generate an effective supervision signal. At the same time, in order to enhance the robustness, the present patent combines pixel-level color loss and block similarity loss SSIM to construct the constraint signal:

[0157]

[0158] where L wl denotes the image projection loss, K denotes the total number of views used to calculate the image projection loss (an artificially designed hyperparameter), and W Li denotes the image projection loss of the i-th view.

[0159] At this point, the model can optimize the matching field unsupervised to efficiently and quickly locate the surface region.

[0160] Further, in order to further improve the efficiency of the model, before using volume rendering to obtain the final color for each ray, a surface sampling strategy is used to efficiently sample more sampling points for the surface region. Since the surface region of each scale has been obtained Therefore, these surface regions can be conveniently used to directly assist the sampling of each ray, so that more sampling points fall in the more valuable area near the surface. Specifically, for the surface interval from the low-resolution scale to the high-resolution scale, by uniformly sampling in these intervals with a decreasing number of sampling points, more sampling points finally fall near the surface region, which can be referred to in Figure 3 In addition, when interpolating features from the multi-scale feature volume, the features of some sampling points outside the surface region are filled with 0, so that importance sampling does not need to be implemented using additional networks and computational complexity as in the existing method.

[0161] Step S63, sparsifying voxels of each target image based on the voxel distance and the first color information to realize unsupervised end-to-end object surface reconstruction.

[0162] Specifically, based on the voxel distance, the number of surface regions into which each voxel falls is determined, and then if the number of surface regions corresponding to the voxel meets a preset region number range, the voxel is taken as a target voxel, and the first color information corresponding to the target voxel is determined, so that a multi-scale feature body is obtained based on the target voxel and the first color information, and volume rendering is performed on the multi-scale feature body to realize unsupervised end-to-end object surface reconstruction.

[0163] The embodiment constructs surface maps corresponding to different scales based on surface positions in surface regions corresponding to voxels of different scales, then projects each voxel into each surface map, calculates the voxel distance between each voxel and a surface region, and sparsifies voxels of each target image based on the voxel distance and the first color information to realize unsupervised end-to-end object surface reconstruction, so that surface maps are constructed at different scales to capture details of object surfaces from coarse to fine, and help to accurately locate the position of a voxel relative to a surface, thereby realizing unsupervised end-to-end surface reconstruction, reducing the dependence on labeled data, and reducing the cost and complexity of data preparation. At the same time, surface region sparsification can reduce unnecessary computational load, improve the efficiency of the algorithm and the efficiency of memory usage. The end-to-end method directly produces a reconstruction result from input data, simplifying the entire processing flow, and even in the sparsification process, the high fidelity of the reconstruction result can be maintained to ensure the visual effect.

[0164] In a feasible implementation, the sparsifying voxels of each target image based on the voxel distance and the first color information to realize unsupervised end-to-end object surface reconstruction comprises:

[0165] Step S71, determining the number of surface regions into which each voxel falls based on the voxel distance;

[0166] It should be noted that the number of surface regions refers to the number of surface regions determined according to the voxel distance calculation in the 3D space.

[0167] Step S72, if the number of surface regions corresponding to the voxel meets a preset region number range, the voxel is taken as a target voxel, and the first color information corresponding to the target voxel is determined;

[0168] It should be noted that the preset range of the number of regions refers to a preset range of the number of regions meeting the target demand, such as more than 2, which is not limited here and can be set according to actual conditions. The target voxel refers to a voxel selected as a voxel belonging to the surface reconstruction target according to the judgment of the surface region number and the preset range of the number of regions. The voxel is considered as a key part of surface reconstruction and will be used in subsequent feature extraction and rendering steps.

[0169] Specifically, because the surface region visible only by a few viewpoints or not visible at all has less value for the reconstructed model, and to ensure consistency between viewpoints and avoid the influence of occluded regions, in an embodiment, only voxels of surface regions visible to at least two viewpoints are retained, as follows:

[0170] Vox j+1 = {v | sum(H j (v)) ≥ 2}

[0171] where sum is a summation operation, Vox j+1 refers to a voxel at the j+1th scale, v refers to a voxel, and H j (v) refers to whether the voxel v is in the surface region at the jth scale (1 if yes, and 0 if no).

[0172] Further, the voxel is taken as a target voxel, and first color information corresponding to the target voxel is determined.

[0173] In addition, for each target voxel, it can be split to aggregate more high-frequency information for the next stage.

[0174] Step S73, based on the target voxel and the first color information, a multi-scale feature body is obtained, and the multi-scale feature body is rendered to realize unsupervised end-to-end object surface reconstruction.

[0175] It should be noted that the multi-scale feature body is a data structure that stores information about a 3D space at different scales, including geometric features, colors, textures, normals, etc., and each scale provides different levels of detail information.

[0176] Specifically, by repeating the sparsification strategy in step S72 for each scale, the multi-scale feature body is finally obtained, which can be represented as:

[0177]

[0178] It can be understood that such a multi-scale strategy is beneficial to reconstructing a surface that is globally smooth and has sufficient details, and can also utilize higher resolution voxels with lower memory occupancy. Meanwhile, because the surface region range adopted by the method is gradually narrowed, the interval at a low resolution scale is wider, so that reliable features can still be extracted from the feature volume at the low resolution scale when there is an error in subsequent surface positioning.

[0179] In this embodiment, the number of surface regions into which each voxel falls is determined based on the voxel distance, and then if the number of surface regions corresponding to the voxel meets a preset region number range, the voxel is taken as a target voxel, and the first color information corresponding to the target voxel is determined, so as to obtain a multi-scale feature volume based on the target voxel and the first color information, and perform volume rendering on the multi-scale feature volume, to realize unsupervised end-to-end object surface reconstruction, and then more accurately identify the surface region of the object. Meanwhile, by performing volume rendering on the multi-scale feature volume, a high-quality 3D surface view can be generated, the calculation efficiency is maintained, and the speed of 3D surface reconstruction is accelerated. Moreover, the entire process from voxel distance determination to volume rendering is realized in an end-to-end automated manner, reducing manual intervention, improving the consistency and repeatability of processing, and having high adaptability to different scenes, that is, adapting to various different 3D data and surface complexities, and having good universality. Even in an efficient calculation process, the high fidelity of the reconstruction result can be maintained, and the visual effect is ensured.

[0180] For example, in order to facilitate understanding of the implementation process of the surface reconstruction method based on unsupervised end-to-end sparsification, please refer to Figure 4 , Figure 4 A brief process example diagram of the surface reconstruction method based on unsupervised end-to-end sparsification provided by the present application is shown in the figure.

[0181] Specifically, F img is an image feature extraction network, V i is a feature volume at the i-th scale, is a 3D regularization network at the i-th scale, is a matching volume at the i-th scale, is a feature volume at the i-th scale, SDF MLP refers to a multi-layer perceptron for predicting a signed distance function, and Blending MLP refers to a multi-layer perceptron for predicting a color.

[0182] It should be noted that the examples in the figure are only used to understand the present application and do not constitute a limitation on the surface reconstruction method based on unsupervised end-to-end sparsification of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0183] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0184] The application also provides a surface reconstruction device based on unsupervised end-to-end sparsification, which refers to Figure 5 The surface reconstruction device based on unsupervised end-to-end sparsification comprises:

[0185] The acquisition module 51 is configured to acquire target images captured by a camera from multiple perspectives.

[0186] The first determination module 52 is configured to determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image.

[0187] The second determination module 53 is configured to determine surface regions and first color information corresponding to voxels of different scales based on the single-channel matching bodies and the multi-channel feature bodies.

[0188] The reconstruction module 54 is configured to sparsify voxels of each target image based on the surface regions and the first color information corresponding to voxels of different scales, so as to realize unsupervised end-to-end object surface reconstruction.

[0189] The surface reconstruction device based on unsupervised end-to-end sparsification is also configured to:

[0190] Extract multi-scale features in each target image through an image feature extraction network.

[0191] Acquire pixel positions corresponding to voxels of different scales.

[0192] Determine fusion weights and multi-perspective features corresponding to voxels of different scales based on the pixel positions and the multi-scale features.

[0193] Determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each target image based on the fusion weights and the multi-perspective features.

[0194] The surface reconstruction device based on unsupervised end-to-end sparsification is also configured to:

[0195] Upsample the multi-scale features based on the pixel positions.

[0196] Calculate the upsampled multi-scale features to obtain multi-perspective features.

[0197] Input the multi-perspective features into a unified network to obtain fusion weights corresponding to voxels of different scales.

[0198] The surface reconstruction device based on unsupervised end-to-end sparsification is further configured to:

[0199] Obtain target sampling points on a plurality of rays;

[0200] Determine surface regions corresponding to voxels of different scales based on the target sampling points and the single-channel matching body;

[0201] Obtain second color information obtained by projecting a reference view and view weights corresponding to each view;

[0202] Based on the second color information and the view weights, calculate first color information corresponding to voxels of different scales.

[0203] The surface reconstruction device based on unsupervised end-to-end sparsification is further configured to:

[0204] Interpolate each target sampling point and the single-channel matching body to obtain an interpolation result;

[0205] Based on the interpolation result, determine the weight distribution of the target ray corresponding to each target sampling point;

[0206] Based on the weight distribution, determine the surface region of the target ray.

[0207] The surface reconstruction device based on unsupervised end-to-end sparsification is further configured to:

[0208] Based on the surface positions in the surface regions corresponding to voxels of different scales, construct surface maps corresponding to different scales;

[0209] Project each voxel into each surface map and calculate the voxel distance between each voxel and the surface region;

[0210] Based on the voxel distance and the first color information, sparsify the voxels of each target image to realize unsupervised end-to-end object surface reconstruction.

[0211] The surface reconstruction device based on unsupervised end-to-end sparsification is further configured to:

[0212] Based on the voxel distance, determine the number of surface regions into which each voxel falls;

[0213] If the number of surface regions corresponding to the voxel meets a predetermined region number range, the voxel is taken as a target voxel, and the first color information corresponding to the target voxel is determined;

[0214] Based on the target voxel and the first color information, obtain a multi-scale feature body, and perform volume rendering on the multi-scale feature body to realize unsupervised end-to-end object surface reconstruction.

[0215] The device for surface reconstruction based on unsupervised end-to-end sparsification provided in the present application adopts the method for surface reconstruction based on unsupervised end-to-end sparsification in the above embodiment, and can solve the technical problems in the background art. Compared with the prior art, the device for surface reconstruction based on unsupervised end-to-end sparsification provided in the present application has the same beneficial effects as the method for surface reconstruction based on unsupervised end-to-end sparsification provided in the above embodiment, and other technical features in the device for surface reconstruction based on unsupervised end-to-end sparsification are the same as the features disclosed in the above embodiment, which will not be repeated here.

[0216] The present application provides a device for surface reconstruction based on unsupervised end-to-end sparsification, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for surface reconstruction based on unsupervised end-to-end sparsification in the above embodiment one.

[0217] Reference will be made to the following description of the embodiments of the present application, taken in conjunction with the accompanying drawings, in which Figure 6 which shows a structural schematic diagram of the device for surface reconstruction based on unsupervised end-to-end sparsification suitable for implementing the embodiments of the present application. The device for surface reconstruction based on unsupervised end-to-end sparsification in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The device for surface reconstruction based on unsupervised end-to-end sparsification shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0218] As Figure 6As shown, the unsupervised end-to-end sparsification based surface reconstruction device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the unsupervised end-to-end sparsification based surface reconstruction device operation are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the unsupervised end-to-end sparsification based surface reconstruction device to communicate wirelessly or wired with other devices to exchange data. Although the unsupervised end-to-end sparsification based surface reconstruction device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0219] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carrying computer program code for performing the methods illustrated by the flowcharts carried on a computer readable medium. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0220] The surface reconstruction device based on unsupervised end-to-end sparsification provided in the application adopts the surface reconstruction method based on unsupervised end-to-end sparsification in the above embodiment, and can solve the technical problems in the background art. Compared with the prior art, the beneficial effects of the surface reconstruction device based on unsupervised end-to-end sparsification provided in the application are the same as those of the surface reconstruction method based on unsupervised end-to-end sparsification provided in the above embodiment, and other technical features in the surface reconstruction device based on unsupervised end-to-end sparsification are the same as those disclosed in the above embodiment method, and will not be repeated here.

[0221] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0222] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0223] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the surface reconstruction method based on unsupervised end-to-end sparsification in the above embodiment.

[0224] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0225] The computer readable storage medium described above may be included in a surface reconstruction device based on unsupervised end-to-end sparsification, or may exist independently without being assembled into the surface reconstruction device based on unsupervised end-to-end sparsification.

[0226] The computer readable storage medium described above carries one or more programs, which, when executed by the surface reconstruction device based on unsupervised end-to-end sparsification, cause the surface reconstruction device based on unsupervised end-to-end sparsification to:

[0227] Obtain target images captured by a camera from a plurality of perspectives;

[0228] Determine single-channel matching bodies and multi-channel feature bodies corresponding to voxels of different scales in each of the target images;

[0229] Determine surface regions and first color information corresponding to voxels of different scales based on the single-channel matching bodies and the multi-channel feature bodies;

[0230] Sparsify voxels of each of the target images based on the surface regions and the first color information corresponding to voxels of different scales, to achieve unsupervised end-to-end object surface reconstruction.

[0231] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0232] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0233] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0234] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the above-mentioned surface reconstruction method based on unsupervised end-to-end sparsification, and can solve the technical problems in the background art. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the surface reconstruction method based on unsupervised end-to-end sparsification provided by the above-mentioned embodiments, which will not be described here.

[0235] The embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described above.

[0236] The computer program product provided by the present application can solve the technical problems in the background art. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the surface reconstruction method based on unsupervised end-to-end sparsification provided by the above-mentioned embodiment, and are not described here.

[0237] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A surface reconstruction method based on unsupervised end-to-end sparsification, characterized in that, include: Acquire target images captured by the camera from several angles; Determine the single-channel matching volume and multi-channel feature volume corresponding to voxels of different scales in each target image; Based on the single-channel matching volume and the multi-channel feature volume, the surface regions corresponding to voxels of different scales and the first color information are determined. Based on the surface regions corresponding to different scales and the first color information, the voxels of each target image are sparsified to achieve unsupervised end-to-end object surface reconstruction.

2. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 1, characterized in that, The determination of single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each of the target images includes: Multi-scale features are extracted from each of the target images using an image feature extraction network; Obtain the pixel positions corresponding to voxels at different scales; Based on the pixel location and the multi-scale features, determine the fusion weights and multi-view features corresponding to voxels of different scales; Based on the fusion weights and the multi-view features, single-channel matching volumes and multi-channel feature volumes corresponding to voxels of different scales in each target image are determined.

3. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 2, characterized in that, The step of determining the fusion weights and multi-view features corresponding to voxels at different scales based on the pixel positions and the multi-scale features includes: Based on the pixel location, the multi-scale features are upsampled; The upsampled multi-scale features are calculated to obtain multi-view features; The multi-view features are input into the fusion network to obtain the fusion weights corresponding to voxels of different scales.

4. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 1, characterized in that, The step of determining the surface regions corresponding to voxels of different scales and the first color information based on the single-channel matching volume and the multi-channel feature volume includes: Acquire target sampling points on several rays; Based on each target sampling point and the single-channel matching body, the surface regions corresponding to voxels of different scales are determined; Obtain the second color information obtained by projecting from the reference viewpoint and the viewpoint weights corresponding to each viewpoint; Based on the second color information and the respective viewpoint weights, the first color information corresponding to voxels of different scales is calculated.

5. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 4, characterized in that, The step of determining the surface regions corresponding to voxels of different scales based on each of the target sampling points and the single-channel matching volume includes: Interpolate each of the target sampling points with the single-channel matching body to obtain the interpolation result; Based on the interpolation results, the weight distribution of the target rays corresponding to each target sampling point is determined; The surface region of the target ray is determined based on the weight distribution.

6. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 1, characterized in that, The step of sparsifying the voxels of each target image based on surface regions corresponding to different scales and first color information to achieve unsupervised end-to-end object surface reconstruction includes: Surface maps corresponding to different scales are constructed based on the surface positions in the surface regions corresponding to voxels of different scales. Project each voxel onto each of the surface maps and calculate the voxel distance between each voxel and the surface region. Based on the voxel distance and the first color information, the voxels of each target image are sparsified to achieve unsupervised end-to-end object surface reconstruction.

7. The surface reconstruction method based on unsupervised end-to-end sparsification as described in claim 6, characterized in that, The step of sparsifying the voxels of each target image based on the voxel distance and the first color information to achieve unsupervised end-to-end object surface reconstruction includes: Based on the voxel distance, determine the number of surface areas that each voxel falls into; If the number of surface regions corresponding to the voxel meets the preset range of region numbers, then the voxel is taken as the target voxel, and the first color information corresponding to the target voxel is determined. Based on the target voxel and the first color information, a multi-scale feature volume is obtained, and the multi-scale feature volume is rendered to achieve unsupervised end-to-end object surface reconstruction.

8. A surface reconstruction device based on unsupervised end-to-end sparsification, characterized in that, include: The acquisition module is used to acquire target images captured by the camera from several perspectives; The first determining module is used to determine the single-channel matching volume and multi-channel feature volume corresponding to voxels of different scales in each of the target images; The second determining module is used to determine the surface regions corresponding to voxels of different scales and the first color information based on the single-channel matching body and the multi-channel feature body. The reconstruction module is used to sparsify the voxels of each target image based on the surface regions corresponding to different scales and the first color information, so as to realize unsupervised end-to-end object surface reconstruction.

9. A surface reconstruction device based on unsupervised end-to-end sparsification, characterized in that, The surface reconstruction apparatus based on unsupervised end-to-end sparsity includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the surface reconstruction method based on unsupervised end-to-end sparsity as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the surface reconstruction method based on unsupervised end-to-end sparsification as described in any one of claims 1 to 7.

Images