System and method for depth completion and three-dimensional reconstruction of image regions

Abstract

According to one aspect of the disclosure, a video processing method is provided. The method may include inputting attribute data associated with an image region and a sparse depth map to a sparse depth completion network. The method may include generating a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network. The method may include performing a three-dimensional (3D) reconstruction process based on the refined dense depth map to generate a point cloud of the image region. The method may include performing a triangular meshing process to generate a mesh model based on the point cloud of the image region. The method may include performing a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. The method may include performing a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region.

Description

Technical Field

[0001] The embodiments disclosed herein relate to video processing. Background Technology

[0002] Digital video has become mainstream and is widely used in various applications, including digital television, video telephony, and teleconferencing. The viability of these digital video applications is due to advancements in computing and communication technologies, as well as efficient video processing techniques. Summary of the Invention

[0003] According to one aspect of this disclosure, a video processing method is provided. The method may include: a processor inputting attribute data associated with an image region and a sparse depth map into a sparse depth completion network. The method may include: the processor using the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map. The method may include: the processor performing a three-dimensional (3D) reconstruction process based on the refined dense depth map to generate a point cloud of the image region. The method may include: the processor performing a triangulation process to generate a mesh model based on the point cloud of the image region. The method may include: the processor performing a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. The method may include: the processor performing a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region.

[0004] According to another aspect of this disclosure, a video processing system is provided. The system may include a processor and a memory storing instructions. The memory stores instructions that, when executed by the processor, cause the processor to: input attribute data associated with an image region and a sparse depth map into a sparse depth completion network. The memory stores instructions that, when executed by the processor, cause the processor to: generate a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network. The memory stores instructions that, when executed by the processor, cause the processor to: perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region. The memory stores instructions that, when executed by the processor, cause the processor to: perform a triangulation process to generate a mesh model based on the point cloud of the image region. The memory stores instructions that, when executed by the processor, cause the processor to: perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. The memory stores instructions that, when executed by the processor, enable the processor to perform the following operations: based on the textured mesh, perform a vertex normal vector calculation process to generate a three-dimensional representation of the image region.

[0005] According to another aspect of this disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium stores instructions. When executed by a processor of a video processing system, the instructions cause the processor to: input attribute data associated with an image region and a sparse depth map into a sparse depth completion network. When executed by a processor of the video processing system, the instructions cause the processor to: generate a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network. When executed by a processor of the video processing system, the instructions cause the processor to: perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region. When executed by a processor of the video processing system, the instructions cause the processor to: perform a triangulation process to generate a mesh model based on the point cloud of the image region. When executed by a processor of the video processing system, the instructions cause the processor to: perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. When executed by the processor of the video processing system, the instructions cause the processor to perform the following operations: based on the textured mesh, perform a vertex normal vector calculation process to generate a three-dimensional representation of the image region.

[0006] These illustrative embodiments are mentioned not to limit or restrict this disclosure, but are provided as examples to aid in understanding the disclosure. Additional embodiments are described in the detailed description, and further description is provided therein. Attached Figure Description

[0007] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, further serve to illustrate the principles of the present disclosure and enable those skilled in the art to make and use the present disclosure.

[0008] Figure 1A A flowchart of an example structure for motion and multi-view stereo reconstruction is shown.

[0009] Figure 1B A flowchart of an example Bundlefusion optimization process is shown.

[0010] Figure 2 The present disclosure illustrates the generation of a single view using an exemplary video processing system according to some embodiments thereof. Figure 3 Flowchart for 3D scene reconstruction.

[0011] Figure 3 Some embodiments according to this disclosure are shown. Figure 2 An exemplary network architecture diagram of the sparse depth completion component of a video processing system.

[0012] Figure 4 Some embodiments according to this disclosure are shown. Figure 2 A diagram of an exemplary multi-affinity matrix convolutional spatial propagation network (CSPN++) component of a video processing system.

[0013] Figure 5 The following are some embodiments of the invention illustrated in this disclosure. Figure 2 An example image of a camera projection pinhole model applied to the 3D reconstruction component of a video processing system.

[0014] Figure 6A The representations shown according to some embodiments of this disclosure use Figure 3 A visual comparison of the sparse depth completion performed on an exemplary network architecture.

[0015] Figure 6B Graphical representations of root mean square error (RMSE) and mean absolute error (MAE) performance for different depths of sparsity are shown according to some embodiments of the present disclosure.

[0016] Figure 6C The diagram illustrates a visual comparison of three-dimensional point cloud scenes with different depth sparsity according to some embodiments of the present disclosure.

[0017] Figure 7 The diagram illustrates a 3D indoor point cloud scene reconstruction based on a single red-green-blue (RGB) depth (D) image according to some embodiments of the present disclosure, wherein the single RGB-D image is used... Figure 3 The exemplary network architecture is used to generate it.

[0018] Figure 8 The diagram illustrates a point cloud and a 3D mesh structure reconstructed from a single RGB-D image in an indoor scene according to some embodiments of the present disclosure, wherein the single RGB-D image is generated by... Figure 3 The example network architecture is used to generate it.

[0019] Figure 9 A diagram illustrating a visual comparison of reconstruction results from different perspectives between point clouds and textured meshes according to some embodiments of the present disclosure, wherein the point clouds and textured meshes are constructed by... Figure 3 The exemplary network architecture is used to generate it.

[0020] Figure 10 A flowchart of an exemplary video processing method according to some embodiments of the present disclosure is shown.

[0021] Figure 11 This is a block diagram illustrating examples of computer systems that can be used to implement the various embodiments set forth in this disclosure.

[0022] Embodiments of this disclosure will be described with reference to the accompanying drawings. Detailed Implementation

[0023] While some configurations and arrangements have been discussed, it should be understood that this is for illustrative purposes only. Those skilled in the art will recognize that other configurations and arrangements may be used without departing from the spirit and scope of this disclosure. It will be apparent to those skilled in the art that this disclosure may also be employed in a variety of other applications.

[0024] It should be noted that references to "one embodiment," "embodiment," "example embodiment," "some embodiments," "certain embodiments," etc., in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but not every embodiment must include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other embodiments will be within the knowledge of those skilled in the art.

[0025] Generally, terms can be understood, at least partially, from their usage in context. For example, the term "one or more," as used herein, depends at least partially on the context and can be used to describe any feature, structure, or characteristic in a singular sense, or in a plural sense, to describe a combination of features, structures, or characteristics. Similarly, again depending at least partially on the context, terms such as "a," "an," or "the" can be understood to convey either a singular or a plural usage. Furthermore, again depending at least partially on the context, the term "based on" can be understood to not necessarily be intended to express an exclusive set of factors, but rather to allow for the presence of additional factors that may not be explicitly described.

[0026] Various aspects of a video processing system will now be described with reference to various apparatuses and methods. These apparatuses and methods will be described in the following detailed embodiments and illustrated in the accompanying drawings by various modules, components, circuits, steps, operations, processes, algorithms, etc. (collectively, “elements”). These elements can be implemented using electronic hardware, firmware, computer software, or any combination thereof. Whether such elements are implemented as hardware, firmware, or software depends on the specific application and the design constraints imposed on the system as a whole.

[0027] The techniques described herein can be used in various video processing applications. As described herein, video processing includes encoding and decoding video. Video encoding and decoding can be performed on a block-by-block basis. For example, encoding / decoding processes such as transform, quantization, prediction, loop filtering, reconstruction, etc., can be performed on encoded blocks, transform blocks, or prediction blocks. As described herein, the block to be encoded / decoded will be referred to as the "current block". For example, depending on the current encoding / decoding process, the current block can represent an encoded block, a transform block, or a prediction block. Furthermore, it should be understood that the term "unit" as used in this disclosure refers to a basic unit used to perform a particular encoding / decoding process, while the term "block" refers to a sample array of a predetermined size. Unless otherwise stated, "block" and "unit" are used interchangeably.

[0028] Two-dimensional (2D) vision technologies primarily involve planar image processing, such as image classification and segmentation. In contrast, 3D vision technologies generate geometric information from natural scenes and use depth maps to understand the entire field of view. 3D reconstruction has been widely applied in applications such as autonomous driving, virtual reality, and 3D printing. Multi-view geometric reconstruction is one such 3D reconstruction method. The basic principle of multi-view geometric reconstruction is that it uses images taken from different angles to capture a target object, and then uses these images to recover the 3D structure and appearance of the target object by analyzing the geometric relationships between these images. To achieve this, camera parameter estimation and dense point cloud reconstruction can be applied.

[0029] Structure-from-motion (SfM) and multi-view stereo (MVS) are two multi-view geometric reconstruction techniques used to recover the spatial structure of an object. SfM is a method that uses feature points from multiple unordered images to reconstruct camera motion trajectories and estimate camera parameters to generate a sparse 3D point cloud. To this end, the SfM pipeline, for example, uses scale-invariant feature transform (SIFT) to extract and match local features between images. The SfM pipeline can then perform sparse point cloud triangulation incrementally or globally. However, the sparse feature point distribution used by the SfM algorithm makes it difficult to recover low-level details. In complex scenes, errors in feature matching and reconstruction may occur, meaning further post-processing and optimization are required. This post-processing and optimization can be performed using multi-view stereo (MVS) methods. For example, MVS relies on matching point clouds from multiple images to achieve high-density 3D reconstruction. To do this, MVS finds the corresponding point in 3D space for each pixel and then uses these corresponding points to obtain additional 3D information through dense matching. Next, MVS interpolates and optimizes the sparse point cloud in SfM to generate a more accurate and complete 3D model. Example operations of the MVS method are as follows: Figure 1A As shown.

[0030] For example, Figure 1AA flowchart of an example structure 100 for motion and MVS reconstruction is shown. MVS reconstruction can begin by collecting a series of overlapping images 101 (e.g., images with overlapping portions) using the same or different viewpoints or camera sensors. Then, meaningful feature points are detected in each image, i.e., keypoint extraction 103. Next, keypoint matching 105 can be performed. During keypoint matching 105, keypoints or features of interest in the overlapping images 101 are matched across different images to compute their correspondences. Next, bundle adjustment 107 can be performed to optimize camera parameters and point positions in the scene. This normalizes the images from multiple viewpoints and improves matching accuracy. The matched keypoints can be converted into points in 3D space to form a sparse point cloud, which implements MVS 109. Finally, using image information from multiple viewpoints and interpolation methods, missing portions of the original point cloud are filled to generate a dense point cloud 111.

[0031] In practical applications, SfM and MVS can be seen as working together. For example, SfM provides camera pose and 3D point cloud, while MVS generates a more accurate and complete 3D model based on the information generated by SfM. However, SfM faces difficulties in handling non-rigid scenes and image noise, while MVS suffers from high computational complexity when dealing with large-scale scenes.

[0032] This is because SfM and MVS are based on 2D image information, resulting in incomplete and unrealistic 3D models. However, with the advent of depth cameras, depth-based 3D scanning and reconstruction techniques have developed. Common RGB-D sensors are now affordable and easy to use, making the development of new technologies easier. RGB-D 3D reconstruction uses color images (RGB) and depth maps to reconstruct scenes. By processing and aligning color and depth images, this technique can accurately reconstruct real-world scenes and objects to generate high-quality 3D models. Bundlefusion is a 3D reconstruction algorithm based on an RGB-D camera that can perform globally consistent modeling in existing scenes. An example flowchart of Bundlefusion global pose optimization 150 is shown below. Figure 1B As shown.

[0033] Reference Figure 1BThe Bundlefusion operation can first acquire a sequence of RGB-D images of an image region from multiple viewpoints using an RGB-D sensor 102. The RGB-D sensor 102 may include a color sensor and a depth sensor (e.g., a LiDAR sensor). The RGB-D image sequence can be output as one or more RGB-D image data 115. For example, the RGB-D image data 115 may include color frames (e.g., color image data) and depth frames (e.g., depth maps of objects in the frames). Then, the correspondence search unit 104 can apply a depth-based feature extraction algorithm to extract feature points from the RGB-D image data 115. The indication of the feature points can be sent as a sparse / dense correspondence signal 117 to the local pose estimation unit 106. The local pose estimation unit 106 can use a feature matching algorithm based on local geometry descriptors to match the feature points extracted from the sparse / dense correspondence signal 117. These extracted feature points can be indicated as chunks 119. The global pose estimation unit 108 can use a surface reconstruction algorithm that continuously merges new point sets into the image region based on geometric information in the point cloud to generate a pose estimate 121. Block 119 and pose estimate 121 can be maintained in a data cache 110. The data cache 110 can send block 119, pose estimate, block update 125, or pose update 127 to the correspondence search unit 104 in a feedback loop 123. Block update 125 and pose update 127 can be sent to the integration / deintegration unit 112. The integration / deintegration unit 112 can apply an optimization framework-based method to jointly optimize the point clouds from all views to obtain a globally consistent 3D representation of the image region.

[0034] However, traditional 3D reconstruction methods and RGB-D camera-based 3D reconstruction methods have various drawbacks. For example, these techniques require multi-view image data and complex steps such as camera calibration, synchronization, view matching, and pose estimation. This leads to high algorithmic complexity. Unfortunately, the computational complexity of the hardware is too high to meet real-time requirements, or it can only provide single-view depth image data instead of multi-view data. This is especially true in mobile devices due to their stringent power consumption requirements.

[0035] Therefore, there is an unmet need for a 3D reconstruction method that can significantly reduce computational and storage costs to enhance its practicality and feasibility in mobile devices.

[0036] To overcome these and other challenges, this disclosure provides an exemplary 3D reconstruction technique that uses single-view color and depth data for 3D reconstruction. Based on the high-precision depth information provided by a single depth image, the geometry and details of an object can be reconstructed, thus avoiding errors and inconsistencies between multiple images. Furthermore, since only a single image pair (e.g., color image data and depth data) is processed, this technique avoids the computational complexity involved in matching and merging multiple images. This allows for a greater focus on depth information processing and improved accuracy to obtain accurate reconstruction results. Therefore, although single-view depth image reconstruction has some limitations in terms of depth estimation error and viewpoint constraints, it remains effective in 3D reconstruction. To this end, this disclosure provides an exemplary network architecture for depth completion to recover a dense depth map from a sparse depth map, guided by RGB attribute data.

[0037] In some embodiments, the exemplary network architecture described herein may include an exemplary sparse depth completion network. In the exemplary sparse depth completion network, the color branch includes two decoders to utilize inter-pixel relationships while extracting depth features. The first decoder in the color branch predicts a depth map for fusion, while the second decoder can be used to extract adaptive features for multi-scale fusion.

[0038] Furthermore, the exemplary sparse depth completion network can combine CSPN operations with guided filtering to fuse features from two modalities (e.g., color and depth) at the encoder-decoder stage of the depth branch. To further refine the depth map, the exemplary sparse depth completion network can fuse multimodal features based on a multi-affinity matrix, which is used to iteratively update the depth map until a refined, dense depth map is obtained.

[0039] Then, 3D reconstruction can be performed using a refined dense depth map to achieve an optimal balance between depth sparsity and completion accuracy. Because the number of valid points in the depth map is reduced through optimal depth sparsity, the exemplary network architecture described below can significantly save power in mobile devices. Furthermore, the exemplary network architecture can reconstruct a point cloud from the depth map using a camera pinhole model, and then perform triangulation and mesh rendering to obtain a realistic 3D reconstruction of the image region. Further details of the exemplary 3D reconstruction network architecture and its exemplary operation are provided below. Figures 2 to 10 supply.

[0040] Figure 2 This illustration demonstrates the generation of a single view using an exemplary video processing system 250 (hereinafter referred to as "video processing system 250") according to some embodiments of the present disclosure. Figure 3A flowchart 200 of a 3D scene. The video processing system 250 may include, for example, a stitcher 202, a sparse depth completion network component 204, a 3D reconstruction component 206, a triangulation component 208, a texture mapping component 210, and a vertex normal vector component 212.

[0041] First, the video processing system 250 can receive attribute data 201 (e.g., color data, RGB data, reflectance data, intensity data, etc.) and a sparse depth map 203 (also referred to herein as "depth data"). In some implementations, image regions can be associated with pictures, subpictures, tiles, slices, or coded blocks. The attribute data 201 and the sparse depth map 203 can be stitched together by a stitcher 202 (e.g., pixel-level stitching). A sparse depth completion network component 204 can perform an RGB-guided depth completion process to generate iterative sparse depth maps, which are updated using a multi-affinity matrix until a refined dense depth map is obtained. Using the refined dense depth map 205, a 3D reconstruction component 206 can generate a point cloud 207 of the 2D image region (e.g., 3D reconstruction). A triangulation component 208 can perform triangulation on the point cloud 207 to generate a mesh model 209. Next, the texture mapping component 210 can apply the mesh model 209 to the attribute data 201 to generate a textured mesh 211 of the image region. Finally, the vertex normal vector component 212 can apply a vertex normal vector algorithm to the textured mesh 211 to generate a three-dimensional representation 213 of the two-dimensional image region. Further details of the exemplary operations performed by the video processing system 250 are provided below. Figures 3 to 10 supply.

[0042] Figure 3 Some embodiments according to this disclosure are shown. Figure 2 A detailed diagram of an exemplary network architecture 300 for sparse depth-complete network component 204. Figure 4 The following are some embodiments of the invention illustrated in this disclosure. Figure 3 Detailed Figure 400 shows the exemplary multi-affinity matrix CSPN++ component 370 used by the sparse depth completion network component 204. It will be described together with the example. Figure 3 and Figure 4 .

[0043] Due to factors such as equipment or the surrounding environment, the depth values ​​obtained by the depth sensor are sparse in two-dimensional space. This sparsity is insufficient to recover rich three-dimensional structural information. For example, if such a depth map, lacking a large amount of depth information, is directly used for 3D reconstruction, the visual effect of the generated 3D point cloud model surface will be significantly incomplete. Therefore, the network architecture 300 is designed to recover a refined, dense depth map 311 from the sparse depth map 303 under the guidance of attribute data 301, such as... Figure 3 As shown.

[0044] Reference Figure 3 Unlike most parallel dual-branch networks, the sparse deep completion network component 204 includes two decoders (e.g., a first decoder 330a and a second decoder 330b) in the RGB decoder stage to predict more adaptive features after processing by the encoder 320. In this way, the sparse deep completion network component 204 can leverage the complementarity of the iterative depth maps predicted simultaneously by the color branch and the depth branch, while avoiding the influence of color branch supervision on the fusion of different modes during the decoder-encoder fusion stage.

[0045] The color stage may include, for example, a first encoder 320a, a first decoder 330a, and a second decoder 330b, while the depth stage may include a second encoder 320b and a third decoder 330c. Attribute data 301 and a sparse depth map 303 (e.g., captured by an RGB-D sensor or an RGB sensor and a LiDAR sensor) may be input to the first encoder 320a. The first decoder 330a may generate a first iterative depth map 340 (and a confidence map) and a multi-affinity matrix 360 based on the input from the RGB stage. On the other hand, the second decoder 330b may be configured to generate multiple adaptive features (e.g., (1) to (5)) after each decoder stage. Each adaptive feature may indicate different inter-pixel relationships between pixels in the sparse depth map 303.

[0046] The sparse depth map 303, the first iterative depth map 340 (generated by the first decoder 330a), and multiple adaptive features (generated by the second decoder 330b) can be inputs to the depth branch. Utilizing these multiple adaptive features, the sparse depth completion network component 204 can fully leverage the inter-pixel relationships in the color branch to extract additional depth features. To further fuse the two different modal features (color and depth), each of the multiple adaptive features is input to a different guided CSPN filter 302 of the second encoder 320b. The third decoder 330c can generate a second iterative depth map 350 (and a confidence map) based on the sparse depth map 303, the first iterative depth map 340, and the multiple adaptive features. By element-wise summing the features of the first iterative depth map 340 and the second iterative depth map 350, a coarse depth map 309 can be generated.

[0047] Guided CSPN filter component 302 (e.g., fusion module) combines CSPN with guided filtering to fuse features captured by the color branch and the depth branch. For example, guided CSPN filter 302 can predict dynamic changes in the convolution kernel from the color branch and then use these changes to extract depth features at different fusion stages according to expression (1).

[0048] (1), in, k determines the neighborhood range of a pixel. ,as well as Calculate based on expressions (2) and (3).

[0049] (2); and (3), in, For the affinity matrix, and To normalize the results and ensure the stability of the guided CSPN filter component 302.

[0050] To avoid over-smoothing of the refined dense depth map 311 after multiple iterations, the multi-affinity matrix CSPN++ component 370 can apply different affinity matrices to update the coarse depth map 309. The multi-affinity matrix 360 can assign different inter-pixel weights associated with its adaptive features. This makes the pixel values ​​of the refined dense depth map 311 more accurate after each iteration.

[0051] Reference Figure 4 The multi-affinity matrix can be adaptively generated via convolutional layers using high-level features at the ends of the network architecture backbone 402. When refining the coarse depth map 309, each affinity matrix 460 can be used to iteratively update pixels at the same spatial location. Since the weights of the multi-affinity matrix take into account inter-pixel relationships, the multi-affinity matrix avoids over-smoothing. This can increase the sharpness and structural detail of the refined dense depth map 311. Finally, the depth values ​​are refined through distant pixels using dilated convolutions with an increased receptive field. The refined dense depth map 311 generated based on the multi-affinity matrix 460 can be described according to expression (4).

[0052] (4), in, It is a CSPN++ function. and These are the corresponding depth maps before and after the update, and... Represents iterative index i The multi-affinity matrix. Additional details of an exemplary 3D reconstruction process performed after sparse depth completion network component 204 are provided below.

[0053] For example, Figure 5 The following are some embodiments of the invention illustrated in this disclosure. Figure 2An exemplary image of a camera projection pinhole model 500 applied to the 3D reconstruction component 206. With the expansion of the mobile device market, more and more consumers are interested in experiencing 3D scenes on mobile devices. Time-of-flight (ToF) stereo depth sensing lenses are an emerging 3D imaging technology that can detect and analyze object distance, shape, and motion with high precision. This can provide a more realistic and immersive experience. However, using depth cameras for 3D reconstruction presents challenges and requires special optimization and control for power-sensitive mobile devices to ensure device stability and battery life. At the same time, power consumption and battery life limit the application of depth cameras. Therefore, when using ToF sensors for 3D reconstruction on mobile devices, an appropriate trade-off between power consumption and performance needs to be struck.

[0054] To make depth cameras more suitable for mobile devices, several measures can be taken to reduce power consumption. For example, in the field of augmented reality (AR), low power consumption and long range have become important technical indicators. In contrast, the sparsity requirement of depth pixel values ​​may be less stringent. Therefore, the video processing system 250 can train sparse depth maps (e.g., iteratively updating a coarse depth map 309 until a refined dense depth map 311 is obtained) to effectively reduce the power consumption requirements of the depth camera. Sparsity refers to retaining less depth information in the depth map, thereby reducing computational complexity and storage requirements. When designing a depth camera, a balance needs to be found between ensuring the accuracy and stability of 3D reconstruction and minimizing power consumption as much as possible. To this end, the video processing system 250 can train and test depth maps with different sparsities to find the ideal sparsity threshold and algorithm parameters. By determining the ideal sparsity, the power consumption of the depth camera can be reduced while maintaining performance, making the depth camera more suitable for use in mobile devices.

[0055] Still refer to Figure 5 Unlike outdoor scenes, indoor scenes typically require close-up image capture within a confined space (e.g., an office or bedroom). In the real world, cameras capture images based on a pinhole model, meaning the camera maps coordinates in three-dimensional space onto the image plane. This mapping process can be handled by a 3D reconstruction component 206, such as... Figure 5 The process is as shown. For example, the 3D reconstruction component 206 can assign each pixel on the image plane to a corresponding point in 3D space. This process can be represented by expression (7).

[0056] (5), in, , , , It's the camera's internal parameters. , It is the camera's external parameters. , These are the coordinates of a point on a two-dimensional plane. , , It corresponds to a point in three-dimensional space.

[0057] Since the video processing system 250 reconstructs a 3D scene from a single view (e.g., generates a point cloud 207), the 3D reconstruction component 206 can be designed to take both camera coordinates and world coordinates into account. Therefore, and Form a matrix as shown in expression (6).

[0058] (6) By combining depth and position information, the 3D reconstruction component 206 can utilize their relationship to transform the refined dense depth map 205 into a point cloud 207 (e.g., a 3D point cloud scene) according to expressions (7) to (9).

[0059] (7); (8); and (9).

[0060] Return to reference Figure 2 The triangulation component 208 can generate a smooth, continuous representation of the surface (e.g., mesh model 209) for further processing and analysis. Triangulation is a technique that uses a mesh composed of many triangles to represent a surface. In Open3D, alpha shape is a point cloud-based 3D surface reconstruction method that can convert discrete point cloud data into a continuous 3D surface model (e.g., mesh model 209). This method uses alpha parameter values ​​to construct a series of nested surfaces, where the alpha parameter is considered as a distance threshold used to construct the surfaces. These operations performed by the triangulation component 208 are summarized below as "Algorithm 1".

[0061] Typically, the goal of the alpha shape method is to find nested triangular faces that form the edges of the alpha complex. As the alpha parameter value increases, the number of edges in the alpha complex increases, while the number of triangular faces decreases. Therefore, the alpha parameter value can control the smoothness and level of detail of the mesh model 209. When using this method for surface reconstruction, the triangular meshing component 208 can apply an appropriate alpha parameter value to obtain optimal results (e.g., mesh model 209).

[0062] Refer again Figure 2 The texture mapping component 210 maps a two-dimensional image (e.g., attribute data 201) onto the surface of a three-dimensional object (e.g., mesh model 209), thereby enhancing the object's realism. This technique not only increases the level of detail and color features of the object's surface but also improves the rendering effect. The texture mapping operation performed by the texture mapping component 210 is shown in "Algorithm 2" below. In practical applications, texture mapping technology can add more detail and patterns to the surface of a three-dimensional object, thus providing a more realistic visual effect for various scenes. Meanwhile, the vertex normal component 212 can use vertex normals, i.e., the normal vectors at each vertex in the textured mesh 211. Vertex normals can be obtained by calculating the average of the normal vectors of the faces surrounding each vertex. Vertex normals can be used to calculate lighting effects to determine the light intensity and color at each vertex. To ensure the quality of the three-dimensional graphics rendering, the vertex normal component 212 can apply the vertex normals to the textured mesh 211 to generate a three-dimensional representation 213.

[0063] Figure 6A The representations shown according to some embodiments of this disclosure use Figure 3 The exemplary network architecture 300 performs a visual comparison of sparse depth completion with the graph of 600. Figure 6B A graphical representation 625 of RMSE and MAE performance based on depth sparsity according to some embodiments of the present disclosure is shown. Figure 6C A diagram 650 showing a visual comparison of different depth sparsity of a 3D point cloud scene according to some embodiments of the present disclosure is shown. Figure 7 Figure 700 illustrates a point cloud 705 generated from a single RGB-D image (e.g., attribute data 701 and a thinned dense depth map 703) according to some embodiments of the present disclosure, wherein the single RGB-D image uses... Figure 3 The exemplary network architecture is used to generate it. Figure 8 Figure 800 illustrates a point cloud and a 3D mesh structure reconstructed from a single RGB-D image according to some embodiments of the present disclosure, wherein the single RGB-D image is generated by... Figure 3 The exemplary network architecture is used to generate it. Figure 9 A diagram showing a visual comparison 900 of point cloud and mesh model reconstruction results from different perspectives according to some embodiments of the present disclosure is illustrated, wherein the point cloud and mesh model are constructed by... Figure 3 The exemplary network architecture is used to generate it.

[0064] Reference Figure 6AThe accuracy of the video processing system 250 was tested using root mean square error (RMSE) and mean absolute error (MAE) as evaluation metrics to select the optimal depth sparsity for 3D reconstruction. As depth sparsity increased, the visible surfaces in the completed dense depth map became clearer. This result demonstrates that the video processing system 250 can handle sparse depth data of varying degrees with improved performance. This is beneficial for 3D reconstruction tasks that may encounter depth data of varying densities, ranging from extremely dense to extremely sparse. Therefore, the video processing system 250 exhibits robustness and versatility in different scenarios.

[0065] Reference Figure 6B To maintain performance and reduce power consumption during 3D reconstruction, the 3D reconstruction component 206 applied an optimal threshold for depth completion in indoor scenes to address the depth sparsity issue. Therefore, quantitative measurements of sparse depth completion were performed based on depth sparsity data, and the results are summarized in Table 1 below. These results demonstrate a positive correlation between depth sparsity and the accuracy of the depth map. This means that the higher the depth sparsity, the more accurate the completed depth map. At a depth sparsity of 300 (0.7%), the obtained RMSE value is less than 100 mm. This indicates that the exemplary depth completion method implemented by the video processing system 250 can obtain a relatively accurate depth map at this level of depth sparsity.

[0066] Table 1: Quantitative Measurement of Sparse Depth Completion Based on Depth Sparsity Reference Figure 6C The images illustrate a visual comparison of sparse depth completion performed by a video processing system 250 using different depth sparsities. In (a), an RGB image and its corresponding true depth map are shown. In (b), a depth sparsity of 100 points (0.1%) is shown. In (c), a depth sparsity of 300 points (0.4%) is shown. In (d), a depth sparsity of 300 points (0.7%) is shown. In (e), a depth sparsity of 1000 points (1.4%) is shown. In (f), a depth sparsity of 3000 points (4.3%) is shown. In (g), a depth sparsity of 5000 points (7.2%) is shown. In (h), a depth sparsity of 10000 points (14%) is shown. Figure 6C The bottom of each image shows its completed depth map.

[0067] from Figure 6B The experimental results shown indicate that depth sparsity affects RMSE and MAE performance. Through observation... Figure 6BThe slope in the graph shows that the slope is relatively large when the depth sparsity is low. This suggests that the performance of the video processing system 250 may be sensitive to depth sparsity. However, as the depth sparsity increases, the slope gradually decreases, which may indicate that the impact of depth sparsity on performance becomes smaller. This means that as the depth sparsity increases, the performance improvement gradually weakens, and there exists an optimal point for achieving the best performance. Figure 6C The 3D point cloud shown further supports this conclusion.

[0068] For example, refer to Figure 6C The reconstructed point cloud in (d) is clearer in detail than those in (b) and (c), but the improvement in clarity in (e), (f), (g), and (h) is relatively insignificant. Therefore, the depth sparsity of (d) (e.g., 500 points (0.7%)) achieves the best balance between reconstruction accuracy and power consumption in 3D reconstruction. These conclusions are very valuable for practical applications because they can help reduce the power consumption of mobile devices.

[0069] Reference Figure 7 The results of 3D indoor point cloud reconstruction from different perspectives are shown (e.g., point cloud 705). This shows that the 3D reconstruction component 206 can generate point cloud 705 with high accuracy based on attribute data 701 and a refined dense depth map 703 (generated by sparse depth completion network component 204).

[0070] like Figure 8 As shown, the proposed framework generates 3D reconstruction results from a single RGB-D image (e.g., RGB image data 802 and sparse depth map 804) in various scenes. The proposed framework utilizes depth information captured by an RGB-D camera and sparse depth completion techniques to obtain 3D data of the scene. The Open3D library can be used to process and analyze the data, thereby generating various types of 3D structures, including point clouds and meshes. For example, based on a refined dense depth map 806 and RGB image data 802, a 3D point cloud 808, a mesh model 810, a textured mesh 812, and a 3D reconstruction 814 (e.g., a textured mesh with normal vectors) can be generated. Figure 9 As shown, meshes offer greater expressive power and better visualization than point clouds because they can simulate and represent 3D surfaces in more detail. By performing point cloud reconstruction, surface reconstruction, and mesh rendering, the video processing system 250 can generate realistic 3D reconstructed models to represent real-world scenes in a virtual environment.

[0071] Figure 10A flowchart of an exemplary video processing method 1000 according to some embodiments of the present disclosure is shown. Method 1000 may be executed by a system, such as a video processing system 250, a sparse depth completion network component 204, a 3D reconstruction component 206, a triangulation component 208, a texture mapping component 210, or a vertex normal vector component 212, to name just a few. Method 1000 may include operations 1002 to 1012 as described below. It should be understood that some steps may be optional, and some steps may be performed simultaneously or in combination with... Figure 10 Execute in different orders as shown.

[0072] Reference Figure 10 At position 1002, the system can input attribute data associated with the image region and a sparse depth map into the sparse depth completion network. For example, refer to... Figure 2 The attribute data 201 of the two-dimensional image region (e.g., color data, RGB data, reflectance data, intensity data, etc.) and the sparse depth map 203 (also referred to as "depth data" in this document) can be input into the video processing system 250.

[0073] At position 1004, the system can use a sparse depth completion network to generate a refined dense depth map based on attribute data and the sparse depth map. For example, refer to... Figure 3Unlike most parallel dual-branch networks, the sparse depth completion network component 204 includes two decoders (e.g., a first decoder 330a and a second decoder 330b) in the RGB decoder stage to predict more adaptive features after processing by the first encoder 320a. Here, the sparse depth completion network component 204 can leverage the complementarity of the iterative depth maps predicted simultaneously by the color branch and the depth branch, while avoiding the influence of color branch supervision on the fusion of different modes in the decoder-encoder fusion stage. The color stage may include, for example, a first encoder 320a, a first decoder 330a, and a second decoder 330b, while the depth stage may include a second encoder 320b and a third decoder 330c. Attribute data 301 and a sparse depth map 303 (e.g., captured by an RGB-D sensor or an RGB sensor and a LiDAR sensor) can be input to the first encoder 320a. The first decoder 330a can generate a first iterative depth map 340 (and a confidence map) and a multi-affinity matrix 360 based on the input from the RGB stage. On the other hand, the second decoder 330b can be configured to generate multiple adaptive features (e.g., (1) to (5)) after each decoder stage. Each adaptive feature can indicate different inter-pixel relationships between pixels in the sparse depth map 303. The sparse depth map 303, the first iterative depth map 340 (generated by the first decoder 330a), and the multiple adaptive features (generated by the second decoder 330b) can be inputs to the depth branch. Using the multiple adaptive features, the sparse depth completion network component 204 can fully utilize the inter-pixel relationships in the color branch to extract additional depth features. To further refine the depth map, each of the multiple adaptive features is input to a different guided CSPN filter 302 of the second encoder 320b. The third decoder 330c can generate a second iterative depth map 350 (and a confidence map) based on the sparse depth map 303, the first iterative depth map 340, and the multiple adaptive features. A coarse depth map 309 can be generated by element-wise summing the features of the first iterative depth map 340 and the second iterative depth map 350. A guided CSPN filter 302 (e.g., a fusion module) combines CSPN with guided filtering to fuse features captured by the color and depth branches at different fusion stages according to expressions (1) through (3). For example, the guided CSPN filter 302 can predict the dynamic changes of the convolutional kernel from the color branch. To avoid over-smoothing of the refined dense depth map 311 after multiple iterations, the multi-affinity matrix CSPN++ component 370 can apply different affinity matrices to update the coarse depth map 309. Each multi-affinity matrix 360 can be assigned different inter-pixel weights associated with its adaptive features. This makes the pixel values ​​of the refined dense depth map 311 more accurate after each iteration. (Refer to...) Figure 4The multi-affinity matrix can be adaptively generated via convolutional layers using high-level features at the ends of the network architecture backbone 402. When refining the coarse depth map 309, the affinity matrix 460 can be used to iteratively update pixels at the same spatial location. Since the weights of the multi-affinity matrix take into account inter-pixel relationships, it avoids over-smoothing. This increases the sharpness and structural detail of the refined dense depth map 311. Finally, the depth values ​​are refined through distant pixels using dilated convolutions with increased receptive fields. The refined dense depth map 311 generated based on the multi-affinity matrix can be described according to expression (4).

[0074] At position 1006, the system can perform a 3D reconstruction process based on a refined dense depth map to generate a point cloud of the image region. For example, referring to... Figure 2 Indoor scenes typically require close-up image capture within a limited space, such as an office or bedroom. In the real world, cameras capture images based on a pinhole model, meaning the camera maps coordinates in three-dimensional space onto the image plane. This mapping process can be performed by the 3D reconstruction component 206. For example, the 3D reconstruction component 206 can assign each pixel on the image plane to its corresponding point in three-dimensional space. This process can be represented by expression (7). Since the video processing system 250 reconstructs a three-dimensional image region from a single view (e.g., generating a point cloud 207), the 3D reconstruction component 206 can be designed to consider both camera coordinates and world coordinates simultaneously. Therefore, and A matrix is ​​formed as shown in expression (6). By combining depth information and position information, the 3D reconstruction component 206 can utilize their relationship to transform the refined dense depth map 205 into a point cloud 207 (e.g., a 3D point cloud scene) according to expressions (7) to (9).

[0075] At point 1008, the system can perform a triangulation process to generate a mesh model based on the point cloud of the image region. For example, referring to... Figure 2The triangulation component 208 can generate a smooth, continuous representation of a surface (e.g., mesh model 209) for further processing and analysis. Triangulation is a technique that uses a mesh composed of many triangles to represent a surface. In Open3D, alpha shape is a point cloud-based 3D surface reconstruction method that can convert discrete point cloud data into a continuous 3D surface model (e.g., mesh model 209). This method uses alpha parameter values ​​to construct a series of nested surfaces, where the alpha parameter is considered a distance threshold used to construct the surfaces. These operations performed by the triangulation component 208 are summarized above as "Algorithm 1". Typically, the goal of the alpha shape method is to find nested triangular faces that form the edges of the alpha complex. As the alpha parameter value increases, the number of edges in the alpha complex increases, while the number of triangular faces decreases. Therefore, the alpha parameter value can control the smoothness and level of detail of the mesh model 209. When using this method for surface reconstruction, the triangulation component 208 can apply appropriate alpha parameter values ​​to obtain optimal results (e.g., mesh model 209).

[0076] At position 1010, the system can perform a texture mapping process based on the point cloud to generate a textured mesh of the image region. For example, refer to... Figure 2 Texture mapping component 210 maps a two-dimensional image (e.g., attribute data 201) onto the surface of a three-dimensional object (e.g., mesh model 209), thereby enhancing the object's realism. This technique not only increases the level of detail and color features on the object's surface but also improves the rendering effect. The texture mapping operation performed by texture mapping component 210 is shown in "Algorithm 2" below. In practical applications, texture mapping technology can add more detail and patterns to the surface of three-dimensional objects, thus providing more realistic visual effects for various scenes.

[0077] At position 1012, the system can perform vertex normal vector calculation based on a textured mesh to generate a 3D representation of the image region. (See reference...) Figure 2 The vertex normal vector component 212 can use vertex normal vectors, which are the normal vectors at each vertex in the textured mesh 211. Vertex normal vectors can be obtained by calculating the average of the normal vectors of the faces surrounding each vertex. Vertex normal vectors can be used to calculate lighting effects to determine the light intensity and color at each vertex. To ensure the quality of 3D graphics rendering, the vertex normal vector component 212 can apply vertex normal vectors to the textured mesh 211 to generate a 3D representation 213.

[0078] One or more computer systems can be used, for example. Figure 11 The computer system 1100 shown is used to implement various embodiments. One or more computer systems 1100 can be used, for example, to implement Figure 10Method 1000. For example, according to various embodiments, computer system 1100 may generate an enhanced image based on first image data captured by a first image sensor at a first FOV and a first resolution, and second image data captured by a second image sensor at a second FOV and a second resolution. Computer system 1100 may be any computer capable of performing the functions described herein.

[0079] Computer system 1100 can be any known computer capable of performing the functions described herein. Computer system 1100 includes one or more processors (also referred to as central processing units or CPUs), such as processor 1104. Processor 1104 is connected to communication infrastructure 1106 (e.g., a bus). The one or more processors 1104 may each be a graphics processing unit (GPU). In embodiments, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. GPUs may have a parallel architecture that is efficient for parallel processing of large blocks of data, such as common mathematically intensive data in computer graphics applications, images, videos, etc.

[0080] The computer system 1100 also includes user input / output devices 1103, such as monitors, keyboards, pointer devices, etc., which communicate with the communication infrastructure 1106 through user input / output interfaces 1102.

[0081] Computer system 1100 also includes main memory (or main memory) 1108, such as random-access memory (RAM). Main memory 1108 may include level 1 or more caches. Control logic (i.e., computer software) and / or data are stored in main memory 1108. Computer system 1100 may also include one or more secondary storage devices or secondary storage units 1110. Secondary storage unit 1110 may include, for example, hard disk drive 1112 and / or removable storage device or drive 1114. Removable storage drive 1114 may be a floppy disk drive, magnetic tape drive, optical disk drive, optical storage device, tape backup device, and / or any other storage device / drive. Removable storage drive 1114 may interact with removable storage unit 1116. Removable storage unit 1116 includes a computer-usable or readable storage device on which computer software (control logic) and / or data are stored. Removable storage unit 1116 may be a floppy disk, magnetic tape, optical disk, DVD, optical storage disk, and / or any other computer data storage device. The removable storage drive 1114 reads from and / or writes to the removable storage cell 1116 in a manner known in the art.

[0082] According to an exemplary embodiment, secondary storage 1110 may include other means, tools, or other methods to allow computer programs and / or other instructions and / or data to be accessed by computer system 1100. Such means, tools, or other methods may include, for example, removable storage unit 1122 and interface 1120. Examples of removable storage unit 1122 and interface 1120 may include program cartridges and cartridge interfaces (e.g., found in video game devices), removable memory chips (e.g., EPROM or PROM) and associated sockets, memory sticks and universal serial bus (USB) ports, memory cards and associated memory card slots, and / or any other removable storage unit and associated interface.

[0083] Computer system 1100 may further include a communication interface 1124. Communication interface 1124 enables computer system 1100 to communicate and interact with any combination (individually and collectively referred to as 1126) of remote devices, remote networks, remote entities, etc. For example, communication interface 1124 may allow computer system 1100 to communicate with remote device 1126 via communication path 1128, which may be wired and / or wireless, and may include any combination of LAN, WAN, Internet, etc. Control logic and / or data may be transmitted to and from computer system 1100 via communication path 1128.

[0084] In embodiments, tangible devices or articles of art that include a tangible computer-usable or readable medium having control logic (software) stored thereon are also referred to herein as computer program products or program storage devices. This includes, but is not limited to, computer system 1100, main memory 1108, secondary memory 1110, and removable storage units 1116 and 1122, and tangible articles of art embodying any combination thereof. When executed by one or more data processing devices (e.g., computer system 1100), such control logic causes such data processing devices to operate as described herein.

[0085] Based on the teachings contained in this disclosure, those skilled in the art will clearly understand how to use it. Figure 11 The embodiments of this disclosure may be made and used using data processing devices, computer systems, and / or computer architectures other than those shown herein. For example, the embodiments may be implemented using software, hardware, and operating systems other than those described herein.

[0086] In all aspects of this disclosure, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as instructions on a non-transitory computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium accessible by a processor (e.g., the processor of video processing system 250). By way of example and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, HDD (e.g., disk storage devices or other magnetic storage devices), flash memory drives, SSDs, or any other medium that can be used to carry or store desired program code in the form of data structures or instructions and is accessible by a processing system (e.g., a mobile device or computer). As used herein, disks and optical discs include CDs, laser discs, optical discs, digital video discs (DVDs), and floppy disks, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0087] According to one aspect of this disclosure, a video processing method is provided. The method may include: a processor inputting attribute data associated with an image region and a sparse depth map into a sparse depth completion network. The method may include: the processor using the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map. The method may include: the processor performing a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region. The method may include: the processor performing a triangulation process to generate a mesh model based on the point cloud of the image region. The method may include: the processor performing a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. The method may include: the processor performing a vertex normal vector calculation process based on the textured mesh to generate a 3D representation of the image region.

[0088] In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: an encoder generating multiple feature maps based on the attribute data and the sparse depth map of the image region. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: the encoder inputting the multiple feature maps to a first decoder and a second decoder. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: the first decoder generating a first iterative depth map and a multi-affinity matrix of the image region based on the multiple feature maps. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: the second decoder generating multiple adaptive features based on the multiple feature maps.

[0089] In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: inputting the sparse depth map, the first iterative depth map, and the plurality of adaptive features into a third decoder. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, including: the third decoder generating a second iterative depth map and the same multi-affinity matrix based on the sparse depth map, the first iterative depth map, and the plurality of adaptive features.

[0090] In some embodiments, the multi-affinity matrix may assign a first set of inter-pixel weights to the pixels of the image region. In some embodiments, the multi-affinity matrix may assign a second set of inter-pixel weights to the pixels of the image region. In some embodiments, the first set of weights is different from the second set of weights.

[0091] In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, and may further include: generating a coarse dense depth map of the image region based on the first iterative depth map and the second iterative depth map. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, and may further include: inputting the multi-affinity matrix and the coarse dense depth map into CSPN++. In some embodiments, the processor uses the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, and may further include: CSPN++ generating the refined dense depth map based on the multi-affinity matrix and the coarse dense depth map.

[0092] In some embodiments, the processor performs a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, which may include receiving a set of coordinates associated with the image region. In some embodiments, the processor performs a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, which may include mapping the set of coordinates associated with the image region to the refined dense depth map. In some embodiments, the processor performs a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, which may include generating the point cloud based on the mapping from the set of coordinates associated with the image region to the refined dense depth map.

[0093] In some embodiments, the processor performing a triangulation process to generate a mesh model based on a point cloud of the image region may include: using the triangulated mesh model to identify multiple nested surfaces in the point cloud. In some embodiments, the processor performing a triangulation process to generate a mesh model based on a point cloud of the image region may include: generating a mesh model of the point cloud based on the multiple nested surfaces.

[0094] In some embodiments, the mesh model may be a three-dimensional surface model. In some embodiments, the triangular mesh model may apply an alpha parameter to identify the plurality of nested surfaces. In some embodiments, the alpha parameter may be associated with a distance threshold of the three-dimensional surface model.

[0095] In some embodiments, the processor performs a texture mapping process based on the point cloud to generate a textured mesh of the image region, which may include mapping the attribute data to the mesh model to generate the textured mesh.

[0096] In some embodiments, the processor performs a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, which may include: calculating a plurality of normal vectors associated with each vertex in the textured mesh based on the attribute data. In some embodiments, the processor performs a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, which may include: generating a three-dimensional representation of the image region based on the textured mesh and the plurality of normal vectors.

[0097] In some embodiments, the image region may be associated with a picture, a sub-picture, a tile, a slice, or a coding block. In some embodiments, the attribute data may include one or more of color data, reflectance data, or intensity data.

[0098] According to another aspect of this disclosure, a video processing system is provided. The system may include a processor and a memory storing instructions. The memory stores instructions that, when executed by the processor, cause the processor to: input attribute data associated with an image region and a sparse depth map into a sparse depth completion network. The memory stores instructions that, when executed by the processor, cause the processor to: generate a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network. The memory stores instructions that, when executed by the processor, cause the processor to: perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region. The memory stores instructions that, when executed by the processor, cause the processor to: perform a triangulation process to generate a mesh model based on the point cloud of the image region. The memory stores instructions that, when executed by the processor, cause the processor to: perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. The memory stores instructions that, when executed by the processor, enable the processor to perform the following operations: based on the textured mesh, perform a vertex normal vector calculation process to generate a three-dimensional representation of the image region.

[0099] In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: the encoder generates a plurality of feature maps based on the attribute data and the sparse depth map of the image region. In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: the encoder inputs the plurality of feature maps to a first decoder and a second decoder. In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: the first decoder generates a first iterative depth map and a multi-affinity matrix of the image region based on the plurality of feature maps. In some embodiments, in order to use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, the memory stores instructions that, when executed by at least one processor, can cause the processor to perform the following operation: the second decoder generates a plurality of adaptive features based on the plurality of feature maps.

[0100] In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: inputting the sparse depth map, the first iterative depth map, and the plurality of adaptive features into a third decoder. In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: the third decoder generates a second iterative depth map and a second multiple affinity matrix based on the sparse depth map, the first iterative depth map, and the plurality of adaptive features.

[0101] In some embodiments, the multi-affinity matrix may assign a first set of inter-pixel weights to the pixels of the image region. In some embodiments, the multi-affinity matrix may assign a second set of inter-pixel weights to the pixels of the image region. In some embodiments, the first set of weights may be different from the second set of weights.

[0102] In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, can cause the processor to perform the following operation: generate a coarse dense depth map of the image region based on the first iterative depth map and the second iterative depth map. In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, can cause the processor to perform the following operation: input the multi-affinity matrix and the coarse dense depth map into CSPN++. In some embodiments, to use the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. The memory stores instructions that, when executed by at least one processor, can cause the processor to perform the following operation: CSPN++ generates the refined dense depth map based on the first multi-affinity matrix, the second multi-affinity matrix, and the coarse dense depth map.

[0103] In some embodiments, for performing a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: receive a set of coordinates associated with the image region. In some embodiments, for performing a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: map the set of coordinates associated with the image region to the refined dense depth map. In some embodiments, for performing a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: generate the point cloud based on the mapping from the set of coordinates associated with the image region to the refined dense depth map.

[0104] In some embodiments, in order to perform a triangulation process to generate a mesh model based on a point cloud of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: identify multiple nested surfaces in the point cloud using the triangulated mesh model. In some embodiments, in order to perform a triangulation process to generate a mesh model based on a point cloud of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: generate a mesh model of the point cloud based on the multiple nested surfaces.

[0105] In some embodiments, the mesh model may be a three-dimensional surface model. In some embodiments, the triangular mesh model may apply an alpha parameter to identify the plurality of nested surfaces. In some embodiments, the alpha parameter may be associated with a distance threshold of the three-dimensional surface model.

[0106] In some embodiments, in order to perform the texture mapping process based on the point cloud to generate a textured mesh of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: mapping the attribute data to the mesh model to generate the textured mesh.

[0107] In some embodiments, in order to perform a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: calculate a plurality of normal vectors associated with each vertex in the textured mesh based on the attribute data. In some embodiments, in order to perform a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to perform the following operation: generate a three-dimensional representation of the image region based on the textured mesh and the plurality of normal vectors.

[0108] In some embodiments, the image region may be associated with a picture, sub-picture, tile, stripe, or coded block. In some embodiments, the attribute data may include one or more of color data, reflectance data, or intensity data.

[0109] According to another aspect of this disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium stores instructions. When executed by a processor of a video processing system, the instructions cause the processor to: input attribute data associated with an image region and a sparse depth map into a sparse depth completion network. When executed by a processor of the video processing system, the instructions cause the processor to: use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map. When executed by a processor of the video processing system, the instructions cause the processor to: perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region. When executed by a processor of the video processing system, the instructions cause the processor to: perform a triangulation process to generate a mesh model based on the point cloud of the image region. When executed by a processor of the video processing system, the instructions cause the processor to: perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region. When executed by the processor of the video processing system, the instructions cause the processor to perform the following operations: based on the textured mesh, perform a vertex normal vector calculation process to generate a three-dimensional representation of the image region.

[0110] In some embodiments, to use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operations: the encoder generates multiple feature maps based on the attribute data and the sparse depth map of the image region. In some embodiments, to use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operations: the encoder inputs the multiple feature maps to a first decoder and a second decoder. In some embodiments, to use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operations: the first decoder generates a first iterative depth map and a multi-affinity matrix of the image region based on the multiple feature maps. In some embodiments, in order to use the sparse depth completion network to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, can cause the processor to perform the following operation: the second decoder generates a plurality of adaptive features based on the plurality of feature maps.

[0111] In some embodiments, to utilize the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. When executed by at least one processor, the instruction may cause the processor to perform the following operation: input the sparse depth map, the first iterative depth map, and the plurality of adaptive features into a third decoder. In some embodiments, to utilize the sparse depth completion network, a refined dense depth map is generated based on the attribute data and the sparse depth map. When executed by at least one processor, the instruction may cause the processor to perform the following operation: the third decoder generates a second iterative depth map and the same multi-affinity matrix based on the sparse depth map, the first iterative depth map, and the plurality of adaptive features.

[0112] In some embodiments, the multi-affinity matrix assigns a first set of inter-pixel weights to the pixels of the image region. In some embodiments, the multi-affinity matrix may assign a second set of inter-pixel weights to the pixels of the image region. In some embodiments, the first set of weights may be different from the second set of weights.

[0113] In some embodiments, to use the sparse depth completion network, to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: generate a coarse dense depth map of the image region based on the first iterative depth map and the second iterative depth map. In some embodiments, to use the sparse depth completion network, to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: input the multi-affinity matrix and the coarse dense depth map into CSPN++. In some embodiments, to use the sparse depth completion network, to generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: CSPN++ generates the refined dense depth map based on the multi-affinity matrix and the coarse dense depth map.

[0114] In some embodiments, in order to perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: receive a set of coordinates associated with the image region. In some embodiments, in order to perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: map the set of coordinates associated with the image region to the refined dense depth map. In some embodiments, in order to perform a 3D reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: generate the point cloud based on the mapping from the set of coordinates associated with the image region to the refined dense depth map.

[0115] In some embodiments, in order to perform a triangulation process to generate a mesh model based on a point cloud of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: identify multiple nested surfaces in the point cloud using the triangulated mesh model. In some embodiments, in order to perform a triangulation process to generate a mesh model based on a point cloud of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: generate a mesh model of the point cloud based on the multiple nested surfaces.

[0116] In some embodiments, the mesh model may be a three-dimensional surface model. In some embodiments, the triangular mesh model applies an alpha parameter to identify the plurality of nested surfaces. In some embodiments, the alpha parameter is associated with a distance threshold of the three-dimensional surface model.

[0117] In some embodiments, in order to perform a texture mapping process based on the point cloud to generate a textured mesh of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: map the attribute data to the mesh model to generate the textured mesh.

[0118] In some embodiments, in order to perform a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: calculate a plurality of normal vectors associated with each vertex in the textured mesh based on the attribute data. In some embodiments, in order to perform a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, the instructions, when executed by at least one processor, may cause the processor to perform the following operation: generate a three-dimensional representation of the image region based on the textured mesh and the plurality of normal vectors.

[0119] In some embodiments, the image region may be associated with a picture, sub-picture, tile, stripe, or coded block. In some embodiments, the attribute data may include one or more of color data, reflectance data, or intensity data.

[0120] The foregoing description of the embodiments will thus reveal the general nature of this disclosure, so that others, by applying knowledge of the art, can readily modify and / or adapt such embodiments to suit various applications without departing from the general concept of this disclosure, without excessive experimentation. Therefore, such adaptive adjustments and modifications are intended to be within the meaning and equivalents of the disclosed embodiments. It should be understood that the wording or terminology herein is for descriptive and not limiting purposes, and that the terminology or terminology of this specification should be interpreted by those skilled in the art based on the teachings and guidance.

[0121] The embodiments of this disclosure have been described above using functional building blocks, which illustrate how specified functions and their relationships are implemented. For ease of description, the boundaries of these functional building blocks are arbitrarily defined herein. Alternative boundaries may be defined provided that the specified functions and their relationships are properly performed.

[0122] The summary and abstract sections may set forth one or more, but not all, exemplary embodiments of this disclosure as conceived by the inventors, and therefore are not intended to limit this disclosure and the appended claims in any way.

[0123] Various functional blocks, modules, and steps have been disclosed above. The arrangements provided are illustrative and not limiting. Therefore, functional blocks, modules, and steps can be rearranged or combined in ways different from the examples provided above. Similarly, some embodiments include only a subset of functional blocks, modules, and steps, and any such subset is permitted.

[0124] The breadth and scope of this disclosure should not be limited by any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method of video processing, comprising: inputting, by a processor, attribute data associated with an image region and a sparse depth map to a sparse depth completion network; generating, by the processor, a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network; performing, by the processor, a three-dimensional (3D) reconstruction process based on the refined dense depth map to generate a point cloud for the image region; performing, by the processor, a triangle meshing process to generate a mesh model based on the point cloud for the image region; performing, by the processor, a texture mapping process based on the mesh model and the attribute data to generate a textured mesh for the image region; and performing, by the processor, a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation for the image region. generating, by the processor, a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network, comprises:

2. The method of claim 1, wherein, generating, by an encoder, a plurality of feature maps based on the attribute data and the sparse depth map for the image region; inputting, by the encoder, the plurality of feature maps to a first decoder and a second decoder; generating, by the first decoder, a first iterative depth map and a plurality of affinity matrices for the image region based on the plurality of feature maps; and generating, by the second decoder, a plurality of adaptive features based on the plurality of feature maps. generating, by the processor, a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network, further comprises:

3. The method of claim 2, wherein, inputting the sparse depth map, the first iterative depth map, and the plurality of adaptive features to a third decoder; and generating, by the third decoder, a second iterative depth map and the same plurality of affinity matrices based on the sparse depth map, the first iterative depth map, and the plurality of adaptive features.

4. The method of claim 3, wherein: the plurality of affinity matrices assign a first set of inter-pixel weights to pixels of the image region, the plurality of affinity matrices assign a second set of inter-pixel weights to pixels of the image region, and the first set of weights is different from the second set of weights. generating, by the processor, a refined dense depth map based on the attribute data and the sparse depth map using the sparse depth completion network, further comprises:

5. The method of claim 3, wherein, generating a coarse dense depth map for the image region based on the first iterative depth map and the second iterative depth map; inputting the plurality of affinity matrices and the coarse dense depth map to a convolution space propagation network (CSPN++); and generating, by the CSPN++, the refined dense depth map based on the plurality of affinity matrices and the coarse dense depth map. performing, by the processor, a three-dimensional reconstruction process based on the refined dense depth map to generate a point cloud for the image region, comprises:

6. The method of claim 1, wherein, receiving a set of coordinates associated with the image region; mapping the set of coordinates associated with the image region to the refined dense depth map; and generating the point cloud based on the mapping of the set of coordinates associated with the image region to the refined dense depth map. ​ 7. The method of claim 1, wherein, The processor performs a triangle meshing process to generate a mesh model based on the point cloud of the image region, including: identifying a plurality of nested surfaces in the point cloud using a triangle mesh model; and generating a mesh model of the point cloud based on the plurality of nested surfaces.

8. The method of claim 7, wherein: the mesh model is a three-dimensional surface model, the triangle mesh model applies an alpha parameter to identify the plurality of nested surfaces, and the alpha parameter is associated with a distance threshold of the three-dimensional surface model.

9. The method of claim 1, wherein, The processor performs a texture mapping process based on the point cloud to generate a textured mesh of the image region, including: mapping the attribute data to the mesh model to generate the textured mesh.

10. The method of claim 1, wherein, The processor performs a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region, including: calculating a plurality of normal vectors associated with each vertex in the textured mesh based on the attribute data; and generating a three-dimensional representation of the image region based on the textured mesh and the plurality of normal vectors.

11. The method of claim 1, wherein: the image region is associated with a picture, sub-picture, tile, slice, or coding block, and the attribute data includes one or more of color data, reflectance data, or intensity data.

12. A video processing system, comprising: a processor; and a memory storing instructions that, when executed by at least one processor, cause the processor to: input attribute data and a sparse depth map associated with an image region to a sparse depth completion network; generate, using the sparse depth completion network, a refined dense depth map based on the attribute data and the sparse depth map; perform a three-dimensional (3D) reconstruction process based on the refined dense depth map to generate a point cloud of the image region; perform a triangle meshing process to generate a mesh model based on the point cloud of the image region; perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region; and perform a vertex normal vector calculation process based on the textured mesh to generate a three-dimensional representation of the image region. To generate, using the sparse depth completion network, a refined dense depth map based on the attribute data and the sparse depth map, the memory stores instructions that, when executed by at least one processor, cause the processor to: generate, by an encoder, a plurality of feature maps based on the attribute data and the sparse depth map of the image region; 13. The system of claim 12, wherein, input, by the encoder, the plurality of feature maps to a first decoder and a second decoder; generate, by the first decoder, a first iterative depth map and a plurality of affinity matrices of the image region based on the plurality of feature maps; and generate, by the second decoder, a plurality of adaptive features based on the plurality of feature maps. ​ ​ ​ 14. The system of claim 13, wherein, To use the sparse depth completion network, generate a refined dense depth map based on the attribute data and the sparse depth map, the memory storing instructions that, when executed by at least one processor, cause the processor to: input the sparse depth map, the first iteration depth map, and the plurality of affinity matrices to a third decoder; and the third decoder generates a second iteration depth map and the same plurality of affinity matrices based on the sparse depth map, the first iteration depth map, and the plurality of affinity matrices.

15. The system of claim 14, wherein: the plurality of affinity matrices assign a first set of inter-pixel weights to pixels of the image region, the plurality of affinity matrices assign a second set of inter-pixel weights to pixels of the image region, and the first set of weights is different from the second set of weights.

16. The system of claim 14, wherein, To use the sparse depth completion network, generate a refined dense depth map based on the attribute data and the sparse depth map, the memory storing instructions that, when executed by at least one processor, cause the processor to: generate a coarse dense depth map of the image region based on the first iteration depth map and the second iteration depth map; input the plurality of affinity matrices and the coarse dense depth map to a convolution space propagation network (CSPN++); and the CSPN++ generates the refined dense depth map based on the plurality of affinity matrices and the coarse dense depth map.

17. The system of claim 12, wherein, To perform a three-dimensional reconstruction process based on the refined dense depth map to generate a point cloud of the image region, the memory storing instructions that, when executed by at least one processor, cause the processor to: receive a set of coordinates associated with the image region; map the set of coordinates associated with the image region to the refined dense depth map; and generate the point cloud based on the mapping of the set of coordinates associated with the image region to the refined dense depth map.

18. The system of claim 12, wherein, To perform a triangular meshing process to generate a mesh model based on the point cloud of the image region, the memory storing instructions that, when executed by at least one processor, cause the processor to: identify a plurality of nested surfaces in the point cloud using a triangular mesh model; and generate a mesh model of the point cloud based on the plurality of nested surfaces.

19. The system of claim 18, wherein: the mesh model is a three-dimensional surface model, the triangular mesh model applies an alpha parameter to identify the plurality of nested surfaces, and the alpha parameter is associated with a distance threshold of the three-dimensional surface model.

20. The system of claim 12, wherein, To perform a texture mapping process based on the point cloud to generate a textured mesh of the image region, the memory storing instructions that, when executed by at least one processor, cause the processor to: map the attribute data to the mesh model to generate the textured mesh.

21. The system of claim 12, wherein, To perform a vertex normal computation process based on the textured mesh to generate a three-dimensional representation of the image region, the memory stores instructions that, when executed by at least one processor, cause the processor to: compute, based on the attribute data, a plurality of normal vectors associated with each vertex in the textured mesh; and generate, based on the textured mesh and the plurality of normal vectors, a three-dimensional representation of the image region.

22. The system of claim 12, wherein: the image region is associated with a picture, a sub-picture, a tile, a slice, or a coding block, and the attribute data includes one or more of color data, reflectance data, or intensity data.

23. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a video processing system, cause the processor to: input attribute data and a sparse depth map associated with an image region to a sparse depth completion network; generate, using the sparse depth completion network, a refined dense depth map based on the attribute data and the sparse depth map; perform a three-dimensional (3D) reconstruction process based on the refined dense depth map to generate a point cloud of the image region; perform a triangle meshing process to generate a mesh model based on the point cloud of the image region; perform a texture mapping process based on the mesh model and the attribute data to generate a textured mesh of the image region; and perform a vertex normal computation process based on the textured mesh to generate a three-dimensional representation of the image region.

24. The non-transitory computer-readable medium of claim 23, wherein, To generate, using the sparse depth completion network, a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, cause the processor to: generate, by an encoder, a plurality of feature maps based on the attribute data and the sparse depth map of the image region; input, by the encoder, the plurality of feature maps to a first decoder and a second decoder; generate, by the first decoder, a first iterative depth map and a plurality of affinity matrices of the image region based on the plurality of feature maps; and generate, by the second decoder, a plurality of adaptive features based on the plurality of feature maps.

25. The non-transitory computer-readable medium of claim 24, wherein, To generate, using the sparse depth completion network, a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, cause the processor to: input the sparse depth map, the first iterative depth map, and the plurality of adaptive features to a third decoder; and generate, by the third decoder, a second iterative depth map and the same plurality of affinity matrices based on the sparse depth map, the first iterative depth map, and the plurality of adaptive features.

26. The non-transitory computer-readable medium of claim 25, wherein: the plurality of affinity matrices assign a first set of inter-pixel weights to pixels of the image region, the plurality of affinity matrices assign a second set of inter-pixel weights to pixels of the image region, and the first set of weights is different from the second set of weights.

27. The non-transitory computer-readable medium of claim 25, wherein, To generate a refined dense depth map based on the attribute data and the sparse depth map, the instructions, when executed by at least one processor, cause the processor to: generate a coarse dense depth map for the image region based on the first iteration depth map and the second iteration depth map; input the multi-affinity matrix and the coarse dense depth map to a convolution space propagation network (CSPN++); and the CSPN++ generates the refined dense depth map based on the multi-affinity matrix and the coarse dense depth map.

28. The non-transitory computer-readable medium of claim 23, wherein, To perform a three-dimensional reconstruction process based on the refined dense depth map to generate a point cloud for the image region, the instructions, when executed by at least one processor, cause the processor to: receive a set of coordinates associated with the image region; map the set of coordinates associated with the image region to the refined dense depth map; and generate the point cloud based on the mapping of the set of coordinates associated with the image region to the refined dense depth map.

29. The non-transitory computer-readable medium of claim 23, wherein, To perform a triangle meshing process to generate a mesh model based on the point cloud for the image region, the instructions, when executed by at least one processor, cause the processor to: identify a plurality of nested surfaces in the point cloud using a triangle mesh model; and generate a mesh model for the point cloud based on the plurality of nested surfaces.

30. The non-transitory computer-readable medium of claim 29, wherein: the mesh model is a three-dimensional surface model, the triangle mesh model applies an alpha parameter to identify the plurality of nested surfaces, and the alpha parameter is associated with a distance threshold of the three-dimensional surface model.

31. The non-transitory computer-readable medium of claim 23, wherein, To perform a texture mapping process based on the point cloud to generate a textured mesh for the image region, the instructions, when executed by at least one processor, cause the processor to: map the attribute data to the mesh model to generate the textured mesh.

32. The non-transitory computer-readable medium of claim 23, wherein, To perform a vertex normal vector computation process based on the textured mesh to generate a three-dimensional representation of the image region, the instructions, when executed by at least one processor, cause the processor to: compute a plurality of normal vectors associated with each vertex in the textured mesh based on the attribute data; and generate the three-dimensional representation of the image region based on the textured mesh and the plurality of normal vectors.

33. The non-transitory computer-readable medium of claim 23, wherein: the image region is associated with a picture, a sub-picture, a tile, a slice, or a coding block, and the attribute data includes one or more of color data, reflectance data, or intensity data.

Images