Mobile monocular three-dimensional scene reconstruction system based on a computing offloading mechanism

Abstract

The application relates to a mobile terminal monocular three-dimensional scene reconstruction system based on a computing offloading mechanism, which comprises the following steps: a user clicks a button to control the mobile terminal to start scanning and collecting three-dimensional reconstruction data, and the three-dimensional reconstruction data is screened. Effective data is used to process and transmit the current obtained RGB pixel image to a server, pre-processing is carried out, and adaptive data is obtained. The effective data is subjected to neural network inference to obtain cube granularity incremental grid model data. Analysis is carried out, a dictionary from a cube number to corresponding grid data is updated, a linear cache for GPU rendering is incrementally updated, and the three-dimensional reconstruction effect is further updated. When the next screen refreshes, the screen displays the grid model update effect of the current three-dimensional reconstruction, the user adjusts the scanning process or angle by himself / herself, continues to generate effective data, and clicks to end the reconstruction. The system can simultaneously meet the requirements of low cost, simple use, real-time interaction, low bandwidth requirement and the like.

Description

Technical Field

[0001] This invention relates to a mobile monocular 3D scene reconstruction system based on a computational offloading mechanism, belonging to the field of computer vision technology. Background Technology

[0002] 3D scene reconstruction is a crucial task in computer vision, with applications in education, gaming, real estate, and urban planning. For example, the popular augmented reality / virtual reality (AR / VR) home viewing applications allow users to view the interior 3D details of a house, obtain information such as size and layout, and even plan their interior design style and preview the finished look, all while receiving AI explanations. This eliminates the need for real estate staff to take users on site visits, saving both time for users and manpower for real estate companies. To realize such AR / VR home viewing applications, 3D scene reconstruction of the building structure is essential. However, real estate professionals may lack the knowledge of 3D scene reconstruction, and some small companies or individuals may not want to purchase relatively expensive professional equipment (such as Microsoft's Kinect sensor or Intel's REALSENSE depth camera) or bear the additional learning costs. Similar challenges exist in other industries; factors such as expertise, cost, and ease of use pose significant obstacles for potential users of 3D scene reconstruction to access and utilize its results.

[0003] Most 3D scene reconstruction methods either rely on hardware support such as RGB-D or LiDAR cameras, or, while they can use only a monocular camera (i.e., a widely used RGB camera with only one lens, capturing data containing camera pose information but not depth information) to calculate depth and reconstruct scene surfaces, they are limited by reconstruction quality and computational resources. Thanks to the introduction of deep learning methods, there are now methods that can perform real-time, high-quality 3D scene reconstruction using monocular camera data. A representative work is NeuralRecon, which skips the step of estimating and fusing depth maps, directly reconstructing the local surfaces corresponding to each video segment, and using gated recurrent units (GRUs) to fuse all local surfaces into a whole. In terms of results, NeuralRecon can already produce good reconstruction results in real time using monocular camera data on servers with strong computing power, enough to reduce the cost of 3D scene reconstruction to an acceptable level for the general public. However, existing 3D scene reconstruction methods, including NeuralRecon, focus on continuously improving reconstruction results on existing datasets (such as ScanNet), neglecting potential problems when popularizing 3D reconstruction methods to the general public, such as how to design a 3D scene reconstruction system that is widely usable. These methods typically require offline 3D reconstruction after user data collection. Users cannot see the reconstruction results of the 3D model in real time during data collection; without immediate feedback, users can only rely on their experience to scan. Imagine a real estate professional performing 3D scene scanning and reconstruction for an AR / VR home viewing application for the first time. They drive dozens of kilometers from their company to the property, but due to lack of experience, they can only scan blindly. Upon returning to the company, they discover that the reconstructed 3D model has many incomplete and missing parts, forcing them to drive dozens of kilometers again to rescan. This undoubtedly results in a poor user experience for 3D scene reconstruction.

[0004] To lower the barrier to entry for 3D scene reconstruction methods and ensure satisfactory reconstruction results for ordinary users, a comprehensive 3D scene reconstruction system covering the entire process from data acquisition to 3D reconstruction and final model output is essential. In terms of effectiveness, users should receive timely feedback or guidance during data acquisition, be aware of the reconstruction progress or intermediate results, and thus control the data acquisition process to ultimately obtain a satisfactory 3D model. Thanks to the application of Simultaneous Localization and Mapping (SLAM) technology on smartphones and tablets, even entry-level devices can acquire data suitable for 3D scene reconstruction, eliminating the need for ordinary users to purchase specialized equipment such as depth cameras. However, smartphones and tablets, as mobile devices, have relatively limited computing power and cannot complete the entire reconstruction process on their own. Therefore, collaboration with edge servers is necessary to offload the heavy computational tasks of 3D scene reconstruction to edge servers, fully utilizing their relatively abundant computing and storage resources. Introducing such an edge-end collaboration model undoubtedly brings new challenges: First, although it is known that the main tasks of the mobile device and the server are data acquisition and 3D reconstruction, respectively, the design of the entire system pipeline requires a more detailed task allocation. For example, it is necessary to consider whether the mobile device should undertake some computing tasks, and responsibilities should be reasonably assigned to the edge and the device. Second, offloading computing means that data needs to be exchanged between the mobile device and the edge server via the network, consuming bandwidth resources. To ensure system availability, the amount of data must be controlled within a range acceptable under laboratory or even commercial conditions. Finally, to meet user experience requirements, the system should not only accurately reflect the reconstruction progress when interacting with users, but also reduce response latency and ensure real-time performance. Summary of the Invention

[0005] Purpose of the invention: In view of the above-mentioned existing problems and shortcomings, the purpose of this invention is to provide a mobile monocular 3D scene reconstruction system based on a computational offloading mechanism.

[0006] Technical solution: To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A mobile monocular 3D scene reconstruction system based on a computational offloading mechanism includes the following steps: Step 1: The user clicks the button to control the mobile device to start scanning, and the mobile device continuously changes the collected 3D reconstruction data; Step 2: The mobile device filters the 3D reconstruction data obtained in Step 1. Data whose offset angle from valid data exceeds a threshold is considered valid, while other data is discarded. Step 3: Use the valid data obtained in Step 2 to scale the current RGB pixel image and perform JPEG encoding, then transmit it to the server via TCP; Step 4: The server preprocesses the data obtained in Step 3 to obtain the adapted data; Step 5: Perform neural network inference on the valid data obtained in Step 4 and update the TSDF data; Step 6: Convert the sparse TSDF data output from the neural network inference in each step 5 into a dense TSDF volume representation to obtain the vertices where the TSDF changes and the corresponding small cubes, i.e., the incremental grid model data at the cube granularity. Step 7: Using the cube number allocator in the incremental mesh model data of cube granularity obtained in Step 6, map the triplet coordinates of each cube to an integer number, then pack the cubes to be deleted and the cubes to be added or modified into byte data, and transmit them to the mobile terminal via TCP. Step 8: After receiving the byte data from Step 7, the mobile device parses it, updates a dictionary from cube number to corresponding mesh data, and incrementally updates the linear cache used for GPU rendering, thereby updating the 3D reconstruction effect; Step 9: The next time the screen refreshes, it displays the updated effect of the current 3D reconstructed mesh model. The user can adjust the scanning process or angle. Steps 1 and 2 continue to generate valid data until the user clicks to end the reconstruction.

[0007] Furthermore, the mobile device used in step 1 is ARKit or ARCore, and the 3D reconstruction data is the current RGB pixel image, camera position and pose, and camera internal parameters.

[0008] Furthermore, in step 3, the RGB pixel image is scaled to a size of 640*360mm.

[0009] Furthermore, the preprocessing in step 4 includes JPEG decoding of the image, conversion of the camera's position and pose data from the physical world coordinate system to the coordinate system under the perspective of the specific 3D scene reconstruction method, and other adaptation processing necessary for the input data.

[0010] Furthermore, in step 6, the incremental grid model data of the cube granularity is generated by the marching cubes algorithm, which contains point and face information and the coordinates of small cubes. If it is a small cube that needs to be deleted from the grid model, only its coordinates are recorded.

[0011] Furthermore, in step 9, if the user does not scan within the scanning time or the scanning speed is too fast when adjusting the scan, the mobile device will provide a corresponding prompt.

[0012] Furthermore, in step 8, the storage space occupied by each cube in the linear cache of GPU rendering is length, which is length = faceCnt * 3 * 6 * 4.

[0013] Furthermore, the threshold in step 2 is 15 degrees.

[0014] Beneficial effects: Compared with the prior art, the present invention has the following advantages: In terms of workflow: This system is designed with a complete pipeline from mobile device pose image data acquisition to server-side 3D reconstruction, and then from server-side incremental mesh generation to mobile device incremental mesh rendering, meeting the user's need to collect complete data during real-time online interaction with the system. The system includes a mobile application and a server process. After the user clicks the start button in the application, the mobile device starts collecting and filtering pose image data. Data that meets the criteria is transmitted to the server via the network. After preprocessing, it is reconstructed by NeuralRecon to obtain a sparse truncated signed distance field (TSDF). Then, it goes through steps such as the cubic-granularity moving cubes (MC) algorithm and cube numbering to obtain cubic-granularity mesh data, which is transmitted to the mobile device for rendering via the network. Through the rendering results on the mobile device, the user can intuitively see the currently reconstructed mesh model, discover areas that have not been scanned or have omissions, and then adjust the data acquisition process themselves. The mobile device also provides prompts to the user to move or slow down the scanning speed.

[0015] In terms of features: Although the system described in this patent is based on NeuralRecon, it is applicable to a class of 3D scene reconstruction methods represented by NeuralRecon that use monocular camera data as input and TSDF data as output, including recent methods that directly regress TSDF from RGB images with pose (such as Atlas and VoRTX) and traditional methods that fuse depth maps after depth estimation (such as DeepVideoMVS). Therefore, the system described in this patent can also serve as an online experimental platform for such methods.

[0016] It is worth emphasizing that the system described in this patent can first and foremost meet the requirements of low cost, ease of use, real-time interaction, and low bandwidth requirements when ordinary users use the 3D scene reconstruction method. Attached Figure Description

[0017] Figure 1 This is a system flowchart of the present invention; Figure 2 This is a schematic diagram of the target TSDF data and synchronous TSDF data in an embodiment of the present invention; Figure 3 This is a schematic diagram of the byte data packaging format of the incremental grid model according to an embodiment of the present invention; Figure 4 These are schematic diagrams illustrating two methods for rendering triangular mesh models according to embodiments of the present invention; Figure 5 This is a screenshot of the mobile terminal of the system according to an embodiment of the present invention during operation. In the image: the left image shows the system prompting the user to "move" when it detects that the user has not moved for a period of time; the right image is a screenshot of the system during the user's normal scanning process. Figure 6 This describes the mesh model effect when different tolerance thresholds are selected in the embodiments of the present invention; Figure 7 This is an embodiment of the present invention. Figure 6 The total amount of intermediate data corresponding to the grid model when different tolerance thresholds are selected. Implementation

[0018] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0019] The main problems solved by the system described in this patent include two aspects: 1) clarifying the functional modules of the mobile terminal and the server respectively, and establishing the operation pipeline of the entire system; 2) eliminating various redundancies and performing special optimizations based on 1, so that the system meets the requirements in terms of feedback quality, running speed and bandwidth requirements.

[0020] The problem will be analyzed from two aspects: I. System Operation Pipeline: The uplink portion of the system pipeline, where the mobile device transmits pose and image data to the server, is relatively straightforward. The key consideration lies in the downlink portion: how the server provides feedback or guidance to the mobile device. One approach is for the server to periodically check the reconstructed portion of the model and transmit the pose of the next scan point to the mobile device, prompting the user to move and supplement data via text or images. However, this method lacks interactivity and is not intuitive enough for the user, failing to give them sufficient control over the scanning process. Another approach is for the server to directly transmit the intermediate reconstruction results—a detailed and visually appealing mesh model—to the mobile device during the 3D reconstruction process. Users can then directly compare the differences between the real-world scene and the reconstructed scene, making corresponding adjustments to the scanning process. Since the mesh model seen by the user is, or very close to, the final mesh model, users can stop scanning when they are satisfied with the reconstruction results, thus meeting the data integrity needs of different users.

[0021] There are several approaches to transmitting detailed, visual mesh models to mobile devices. One approach is to not transmit the actual mesh model, but instead, the server acquires the mobile device's current viewpoint in real time, renders the mesh model from that viewpoint as a 2D planar image, and then transmits the data to the mobile device as a video stream. The mobile device actually receives an image that can be directly displayed on the screen. However, this method suffers from severe latency issues. For example, if the mobile device sends its current pose to the server at a certain moment, and only receives the corresponding 2D mesh model image a second later, by which time the mobile device has already changed its viewpoint, this will cause a misalignment between the real-world image on the screen and the 3D mesh model image. Even though some trajectory prediction algorithms exist to address this problem, they cannot completely eliminate it. Secondly, instead of transmitting the actual mesh model, the TSDF is transmitted to the mobile device, which then generates and renders the mesh model. However, this obviously places a significant burden on mobile devices with limited computing power. Thirdly, the actual mesh model is transmitted. The server uses the marching cubes algorithm to generate the mesh model each time the intermediate TSDF of the reconstruction is updated, and then directly transmits the mesh model to the mobile device for rendering. However, such a mesh model will gradually increase in size with the scanning scale until it exceeds the acceptable transmission time and bandwidth. Fourthly, based on the third approach, 3D data encoding and decoding tools such as Draco are applied. Although this can relatively reduce the amount of data transmitted, it cannot fundamentally solve the problem of the increasing size of the transmitted mesh model and will introduce additional encoding and decoding latency.

[0022] In summary, designing a pipeline for a mobile 3D scene reconstruction system based on a computational offloading mechanism that meets expectations faces the following challenges: Model consistency. The mesh model rendered on the mobile screen should be exactly the same as or very close to the mesh model generated after 3D reconstruction on the server, reflecting consistent shape features and allowing users to clearly see the refined 3D reconstruction results.

[0023] Low latency. The latency between sending valid 3D reconstruction data from the mobile device to the server and receiving the corresponding response from the server on the mobile device should be low enough that the real-world image displayed on the mobile device screen and the 3D mesh model image should change synchronously.

[0024] The load should be reasonable. Mobile devices and servers should be allocated appropriate loads, taking into account the computing power of each device, the power consumption and battery life of the mobile device, and the network's data transmission capacity.

[0025] No redundancy. The reason why the third and fourth approaches mentioned above are not feasible is that the transmitted data contains a lot of redundancy. Much data that has been reconstructed and no longer changes is repeatedly transmitted. Eliminating this redundancy can solve the problem of the transmitted grid model increasing with the scan size.

[0026] As mentioned earlier, there is currently no work addressing the application requirements of 3D scene reconstruction methods in edge-end collaborative systems based on computational offloading mechanisms, while the work of this patent will solve these problems for the first time.

[0027] II. Elimination of data redundancy: Once the system design challenges are identified, the most difficult and critical aspect lies in eliminating data redundancy. This redundancy can occur during network transmission, mesh model fusion, and mesh model rendering. Due to redundancy, the system's performance is likely to decline continuously as the scale of 3D reconstruction data scanning increases, potentially rendering it unusable. Specifically, eliminating data redundancy presents three main challenges: Redundancy in transmitting mesh models that are no longer updated. During the same scan, after the user finishes scanning area A, they start scanning area B. At this point, the server's 3D reconstruction result for area A has not changed. If the server transmits the global reconstruction result (which includes both areas A and B) to the mobile device after performing 3D reconstruction of area B, it will cause redundancy in the transmitted data.

[0028] Redundancy of mesh models within the same area at different times. Even if the server transmits mesh models divided by region to the mobile device, for example, the server only transmits the mesh model of region A after rebuilding region A, and only transmits the mesh model of region B after rebuilding region B, at the boundary between regions A and B, if the fusion method of the two mesh models is simply superposition, then mesh models from different times within the same area will coexist, which not only brings redundancy but may also confuse users.

[0029] Redundancy during render cache updates. When the mesh model data from the server arrives on the mobile device, due to deletion and modification operations and the fact that GPU rendering requires linear cache input, simply reassembling all the data into a linear render cache involves mesh model data that is no longer being updated. This causes the assembly time to increase as the size of the mesh model grows, ultimately resulting in unacceptable time overhead.

[0030] This method proposes a series of incremental algorithms, including a cubic-granularity marching cubes algorithm, a difference-comparison marching cubes algorithm, and an incremental update algorithm for the rendering cache, which addresses the aforementioned challenges in this part.

[0031] In short, the main contents of this patent consist of two parts: A workflow for a mobile monocular 3D scene reconstruction system based on a computational offloading mechanism was designed. It simultaneously addresses challenges such as model consistency, low latency, reasonable load, and no redundancy. Through a user-friendly human-computer interaction, it enables ordinary users without background knowledge to use state-of-the-art 3D scene reconstruction methods.

[0032] For the aforementioned pipelines, a series of supporting algorithms and grid model data transmission formats were designed to minimize system runtime overhead and required network bandwidth resources, achieving satisfactory response speed and model reconstruction feedback quality for users.

[0033] Detailed system design overview: Figure 1 The workflow of the system described in this patent is illustrated. The system consists of a mobile device (smartphone or tablet) and a server device (GPU server or workstation). The following steps describe the operation of the system: The user clicks a button to start scanning. Using AR development frameworks provided by mobile operating systems such as ARKit or ARCore, the mobile device can continuously collect data that can be used for 3D reconstruction, including RGB pixel images, camera position and pose, and camera internal parameters at the current moment. This data is collectively referred to as a frame of data corresponding to that moment. After a frame of data is collected, the mobile device filters it. Only when the change in camera movement distance or camera offset angle from the previous frame of "valid data" exceeds a certain threshold will it be selected as a new frame of valid data and the subsequent process will continue. Data that is not considered valid will be discarded directly. The selected frame of valid data will first be scaled (usually cropped to 640*360 pixels) and JPEG encoded, and then transmitted to the server via TCP. After receiving the data, the server first performs data preprocessing, including JPEG decoding of the image, conversion of the camera's position and pose data from the physical world coordinate system to the coordinate system under the perspective of the specific 3D scene reconstruction method, and other adaptation processing necessary for the input data. After collecting a sufficient amount of valid data, the server performs a neural network inference and updates the TSDF data accordingly. Generally, only the TSDF within a specific region changes. Imagine a fixed-size cube enclosing this region, and then dividing this large space (hereinafter referred to as the TSDF volume) into smaller cubes. Each vertex of a small cube has a TSDF value representing its distance from the surface of the topological shape. Each neural network inference changes only the TSDF values ​​at the vertices of a small cube within that region, and not all vertex TSDF values ​​change. After each neural network inference, we first convert the output sparse TSDF data into a dense TSDF volume representation. Then, we only consider the vertices whose TSDF changes and their corresponding small cubes. For these small cubes, we generate mesh data containing point and face information and the small cube coordinates using the improved marching cubes algorithm described later. If a small cube needs to have its mesh model deleted, we only record its coordinates. After obtaining the incremental mesh model data at the cube granularity, the triplet coordinates of each cube are mapped to an integer number through the cube number allocator. Then, the cubes to be deleted and the cubes to be added or modified are packaged into byte data and transmitted to the mobile terminal via TCP. After receiving the data, the mobile device parses it, updates a dictionary from cube number to corresponding grid data, and incrementally updates the linear cache used for GPU rendering. On the next screen refresh, users can visually see the current 3D reconstructed mesh model and adjust the scanning process or angle accordingly, causing steps 1 and 2 to continue generating valid data. If the user does not scan within a certain time (e.g., 5 seconds) or the scanning speed is too fast, the mobile device will provide a corresponding prompt. When the user is satisfied with the 3D reconstruction results, they can click the button to stop data acquisition.

[0034] Having clarified the pipeline of the system described in this patent, the key algorithms and data structures will be explained in detail below, including the cubic granularity marching cubes algorithm, the difference comparison marching cubes algorithm, the packing (transmission) format of mesh model data, and the incremental rendering cache update algorithm.

[0035] Both the cube-granularity marching cubes algorithm and the difference-comparison marching cubes algorithm are improvements on the classic marching cubes algorithm proposed by Lorensen et al., applied respectively when obtaining TSDF data during the first inference of the 3D scene reconstruction method and when updating TSDF data in all subsequent inference rounds. In the classic marching cubes algorithm, TSDF is input into the algorithm as a dense 3D matrix. This 3D matrix can be viewed as a space containing a series of small cubes, each with a TSDF value representing the distance to the topological shape surface at its vertex. The algorithm traverses all small cubes, determining the shape of the mesh model (triangles) within each small cube by the sign and magnitude of the TSDF values ​​of the 8 vertices on each small cube, and calculating data including triangle vertex coordinates, normal vectors, and vertex numbers that make up the triangles. However, all the data calculated by the classic marching cubes algorithm is global data; that is, the vertex coordinates of the triangles are global coordinates, and the vertex numbers that make up the triangles are also global numbers, making it inconvenient to perform deletion or update operations at a finer granularity.

[0036] To eliminate spatial and temporal redundancy, we aim to manipulate the mesh model at a finer granularity. Fortunately, triangles generated by the classic marching cubes algorithm are confined to a single small cube; triangles do not span two or more small cubes. Therefore, in addition to the vertex coordinates, vertex normals, vertex numbers of the triangles, and vertex values ​​for visualization, we add the cube coordinates of each triangle to the output of the classic marching cubes algorithm. This allows us to access all triangles within the cube using the triplet coordinates, making it possible to add, delete, or modify the mesh model at the cube level. This is the cube-level marching cubes algorithm, which accepts a single 3D matrix (TSDF volume) as input. Furthermore, to ensure that the intermediate mesh model generated by this algorithm is consistent with the mesh model generated by applying the classic marching cubes algorithm to the final TSDF from the 3D reconstruction, upper and lower bounds of the TSDF data in three dimensions must be provided. Cubes outside these bounds are invalid cubes and cannot generate a mesh model even if they are inside the TSDF volume.

[0037] The cubic-level marching cubes algorithm is only applied after the first inference in the 3D scene reconstruction method. This is because there was no TSDF data prior to this, and the first inference only adds TSDF data without any deletions or modifications. However, after the first inference, TSDF data may already exist in the region involved in the inference, and operations such as adding, deleting, and modifying vertex TSDF values ​​may occur. Therefore, the marching cubes algorithm with a single TSDF volume as input is no longer used. Instead, a marching cubes algorithm that compares the differences between two different TSDF volumes representing the same region before and after inference is used as input.

[0038] The difference-comparison marching cubes algorithm also outputs data at the cube level, but it requires two identical 3D matrices as input. These matrices represent the TSDF volume before inference (used for synchronization with the mobile mesh model) and the TSDF volume after inference (to make the mobile mesh model more similar). The algorithm maintains a table with the same shape as these two TSDF volumes, recording whether the change in the TSDF value of each vertex is sufficient to warrant modification. Based on our observations, assuming the TSDF values ​​of the eight vertices of a small cube remain unchanged, small changes in the TSDF values ​​do not significantly affect the shape of the generated triangles. Therefore, when the change in the TSDF value is sufficiently small, a mesh model can be avoided for the small cube, thus reducing the amount of data generated and transmitted. Similar to the classic marching cubes algorithm and the cube-level marching cubes algorithm, the difference-comparison marching cubes algorithm still iterates through the small cubes one by one, but it records the deletion of triangles in each small cube. A mesh model is only generated for the cube if there is a vertex among its eight vertices whose TSDF value change is sufficient to warrant modification. Furthermore, the algorithm will only record enough TSDF data to warrant modification for updating the TSDF volume synchronized with the mobile mesh model. The following pseudocode describes the execution process of the marching cubes algorithm for comparing differences: This section requires further clarification regarding the application scenarios of the marching cubes algorithm for comparing differences. The inputs are a synchronized TSDF volume and a target TSDF volume, a tolerance threshold, and upper and lower bounds for mesh generation. For each cube at the same position within the upper and lower bounds of mesh generation, the TSDF values ​​at their eight vertices are compared. If any pair of TSDF differences exceeds the tolerance threshold or if the differences are opposite in sign, the cube undergoes a mesh extraction operation. The deleted or added information is added to the incremental mesh data, and the corresponding TSDF values ​​and coordinates are recorded as valid TSDF data. The algorithm ultimately outputs the incremental mesh data and all valid TSDF data. Figure 2As shown, after the initial inference of the 3D scene reconstruction method, the server has two TSDF data sets, both stored as sparse coordinates and values. These can be converted into dense TSDF volumes and used as input to apply the marching cubes algorithm to generate a mesh model. One of these TSDF data sets is the TSDF data obtained after direct inference and updating by the server (target TSDF data), while the other corresponds to the mesh model rendered on the mobile device (synchronous TSDF data). That is, the mesh model generated by applying the classic marching cubes algorithm to this TSDF data set is the same as the mesh model rendered on the mobile screen. These two TSDF data sets can be identical or different, depending on the tolerance threshold parameter of the marching cubes algorithm. When this parameter is 0.0, meaning that a change in the TSDF value of a vertex is sufficient to warrant modification, the two TSDF data sets are identical. In other cases, the two TSDF data sets show slight differences, with the overall degree of difference increasing as the tolerance threshold increases. In other words, the TSDF data synchronized with the mobile device can be considered to be converging towards another piece of TSDF data, but not necessarily converging to be identical. On the other hand, in order to update the TSDF data synchronized with the mobile device, the marching cubes algorithm that compares the differences also needs to feed back the TSDF data that needs to be updated. The direct motivation for this design comes from the idea of ​​trading a slightly lower amount of transmitted data at the cost of a slight loss of mesh model quality.

[0039] The incremental mesh model data generated by the cube-granularity marching cubes algorithm or the difference-comparison marching cubes algorithm is first assigned a corresponding integer number to each cube by a cube numbering allocator, and then packaged and sent to the mobile device. The word "incremental" in "incremental mesh model data" indicates that this data only includes the changes in the mesh model before and after a single 3D reconstruction inference (including both additions and modifications to the mesh model, as well as deletions of the mesh model), and does not include unchanged mesh models, thus eliminating the redundancy of mesh models that are no longer updated. This elimination of redundancy is a direct result of the two improved marching cubes algorithms in the system described in this patent. The cube numbering allocator is implemented directly using a hash table. All cube numbers start from 1. If, when assigning a number to a cube, its triple coordinates have never appeared before, the current maximum number is directly incremented by 1 and assigned to the cube, and the current maximum number is also updated to the cube's number. If, when the cube's triple coordinates have appeared before, the previously assigned number is directly found, and this number remains valid throughout the scanning process. The handling of hash collisions is not the focus of the system described in this patent; it is the responsibility of the underlying implementation of the programming language.

[0040] After the cube numbering allocator converts the triplet coordinates in the incremental mesh model data into integer numbers, the data can be packaged. The system described in this patent designs two packaging formats for the incremental mesh model data: one uses UTF-8 encoding and then converts it to byte data transmission, while the other directly packages it as byte data transmission. These two packaging formats are essentially the same, both containing cube information for mesh models that need to be deleted and cube information for mesh models that need to be added or modified (note that there is no distinction between add and modify operations during packaging, as both have corresponding mesh models, hence they are collectively referred to as "add / modify" operations). The only difference is that directly packaging as byte data results in less data compared to using UTF-8 encoding and then converting to byte data; therefore, the system defaults to directly packaging the incremental mesh model as byte data, and this packaging format is only described here.

[0041] like Figure 3As shown, in the packaged incremental mesh model data, the first byte is interpreted as a single-byte integer, denoted as flag, indicating whether the data contains information about cubes to be deleted. A value of 0 indicates that information about cubes to be added or modified follows immediately, while a value of 1 indicates that information about cubes to be deleted follows immediately. Information about cubes to be added or modified follows the information about cubes to be deleted. In the information about cubes to be deleted, the first 4 bytes are interpreted as a 4-byte unsigned integer, representing the number of cubes to be deleted, denoted as numCubeIdxDelete. The following 4 * numCubeIdxDelete bytes are interpreted as numCubeIdxDelete four 4-byte unsigned integers, representing the numbers of the cubes to be deleted (these numbers were previously assigned by the cube number allocator). This concludes the information about cubes to be deleted. In the information about cubes to be added or modified, the first 8 bytes are interpreted as two 4-byte unsigned integers, representing the number of triangle vertices and the number of triangles to be added or deleted, denoted as vertCount and faceCount, respectively. The next 3 * 4 * vertCount bytes are interpreted as 3 * vertCount 4-byte floating-point data, with each 3-byte floating-point data representing the 3D coordinates of a triangle vertex. The following 3 * 4 * vertCount bytes are also interpreted as 3 * vertCount 4-byte floating-point data, with each 3-byte floating-point data representing the normal vector of a triangle vertex. The next 3 * 4 * faceCount bytes are interpreted as 3 * faceCount 4-byte unsigned integer data, with each 3-byte integer data representing the number of the three vertices that make up a triangle (these numbers are not previously assigned by the cube number allocator, but are the vertex numbers of the triangle, starting from 0, with a maximum value of vertCount - 1). The last 4 * faceCount bytes are interpreted as faceCount 4-byte unsigned integer data, representing the cube number of each triangle, which was assigned by the cube number allocator. Separating the coordinates and normals of triangle vertices, and separating the vertex numbers and cube numbers of triangles, seems a bit strange. However, in practice, this packing format was found to be faster than the interleaved packing format (although this depends to some extent on the output format of the marching cubes algorithm). Therefore, it was ultimately decided to use the non-interleaved data packing format.

[0042] After receiving the incremental mesh model data from the server, the mobile device parses it according to the format described above. To update the mesh model at the cube level, the mobile device maintains a dictionary from cube numbers to mesh model data. This dictionary is updated after parsing the incremental mesh model data, with deletions or additions made based on the parsing results. The mesh model data in the dictionary only contains the mesh model within the corresponding cube, including the vertex coordinates and normal vectors of each triangle. It's important to note that the transmitted data includes the global numbers of the vertices forming each triangle, while the parsed data does not contain either the global or local numbers of the triangle vertices, only their coordinates. Since the same vertex may be used by multiple triangles, its coordinates and normal vectors may appear more than once. However, since each cube contains a maximum of five triangles, and most of the time only two or three, the storage space required for using local numbers and repeating vertex coordinates and normal vectors is not large. Furthermore, using the method of repeating vertex coordinates and normal vectors for mesh model data storage has an additional advantage: it allows for more convenient incremental updates to the linear rendering cache.

[0043] After updating the dictionary from cube numbers to mesh model data, we need to use this data to render the mesh model. However, the GPU only accepts linear caches as input, and dictionary-style data cannot be directly used as input for GPU rendering. Therefore, we need to assemble the dictionary data into a linear cache. Figure 4 As shown (only vertex coordinates are displayed, not normals, but the principle is the same), without considering strips, there are generally two ways to render a triangular mesh model: one is to list the coordinates and normals of all vertices and the global vertex numbers that make up all triangles (numbered method), which is not convenient for modifying local data; the other is to arrange the information of all vertices of all triangles, where the information of the same vertex may appear multiple times in different triangles (unnumbered method). In general scenarios, the second method usually requires more storage space than the first method, so it is not recommended; however, in the system described in this patent, the mesh model needs to be updated at the cube granularity, so data redundancy in the mobile mesh model is unavoidable. The storage space disadvantage of the second method compared to the first method is no longer obvious, and it is actually more convenient for incremental updates of the rendering cache because the data length in each cube is only related to the number of triangles.

[0044] The basic idea of ​​incremental update of the rendering cache is as follows: In the above mesh model storage method, the length of the storage space occupied by each cube is only related to the number of triangles faceCnt, that is, length = faceCnt * 3 * 6 * 4. Each triangle contains three vertices, and each vertex contains 24 bytes of coordinate and normal vector data. Since there can be at most 5 triangles in a cube, there are only 5 possible lengths of storage space occupied by the mesh model within all cubes. The difficulty of incremental update of the rendering cache lies in the handling of deletion operations. Given that the possible lengths of storage space affected by each deletion operation are certain, deletion operations can be performed first to create "holes" in the linear rendering cache. Then, if there is an increase or modification operation of a suitable length, the data is filled into the holes, thus ensuring that other parts of the linear cache do not need to be changed. The update speed of the linear cache is only related to the amount of change in the cube mesh model, and not to the total amount of cube mesh models. The following pseudocode describes the specific incremental update process of the rendering cache: The incremental update process for the linear rendering cache is as follows: Input is an array of cube numbers to be deleted and cube mesh models to be added or modified. For each cube to be deleted, the hash map is consulted based on its number to find its position in the rendering cache. The data at the corresponding position is cleared, and the starting position is added to a list for future use. For a cube to be added, the list is searched based on the number of triangles within the cube. If an available starting position exists, the data is filled into the corresponding position; otherwise, it is directly appended to the end of the rendering cache. The same process applies to cubes to be modified: if the number of triangles remains unchanged before and after modification, it is modified in place; otherwise, the deletion and addition operations are performed according to the aforementioned process. The algorithm finally outputs the updated rendering cache.

[0045] Therefore, in the system described in this patent, the mobile terminal only undertakes data acquisition, mesh model rendering, and a small amount of computation, while other heavyweight tasks are handled by the relatively powerful server. This pipeline design allows for a relatively reasonable load distribution between the two. Furthermore, thanks to a series of supporting incremental algorithms and data packaging format designs, the system achieves satisfactory results in terms of feedback quality, response speed, time and bandwidth resource usage.

[0046] The system described in this patent has been implemented and comprehensively evaluated. Under laboratory conditions, the system can provide users with a 3D scene reconstruction mesh model that adequately reflects topological features and even has an identical shape, while meeting the requirements of low latency response and low bandwidth consumption. Figure 5The screenshots show the mobile device running the system described in this patent. The mesh model transmitted from the server is rendered and overlaid on the surface of the real object. Users can compare the differences between the mesh model on the screen and the real-world object to determine which areas of data were not fully collected, thus allowing them to autonomously control the next step of the scan. Experimental results show that in a real online scanning and reconstruction feedback process, for a 3D reconstructed intermediate mesh model, the average latency from the user acquiring the last frame of valid data to the mesh model being rendered on the mobile screen is less than 1 second, and the average bandwidth required for transmitting the intermediate mesh model is approximately 4 Mbps.

[0047] Specifically: Usability and model consistency: Users do not need to have relevant background knowledge when using the system described in this patent. Figure 5 As shown, after the user clicks the start button, if the system detects that the user has not collected valid data for a period of time (5 seconds), a "Move Scanning Device" prompt will appear. Once the user begins collecting valid data, the progress bar in the upper right corner will scroll accordingly, indicating the progress towards the next 3D reconstruction inference. Encouraged by the progress bar, even if the user does not understand its meaning, they will continue scanning following the progress bar or text prompts. When the progress bar is full, the server performs a 3D reconstruction inference, generates an intermediate mesh model, and transmits it to the mobile device for rendering. The user can intuitively see the mesh model appear on the screen and the progress bar reset to zero, thus continuing to scan and collect data. This interaction method ensures the system's ease of use; even if the user does not understand the specific meaning of the progress bar or text prompts, they can easily perform the scanning work.

[0048] The mesh model rendered on the mobile device corresponds to the synchronized TSDF volume on the server, and its specific shape is related to the tolerance threshold parameter of the marching cubes algorithm used for comparison. When the tolerance threshold is 0.0, the mesh model rendered on the mobile device should have a completely consistent shape with the mesh model generated by directly applying the classic marching cubes algorithm to the TSDF volume obtained from the server's 3D reconstruction inference. We verified this manually. When the tolerance threshold is not 0.0, the mesh model rendered on the mobile device suffers a loss, but the total amount of data transmitted is reduced accordingly. We conducted experiments on data collected during the same scan process, selecting several different tolerance threshold values ​​(0.0, 0.05, 0.1, 0.2, 0.5, 1.0). Figure 6As shown, when the selected tolerance threshold is small (<0.1), the quality of the mesh model does not change significantly with increasing tolerance threshold, but the total amount of intermediate data to be transmitted decreases dramatically (the total amount of intermediate data decreases to 56.75% when the tolerance threshold is 0.05 compared to 0.0, and to 46.13% when the tolerance threshold is 0.1 compared to 0.0). However, when the selected tolerance threshold is large, the quality of the mesh model decreases noticeably with increasing tolerance threshold, while the decrease in the total amount of intermediate data is not significant. Therefore, it is recommended to set the tolerance threshold to 0.05 or 0.1 in practical applications to achieve a balance between reducing network bandwidth consumption and maintaining mesh model quality.

[0049] System performance: The performance evaluation of the system described in the patent is divided into several aspects, including the time overhead of each part of the pipeline, network bandwidth overhead, and transmission frame rate. The system performance was evaluated on an Apple iPhone 11 and a server equipped with an NVIDIA GeForce RTX 2080 Ti GPU. The test was conducted under laboratory conditions, with a scan lasting approximately 5 minutes performed in a conference room setting. A total of 1108 keyframes and 123 intermediate mesh models were transmitted during the scan. The table below shows the performance test results of the above experiment. The average time per round of the system described in the patent (from the selection of the last keyframe constituting the intermediate mesh model to the rendering of that intermediate mesh model on the mobile screen) is less than 1 second. Considering that the downlink intermediate mesh model transmission frame rate is only 0.396, this is an acceptable speed. The time taken for all critical steps is listed in Table 1 below; other components of the round time, such as transmission time and JPEG compression time, are not included.

[0050] Minimum value 587.00 0.00 20.00 40.00 137.00 23.00 42.00 65.00 Maximum value 1340.00 1.00 62.00 237.00 292.00 212.00 274.00 314.00 average value 940.74 0.49 31.03 50.95 216.35 140.71 149.87 174.63 .

Claims

1. A mobile monocular 3D scene reconstruction system based on a computational offloading mechanism, characterized in that: Includes the following steps: Step 1: The user clicks the button to control the mobile device to start scanning, and the mobile device continuously changes the collected 3D reconstruction data; Step 2: The mobile device filters the 3D reconstruction data obtained in Step 1. Data whose offset angle from valid data exceeds a threshold is considered valid, while other data is discarded. Step 3: Use the valid data obtained in Step 2 to scale the current RGB pixel image and perform JPEG encoding, then transmit it to the server via TCP; Step 4: The server preprocesses the data obtained in Step 3 to obtain the adapted data; Step 5: Perform neural network inference on the valid data obtained in Step 4 and update the TSDF data; Step 6: Convert the sparse TSDF data output from the neural network inference in each step 5 into a dense TSDF volume representation to obtain the vertices where the TSDF changes and the corresponding small cubes, i.e., the incremental grid model data at the cube granularity. Step 7: Using the cube number allocator in the incremental mesh model data of cube granularity obtained in Step 6, map the triplet coordinates of each cube to an integer number, then pack the cubes to be deleted and the cubes to be added or modified into byte data, and transmit them to the mobile terminal via TCP. Step 8: After receiving the byte data from Step 7, the mobile device parses it, updates a dictionary from cube number to corresponding mesh data, and incrementally updates the linear cache used for GPU rendering, thereby updating the 3D reconstruction effect; Step 9: The next time the screen refreshes, it displays the updated effect of the current 3D reconstructed mesh model. The user can adjust the scanning process or angle. Steps 1 and 2 continue to generate valid data until the user clicks to end the reconstruction.

2. The mobile monocular 3D scene reconstruction system based on computational offloading mechanism according to claim 1 includes the following steps: the mobile terminal used in step 1 is ARKit or ARCore, and the 3D reconstruction data is the RGB pixel image at the current moment, the camera position and pose, and the camera internal parameters.

3. The mobile monocular 3D scene reconstruction system based on computational offloading mechanism according to claim 1 includes the following steps: in step 3, the RGB pixel image is scaled to a size of 640*360mm.

4. The mobile monocular 3D scene reconstruction system based on the computational offloading mechanism according to claim 1 includes the following steps: the preprocessing in step 4 includes JPEG decoding of the image, conversion of the camera's position and pose data from the physical world coordinate system to the coordinate system under the perspective of the specific 3D scene reconstruction method, and other adaptation processing necessary for the input data.

5. The mobile monocular 3D scene reconstruction system based on the computational unloading mechanism according to claim 1 includes the following steps: In step 6, the incremental mesh model data of the cube granularity is generated by the marching cubes algorithm, which contains point and face information and small cube coordinates. If it is a small cube that needs to be deleted from the mesh model, only its coordinates are recorded.

6. The mobile monocular 3D scene reconstruction system based on the computational unloading mechanism according to claim 1 includes the following steps: In step 9, if the user adjusts the scanning time without scanning or the scanning speed is too fast, the mobile terminal will give a corresponding prompt.

7. The mobile monocular 3D scene reconstruction system based on computational offloading mechanism according to claim 1, comprising the following steps: the storage space occupied by each cube in the linear cache rendered by the GPU in step 8 is length, which is length = faceCnt * 3 * 6 * 4.

8. The mobile monocular 3D scene reconstruction system based on computational offloading mechanism according to claim 1, comprising the following steps: the threshold in step 2 is 15 degrees.

Images