A method and device for displaying a three-dimensional model

Abstract

This application relates to the field of VR technology, providing a method and device for displaying 3D models. The acquisition terminal divides the high-precision and low-precision data of the 3D model into first sub-models and second sub-models, respectively, and sends the complete low-precision data of each second sub-model to the rendering terminal. This ensures the integrity of the 3D model, and the smaller data size of each second sub-model solves the problem of insufficient frame rate in 3D model animations caused by insufficient bandwidth. This ensures that the normal rendering and display of the model is not affected by data volume and bandwidth limitations, thereby reducing video stuttering. The rendering terminal downloads the high-precision data of the target sub-models within the field of view in order of priority, allowing the display of high-precision sub-models in areas of interest to be prioritized, improving the display quality of the 3D model. For invisible areas, there is no need to download high-precision data, reducing the download of redundant data and improving the overall rendering and display efficiency of the 3D model.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and more particularly to a method and device for displaying a three-dimensional model. Background Technology

[0002] 3D reconstruction refers to the process of reconstructing 3D information from single-view or multi-view images. For static scenes and objects, 3D models of the scene and objects are reconstructed using static 3D reconstruction algorithms and then rendered and displayed in the 3D scene. For dynamic objects (such as people), because they are dynamically changing, it is often necessary to create a series of 3D models, and then render and display these 3D models in sequence to describe the object's motion state.

[0003] The display of 3D reconstruction data, as a key technology in real-time remote 3D reconstruction social systems, directly impacts the user's immersive experience. Because 3D reconstruction data is dynamically generated, each frame is different. Therefore, when a terminal simultaneously receives multiple streams of 3D reconstruction data corresponding to the current frame, the data volume is enormous, often requiring hundreds of megabits of network bandwidth. Typical home networks or even dedicated lines struggle to meet the real-time transmission requirements, leading to insufficient frame rates in the rendered 3D model and visually causing stuttering in the 3D model's movements.

[0004] Currently, most methods reduce the amount of data transmitted by lowering the accuracy of the 3D model. However, if the accuracy of the reconstructed model is reduced, the level of detail in the model will decrease, affecting the display effect. Summary of the Invention

[0005] This application provides a method and device for displaying a three-dimensional model, which can improve the rendering and display efficiency of the three-dimensional model while ensuring the accuracy of the three-dimensional model, thereby improving the real-time performance of the three-dimensional reconstruction.

[0006] In a first aspect, embodiments of this application provide a method for displaying a three-dimensional model, comprising:

[0007] Based on the depth and color images of the target object, a three-dimensional model of the target object is reconstructed, and the skeleton of the three-dimensional model is obtained;

[0008] The initial reconstruction data of the three-dimensional model is downsampled to obtain the target reconstruction data;

[0009] Based on the preset priority of the skeleton nodes, the priority of each first sub-model in the model corresponding to the initial reconstruction data is determined respectively.

[0010] The priority data of each first sub-model and the data of each second sub-model in the model corresponding to the target reconstruction data are sent to the rendering terminal, and the first sub-model and the second sub-model correspond one-to-one.

[0011] The system receives a data acquisition request sent by the rendering terminal. The data acquisition request carries the identifiers of each target sub-model in descending order of priority. The identifiers of the target sub-models are determined based on the spatial relationship between the bounding boxes of each second sub-model and the view frustum of the rendering terminal.

[0012] According to the identifiers of each target sub-model sorted in the data acquisition request, the initial reconstruction data corresponding to each target sub-model is sent to the rendering terminal in sequence.

[0013] Secondly, embodiments of this application provide a method for displaying a three-dimensional model, including:

[0014] The system receives priority data of each first sub-model sent by the acquisition terminal, as well as data of each second sub-model in the model corresponding to the target reconstruction data; wherein, the priority data is determined according to the priority of the preset skeleton node, the first sub-model is a sub-model in the model corresponding to the initial reconstruction data of the target object, the target reconstruction data is obtained by downsampling the initial reconstruction data, and the first sub-model and the second sub-model correspond one-to-one.

[0015] Generate bounding boxes for each of the second sub-models, and determine the identifiers of each target sub-model contained within the view frustum based on the spatial relationship between each bounding box and the view frustum.

[0016] The identifiers of each target sub-model are sorted in descending order of priority.

[0017] Send a data acquisition request to the acquisition terminal, the data acquisition request carrying the identifiers of each sorted target sub-model;

[0018] Receive the initial reconstruction data of the corresponding target sub-models sent sequentially by the acquisition terminal according to the identifiers of each target sub-model;

[0019] Based on the initial reconstruction data of each target sub-model and the target reconstruction data of a second sub-model with a different identifier from each target sub-model, the three-dimensional model of the target object is rendered and displayed.

[0020] Thirdly, this application provides a data acquisition terminal, including an RGBD camera and a host, wherein the host includes at least a processor, a memory and a communication interface, and the memory and the communication interface are connected to the processor via a bus;

[0021] The RGBD camera is used to acquire depth and color images of the target object; the memory stores a computer program, and the processor performs the following operations according to the computer program:

[0022] Based on the depth and color images of the target object, a three-dimensional model of the target object is reconstructed, and the skeleton of the three-dimensional model is obtained;

[0023] The initial reconstruction data of the three-dimensional model is downsampled to obtain the target reconstruction data;

[0024] Based on the preset priority of the skeleton nodes, the priority of each first sub-model in the model corresponding to the initial reconstruction data is determined respectively.

[0025] Through the communication interface, the priority data of each first sub-model and the data of each second sub-model in the model corresponding to the target reconstruction data are sent to the rendering terminal, and the first sub-model and the second sub-model correspond one-to-one;

[0026] Through the communication interface, a data acquisition request sent by the rendering terminal is received. The data acquisition request carries the identifiers of each target sub-model in descending order of priority. The identifiers of the target sub-models are determined based on the spatial relationship between the bounding boxes of each second sub-model and the view frustum of the rendering terminal.

[0027] According to the identifiers of each target sub-model sorted in the data acquisition request, the initial reconstruction data corresponding to each target sub-model is sent sequentially to the rendering terminal through the communication interface.

[0028] Fourthly, embodiments of this application provide a rendering terminal, including a processor, a memory, a display, and a communication interface, wherein the memory, the display, and the communication interface are connected to the processor via a bus;

[0029] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0030] Through the communication interface, priority data of each first sub-model and data of each second sub-model in the model corresponding to the target reconstruction data are received from the acquisition terminal; wherein, the priority data is determined according to the priority of the preset skeleton node, the first sub-model is the sub-model in the model corresponding to the initial reconstruction data of the target object, the target reconstruction data is obtained by downsampling the initial reconstruction data, and the first sub-model and the second sub-model correspond one-to-one.

[0031] Generate bounding boxes for each of the second sub-models, and determine the identifiers of each target sub-model contained within the view frustum based on the spatial relationship between each bounding box and the view frustum.

[0032] The identifiers of each target sub-model are sorted in descending order of priority.

[0033] The data acquisition request is sent to the acquisition terminal through the communication interface, and the data acquisition request carries the identifiers of each sorted target sub-model;

[0034] Through the communication interface, the initial reconstruction data of the corresponding target sub-models, which are sent sequentially by the acquisition terminal according to the identifiers of each target sub-model, are received.

[0035] Based on the initial reconstruction data of each target sub-model and the target reconstruction data of a second sub-model with a different identifier from each target sub-model, a three-dimensional model of the target object is rendered and displayed on the monitor.

[0036] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the method for displaying a three-dimensional model provided in the embodiments of this application.

[0037] In the above embodiments of this application, the acquisition terminal reconstructs the 3D model of the target object based on the depth image and color image of the target object. According to the preset priority of the skeleton nodes, it determines the priority of each first sub-model in the model corresponding to the initial reconstruction data. Each first sub-model corresponds one-to-one with a second sub-model in the model corresponding to the target reconstruction data. The priority data of each first sub-model and the target reconstruction data of each second sub-model are sent to the rendering terminal to ensure the integrity of the 3D model and the continuity of its actions. Furthermore, since the target reconstruction data is obtained by downsampling the initial reconstruction data, the pressure on network bandwidth is reduced. The rendering terminal determines the identifier of each target sub-model within the field of view based on the spatial relationship between the bounding box of each second sub-model and the view frustum. Since the first and second sub-models correspond one-to-one, the priority of the first sub-model can be used as the priority of the target sub-model. Therefore, the identifiers of each target sub-model are sorted in descending order of priority, and the sorted identifiers are sent in the data acquisition request. After receiving a data acquisition request, the acquisition terminal sends the initial reconstruction data of each target sub-model sequentially to the rendering terminal according to the identifiers of the sorted target sub-models. High-precision first sub-model data within invisible areas is not transmitted, reducing the amount of data transmitted and improving rendering efficiency. Upon receiving the initial reconstruction data of each target sub-model, the rendering terminal combines it with the target reconstruction data of the second sub-models, which have different identifiers from the target sub-models, to perform real-time rendering and display of the 3D model. Because the target sub-models are within the field of view and the data has high precision, the display quality of the 3D model is improved. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 An exemplary structural diagram of a real-time remote 3D reconstruction social system provided by an embodiment of this application is shown;

[0040] Figure 2 The examples illustrate application scenarios to which the embodiments of this application are applicable;

[0041] Figure 3 An exemplary illustration shows the bounding box of a three-dimensional model provided in an embodiment of this application;

[0042] Figure 4 An exemplary flowchart illustrates a method for displaying a 3D model on the acquisition terminal side according to an embodiment of this application;

[0043] Figure 5 An exemplary diagram illustrates the equal division of the bounding box provided in an embodiment of this application;

[0044] Figure 6 An exemplary illustration shows a spatial diagram of a cuboid divided by model vertices and bounding boxes provided in an embodiment of this application;

[0045] Figure 7 An exemplary illustration shows a spatial diagram of a cuboid divided by a skeleton node and a bounding box, provided in an embodiment of this application.

[0046] Figure 8 An exemplary flowchart illustrates a method for displaying a 3D model on a rendering terminal side according to an embodiment of this application;

[0047] Figure 9 An exemplary schematic diagram illustrating the display effect of a three-dimensional model provided in an embodiment of this application is shown.

[0048] Figure 10 An exemplary flowchart illustrates the interaction between the acquisition terminal and the rendering terminal in displaying a 3D model, as provided in an embodiment of this application.

[0049] Figure 11 An exemplary structural diagram of the data acquisition terminal provided in an embodiment of this application is shown;

[0050] Figure 12 An exemplary structural diagram of a rendering terminal provided in an embodiment of this application is shown. Detailed Implementation

[0051] Real-time social interaction requires the transmission of data such as vertices, faces, and textures of the reconstructed model for each frame. As a result, when rendering virtual social scenes, the amount of data of the 3D model transmitted for each frame is huge. Under the condition of limited network bandwidth, it takes a long time to load a complete frame of 3D model data. When the rendering terminal has insufficient action frame rate, it will cause the 3D model to stutter. On the other hand, reducing the accuracy of the 3D model will reduce the rendering quality and affect the user experience.

[0052] Therefore, to reduce network bandwidth consumption for data transmission while ensuring the fidelity of the reconstructed model, thereby meeting the real-time requirements of social systems and improving the rendering efficiency of the rendering terminal, this application provides a method and device for displaying a 3D model. The acquisition terminal downsamples the geometric and texture data of the 3D model (e.g., reducing vertex data and texture resolution) to generate a low-precision 3D model data set. This low-precision 3D model data is then transmitted to the rendering terminal in real time to ensure the integrity of the 3D model data and the frame rate of the animation in the rendering terminal. Simultaneously, when the acquisition terminal calculates the bounding box of the 3D model, it divides the bounding box into several cuboids, and then cuts the 3D model into several sub-models according to each cuboid. The priority of each sub-model is calculated based on the pre-set priority of the 3D model's skeletal data. The rendering terminal requests high-precision sub-model data based on the visibility and priority of each sub-model. Combined with the low-precision sub-model data, it renders and displays the 3D model. In this way, the entire 3D model is composed of sub-models of different precision. Therefore, under limited network bandwidth conditions, more high-precision sub-model reconstruction data can be transmitted. Compared to transmitting a single frame of high-precision reconstruction data of a complete 3D model, this reduces network bandwidth pressure, improves rendering and display efficiency, and ensures the integrity of the model and the continuity of its movements. Furthermore, since the high-precision sub-model reconstruction data is obtained based on visibility and priority, the accuracy of the model within the human eye's field of vision can be guaranteed, ensuring the normal display of the model.

[0053] In this embodiment, high precision and low precision are relative to the number of vertices and texture resolution of the model.

[0054] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0055] Figure 1 An exemplary diagram of a real-time remote 3D reconstruction social system architecture provided by an embodiment of this application is shown. Figure 1 As shown, the system includes a data acquisition terminal 101, a transmission terminal 102, and a rendering terminal 103.

[0056] The acquisition terminal 101 includes an RGBD camera 101_1 and a host 101_2. The RGBD camera 101_1 is used to acquire depth images and RGB images of the user and the scene in which the user is located. The host 101_2 performs relevant calculations on the acquired depth images and RGB images of the user to reconstruct a three-dimensional human body model. The reconstructed data is then encoded according to a custom data structure and sent to the transmission terminal 102.

[0057] The transmission terminal 102 can be a cloud server, used to distribute the reconstruction data sent by the acquisition terminal 101 to the rendering terminal 103. The transmission terminal 102 can also re-encode the reconstruction data sent by the acquisition terminal 101 using lossless encoding before distributing it to the rendering terminal 103.

[0058] The rendering terminal 103 can be an interactive device such as a VR / AR head-mounted display, television, mobile phone, computer, or tablet. It can render a 3D human body model based on the reconstructed data obtained from the transmission terminal 102 and display it immersively. During the display process, it can also be connected to display devices such as televisions, mobile phones, and computers for 2D display.

[0059] exist Figure 1 In the system shown, after the acquisition terminal reconstructs the 3D model, it transmits it to the rendering terminal for rendering and display. Taking the application of the above system in a virtual social scene as an example, the acquisition terminal transmits the dynamic 3D model of its corresponding user to the rendering terminal through 3D reconstruction. After receiving this reconstructed data, the rendering terminal renders and displays it to achieve an immersive social experience.

[0060] like Figure 2 As shown, users 1 to 4 engage in real-time remote 3D communication, with terminals deployed for each user. During the remote 3D communication, terminal 1 uploads user 1's 3D reconstruction data to a cloud server, while terminals 2 to 4 download user 1's 3D reconstruction data from the cloud server and simultaneously display the 3D human models of users 1 to 4. Similarly, the 3D human models of users 2, 3, and 4 can also be simultaneously displayed on terminal 1 corresponding to user 1, and so on.

[0061] It should be noted that the acquisition terminal and rendering terminal in this embodiment are relative to the user. For example, Figure 2 Terminal 1 is the acquisition terminal for user 1, and the rendering terminal for users 2, 3 and 4.

[0062] In remote 3D communication, the 3D reconstruction of the human body is crucial. Typically, the skeleton of a 3D human model consists of interconnected "skeletal joints" (i.e., skeletal nodes). The skeleton is embedded within the 3D model and represents its shape. The relationship between the 3D model and its corresponding skeleton is as follows: Figure 3 As shown, this skeleton includes 19 joints. This application uses this skeleton as an example for description in its embodiments.

[0063] In the embodiments of this application, the display priority of each joint in the skeleton corresponding to the 3D model is preset. An optional implementation is to set priorities according to body parts. Specifically, for a 3D human body model, the head is the most important part and a key part for identifying a person, and should be rendered and displayed with the highest priority; the chest and abdomen are large areas of the human body, usually covered by clothing, and the patterns on the clothing are also important personal characteristics, therefore, the chest and abdomen are set to have the second highest priority after the head; generally, the upper limbs are more likely to receive attention, therefore, the upper limbs are set to have the lowest rendering and display priority after the chest and abdomen and before the lower limbs.

[0064] At the same time, different types of skeletal nodes in the same part are assigned the same priority. For example, the left and right hands, and the left and right feet have the same priority.

[0065] Based on the above rules, for a skeleton composed of 19 joints, the priority settings for each node are as follows: {5, 10, 11, 12, 10, 11, 12, 4, 3, 2, 1, 6, 7, 8, 9, 6, 7, 8, 9}. The priority of each skeleton node is arranged in order from 0 to 18, with a smaller priority number indicating a higher priority. For example, the skeleton node with the sequence number 10 has a priority number of 1, representing the highest priority.

[0066] Based on the priorities of each joint in the pre-defined model skeleton, the following describes the method for displaying the 3D model in this embodiment of the application, using a data acquisition terminal and a rendering terminal as an example. (See also...) Figure 4 This method is executed by the data acquisition terminal and mainly includes the following steps:

[0067] S401: Reconstruct the 3D model of the target object based on the depth and color images of the target object, and obtain the skeleton of the 3D model.

[0068] In one optional implementation, during step S401, the acquisition terminal acquires the depth image and color image of the target object captured by the RGBD camera. The host converts the depth image into a point cloud dataset, reconstructs the geometric model of the target object, and extracts texture data from the color image based on the mapping relationship between the depth image and the color image to apply texture mapping to the geometric model, thereby obtaining a three-dimensional model of the target object. Further, the skeleton embedded within this three-dimensional model is acquired, and the joint information contained in the skeleton is obtained, including the number of nodes, spatial coordinates, priority, and other data.

[0069] S402: Downsample the initial reconstruction data of the 3D model to obtain the target reconstruction data.

[0070] Since the 3D model is reconstructed based on collected real data, the initial reconstruction data (including geometric and texture data) contains accurate information such as the number of vertices and texture resolution, ensuring the accuracy of the 3D model reconstruction. This initial reconstruction data can be used as high-precision reconstruction data. However, high-precision reconstruction data requires high network bandwidth. Without effective compression encoding methods, existing networks struggle to achieve real-time transmission. Therefore, to meet the real-time requirements of remote 3D communication, the initial reconstruction data needs to be downsampled.

[0071] In one optional implementation, when executing S402, the number of vertices of the 3D model in the initial reconstruction data is reduced and the resolution of the texture data in the initial reconstruction data is reduced according to the preset sampling coefficients to obtain the downsampled target reconstruction data, thereby reducing the amount of data transmitted over the network, improving the transmission speed, and ensuring the continuity of model actions; and since the target reconstruction data is only obtained by downsampling the initial reconstruction data, it only reduces the accuracy of the reconstruction data, and the data is still complete, ensuring the integrity of the model.

[0072] It should be noted that the sampling coefficients can be set according to actual needs, and this application embodiment does not impose any limiting requirements. For example, they can be set according to the actual network bandwidth. The sampling coefficients of the vertices and the sampling coefficients of the texture data can be the same or different.

[0073] S403: Based on the preset priority of the skeleton nodes, determine the priority of each first sub-model in the model corresponding to the initial reconstruction data.

[0074] In the embodiments of this application, the model corresponding to the initial reconstruction data has higher accuracy than the model corresponding to the target reconstruction data, but the skeleton structure of the two models is the same. Therefore, the two models are divided in the same way, and each first sub-model corresponds to a second sub-model. That is, the first sub-model and the second sub-model have the same spatial size and the same number of divisions.

[0075] In computer graphics and computational geometry, the bounding box of a 3D model is a closed space that completely encloses the model. Encapsulating complex models within simple bounding boxes, and using the shape of the simple bounding box to approximate the shape of the complex geometry, can improve the efficiency of geometric calculations. One possible implementation is that, during execution S403, the complete model is divided into sub-models based on the bounding box corresponding to the model.

[0076] In practice, the acquisition terminal generates the bounding box corresponding to the 3D model of the target object, such as... Figure 3As shown, the outer layer is the cuboid bounding box corresponding to the 3D model, and the inner layer is the 3D model and its corresponding skeleton structure. The generated bounding box is divided equally in the X, Y, and Z directions to obtain several cuboids of the same size, each cuboid corresponding to a unique number.

[0077] It should be noted that the embodiments of this application do not impose a limiting requirement on the number of cuboids to be divided, and can be set according to the actual situation.

[0078] like Figure 5 The diagram shows a bounding box segmentation of a 3D model. The model is divided into 6, 6, and 1 equal parts in the X, Y, and Z directions, respectively, generating a total of 36 identical cuboids, numbered from 0 to 35.

[0079] Furthermore, the model vertices corresponding to the initial reconstructed data and the model vertices corresponding to the target reconstructed data are traversed separately. Based on the spatial coordinates of each vertex, the cuboid in which each vertex is located is determined. Then, based on the cuboid in which each vertex is located, the model corresponding to the initial reconstructed data and the model corresponding to the target reconstructed data are subjected to spatial geometric segmentation. In other words, the spatial geometric segmentation process of the model is to classify and divide all vertices of the two models according to the cuboid space.

[0080] In the embodiments of this application, the model vertices corresponding to the initial reconstructed data and the model vertices corresponding to the target reconstructed data are processed separately and are independent of each other, making full use of the parallel processing capability of the CPU and achieving a faster processing speed. For each vertex in the same cuboid space in both models, a new topological relationship is established, and texture segmentation is performed on the initial reconstructed data and the target reconstructed data based on the new topological relationship. Then, based on the initial reconstructed data after spatial geometric segmentation and texture segmentation, each first sub-model is obtained, and based on the target reconstructed data after spatial geometric segmentation and texture segmentation, each second sub-model is obtained. The number of each sub-model is consistent with the cuboid number, i.e., the number (identifier) ​​of the first sub-model ranges from 0 to 35, and the number (identifier) ​​of the second sub-model ranges from 0 to 35. After geometric segmentation and texture segmentation, two sets of sub-models, one high-precision and one low-precision, are obtained.

[0081] Figure 6 This is a schematic diagram of the segmentation of a three-dimensional model provided in an embodiment of this application. Figure 6In the diagram, black dots represent the vertices of the model to visually represent the segmentation process. Since the high-precision and low-precision models are segmented in the same way, the first and second sub-models are numbered identically, corresponding to the numbers of the cuboids. Here, we uniformly specify the sub-model numbers as {2, 3, 6, 7, 8, 9, 10, 11, 14, 15, 20, 21, 26, 27, 32, 33}. The cuboids with other numbers do not contain vertices, therefore they do not have corresponding sub-model numbers.

[0082] In some embodiments, other methods may be used to segment the high-precision and low-precision models, such as segmenting the model according to its parts.

[0083] In step S403, after segmenting the model corresponding to the initial reconstruction data and the model corresponding to the target reconstruction data, the priority of each first sub-model is determined according to the preset priority of the skeleton nodes. The method for setting the priority of the skeleton nodes is described in the previous embodiment and will not be repeated here.

[0084] The following example, using any one of the first sub-models, illustrates how the priority of a sub-model is determined. The method for determining the priority of a sub-model varies depending on the specific circumstances.

[0085] Scenario 1

[0086] When there is a skeleton node in the cuboid corresponding to the sub-model, the priority of the skeleton node is used as the priority of the sub-model.

[0087] For example, with Figure 7 For example, if cuboid number 3 contains only one skeleton node 10, then the priority 1 of skeleton node 10 is used as the priority of sub-model 3 corresponding to cuboid number 3; if cuboid number 32 contains only one skeleton node 6, then the priority 12 of skeleton node 6 is used as the priority of sub-model 32 corresponding to cuboid number 32; if cuboid number 33 contains only one skeleton node 3, then the priority 12 of skeleton node 3 is used as the priority of sub-model 33 corresponding to cuboid number 33.

[0088] Scenario 2

[0089] When there are different skeleton nodes within the cuboid corresponding to the sub-model, the highest priority of each skeleton node is used as the priority of the sub-model.

[0090] For example, with Figure 7For example, cuboid number 7 contains two skeleton nodes 17 and 18, with priorities of 8 and 9 respectively. According to the rule of "the smaller the level, the higher the priority" specified in the embodiments of this application, the priority of sub-model 7 is 8. Cuboid number 8 contains two skeleton nodes 15 and 16, with priorities of 6 and 7 respectively, so the priority of sub-model 8 is 6. Cuboid number 9 contains four skeleton nodes 8, 9, 11, and 12, with priorities of 3, 2, 6, and 7 respectively, so the priority of sub-model 9 is 2. Cuboid number 10 contains two skeleton nodes 13 and 14, with priorities of 8 and 9 respectively, so the priority of sub-model 10 is 8. Cuboid number 15 contains two skeleton nodes 0 and 7, with priorities of 5 and 4 respectively, so the priority of sub-model 15 is 4.

[0091] Scenario 3

[0092] When there are no skeleton nodes but there are vertices in the cuboid corresponding to the sub-model, the priority of the skeleton node that is closest to the cuboid among all skeleton nodes is used as the priority of the sub-model.

[0093] For example, with Figure 7 For example, cuboids numbered 2, 6, 11, and 14 contain vertices but not skeleton nodes. The skeleton nodes closest to the center point of these four cuboids are 10, 18, 14, and 7, respectively. Therefore, the priorities of sub-models 2, 6, 11, and 14 are 1, 9, 9, and 4, respectively.

[0094] After traversing all sub-models, the priority of the first sub-model {2, 3, 6, 7, 8, 9, 10, 11, 14, 15, 20, 21, 26, 27, 32, 33} is {1, 1, 9, 8, 6, 2, 8, 9, 4, 4, 10, 10, 11, 11, 12, 12}.

[0095] S404: Send the priority data of each first sub-model and the data of each second sub-model in the model corresponding to the target reconstruction data to the rendering terminal.

[0096] In the embodiments of this application, since the target object's pose varies at different times, the sub-models containing vertex data are different. Therefore, the acquisition terminal sends the priority data of each first sub-model to the rendering terminal in real time. Simultaneously, the data of each second sub-model is completely uploaded to the cloud server in real time. The rendering terminal downloads the target reconstruction data of the current target object from the cloud server to ensure the integrity of the 3D model and the continuity of its movements.

[0097] It should be noted that in S404, the acquisition terminal sends a complete set of low-precision target reconstruction data, which is composed of each second sub-model in sequence, to the rendering terminal. The target reconstruction data of each second sub-model is not sent independently. Furthermore, since the target reconstruction data is obtained by downsampling and the data volume is small, the existing network bandwidth can meet the needs of real-time transmission. Therefore, it is not necessary to send them in order of priority.

[0098] S405: Receives data acquisition requests sent by the rendering terminal.

[0099] In the embodiments of this application, since the first sub-model and the second sub-model are in one-to-one correspondence, that is, they have the same space size and the same number, the difference lies in the different data size of the sub-models with the same number (e.g., the data size of the first sub-model with number 0 is greater than the data size of the second sub-model with number 0). Therefore, the priority of the first sub-model can be used as the priority of the second sub-model with the same number.

[0100] In S405, the rendering terminal sends a data acquisition request, which carries identifiers of each target sub-model in descending order of priority. The identifiers of the target sub-models are determined based on the spatial relationship between the bounding boxes of each second sub-model and the rendering terminal's view frustum. The cloud server forwards the data acquisition request to the acquisition terminal, enabling the acquisition terminal to upload high-precision reconstructed data according to the request.

[0101] S406: According to the identifiers of each target sub-model sorted in the data acquisition request, send the initial reconstruction data corresponding to each target sub-model to the rendering terminal in sequence.

[0102] In S406, after receiving a data acquisition request, the acquisition terminal, under the condition of existing network bandwidth, sequentially sends the initial reconstruction data of the corresponding target sub-models (i.e., the initial reconstruction data of the first sub-model with the same identifier) ​​to the rendering terminal according to the identifier of each target sub-model after sorting, thereby reducing the amount of data transmitted over the network, reducing transmission latency, and improving rendering and display efficiency.

[0103] Figure 4 The operations performed on the acquisition terminal side in the three-dimensional model display method provided in this application embodiment correspond to the acquisition terminal. Figure 8 The method flow for rendering on the terminal side mainly includes the following steps:

[0104] S801: Receives priority data of each first sub-model sent by the acquisition terminal, as well as target reconstruction data of each second sub-model in the model corresponding to the target reconstruction data.

[0105] In the embodiments of this application, the methods for dividing each first sub-model and each second sub-model, the methods for determining the priority of each first sub-model, and the methods for generating the target reconstruction data are detailed in [reference needed]. Figure 4 The description in [the document] will not be repeated here.

[0106] In S801, after the rendering terminal receives the target reconstruction data of each second sub-model, it decodes the target reconstruction data and stores the decoded target reconstruction data in the rendering engine's rendering queue. This is so that if the initial reconstruction data of the first sub-model is missing, the target reconstruction data of the second sub-model with the same identifier can be used to replace it, thus ensuring the integrity of the 3D model and the continuity of the action.

[0107] S802: Generate bounding boxes for each second sub-model, and determine the target sub-models contained within the view volume based on the spatial relationship between each bounding box and the view volume.

[0108] In the embodiments of this application, since the first sub-model and the second sub-model have the same spatial size and number, their bounding boxes have the same size and number. Since the data for the first sub-model has not yet been received, in S802, bounding boxes for each second sub-model are generated. For any bounding box among these bounding boxes, its spatial relationship with the rendering terminal's view volume is determined. If the bounding box intersects with the view volume or is within the view volume, it indicates that the sub-model with the same identifier as the bounding box is visible. Then, the identifier of the second sub-model corresponding to that bounding box is used as the identifier of a target sub-model within the view volume, and the identifier of the target sub-model is placed in the high-precision sub-model data request queue.

[0109] S803: Sort the identifiers of each target sub-model according to their priority from high to low.

[0110] In the embodiments of this application, since the first sub-model and the second sub-model correspond one-to-one, the priority of the first sub-model can be used as the priority of the second sub-model with the same identifier. That is, the priority of each target sub-model can be determined based on the priority of each first sub-model. When executing S803, the rendering terminal determines the priority of each target sub-model based on the priority data of each first sub-model sent by the acquisition terminal, and sorts the identifiers of each target sub-model in descending order. Since a higher priority indicates a higher level of attention to the target sub-model, sorting the identifiers of the target sub-models in descending order of priority allows for priority request of the initial reconstruction data of high-priority target sub-models, thereby ensuring the display accuracy of the 3D model within the field of view.

[0111] S804: Sends a data acquisition request to the data acquisition terminal.

[0112] In S804, the rendering terminal sends the identifiers of each sorted target sub-model to the cloud server in the data acquisition request, and the cloud server forwards it to the acquisition terminal so that the acquisition terminal can upload high-precision reconstructed data according to the data acquisition request.

[0113] S805: Receives the initial reconstruction data of the corresponding target sub-models sent sequentially by the acquisition terminal according to the identifiers of each target sub-model.

[0114] In the embodiments of this application, after receiving a data acquisition request, the acquisition terminal uses the initial reconstruction data of the first sub-model with the same identifier as the initial reconstruction data of the target sub-model according to the identifier of each target sub-model, and uploads the initial reconstruction data of the target sub-model to the cloud server in sequence, so that the rendering terminal downloads the initial reconstruction data of the target sub-model with the corresponding identifier from the cloud server in sequence.

[0115] It should be noted that in S805, unlike the complete low-precision target reconstruction data corresponding to each second sub-model, the high-precision initial reconstruction data corresponding to each target sub-model is sent independently to the rendering terminal to achieve fast rendering and display of the high-precision sub-model.

[0116] S806: Render and display the 3D model of the target object based on the initial reconstruction data of each target sub-model and the target reconstruction data of the second sub-model with a different identifier from each target sub-model.

[0117] In S806, the data of each target sub-model and the second sub-model with different identifiers are combined to form a complete reconstruction data. The accuracy of the initial reconstruction data of the target sub-model is higher than that of the target reconstruction data of the second sub-model. In this way, the rendered 3D model is a complete 3D model stitched together from sub-models with different accuracies.

[0118] like Figure 9 The diagram shows a 3D model assembled from sub-models of varying precision. The bounding box of the entire human 3D model is divided into 36 cuboid regions. Of these, 16 regions, including those whose vertices can be further divided into sub-models, are numbered {2, 3, 6, 7, 8, 9, 10, 11, 14, 15, 20, 21, 26, 27, 32, 33}. The high-precision data of the first sub-model (numbered 26, 27, 32, and 33) is not fully loaded and is rendered using the data of the lower-precision second sub-model with the same identifier. The first sub-model, which has already been loaded, is rendered using the high-precision data. After rendering, the human 3D model remains a complete 3D model.

[0119] As the high-precision target sub-model data is downloaded, the corresponding area is displayed as a high-precision sub-model instead of a low-precision one. Furthermore, sub-models with higher priority are displayed first, ensuring the display quality of the 3D model.

[0120] It's important to note that for the rendering terminal, the main thread sequentially retrieves the reconstructed data of the sub-models from the rendering queue to complete the rendering of one frame. The download of all sub-model reconstructed data occurs in a background thread and does not block the main thread from rendering the 3D model. Because the rendering queue contains complete reconstructed data of each sub-model, differing only in their level of precision, the 3D model rendered and displayed by the rendering terminal is complete.

[0121] In the above embodiments of this application, the acquisition terminal downsamples the initial reconstruction data of the 3D model to obtain target reconstruction data. It then segments the high-precision model corresponding to the initial reconstruction data and the low-precision model corresponding to the target reconstruction data to obtain first and second sub-models. The target reconstruction data of each second sub-model is then completely sent to the rendering terminal. Since the target reconstruction data is a complete set of reconstruction data, the integrity of the 3D model is guaranteed. Furthermore, the target reconstruction data has lower precision and smaller data volume, which can solve the problem of insufficient frame rate for 3D model actions caused by insufficient bandwidth. This ensures that the normal rendering and display of the model is not affected by data volume and bandwidth limitations, reducing video stuttering. Moreover, by determining the priority of each first sub-model through the acquisition terminal, the rendering terminal can download the initial reconstruction data of the target sub-models within the field of view in order of priority. Under existing network bandwidth conditions, this ensures that the areas of interest in the 3D model are prioritized for displaying high-precision sub-models, improving the display quality of the 3D model. For invisible areas, there is no need to download high-precision reconstruction data, reducing redundant data downloads, lowering network bandwidth pressure, and improving the overall rendering and display efficiency of the 3D model.

[0122] In the embodiments of this application, the acquisition terminal and the rendering terminal establish a communication connection through a cloud server, based on the interaction process between the acquisition terminal and the rendering terminal. Figure 10 This example illustrates a complete method flow for displaying a 3D model provided in an embodiment of this application, such as... Figure 10 As shown, this process is described using the rendering and display of a single frame of data as an example, and mainly includes the following steps:

[0123] S1001: The acquisition terminal reconstructs the 3D model of the target object and obtains the skeleton information of the 3D model.

[0124] S1002: The acquisition terminal downsamples the initial reconstruction data of the 3D model to obtain low-precision target reconstruction data.

[0125] S1003: The acquisition terminal generates a bounding box for the 3D model and divides the bounding box into several cuboids.

[0126] S1004: The acquisition terminal divides the high-precision model corresponding to the initial reconstruction data into first sub-models and the low-precision model corresponding to the target reconstruction data into second sub-models based on the cuboids where each vertex of the 3D model is located.

[0127] S1005: The acquisition terminal determines the priority of each first sub-model according to the preset priority of the skeleton node.

[0128] S1006: The acquisition terminal sends the priority data of each first sub-model and a complete frame of target reconstruction data of each second sub-model to the rendering terminal through the cloud server.

[0129] S1007: The rendering terminal receives the priority data of each first sub-model and the target reconstruction data of each second sub-model sent by the acquisition terminal, decodes the target reconstruction data, and stores it in the rendering queue.

[0130] S1008: The rendering terminal generates the bounding boxes for each second sub-model.

[0131] S1009: The rendering terminal determines whether the bounding boxes of each second sub-model are visible. If they are visible, S1010 is executed; otherwise, S1015 is executed.

[0132] S1010: The rendering terminal uses the identifier of the visible second sub-model as the identifier of the target sub-model, and determines the priority of each target sub-model based on the priority data of each first sub-model.

[0133] S1011: The rendering terminal sorts the identifiers of each target sub-model according to their priority and generates a request queue for high-precision sub-model data.

[0134] S1012: The rendering terminal sends a data acquisition request to the cloud server. The data acquisition request carries a request queue of high-precision sub-model data generated by the identifiers of each target sub-model.

[0135] S1013: The cloud server sequentially retrieves the initial reconstruction data of the target sub-model corresponding to the corresponding identifier from the acquisition terminal according to the request queue carried by the data request, and sends it to the rendering terminal.

[0136] S1014: The rendering terminal uses the initial reconstruction data of the received target sub-model to sequentially replace the target reconstruction data of the second sub-model with the same identifier, and updates the rendering queue.

[0137] S1015: The rendering terminal renders and displays the 3D model based on the reconstruction data of each sub-model in the rendering queue.

[0138] Based on the same technical concept, this application provides a data acquisition terminal that can execute the data acquisition terminal side process in the three-dimensional model display method provided in this application and achieve the same technical effect, which will not be repeated here.

[0139] See Figure 11 The acquisition terminal includes an RGBD camera 1101 and a host 1102. The RGBD camera is used to acquire depth and color images of the target object. The host 1102 includes at least a processor 1102_1, a memory 1102_2, and a communication interface 1102_3. The communication interface 1102_3 and the memory 1102_2 are connected to the processor 1102_1 via a bus 1102_4. The communication interface 1102_3 is used to communicate and transmit data with the rendering terminal. The memory 1102_2 stores a computer program. The processor 1102_1 executes the acquisition terminal method in the above embodiment according to the computer program.

[0140] It should be noted that, Figure 11 Only the hardware necessary for the acquisition terminal to execute the display method of the 3D model is shown. In addition, the acquisition terminal may also include conventional hardware of the display terminal, such as the display 1102_5.

[0141] Based on the same technical concept, this application provides a rendering terminal that can execute the rendering terminal side process in the three-dimensional model display method provided in this application and achieve the same technical effect, which will not be repeated here.

[0142] See Figure 12 The acquisition terminal includes a processor 1201, a memory 1202, a display 1203, and a communication interface 1204. The communication interface 1204, the display 1203, and the memory 1202 are connected to the processor 1201 via a bus 1205. The communication interface 1204 is used for communication and data transmission with the acquisition terminal. The display 1203 is used to display a three-dimensional model. The memory 1202 stores a computer program. The processor 1201 executes the rendering terminal side method in the above embodiment according to the computer program.

[0143] It should be noted that, Figure 12 Only the hardware necessary for the rendering terminal to execute the display method of the 3D model is shown. In addition, the rendering terminal may also include conventional hardware of the display terminal, such as the RGBD camera 1206.

[0144] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the methods of the foregoing embodiments.

[0145] This application also provides a computer program product for storing a computer program that performs the methods described in the foregoing embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0147] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A method for displaying three-dimensional reconstruction, characterized in that, Applied to data acquisition terminals, including: Based on the depth and color images of the target object, a three-dimensional model of the target object is reconstructed, and the skeleton of the three-dimensional model is obtained; The initial reconstruction data of the three-dimensional model is downsampled to obtain the target reconstruction data; Based on the preset priority of the skeleton nodes, the priority of each first sub-model in the model corresponding to the initial reconstruction data is determined respectively. The priority data of each first sub-model and the data of each second sub-model in the model corresponding to the target reconstruction data are sent to the rendering terminal, and the first sub-model and the second sub-model correspond one-to-one. The system receives a data acquisition request sent by the rendering terminal. The data acquisition request carries the identifiers of each target sub-model arranged in descending order of priority. The identifier of each target sub-model is determined based on the spatial relationship between the bounding boxes of each second sub-model and the view frustum of the rendering terminal. For any bounding box of each second sub-model, if the spatial relationship between the bounding box and the view frustum is that the bounding box intersects with the view frustum or is within the view frustum, then the identifier of the second sub-model corresponding to the bounding box is the identifier of a target sub-model. According to the identifiers of each target sub-model sorted in the data acquisition request, the initial reconstruction data corresponding to each target sub-model is sent to the rendering terminal in sequence.

2. The method as described in claim 1, characterized in that, The respective first sub-models and the respective second sub-models are obtained in the following way: Generate a bounding box corresponding to the 3D model of the target object, and divide the bounding box into several cuboids; Based on the cuboid where the model vertices corresponding to the initial reconstructed data are located, the model corresponding to the initial reconstructed data is spatially geometrically segmented; and based on the cuboid where the model vertices corresponding to the target reconstructed data are located, the model corresponding to the target reconstructed data is spatially geometrically segmented. For each vertex within the same cuboid space, a new topological relationship is established, and texture segmentation is performed on the initial reconstructed data based on the topological relationship, and texture segmentation is performed on the target reconstructed data based on the topological relationship; Each first sub-model is obtained based on the initial reconstruction data after spatial geometric segmentation and texture segmentation, and each second sub-model is obtained based on the target reconstruction data after spatial geometric segmentation and texture segmentation.

3. The method as described in claim 1 or 2, characterized in that, The step of determining the priority of each first sub-model in the model corresponding to the initial reconstruction data according to the preset priority of the skeleton nodes includes: For each of the first sub-models, perform the following operations: When there is a skeleton node within the cuboid corresponding to the sub-model, the priority of the skeleton node is used as the priority of the sub-model; or When there are different skeleton nodes within the cuboid corresponding to the sub-model, the highest priority of each skeleton node is used as the priority of the sub-model; or When the cuboid corresponding to the sub-model has no skeleton nodes but has vertices, the priority of the skeleton node that is closest to the cuboid among all skeleton nodes is used as the priority of the sub-model.

4. The method as described in claim 3, characterized in that, Among the priorities of the skeleton nodes, skeleton nodes of the same type in different locations have the same priority.

5. A method for displaying a three-dimensional model, characterized in that, Applied to rendering terminals, including: The system receives priority data of each first sub-model sent by the acquisition terminal, as well as data of each second sub-model in the model corresponding to the target reconstruction data; wherein, the priority data is determined according to the priority of the preset skeleton node, the first sub-model is a sub-model in the model corresponding to the initial reconstruction data of the target object, the target reconstruction data is obtained by downsampling the initial reconstruction data, and the first sub-model and the second sub-model correspond one-to-one. Generate the bounding boxes of each of the second sub-models and determine the spatial relationship between each bounding box and the view volume; For any one of the bounding boxes, when the spatial relationship is that the bounding box intersects with the view volume or is within the view volume, the identifier of the second sub-model corresponding to the bounding box is used as the identifier of a target sub-model within the view volume. The identifiers of each target sub-model are sorted in descending order of priority. Send a data acquisition request to the acquisition terminal, the data acquisition request carrying the identifiers of each sorted target sub-model; Receive the initial reconstruction data of the corresponding target sub-models sent sequentially by the acquisition terminal according to the identifiers of each target sub-model; Based on the initial reconstruction data of each target sub-model and the target reconstruction data of a second sub-model with a different identifier from each target sub-model, the three-dimensional model of the target object is rendered and displayed.

6. The method as described in claim 5, characterized in that, The step of determining the identifiers of each target sub-model contained within the view frustum based on the spatial relationship between each bounding box and the view frustum includes: For each of the bounding boxes, if the bounding box intersects with the view volume, or if the bounding box is within the view volume, then the identifier of the second sub-model corresponding to the bounding box is used as the identifier of the target sub-model within the view volume.

7. A data acquisition terminal, characterized in that, The system includes an RGBD camera and a host computer, wherein the host computer includes at least a processor, a memory, and a communication interface, and the memory and the communication interface are connected to the processor via a bus. The RGBD camera is used to acquire depth and color images of the target object; the memory stores a computer program, and the processor performs the following operations according to the computer program: Based on the depth and color images of the target object, a three-dimensional model of the target object is reconstructed, and the skeleton of the three-dimensional model is obtained; The initial reconstruction data of the three-dimensional model is downsampled to obtain the target reconstruction data; Based on the preset priority of the skeleton nodes, the priority of each first sub-model in the model corresponding to the initial reconstruction data is determined respectively. Through the communication interface, the priority data of each first sub-model and the data of each second sub-model in the model corresponding to the target reconstruction data are sent to the rendering terminal, and the first sub-model and the second sub-model correspond one-to-one; Through the communication interface, a data acquisition request sent by the rendering terminal is received. The data acquisition request carries the identifiers of each target sub-model arranged in descending order of priority. The identifier of each target sub-model is determined based on the spatial relationship between the bounding boxes of each second sub-model and the view frustum of the rendering terminal. For any bounding box of each second sub-model, if the spatial relationship between the bounding box and the view frustum is that the bounding box intersects with the view frustum or is within the view frustum, then the identifier of the second sub-model corresponding to the bounding box is the identifier of a target sub-model. According to the identifiers of each target sub-model sorted in the data acquisition request, the initial reconstruction data corresponding to each target sub-model is sent sequentially to the rendering terminal through the communication interface.

8. The data acquisition terminal as described in claim 7, characterized in that, The processor obtains each first sub-model and each second sub-model in the following manner: Generate a bounding box corresponding to the 3D model of the target object, and divide the bounding box into several cuboids; Based on the cuboid where the model vertices corresponding to the initial reconstructed data are located, the model corresponding to the initial reconstructed data is spatially geometrically segmented; and based on the cuboid where the model vertices corresponding to the target reconstructed data are located, the model corresponding to the target reconstructed data is spatially geometrically segmented. For each vertex within the same cuboid space, a new topological relationship is established, and texture segmentation is performed on the initial reconstructed data based on the topological relationship, and texture segmentation is performed on the target reconstructed data based on the topological relationship; Each first sub-model is obtained based on the initial reconstruction data after spatial geometric segmentation and texture segmentation, and each second sub-model is obtained based on the target reconstruction data after spatial geometric segmentation and texture segmentation.

9. The data acquisition terminal as described in claim 7 or 8, characterized in that, The processor determines the priority of each first sub-model in the model corresponding to the initial reconstruction data according to the preset priority of the skeleton nodes. The specific operation is as follows: For each of the first sub-models, perform the following operations: When there is a skeleton node in the cuboid corresponding to the sub-model, the priority of the skeleton node is used as the priority of the sub-model. When there are different skeleton nodes in the cuboid corresponding to the sub-model, the highest priority of each skeleton node is taken as the priority of the sub-model. or When the cuboid corresponding to the sub-model has no skeleton nodes but has vertices, the priority of the skeleton node that is closest to the cuboid among all skeleton nodes is used as the priority of the sub-model.

10. A rendering terminal, characterized in that, It includes a processor, a memory, a display, and a communication interface, wherein the memory, the display, and the communication interface are connected to the processor via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Through the communication interface, priority data of each first sub-model and data of each second sub-model in the model corresponding to the target reconstruction data are received from the acquisition terminal; wherein, the priority data is determined according to the priority of the preset skeleton node, the first sub-model is the sub-model in the model corresponding to the initial reconstruction data of the target object, the target reconstruction data is obtained by downsampling the initial reconstruction data, and the first sub-model and the second sub-model correspond one-to-one. Generate the bounding boxes of each of the second sub-models and determine the spatial relationship between each bounding box and the view volume; For any one of the bounding boxes, when the spatial relationship is that the bounding box intersects with the view volume or is within the view volume, the identifier of the second sub-model corresponding to the bounding box is used as the identifier of a target sub-model within the view volume. The identifiers of the target sub-models are sorted in descending order of priority. The data acquisition request is sent to the acquisition terminal through the communication interface. The data acquisition request carries the identifiers of each sorted target sub-model. Through the communication interface, the initial reconstruction data of the corresponding target sub-models, which are sent sequentially by the acquisition terminal according to the identifiers of each target sub-model, are received. Based on the initial reconstruction data of each target sub-model and the target reconstruction data of a second sub-model with a different identifier from each target sub-model, a three-dimensional model of the target object is rendered and displayed on the monitor.

Images