Video call method and device, equipment, storage medium and program product

Abstract

The invention discloses a video call method and device, equipment, a storage medium and a program product. The method comprises the following steps: receiving point cloud data of a target object sent by a first terminal; wherein the target object is an object in video data transmitted between the first terminal and a second terminal; performing coordinate conversion on the point cloud data, and establishing a standard coordinate system taking the target object as a center; dispersing the point cloud data after coordinate conversion into voxel grids by using the standard coordinate system, and rendering the voxel grids according to a set level; packaging the rendering result of each level to obtain modeling data; and sending the modeling data to the second terminal. By adopting the embodiment of the invention, real-time and efficient 3D modeling rendering and transmission in the call can be realized, and the communication requirement of a user is met.

Description

Technical Field

[0001] This invention relates to the field of communications, and more particularly to a video calling method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the widespread use of communication devices, real-time video calls have become an important means of communication for modern people. Such call scenarios require substantial data transmission support. In the current network environment, the transmission services running concurrently with video calls mainly rely on the dynamic allocation of radio resource blocks at the base station's MAC (Medium Access Control) layer. Video calls consume fixed resources, with the remaining resources used for other data transmissions, such as file transfers, image transfers, or simple text interactions. While real-time video calls have become the mainstream communication method, limitations imposed by weak network environments or the concurrency constraints of VoLTE (Voice over Long-Term Evolution) and non-IMS (IP Multimedia Subsystem) services often result in video calls and other data transmissions being unable to occur simultaneously, leading to the inability to efficiently transmit data in real-time during calls. Therefore, traditional video call methods can only provide planar visual information and cannot convey 3D spatial information of objects, making it difficult to meet communication needs in many scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a video call method, apparatus, device, storage medium, and program product that can achieve real-time and efficient 3D modeling, rendering, and transmission during a call, thereby meeting the user's communication needs.

[0004] To achieve the above objectives, embodiments of the present invention provide a video call method, including: Receive point cloud data of a target object sent by a first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal; The point cloud data is subjected to coordinate transformation, and a standard coordinate system centered on the target object is established; The point cloud data after coordinate transformation is discretized into a voxel grid using the standard coordinate system, and the voxel grid is rendered according to a set level. The rendering results of each level are encapsulated to obtain modeling data; The modeling data is sent to the second terminal.

[0005] As an improvement to the above scheme, establishing a standard coordinate system centered on the target object includes: Obtain the terminal coordinate matrix corresponding to the target object, and obtain the initial object coordinate matrix generated after coordinate transformation of the point cloud data; The center coordinates of the target object are determined using the terminal coordinate matrix. Based on the initial object coordinate matrix and the object center coordinates, a standard coordinate system centered on the target object is established.

[0006] As an improvement to the above scheme, the step of establishing a standard coordinate system centered on the target object based on the initial object coordinate matrix and the object center coordinates includes: The initial object coordinate matrix is ​​moved to a coordinate system with the center coordinates of the object as the origin to obtain a centered object coordinate matrix; Calculate the covariance matrix based on the centered object coordinate matrix, and solve for the eigenvalues ​​and eigenvectors in the covariance matrix; Construct an odd-order transformation matrix based on the eigenvalues ​​and eigenvectors; Multiplying the centered object coordinate matrix with the odd transformation matrix yields a standard coordinate system centered on the target object.

[0007] As an improvement to the above scheme, the rendering of the voxel mesh according to a set hierarchy includes: Seed points are selected from all coordinate points within each voxel grid at the current level; Determine the diffusion radius of each seed point; The seed points are merged using the diffusion radius until a preset stopping condition is reached, at which point the rendering of the voxel mesh at the current level is completed. The voxel mesh is refined according to a preset resolution level to obtain another level of voxel mesh. The voxel mesh of the other level is rendered until the preset highest resolution level is reached.

[0008] As an improvement to the above scheme, the step of selecting seed points from all coordinate points within each voxel grid at the current level includes: Calculate the density gradient at each coordinate point within each voxel grid, and use the coordinate point with the largest density gradient as the seed point for the current voxel grid.

[0009] As an improvement to the above scheme, determining the diffusion radius of each seed point includes: Calculate the density kernel of the seed point based on the coordinates of all coordinate points within each voxel grid and the coordinates of the seed point; Map the density kernel to a specified interval; The diffusion radius of each seed point is calculated using the mapped density kernel and the preset minimum diffusion radius.

[0010] As an improvement to the above scheme, the step of merging the seed points using the diffusion radius includes: Based on the diffusion radius corresponding to each seed point, determine the diffusion area in three-dimensional space that each seed point diffuses into its surroundings. Traverse the diffusion regions of all seed points and calculate the spatial distance between any two seed points; If the spatial distance between two seed points is less than a preset distance threshold, they are merged into a new seed point.

[0011] As an improvement to the above scheme, the stopping condition is: the number of seed points after merging reaches the target number of rendering points; wherein, the target number of rendering points is determined based on the total number of coordinate points of all voxel meshes in the current level.

[0012] As an improvement to the above scheme, after rendering the voxel mesh at the current level is completed, the method further includes: The rendered voxel mesh is subjected to texture processing operations for transparent objects; wherein the texture processing operations include at least one of displacement distortion processing and random noise addition.

[0013] As an improvement to the above solution, after receiving the modeling data, the second terminal parses the hierarchical information in the modeling data and displays the modeled target object in hierarchical order.

[0014] As an improvement to the above solution, the second terminal receives control commands transmitted by the first terminal through the data channel, updates the locally built-in state description file according to the control commands, and performs synchronous control operations on the modeled target object based on the updated state description file.

[0015] As an improvement to the above scheme, the control command carries a timestamp, session identifier, operation data, and the principal axis vector of the modeled target object.

[0016] As an improvement to the above solution, before receiving the point cloud data of the target object sent by the first terminal, the method further includes: Upon receiving the real-time modeling start command sent by the first terminal, the real-time modeling application is sent to the first terminal and the second terminal through the data channel, so that the first terminal and the second terminal can use the real-time modeling application to perform modeling interaction.

[0017] To achieve the above objectives, embodiments of the present invention also provide a video call device, comprising: A point cloud data receiving module is used to receive point cloud data of a target object sent by a first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal. A coordinate transformation module is used to perform coordinate transformation on the point cloud data; The standard coordinate system establishment module is used to establish a standard coordinate system centered on the target object; The point cloud data discretization module is used to discretize the coordinate-transformed point cloud data into a voxel grid using the standard coordinate system. The voxel mesh rendering module is used to render the voxel mesh according to a set level; The modeling data generation module is used to encapsulate the rendering results of each level to obtain modeling data; The modeling data sending module is used to send the modeling data to the second terminal.

[0018] To achieve the above objectives, embodiments of the present invention also provide a video call device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the video call method as described in any of the above embodiments.

[0019] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the video call method as described in any of the above embodiments.

[0020] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the video call method as described in any of the above embodiments.

[0021] Compared to existing technologies, the video call method, apparatus, device, storage medium, and program products disclosed in this invention can achieve real-time and efficient 3D modeling, rendering, and transmission during calls, meeting users' communication needs. By receiving point cloud data associated with video data from a first terminal, and combining coordinate transformation with the establishment of a centralized standard coordinate system for the target object, the spatial consistency of the point cloud data and the accuracy of the modeling benchmark are ensured, laying the foundation for the accurate generation of subsequent 3D models. At the same time, by adopting a hierarchical voxel mesh rendering and refinement strategy, combined with seed point selection, adaptive diffusion merging, and other fine-tuning processes, a progressive generation of the model from coarse to fine is achieved, ensuring both modeling efficiency and improving the detail reproduction of the model. In particular, the special texture processing of transparent objects further optimizes the visual presentation effect of objects with special materials. Furthermore, by encapsulating the rendering results at each level into standardized modeling data and transmitting it to the second terminal, not only is efficient and standardized transmission of modeling data achieved, but the second terminal also supports the simultaneous display of video streams and progressive models. Combined with the control command synchronization mechanism of the dedicated data channel for video calls, both users can view and interact with 3D models in real time during video calls. This breaks the limitation of traditional video calls that can only transmit 2D images, significantly improving the intuitiveness of object display and the real-time nature of interaction during calls. It provides richer information transmission methods for scenarios such as remote communication, product demonstrations, and scene sharing, enhancing the practicality and user experience of video calls. Attached Figure Description

[0022] Figure 1 This is a flowchart for generating a 3D model provided in the existing technology; Figure 2 This is a flowchart of the video call method provided in an embodiment of the present invention; Figure 3 This is another flowchart of the video call method provided in the embodiments of the present invention; Figure 4 This is a flowchart of establishing a standard coordinate system provided in an embodiment of the present invention; Figure 5 This is another flowchart for establishing a standard coordinate system provided in an embodiment of the present invention; Figure 6 This is a flowchart of voxel mesh rendering provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of the video call device provided in an embodiment of the present invention; Figure 8 This is a structural block diagram of the video call device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] 3D modeling over 5G networks still heavily relies on hardware and professional personnel. Firstly, a single-line LiDAR is required to collect tens of thousands of depth points through depth point sampling. This is then combined with SLAM (Simultaneous Localization and Mapping) technology to achieve bidirectional localization, completing the basic positioning of space and objects. (See [link to relevant documentation]). Figure 1 , Figure 1 This is a flowchart of the existing technology for generating 3D models. During the 3D modeling process, a high-resolution camera needs to be activated simultaneously to capture and restore the light details under different lighting conditions through HDR (High Dynamic Range) shooting. Then, the depth data collected by the LiDAR and the light information captured by the camera are uniformly summarized into the SLAM module to complete the material acquisition. After the material acquisition is completed, it still needs to go through point cloud stitching, data processing and other steps. Finally, a professional modeler will make fine adjustments to generate a complete 3D model.

[0025] Therefore, current 3D modeling and reconstruction require professional software for modeling and reconstruction. They cannot automatically construct based on transmitted data, lacking intelligence, and are time-consuming. Furthermore, they are limited by scanning technology and image acquisition methods, such as insufficient lighting, surface reflections, and occlusion, leading to incomplete or poor-quality data, thus affecting the accuracy of reconstruction results and preventing the construction of images. Traditional modeling techniques also fail to provide users with an intuitive, progressive demonstration of the model rendering process, presenting it only after the entire model has been created. General automated modeling techniques also cannot solve the problems of long modeling times and the inability to output intermediate results. The long modeling time and the inability to output only the final modeling result exacerbate the user's perception of long waiting times, resulting in a poor user experience and making them unsuitable for real-time communication scenarios.

[0026] To address the aforementioned shortcomings, this invention proposes a 3D modeling technology based on 5G new calls. During a 5G new call, users can choose to enable real-time modeling, utilizing the rear camera function on their mobile phones. Processed modeling data is transmitted in real-time during the call. Upon data arrival, preprocessing, coordinate transformation, and staged rendering are performed to construct a rough model location. Sensor data from LiDAR and other sensors are overlaid to build a physical property layer. Combined with 3D modeling technology, the solid model is rendered layer by layer from simple to complex. For special parts, such as glass or other objects with unique optical properties, special algorithmic processing is used for correction and confirmation, thus achieving a real-time modeling, rendering, and transmission solution. This allows the model to be rendered and displayed gradually, enabling users to experience objects more quickly in real-time call scenarios. Furthermore, this invention defines AR (Augmented Reality) commands, allowing both users to synchronize their operations on the model, enabling users to more intuitively and deeply understand and interact with surrounding real-world objects.

[0027] See Figure 2 , Figure 2 This is a flowchart of a video call method provided in an embodiment of the present invention, the video call method including steps S1 to S5. See also... Figure 3 , Figure 3 This is another flowchart of the video call method provided in the embodiments of the present invention, which is described below in conjunction with... Figure 3 Steps S1 to S5 are explained in detail.

[0028] Before performing step S1, the method further includes: when receiving a real-time modeling start command sent by the first terminal, sending a real-time modeling application to the first terminal and the second terminal through a data channel, so that the first terminal and the second terminal can use the real-time modeling application to perform modeling interaction.

[0029] For example, during a 5G new call, user A triggers a real-time modeling application selection operation through the interface of a first terminal. The first terminal then sends a real-time modeling start command to the server. Upon receiving the command, the server synchronously pushes the real-time modeling application to both the first and second terminals through the dedicated 5G new call data channel (DC), ensuring that both terminals complete the application loading and startup, thus establishing a unified technical environment for subsequent bidirectional modeling interaction. After the application starts, user A takes a video around the target object. The first terminal simultaneously activates the LiDAR component to collect spatial data, and then converts the 3D spatial points obtained by LiDAR scanning into point cloud data in a unified world coordinate system. For example, the first terminal uses ARKit technology to convert the 3D spatial points obtained by LiDAR scanning into point cloud data in the ARKit world coordinate system. The first terminal encapsulates the collected video stream and the point cloud data in real time and transmits them synchronously to the server through the 5G network. At the same time, the server forwards the video stream taken by user A to the second terminal in real time, allowing user B to view the captured image simultaneously.

[0030] In this embodiment of the invention, end users can initiate real-time modeling as needed, improving the flexibility of real-time modeling. Furthermore, by synchronously pushing the real-time modeling application to both terminals via a data channel, application deployment and collaborative preparation can be completed in advance, avoiding modeling process delays due to missing applications or loading delays. A unified application environment also lays a solid technical foundation for subsequent modeling interactions between the two terminals, ensuring seamless integration of point cloud acquisition, model rendering, and operation synchronization.

[0031] S1. Receive point cloud data of a target object sent by the first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal.

[0032] For example, when the server receives video data and point cloud data transmitted from the first terminal, it immediately parses and processes the two types of data to form a standardized dataset containing mobile phone coordinate data, point cloud coordinate data, and color data. Specifically, the point cloud data in ARKit world coordinates has automatically separated the terminal motion trajectory information from the pure point cloud data obtained by LiDAR scanning, without the need for additional separation operations. Secondly, the server directly obtains the point cloud data that has been aligned with RGB color information by calling the ARPointCloud interface. This data has achieved accurate matching between point cloud coordinates and video screen colors, laying the foundation for subsequent coordinate transformation and modeling rendering.

[0033] S2. Perform coordinate transformation on the point cloud data and establish a standard coordinate system centered on the target object.

[0034] For example, Principal Component Analysis (PCA) is used to determine the principal axes using the terminal coordinates, transforming the point cloud coordinates in the world coordinate system into a standard coordinate system centered on the object. PCA is a commonly used multivariate statistical analysis and data dimensionality reduction technique. Through linear transformation, it maps the original data scattered across multiple dimensions to a new orthogonal coordinate system (principal component space). The principal axis refers to the main characteristic direction axis of the three-dimensional spatial distribution of the target object, calculated by PCA. This principal axis accurately reflects the core extension direction of the target object and serves as the benchmark for constructing a standard coordinate system centered on the target object.

[0035] It's important to note that coordinate transformation of point cloud data is performed to eliminate differences in the coordinate systems of the first terminal (such as different mobile phone models or devices equipped with LiDAR). This transforms the ARKit world coordinate point cloud data into a universal coordinate format compatible with server processing, preventing data misalignment caused by hardware differences. The standard coordinate system is constructed centered on the target object, providing a unified spatial reference for collaborative modeling in 5G new call scenarios. This ensures that point cloud acquisition on the first terminal, voxel rendering on the server, and model display and interactive operations on the second terminal are all based on the same coordinate system, achieving precise cross-device data alignment. This lays a stable foundation for subsequent hierarchical rendering and real-time control command synchronization, guaranteeing smooth modeling interaction between the two terminals.

[0036] Further, see Figure 4 , Figure 4 This is a flowchart of establishing a standard coordinate system provided in an embodiment of the present invention. The establishment of the standard coordinate system centered on the target object includes steps S21 to S23.

[0037] S21. Obtain the terminal coordinate matrix corresponding to the target object, and obtain the initial object coordinate matrix generated after coordinate transformation of the point cloud data.

[0038] For example, assuming the terminal coordinates have n 3D points and the object coordinates have m 3D points, first construct an n×3 terminal coordinate matrix. and an initial object coordinate matrix of m×3 The two matrices are represented as follows: (1); (2); in, For terminal coordinate data, This refers to the object's coordinate data, specifically the point cloud data after coordinate transformation.

[0039] S22. Use the terminal coordinate matrix to determine the center coordinates of the target object.

[0040] For example, using the terminal coordinate matrix The geometric center (mean) is calculated, which is the coordinate of the object center of the target object. The calculation process satisfies the following formula: (3).

[0041] S23. Based on the initial object coordinate matrix and the object center coordinates, establish a standard coordinate system centered on the target object.

[0042] In this embodiment of the invention, by synchronously acquiring the terminal coordinate matrix corresponding to the target object and the initial object coordinate matrix after coordinate transformation, the coordinate association between the terminal device space and the object's own space is realized, ensuring accurate matching between the modeling benchmark and the terminal acquisition perspective. Simultaneously, the object's center coordinates are locked based on the terminal coordinate matrix, and then combined with the initial object coordinate matrix to complete centralized anchoring, ensuring that the standard coordinate system always takes the target object as its spatial core. This effectively avoids the modeling deformation problem caused by benchmark offset in traditional coordinate systems. This not only ensures the consistency of point cloud data under a unified spatial benchmark, providing accurate spatial reference for subsequent voxel mesh discretization and layered rendering, but also allows the 3D model to always revolve around the object itself, improving the morphological fit between the model and the real object, and laying a stable coordinate foundation for cross-terminal synchronous display and interaction.

[0043] Further, see Figure 5 , Figure 5 This is another flowchart for establishing a standard coordinate system provided in an embodiment of the present invention. Step S23 specifically includes steps S231 to S234.

[0044] S231. Move the initial object coordinate matrix to a coordinate system with the center coordinates of the object as the origin to obtain a centered object coordinate matrix.

[0045] For example, moving each point in the initial object coordinate matrix to a coordinate system with the object's center coordinates as the origin satisfies the following formula: (4); in, It is the centered object coordinate matrix.

[0046] S232. Calculate the covariance matrix based on the centered object coordinate matrix, and solve for the eigenvalues ​​and eigenvectors in the covariance matrix.

[0047] For example, the spatial distribution characteristics of point clouds are analyzed using the covariance matrix to determine the principal axis direction of an object. First, based on... and its transpose matrix The covariance of a point cloud is calculated according to the following formula: (5); in, Let be the covariance matrix.

[0048] Then, the eigenvalue decomposition equation is introduced to solve the covariance matrix, obtaining the eigenvalues ​​and eigenvectors. The solution process satisfies the following formula: (6); Among them, formula (6) is the characteristic decomposition equation. For the first 1 eigenvalue, To and The corresponding number eigenvectors, eigenvalues The size represents the point cloud in The variance (dispersion) in a direction indicates that the larger the eigenvalue, the more dispersed the distribution of the point cloud in that direction.

[0049] S233. Construct an odd-order transformation matrix based on the eigenvalues ​​and eigenvectors.

[0050] For example, firstly, a principal axis matrix is ​​constructed based on the eigenvalues ​​and the eigenvectors, satisfying the following formula: (7); in, Using the principal axis matrix, formula (7) represents sorting the eigenvalues ​​in descending order, and the corresponding eigenvectors. , , These are the first, second, and third principal axes, respectively. The first principal axis is the direction in which the point cloud distribution is most dispersed, corresponding to the core extension direction of the object.

[0051] Then, based on the principal axis matrix Construct an odd-order transformation matrix that satisfies the following formula: (8); in, It is an odd-order transformation matrix with a shape of 4x4, used to simultaneously perform rotation and translation transformations in homogeneous coordinates; is the transpose of the zero vector, i.e., [0,0,0]; 1 is the constant term of the homogeneous coordinates, used to accommodate translation transformations.

[0052] S234. Multiply the centered object coordinate matrix with the odd transformation matrix to obtain a standard coordinate system centered on the target object.

[0053] For example, the multiplication process satisfies the following formula: (9); in, This is the object coordinate matrix in the standard coordinate system with the object's center as the origin.

[0054] It should be noted that odd-order transformation matrices It is a homogeneous transformation matrix of pure rotation (because the translation term is 0), which transforms the centered object coordinate matrix. With the odd-order transformation matrix Multiplication, representing the process of transforming a homogeneous matrix. Rotate the centered point cloud so that its coordinate axes are aligned with the principal axis of the object. The resulting coordinates are those in the standard coordinate system with the object's center as the origin and the object's principal axis as the coordinate axes.

[0055] In this embodiment of the invention, the initial object coordinate matrix is ​​translated to a point with the object's center coordinates as the origin, eliminating the influence of coordinate reference offset on modeling and ensuring that the coordinate system always centers on the target object, providing a unified and stable spatial anchor point for subsequent modeling. By calculating the covariance matrix and solving for eigenvalues ​​and eigenvectors through the centered coordinate matrix, the three-dimensional spatial distribution characteristics and main extension direction of the target object can be accurately captured, making the constructed coordinate system highly compatible with the object's own shape and avoiding the model stretching and deformation problems caused by directional deviations in traditional coordinate systems. In addition, by constructing an odd-order transformation matrix using eigenvalues ​​and eigenvectors and completing coordinate mapping, the standardization and normalization of object coordinates are achieved. This not only ensures the consistency and comparability of point cloud data in spatial dimensions, providing accurate spatial references for subsequent steps such as voxel mesh discretization and layered rendering, but also ensures that the shape of the three-dimensional model highly matches the geometric features of the real object. This lays a solid foundation for the stability and accuracy of the model during cross-terminal synchronous display and real-time interaction, significantly improving the reliability and visual fidelity of three-dimensional modeling.

[0056] S3. Discretize the point cloud data after coordinate transformation into a voxel grid using the standard coordinate system, and render the voxel grid according to the set level.

[0057] For example, the point cloud data after coordinate transformation is first discretized into a coarsely segmented voxel mesh of 64×64×64 (for example only). A voxel mesh is a mesh structure formed by dividing a continuous three-dimensional space into regular cubic units. Each voxel corresponds to a fixed small cubic region in space. By marking whether the region contains point cloud data (or recording information such as point cloud density and color within the region), the discrete point cloud data is transformed into structured three-dimensional mesh data. In this embodiment of the invention, a 64×64×64 partitioning method is used, which uniformly divides the standard coordinate system space centered on the object into 64 voxel units of equal size along the X, Y, and Z axes, forming a three-dimensional mesh composed of 64×64×64=262144 small cubes. This coarsely segmented voxelization process can preserve the core spatial shape of the object and significantly reduce the amount of data, adapting to the computing power and transmission efficiency requirements of real-time modeling in the new 5G call scenario.

[0058] Furthermore, the formula for calculating the coordinates of the center point of each voxel is as follows: (10); (11); in, , This is the index (serial number) of the voxel in the 3D mesh. This range of values ​​covers exactly 64 elements, that is, from 0 to 63, a total of 64 indices, corresponding to all elements of a 64×64×64 voxel mesh. For example, when... At that time, it corresponds to the first voxel on the X-axis, the first voxel on the Y-axis, and the first voxel on the Z-axis in the grid; This is the voxel fineness coefficient matrix. , , , , , The maximum and minimum values ​​corresponding to the three-dimensional coordinates. The coordinates of the center point of each voxel grid.

[0059] Further, see Figure 6 , Figure 6 This is a flowchart of rendering a voxel mesh according to an embodiment of the present invention. The rendering of the voxel mesh according to a set level includes steps S31 to S34.

[0060] S31. Select seed points from all coordinate points within each voxel grid of the current level.

[0061] For example, the current level refers to a certain stage in the layered refinement process during 3D modeling. For instance, the 64×64×64 coarse voxel mesh in this case corresponds to the initial coarse level. Different levels correspond to different voxel mesh precision: the earlier the level, the larger the voxel unit size and the lower the mesh precision; the later the level, the more refined the voxel units and the higher the precision. Seed points are key sampling points selected from the coordinate points within each voxel mesh that represent the core spatial characteristics of that voxel, such as the geometric center point within the voxel or the point with the highest point cloud density. Their function is to serve as core anchor points for subsequent voxel mesh rendering and detail supplementation, ensuring that the core morphological information of the voxel region is preserved while controlling the computational load.

[0062] Further, step S31 specifically includes: calculating the density gradient of each coordinate point within each voxel grid, and using the coordinate point with the largest density gradient as the seed point of the current voxel grid.

[0063] For example, then for each voxel mesh, candidate initial seed points are selected based on density gradients, allowing for the rendering of key object parts first. The calculation is performed on each point within the voxel mesh, and the initial seed point is selected based on the highest value obtained from the following formula: (12); in, is a plane function formed by any point within the voxel grid and its two nearest points; The density gradient at the current point; , , respectively plane functions For each variable , , The partial derivative of D represents the rate of change of D in the corresponding coordinate direction. , , This corresponds to the coordinate dimensions in three-dimensional space. Formula (12) calculates the plane function. The partial derivatives in each direction in three-dimensional space are used to obtain the density gradient magnitude of the current point, which is used to measure the degree of change in point cloud density near the point and serves as a basis for selecting seed points.

[0064] In this embodiment of the invention, the points with the maximum density gradient often correspond to key feature regions such as critical nodes of the object's contour and areas of abrupt changes in detail. Using these as seed points avoids the problems of missing core features and unbalanced seed point distribution caused by traditional random selection or uniform sampling. Simultaneously, the density gradient-based selection logic ensures that the selection of seed points has clear physical meaning and spatial orientation, providing a precise and consistent core benchmark for subsequent adaptive diffusion radius calculation, seed point merging, and hierarchical rendering. This effectively reduces redundant calculations in subsequent rendering processes and improves the accuracy of model detail restoration and rendering efficiency.

[0065] S32. Determine the diffusion radius of each seed point.

[0066] For example, the diffusion radius refers to the effective range threshold for expanding morphological details within the voxel mesh centered on the current seed point.

[0067] Further, step S32 specifically includes: calculating the density kernel of the seed point based on the coordinates of all coordinate points in each voxel grid and the coordinates of the seed point; mapping the density kernel to a specified interval; and calculating the diffusion radius of each seed point using the mapped density kernel and a preset minimum diffusion radius.

[0068] For example, kernel density refers to a local density index obtained by weighting and aggregating the discrete point cloud data around a seed point using a function. Essentially, it is a quantification of the degree of aggregation of the point cloud near the target point. The higher the kernel density, the denser the point cloud distribution around the seed point, and the more accurately it reflects the core spatial features of the corresponding voxel region. Conversely, seed points with low kernel density may be located in sparse point cloud regions, have weaker representativeness, and can be removed or adjusted later. This ensures the morphological accuracy of the 3D model while controlling the computational load, meeting the high-efficiency requirements of 5G real-time modeling.

[0069] First, for the seed point Density kernel calculations are performed, and the calculation process satisfies the following formula: (13); in, , indicating the index range of the voxel grid in the corresponding dimension; This is the density kernel of the seed point of the current voxel; It is a set containing all coordinates of the current voxel; For set A coordinate point in the data; Point With seed point The Euclidean distance is used to measure similarity; Let be the standard deviation of the Gaussian kernel, which we take here. =0.01, used to control the decay rate of the Gaussian function; For Gaussian kernel function, it means that the closer the point is, the greater its contribution to the density kernel; the farther away the point is, the faster the contribution decays.

[0070] Then, the density kernel is mapped to the [0,1] interval, and the mapping process satisfies the following formula: (14); in, ; The normalized density kernel; and These are the maximum and minimum values ​​in the original density kernel of all seed points, respectively.

[0071] Finally, using the mapped density kernel and the preset minimum diffusion radius, the diffusion radius of each seed point is calculated. The calculation process satisfies the following formula: (15); in, ; The diffusion radius of the seed point; The minimum diffusion radius can take values ​​of [value]. ; The adjustment coefficient controls the magnitude of the diffusion radius variation with the density nucleus; it can take values ​​ranging from [value missing]. .

[0072] In this embodiment of the invention, by combining the coordinates of all coordinate points within the voxel mesh with the coordinates of the seed point to calculate the density kernel, the spatial density distribution characteristics of the region where the seed point is located are accurately captured. Then, through mapping the density kernel to a specified interval, a standardized conversion of density information is achieved. Finally, the mapped density kernel is used in conjunction with a preset minimum diffusion radius for calculation, allowing the diffusion radius of each seed point to dynamically adapt to the density characteristics of its region. This adaptive design avoids the loss of model details or waste of computational resources caused by a fixed diffusion radius, and provides a precise range basis for subsequent seed point merging and hierarchical rendering, significantly improving the detail reproduction and overall efficiency of voxel mesh rendering, making the generated 3D model more closely resemble the spatial structural characteristics of real objects.

[0073] S33. Merge the seed points using the diffusion radius until a preset stopping condition is reached, and then complete the rendering of the voxel mesh at the current level.

[0074] For example, merging seed points is used to eliminate redundancy and overlap of seed points in the current level of voxel mesh. Since the diffusion radii of different seed points may cover the same spatial area, the merging operation can integrate seed points that are close to each other and have overlapping diffusion ranges into a more representative core point, avoiding voxel shape redundancy or data waste caused by repeated rendering. Specifically, the merging process uses the seed point as the center and combines its adaptive diffusion radius to determine whether the coverage area of ​​adjacent seed points meets the merging conditions. If it does, these seed points are merged into a new seed point, and its diffusion radius is updated. This operation continues until a preset stopping condition is met, ultimately completing the shape rendering of the current level of voxel mesh. This ensures that the voxel mesh can completely cover the object features while controlling the amount of data and rendering complexity.

[0075] Furthermore, the step of merging the seed points using the diffusion radius includes: determining the diffusion region of each seed point in three-dimensional space according to the diffusion radius corresponding to each seed point; traversing the diffusion regions of all seed points and calculating the spatial distance between any two seed points; if there is a spatial distance between two seed points that is less than a preset distance threshold, then they are merged into a new seed point.

[0076] For example, firstly, based on the adaptive diffusion radius corresponding to each seed point, a dedicated three-dimensional spherical diffusion region is defined for it, clarifying the spatial influence range of various seed points; secondly, the diffusion regions of all seed points in the current level are traversed, and the spatial straight-line distance between any two seed points is calculated using the Euclidean distance algorithm to accurately quantify the positional relationship between the two points; finally, the calculated spatial distance is compared with a preset threshold (e.g., 0.03 units, approximately 2 bold voxels). If the distance is less than the threshold, it indicates that the diffusion regions of the two points overlap and are spatially close, belonging to redundant data. In this case, the two are merged into a new seed point.

[0077] In this embodiment of the invention, the three-dimensional spatial diffusion area of ​​various sub-points is first defined based on the diffusion radius to clarify the boundary of the influence range of the seed points. Then, by traversing and calculating the spatial distance between any two types of sub-points, an accurate overlap judgment standard is established, which can efficiently eliminate redundant overlapping seed points and avoid the waste of computing resources and excessive redundancy of model details caused by dense seed points in the subsequent rendering process.

[0078] Furthermore, the stopping condition is: the number of merged seed points reaches the target number of rendering points; wherein, the target number of rendering points is determined based on the total number of coordinate points of all voxel meshes in the current level.

[0079] For example, when the number of merged seed points reaches the target number of rendering points, the rendering of this layer ends. Since different levels of voxel meshes correspond to different levels of detail precision, the target number of rendering points is a reasonable number of seed points that matches the precision of the current level. Too many seed points will lead to detail redundancy, such as exceeding the precision requirements of the current level. Too few seed points will not be able to cover the core features of the object. Therefore, when the target number of points is reached, the shape rendering of the current level has met the precision requirements.

[0080] Furthermore, the calculation process for the target rendering point count satisfies the following formula: (16); in, Render the target number of points; This represents the total number of coordinate points for all voxel meshes at the current level.

[0081] In this embodiment of the invention, the stopping condition is based on the total number of coordinate points of all voxel meshes in the current level to determine the target rendering points, which provides a clear quantitative termination standard for seed point merging, effectively avoiding the problem of excessive merging caused by the lack of a unified reference during the merging process, and ensuring that model details are not lost.

[0082] S34. Refine the voxel mesh according to the preset resolution level to obtain another level of voxel mesh, and render the other level of voxel mesh until the preset highest resolution level is reached.

[0083] For example, after one layer of rendering is completed, the resolution is gradually refined from 64×64×64 to 512×512×512, that is, multi-level refinement rendering is performed according to the rule of doubling the resolution at each level (64→128→256→512), and then the above steps S31~S34 are repeated. At this time, the formula for calculating the voxel refinement coefficient matrix is ​​updated to: (17); in, Given the current voxel side length, To refine the hierarchy, such as from level 1 to level 3.

[0084] In this embodiment of the invention, seed points are selected based on coordinates within the voxel mesh, establishing a precise core benchmark for each rendering level and avoiding resource waste caused by indiscriminate rendering. By selectively configuring the diffusion radius of seed points and merging them based on overlap judgment of diffusion areas, redundant seed points are eliminated, computational efficiency is optimized, and uniform coverage of model details is ensured, avoiding problems such as missing or overly dense local details. Secondly, a preset stopping condition controls the termination of rendering at a single level, ensuring that the detail richness of each level meets the standard. Then, the voxel mesh is gradually refined through preset resolution levels, and the rendering process is repeated, achieving a gradual improvement in model accuracy. This balances the efficiency requirements of real-time modeling, avoids computational overload from one-time high-resolution rendering, and ensures the detail reproduction of the final model. The overall process takes into account both rendering efficiency and model quality. Through hierarchical iteration and refined seed point processing, the 3D model can quickly present its basic form and gradually supplement details in real-time interactive scenarios, providing efficient and stable technical support for cross-terminal synchronous display and real-time operation, and significantly improving the real-time performance and visual effects of 3D modeling in video calls.

[0085] Furthermore, after rendering the voxel mesh at the current level is completed, the method further includes: performing a texture processing operation on the rendered voxel mesh to represent transparent objects; wherein the texture processing operation includes at least one of displacement distortion processing and random noise addition.

[0086] For example, a transparent object is an object that is visually translucent and allows light to pass through, allowing the view of objects behind it, such as glass, water, and plastic film. Because the optical properties of transparent objects (such as refraction and reflection) are quite complex, direct accurate modeling and rendering using point cloud data often yields poor results or incurs excessive computational costs. Therefore, in real-time modeling scenarios, specific texture processing techniques are often used to simulate their appearance.

[0087] It's important to note that displacement distortion processing involves calculating a series of displacement values ​​using mathematical algorithms (such as NumPy's trigonometric functions). These displacement values ​​form ripple-like or uneven patterns. Then, OpenCV's `remap` function is used to remap the pixel positions of the rendered image (or texture map) based on these calculated displacement values. This visually adds a distorted texture to the surface of the transparent object, simulating refraction or ripple effects, making it look more realistic and three-dimensional. Random noise addition involves overlaying a layer of random brightness or color variations onto the rendered image of the transparent object using a random number generator. To ensure these random values ​​are within the valid range of the image (usually grayscale or RGB values ​​from 0 to 255), the `clip` function is used to limit the generated random values ​​to this range. Adding random noise can simulate minor imperfections, dust, or uneven refractive index on the surface of a transparent object, helping to break the plasticky look that perfect rendering might produce, making its appearance more natural and realistic.

[0088] In this embodiment of the invention, after voxel mesh rendering is completed, targeted optimization techniques of displacement distortion and random noise addition are introduced. Displacement distortion simulates the natural ripple texture of transparent materials (such as glass and crystal) caused by light refraction, while random noise restores the realistic characteristics of subtle uneven reflections on transparent surfaces. This effectively compensates for the shortcomings of traditional voxel rendering, which results in monotonous and textureless images of transparent objects. The flexible combination of these two processing methods enriches the detail representation of transparent objects and enhances the visual hierarchy and realism of the rendered image. This makes the transparent materials in the 3D model more closely resemble real-world scenes, significantly improving the overall visual effect and user experience of voxel mesh rendering, and providing strong support for the intuitive display of the structure and material properties of transparent objects in educational scenarios.

[0089] S4. Encapsulate the rendering results of each level to obtain modeling data.

[0090] For example, the 3D model data obtained after hierarchical rendering is converted and packaged into a standard, efficient, and easily transmitted and parsed file format. For instance, the glTF (GL Transmission Format) format can be used for encapsulation. Specifically, the encapsulation process includes: organizing the rendered vertex, index, texture coordinates, normals, and other information; associating the material parameters determined during rendering, as well as possible special textures (such as displacement distortion textures for transparent objects, noise textures), with the geometric data; creating a glTF JSON description file, which contains the scene structure, node hierarchy, camera, lighting, material references, and references to binary data for the entire 3D model; and simultaneously separating the massive binary content, such as geometric vertex data and texture image data, and storing it as one or more bin (binary) files. The advantage of this approach is that glTF files are small in size, facilitating fast transmission and parsing, while bin files efficiently store a large amount of binary data. The final combination of glTF and bin files constitutes the modeling data.

[0091] S5. Send the modeling data to the second terminal.

[0092] For example, the encapsulated glTF format modeling data is transmitted to a second terminal that needs to display the model.

[0093] In this embodiment of the invention, by receiving point cloud data associated with video data from the first terminal, and combining coordinate transformation with the establishment of a centralized standard coordinate system for the target object, the spatial consistency of the point cloud data and the accuracy of the modeling benchmark are ensured, laying the foundation for the accurate generation of the subsequent 3D model. At the same time, by adopting a hierarchical voxel mesh rendering and refinement strategy, combined with seed point selection, adaptive diffusion merging and other fine processing, the model is generated progressively from coarse to fine, which not only ensures modeling efficiency but also improves the detail reproduction of the model. In particular, the visual presentation effect of objects with special materials is further optimized through special texture processing of transparent objects. Furthermore, by encapsulating the rendering results at each level into standardized modeling data and transmitting it to the second terminal, not only is efficient and standardized transmission of modeling data achieved, but the second terminal also supports the simultaneous display of video streams and progressive models. Combined with the control command synchronization mechanism of the dedicated data channel for video calls, both users can view and interact with 3D models in real time during video calls. This breaks the limitation of traditional video calls that can only transmit 2D images, significantly improving the intuitiveness of object display and the real-time nature of interaction during calls. It provides richer information transmission methods for scenarios such as remote communication, product demonstrations, and scene sharing, enhancing the practicality and user experience of video calls.

[0094] Furthermore, after executing step S5, the second terminal receives the modeling data, parses the hierarchical information in the modeling data, and displays the modeled target object in hierarchical order.

[0095] For example, the second terminal can continuously receive glTF block modeling data sent from the first terminal via a mini-program. It first parses key information such as layer identifiers and data block relationships in the glTF description file to clarify the modeling layer corresponding to each data block (e.g., initial coarse layer, intermediate refinement layer, final high-precision layer). Simultaneously, it parses the substantive content such as geometric data and texture data in the bin file. Based on the parsed layer information, the modeling data of each layer is sorted in a preset order from coarse to fine. That is, the basic model data corresponding to the 64×64×64 coarse block voxel mesh is processed first, and then the supplementary data of subsequent refinement layers is loaded sequentially, ensuring that the display process is consistent with the hierarchical flow of modeling. The mini-program first renders the core outline of the target object based on the first layer (coarse layer) modeling data, allowing user B to initially identify the overall structure of the object. As subsequent layer data is continuously received and parsed, details are gradually added to the rendered model, such as fine textures filling the voxel mesh and special effects for transparent objects, ultimately presenting a high-precision complete model. At the same time, the mini program will simultaneously overlay the real-time video footage captured by user A, making the model echo the real scene. This allows user B to clearly see the complete rendering process from rough outline to fine form, reducing the feeling of waiting and enhancing the immersive experience of real-time interaction, fully adapting to the real-time visualization needs of 5G new calls.

[0096] In this embodiment of the invention, the second terminal displays the target object in a hierarchical order, which can clearly present the model's construction process and spatial structure logic from rough to detailed, enabling a more intuitive understanding of the progressive principle of modeling and improving the user experience.

[0097] Furthermore, this invention proposes a method for defining, transmitting, and synchronizing AR commands based on model operations. It defines a general format for encapsulating AR commands. At regular intervals, the user terminal transmits the current operation's AR commands in XML (Extensible Markup Language) format to another user terminal via a DC channel. Upon receiving the file, the other user terminal updates its state file. Once the 5G new call program detects the state file change, it smoothly synchronizes the current position and zoom level, as well as any newly added doodles, at the modeling and display level. This asynchronous synchronization method leverages the low latency and high bandwidth of the 5G network, making operation synchronization between users faster, more accurate, and easier, while simultaneously enhancing the user experience.

[0098] At this time, the second terminal receives the control command transmitted by the first terminal through the data channel, updates the locally built-in state description file according to the control command, and performs synchronous control operation on the modeled target object based on the updated state description file; wherein, the control command carries a timestamp, session identifier, operation data and the principal axis vector of the modeled target object.

[0099] For example, user A can perform operations on the modeled target object, such as rotating the model, marking it, zooming in and out, etc. The first terminal will generate corresponding control commands in real time. The control commands are encapsulated in XML format, with parent elements... <ar-command>The protocol type is specified, and the control command carries the current timestamp, session identifier, operation data, and the principal axis vector of the target object. The operation data includes key information such as the operation type (e.g., drawing, moving, zooming in / out) and operation parameters (e.g., drawing color, base64 encoding of touchscreen position), and is transmitted to the second terminal via the DC data channel at 20ms intervals (consistent with the audio sampling rate). The second terminal's 5G new calling mini-program has a built-in XML-formatted status description file, with the parent element... <ar:modelstate>The identifier includes the session ID, timestamp, and AR version. <ar:metadata>Record the principal axis position and scaling ratio of the model. <ar:transform>Record the status of your doodles (start and end points, brush thickness, etc.). <ar:annotations>Sub-elements.

[0100] When the second terminal receives the control command, it parses the operation information in the command and synchronously updates the local state description file. For example, if the command is a drawing operation, then... <ar:annotations>Add corresponding child elements to supplement the data such as the color and position of the graffiti; if the instruction is to rotate the model, then update... <ar:transform>The first terminal sends a control command carrying the new spindle position information of the model. The second terminal's mini-program parses the updated status description file and translates the corresponding operations into real-time actions of the model. For example, if user A rotates the model, the first terminal sends a control command carrying the model's new spindle position, and the second terminal updates... <ar:transform>After the principal axis vector is set, the local model will smoothly rotate to the corresponding angle; user B performs a graffiti marking operation, the second terminal sends a command carrying the graffiti color and touch screen position, and the first terminal updates... <ar:annotations>Then, the same marker symbol will be rendered at the corresponding position on the model to achieve zero-delay synchronization of operations at both ends and ensure consistent interaction.

[0101] In this embodiment of the invention, the second terminal receives and executes control commands from the first terminal, achieving real-time alignment of the object states modeled by both terminals. This ensures precise synchronization of operations (such as rotation and marking) on ​​the target object by both parties, avoiding state deviations during collaboration. It clearly presents the logical closed loop of dual-terminal collaborative control and improves the stability and reliability of remote interaction. Furthermore, the control commands integrate key information such as timestamps, session identifiers, operation data, and the principal axis vector of the target object, providing comprehensive data support for dual-terminal synchronization. The collaborative carrying of multiple types of information enables precise parsing and execution of operation commands and ensures a high degree of synchronization of the object states modeled by both terminals.

[0102] Existing 4G networks employ a centralized core network, with data processing heavily reliant on central nodes. Video calls consume a significant portion of the bandwidth, resulting in extremely slow data transmission speeds for remaining bandwidth. This can also negatively impact video call quality, causing frame drops and significant resource contention. This invention addresses this by utilizing 5G network transmission and sensor collaboration to perform network slicing, dynamically isolating these two services to prevent interference. Furthermore, it combines distributed edge computing to provide data support for real-time AI rendering and modeling. Based on video information transmitted from sensors and different angles, algorithms are used for AI-driven in-depth rendering and real-time rendering.

[0103] Furthermore, traditional industry modeling capabilities require high-quality equipment, such as high-frequency cameras and high-precision detectors, to acquire massive amounts of data to calculate a camera's location and build 3D models based on different camera perspectives. They also require professional modelers to operate and modify models using fixed software. This significantly raises the barrier to entry for users of 3D modeling capabilities, limiting its application scenarios. Additionally, the real-time performance of traditional industry modeling is unsatisfactory. This invention, by adding certain functionalities to video calls, enhances the interaction between callers under 5G, enabling real-time object transfer in the network world. It combines traditional industry expertise with AI and 5G development, deeply integrating into users' lives, expanding application scenarios, and providing convenience for users to quickly and easily perceive remote objects in real time.

[0104] See Figure 7 , Figure 7 This is a structural block diagram of a video call device 100 provided in an embodiment of the present invention. The video call device 100 includes: The point cloud data receiving module 11 is used to receive point cloud data of a target object sent by the first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal. Coordinate transformation module 12 is used to perform coordinate transformation on the point cloud data; Standard coordinate system establishment module 13 is used to establish a standard coordinate system centered on the target object; Point cloud data discretization module 14 is used to discretize the coordinate-transformed point cloud data into a voxel grid using the standard coordinate system; The voxel mesh rendering module 15 is used to render the voxel mesh according to a set level; The modeling data generation module 16 is used to encapsulate the rendering results of each level to obtain modeling data; The modeling data sending module 17 is used to send the modeling data to the second terminal.

[0105] Specifically, the video call device 100 further includes: The real-time modeling application sending module is used to send the real-time modeling application to the first terminal and the second terminal through a data channel when it receives the real-time modeling start command sent by the first terminal, so that the first terminal and the second terminal can use the real-time modeling application to perform modeling interaction.

[0106] It is worth noting that the working process of each module in the video call device 100 described in the embodiments of the present invention can refer to the working process of the video call method described in the above embodiments, and will not be repeated here.

[0107] See Figure 8 , Figure 8 This is a structural block diagram of a video call device 200 provided in an embodiment of the present invention. The video call device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the various video call method embodiments described above.

[0108] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the video conferencing device 200.

[0109] The video call device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the video call device 200 and does not constitute a limitation on the video call device 200. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the video call device 200 may also include input / output devices, network access devices, buses, etc.

[0110] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the video conferencing device 200, connecting all parts of the video conferencing device 200 via various interfaces and lines.

[0111] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the video calling device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0112] If the modules / units integrated into the video calling device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0113] Furthermore, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the video call method as described in any of the above embodiments.

[0114] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.< / ar:annotations> < / ar:transform> < / ar:transform> < / ar:annotations> < / ar:annotations> < / ar:transform> < / ar:metadata> < / ar:modelstate>

Claims

1. A video call method, characterized in that, include: Receive point cloud data of a target object sent by a first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal; The point cloud data is subjected to coordinate transformation, and a standard coordinate system centered on the target object is established; The point cloud data after coordinate transformation is discretized into a voxel grid using the standard coordinate system, and the voxel grid is rendered according to a set level. The rendering results of each level are encapsulated to obtain modeling data; The modeling data is sent to the second terminal.

2. The video call method as described in claim 1, characterized in that, The establishment of a standard coordinate system centered on the target object includes: Obtain the terminal coordinate matrix corresponding to the target object, and obtain the initial object coordinate matrix generated after coordinate transformation of the point cloud data; The center coordinates of the target object are determined using the terminal coordinate matrix. Based on the initial object coordinate matrix and the object center coordinates, a standard coordinate system centered on the target object is established.

3. The video call method as described in claim 2, characterized in that, The step of establishing a standard coordinate system centered on the target object based on the initial object coordinate matrix and the object center coordinates includes: The initial object coordinate matrix is ​​moved to a coordinate system with the center coordinates of the object as the origin to obtain a centered object coordinate matrix; Calculate the covariance matrix based on the centered object coordinate matrix, and solve for the eigenvalues ​​and eigenvectors in the covariance matrix; Construct an odd-order transformation matrix based on the eigenvalues ​​and eigenvectors; Multiplying the centered object coordinate matrix with the odd transformation matrix yields a standard coordinate system centered on the target object.

4. The video call method as described in claim 1, characterized in that, The rendering of the voxel mesh according to the set hierarchy includes: Seed points are selected from all coordinate points within each voxel grid at the current level; Determine the diffusion radius of each seed point; The seed points are merged using the diffusion radius until a preset stopping condition is reached, at which point the rendering of the voxel mesh at the current level is completed. The voxel mesh is refined according to a preset resolution level to obtain another level of voxel mesh. The voxel mesh of the other level is rendered until the preset highest resolution level is reached.

5. The video call method as described in claim 4, characterized in that, The process of selecting seed points from all coordinate points within each voxel grid at the current level includes: Calculate the density gradient at each coordinate point within each voxel grid, and use the coordinate point with the largest density gradient as the seed point for the current voxel grid.

6. The video call method as described in claim 4, characterized in that, Determining the diffusion radius of each seed point includes: Calculate the density kernel of the seed point based on the coordinates of all coordinate points within each voxel grid and the coordinates of the seed point; Map the density kernel to a specified interval; The diffusion radius of each seed point is calculated using the mapped density kernel and the preset minimum diffusion radius.

7. The video call method as described in claim 4, characterized in that, The process of merging the seed points using the diffusion radius includes: Based on the diffusion radius corresponding to each seed point, determine the diffusion area in three-dimensional space that each seed point diffuses into its surroundings. Traverse the diffusion regions of all seed points and calculate the spatial distance between any two seed points; If the spatial distance between two seed points is less than a preset distance threshold, they are merged into a new seed point.

8. The video call method as described in claim 4, characterized in that, The stopping condition is: the number of seed points after merging reaches the target number of rendering points; wherein, the target number of rendering points is determined based on the total number of coordinate points of all voxel meshes in the current level.

9. The video call method as described in claim 4, characterized in that, After completing the rendering of the voxel mesh at the current level, the method further includes: The rendered voxel mesh is subjected to texture processing operations for transparent objects; wherein the texture processing operations include at least one of displacement distortion processing and random noise addition.

10. The video call method as described in claim 1, characterized in that, After receiving the modeling data, the second terminal parses the hierarchical information in the modeling data and displays the modeled target object in hierarchical order.

11. The video call method as described in claim 10, characterized in that, The second terminal receives control commands transmitted by the first terminal through the data channel, updates its locally built-in state description file according to the control commands, and performs synchronous control operations on the modeled target object based on the updated state description file.

12. The video call method as described in claim 11, characterized in that, The control command carries a timestamp, session identifier, operation data, and the principal axis vector of the modeled target object.

13. The video call method as described in claim 1, characterized in that, Before receiving the point cloud data of the target object sent by the first terminal, the method further includes: Upon receiving the real-time modeling start command sent by the first terminal, the real-time modeling application is sent to the first terminal and the second terminal through the data channel, so that the first terminal and the second terminal can use the real-time modeling application to perform modeling interaction.

14. A video call device, characterized in that, include: A point cloud data receiving module is used to receive point cloud data of a target object sent by a first terminal; wherein the target object is an object in the video data transmitted between the first terminal and the second terminal. A coordinate transformation module is used to perform coordinate transformation on the point cloud data; The standard coordinate system establishment module is used to establish a standard coordinate system centered on the target object; The point cloud data discretization module is used to discretize the coordinate-transformed point cloud data into a voxel grid using the standard coordinate system. The voxel mesh rendering module is used to render the voxel mesh according to a set level; The modeling data generation module is used to encapsulate the rendering results of each level to obtain modeling data; The modeling data sending module is used to send the modeling data to the second terminal.

15. A video call device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the video call method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the video call method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the video call method as described in any one of claims 1 to 13.

Images