Computer-implemented method, electronic device, and computer-readable storage medium for simulating deformation in real-world scenes

Abstract

The present disclosure provides a computer-implemented method. The method includes: (101) acquiring 3D volume data for representing a real-world scene; (102) in response to an activation action on the 3D volume data, acquiring a current active 3D volume data set from the 3D volume data; (103) updating multiple active 3D volume data sets based on the current active 3D volume data set, the multiple active 3D volume data sets being acquired from the 3D volume data; (104) constructing a surface based on the multiple updated active 3D volume data sets; and (105) simulating a deformation of a portion of the real-world scene based on the surface, the portion of the real-world scene corresponding to the multiple updated active 3D volume data sets. In addition, the present disclosure also provides an electronic device and a storage medium.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. patent application serial number 62 / 957,702, filed in the U.S. Patent and Trademark Office on January 6, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of augmented reality technology, and more specifically, to a computer-implemented method for simulating a deformation in a real-world scene, an electronic device, and a computer-readable storage medium. Background Art

[0004] In an augmented reality (AR) application presented on many frameworks and many devices such as Hololens, Magic Leap One, ARcore and ARkit, the geometric information of a real-world static scene can be scanned and stored in advance, and a scene map can be established based on it, i.e., three-dimensional (3D) reconstruction.

[0005] One AR application is to enhance (many) real objects in the scene with physical simulation (e.g., deformation). Specifically, the simulated (many) visual results can be added to the pre-scanned scene, for example, virtual drilling in a wall and virtual cracking in the floor. Physical simulation of virtual deformation of many real objects, for example, deforming a part of a sculpture, and enhancing the real sculpture with the simulation results for display, can be applied to many fields, such as art, games and medical care. Summary of the invention

[0006] A first aspect of the present disclosure provides a computer-implemented method, comprising: obtaining three-dimensional (3D) volume data for representing a real-world scene; in response to an activation action on the 3D volume data, obtaining a current active 3D volume data set from the 3D volume data; updating multiple active 3D volume data sets based on the current active 3D volume data set, the multiple active 3D volume data sets being obtained from the 3D volume data; constructing a surface based on the multiple updated active 3D volume data sets; and simulating a deformation of a portion of the real-world scene based on the surface, the portion of the real-world scene corresponding to the multiple updated active 3D volume data sets.

[0007] A second aspect of the present disclosure provides an electronic device, comprising: a memory; a processor; and a computer program stored in the memory and operable on the processor. When the program is executed by the processor, the method according to the first aspect is implemented.

[0008] A third aspect of the present disclosure provides a computer-readable storage medium on which a computer-readable storage instruction is stored. The computer-readable storage instruction is configured to cause a computer to implement the method according to the first aspect.

[0009] Many additional aspects and advantages of the various embodiments of the present disclosure will be given in part in the following description, and in part will be obvious from the following description, or will be learned from the practice of the various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the various embodiments of the present disclosure will become apparent and more easily understood from the following description made with reference to the accompanying drawings, in which:

[0011] Figure 1 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0012] Figure 2 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0013] Figure 3 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0014] Figure 4 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0015] Figure 5 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0016] Figure 6 is a schematic diagram of a pixel grid according to an embodiment of the present disclosure.

[0017] Figure 7 It is a block diagram of a device for an augmented reality scene according to an embodiment of the present disclosure.

[0018] Figure 8 It is a block diagram of a device for an augmented reality scene according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Many embodiments of the present disclosure will be described in detail and many examples of many embodiments are illustrated in the accompanying drawings. Throughout the description, the same or similar elements and elements with the same or similar functions are marked with the same reference numerals. The many embodiments described with reference to the accompanying drawings are illustrative and are used to explain the present disclosure. The many embodiments are not to be construed as limiting the many embodiments of the present disclosure.

[0020] In many AR applications, the geometric information of real-world static scenes may be scanned and stored in advance. The scanned geometric information may be represented by data in an implicit data structure. Many algorithms may be executed on the data in the implicit data structure to convert the data in the implicit data structure into data in an explicit data structure. Object segmentation may be performed based on the data in the explicit data structure to obtain a plurality of segmented objects. Physical simulation, such as deformation simulation, may be performed on the plurality of segmented objects, for example, cloth deformation or soft body deformation.

[0021] However, when an object contains a lot of geometric details, such as simulating the deformation of a sculpture object, or simulating the deformation of a wall as a piece of fabric, the physical simulation of the entire object is very time-consuming. In addition, the situation of simultaneous simulation and scanning in the AR scene is not excluded. In addition, the physical simulation requires generating new data in the explicit data structure after scanning each frame, which will increase the processing time of each frame, resulting in slow simulation updates of many objects with complex geometric information.

[0022] The purpose of the present disclosure is to provide a method for an augmented reality scene, which aims to solve the technical problem that the deformation simulation of the object in the prior art is time-consuming and has a slow simulation speed.

[0023] In this method, when it is detected that the virtual object collides with the real scene (i.e., a user places the virtual object into the augmented reality scene, such as dragging a virtual person or a virtual ball into the augmented reality scene, or controls the virtual object in the augmented reality scene to interact with the augmented reality scene, such as the user hitting a virtual ball), a region of interest (ROI) corresponding to the current collision in the scene is extracted. If a set of global regions can be obtained for a set of collisions, when the ROI overlaps with one or more existing global regions, the ROI is combined with the existing global regions. When there is no overlap between the ROI and each existing global region, the ROI will be regarded as a new global region. In addition, if there is no global region, the ROI will be regarded as a new global region. Finally, based on the multiple global regions, the deformations corresponding to the multiple collisions are simulated to generate a deformation simulation result. The external force driving the deformation can be simulated using the collision penetration distance based on the multiple global regions to achieve deformation simulation of the multiple global regions. Thus, the deformation simulation can be calculated for the global areas instead of all objects and all data structures of the scene, so that the simulation speed is greatly improved.

[0024] Figure 1 The present invention is a flowchart of a method for an augmented reality scene according to an embodiment of the present invention.

[0025] Many embodiments of the present disclosure provide a device for an augmented reality scene to implement the method for an augmented reality scene. The device can be applied to any electronic device, so that the electronic device can perform the above-mentioned function of simulating the deformation in the augmented reality scene. The electronic device can be, for example, a hardware device with various operating systems, touch screens and / or display screens, such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device or the like.

[0026] As in Figure 1 As illustrated, the method includes the following actions in the following blocks.

[0027] At block 101 , three-dimensional (3D) volumetric data representing a real-world scene is acquired.

[0028] Depth data for the real-world scene may be captured by a depth camera, and the 3D volume data may be acquired based on the captured depth data.

[0029] In one embodiment, the 3D volume data may include a set of voxels in a signed distance function (SDF) or a truncated signed distance function (TSDF). For example, if a red-green-blue-depth (RGB-D) camera is used to capture the scene, the captured depth data may be represented by the data in the SDF or TSDF to reconstruct the three-dimensional scene.

[0030] SDF is a 3D geometric data structure. The input of the SDF is a spatial coordinate of a point, and the output of the SDF is a directional distance, which represents the closest distance from the input point to the nearest 3D geometric surface. The idea of ​​the SDF is to use a large volume as a 3D map (or model) to be reconstructed, and express the volume as many small voxels. Each voxel corresponds to a point in the real-world scene. According to the SDF, a voxel has a vertex or a center point, and the vertex or the center point contains the nearest Euclidean distance relative to the nearest 3D geometric surface. TSDF is an improved version of SDF. TSDF only saves (many) directional distances near the geometric surface and cuts off the remaining area.

[0031] SDF and TSDF are implicit data structures. A Marching Cubes algorithm can be applied to the data in the implicit data structure to generate an explicit three-dimensional representation, such as a triangular mesh or point cloud. The basic unit contained in the SDF or TSDF, the voxel, is typically a cube with metric information. If the cube contains a portion of the surface of the scanned scene, each vertex stores a Euclidean distance relative to the surface. If the cube does not contain a portion of the surface of the scanned scene, each vertex is assigned a default don't care value.

[0032] At block 102 , in response to an activating action on the 3D volumetric data, a current set of active 3D volumetric data is obtained from the 3D volumetric data.

[0033] In one embodiment, the activation action includes: interaction of an input device with one or more objects in the real-world scene, or interaction of a virtual object existing in the real-world scene or input into the real-world scene with one or more objects in the real-world scene.

[0034] For example, in the AR scene (i.e., the 3D reconstructed static scene for the real world scene), a simulation program (on an AR device) can obtain user input (e.g., gesture selection) or interaction between a virtual object and the augmented reality scene (e.g., a bouncing ball). Specifically, when the user inputs an operation through the input device such as a remote control or a gesture, such as dragging a portion of an object in the AR scene, the activation action is detected. Alternatively, when the virtual object is placed in the AR scene, the virtual object collides with the real world scene and the activation action is detected.

[0035] In this case, a fast collision detection is performed to detect the activation action. The fast collision detection can be performed as follows. The virtual object (or the input device) can be represented by data of the explicit data structure such as a triangular mesh or a point cloud. Based on the data of the explicit data structure, a bounding box of the data of the explicit data structure containing the virtual object can be obtained, and the bounding box can be represented by corresponding 3D volume data. The collision detection can be implemented by querying the bounding box in the 3D volume data used to represent the real world scene. In response to the bounding box being queried in the 3D volume data used to represent the real world scene, the activation action is detected.

[0036] The currently active 3D volume data set may correspond to the ROI in the AR scene.

[0037] At block 103, a plurality of active 3D volume data sets are updated based on the current active 3D volume data set. The plurality of active 3D volume data sets are acquired from 3D volume data.

[0038] At block 104, a surface is constructed based on the plurality of updated active 3D volume data sets.

[0039] In one embodiment, a MarchingCubes algorithm may be executed on the plurality of updated active 3D volume data sets to construct the surface. The constructed surface may be represented by a surface mesh or a surface point cloud, which is not limited herein.

[0040] At block 105, a deformation of a portion of the real-world scene is simulated based on the surface. The portion of the real-world scene corresponds to the plurality of updated active 3D volume data sets.

[0041] In the method, the active 3D volume data is used to construct the surface instead of all 3D volume data, and the deformation corresponding to the portion of the real world scene is simulated based on the constructed surface, thereby greatly improving the speed of the simulation.

[0042] In one embodiment, the plurality of active 3D volume data sets are empty. In this case, the current 3D volume data set can be directly utilized to construct the surface.

[0043] In one embodiment, the plurality of active 3D volume data sets are not empty. That is, there may be a set of global regions in the scene that can be acquired for a set of collisions. In this case, an embodiment can be illustrated in Figure 2 middle.

[0044] As in Figure 2 As illustrated, the method includes the following actions in the following blocks.

[0045] At block 201 , 3D volumetric data representing a real-world scene is acquired.

[0046] At block 202 , in response to an activation action on the 3D volume data, a current active 3D volume data set is obtained from the 3D volume data.

[0047] At block 203 , it is determined whether the current active 3D volume data set overlaps with each of the multiple active 3D volume data sets.

[0048] At box 204, in response to the current active 3D volume data set overlapping with one or more active 3D volume data sets in the multiple active 3D volume data sets, the current active 3D volume data set is combined with the one or more active 3D volume data sets to form a new active 3D volume data set in the multiple active 3D volume data sets.

[0049] At block 205 , in response to the current active 3D volume data set not overlapping each of the plurality of active 3D volume data sets, the current active 3D volume data set is inserted into the plurality of active 3D volume data sets.

[0050] If multiple collisions are detected in the scene, the multiple regions corresponding to the multiple collisions need to be merged. If there is an overlap between the ROI corresponding to a current collision and one of the multiple existing global regions, the ROI is combined with the existing global region. If there is no overlap between the ROI and each of the existing global regions, the ROI will be regarded as a new region and added to the multiple global regions.

[0051] For example, due to the user's gesture operation, there are 3 areas on a wall that need to be deformed. If the user's new gesture operation is input, the new area in the scene becomes an ROI. If the ROI overlaps with one of the 3 areas, the overlapped area may be combined with the ROI to form a new area, so the 3 new areas need to be deformed. If the ROI does not overlap with each of the 3 areas, the ROI is regarded as a new area to obtain 4 areas to be deformed.

[0052] At block 206 , a surface is constructed based on the plurality of updated active 3D volume data sets.

[0053] At block 207, a deformation of a portion of the real-world scene is simulated based on the surface. The portion of the real-world scene corresponds to the plurality of updated active 3D volume data sets.

[0054] In the method, the plurality of active 3D volume data sets are updated based on the overlap, thereby further improving the simulation speed.

[0055] In addition, the deformation is simulated to generate a deformation result, and based on the deformation simulation result, the augmented reality scene will be colored and rendered. In the embodiments of the present disclosure, after the deformation simulation result is obtained, the deformation simulation result can be rendered as a texture mapping in a shader, that is, the deformation simulation result is added to the current AR display by the shader.

[0056] As a possible implementation, when the virtual object collides with the real scene, all collision voxels are determined, and a bounding box is generated based on all collision voxels. The smallest bounding box is used as the ROI. Figure 3 The above process is described in detail with reference to the embodiments in the accompanying drawings.

[0057] Figure 3 It is a flow chart of a method for an augmented reality scene according to an embodiment of the present disclosure.

[0058] As in Figure 3As illustrated, the method includes the following actions in the following blocks.

[0059] At block 301 , 3D volumetric data representing a real-world scene is acquired.

[0060] At block 302 , in response to an activation action on the 3D volume data, a current active 3D volume data set is obtained from the 3D volume data.

[0061] At block 303, a plurality of active 3D volume data sets are updated based on the current active 3D volume data set. The plurality of active 3D volume data sets are acquired from the 3D volume data.

[0062] At block 304, a bounding box is generated for a plurality of voxels surrounding each of the plurality of updated active 3D volume data sets.

[0063] As a possible implementation scheme, when the physical elasticity of the collided virtual object is different, the generated bounding box may be different. For example, when the physical elasticity of the virtual object is strong, the bounding box may be larger, and when the physical elasticity of the virtual object is weak, the bounding box may be smaller. Therefore, in the present disclosure, the material properties of the virtual object may be first acquired or defined, and then the elastic coefficient k of the virtual object may be acquired based on the material properties. Then the bounding box is generated based on the multiple collision voxels and the elastic coefficient. In this way, an initial bounding box may be determined based on the multiple collision voxels, and the radius of the initial bounding box may be multiplied by the coefficient k to obtain the generated bounding box. k is greater than 1 and is controlled by many simulation parameters of the virtual object (such as the elastic coefficient of the virtual object). The many simulation parameters of the virtual object may be set by the user or by default. The stronger the physical elasticity, the larger the k.

[0064] At block 305, a Marching Cubes algorithm is performed on the bounding box to construct the surface.

[0065] At block 306, a deformation of a portion of the real-world scene is simulated based on the surface. The portion of the real-world scene corresponds to the plurality of updated active 3D volume data sets.

[0066] As a possible implementation method, the following is combined Figure 4 The embodiments in the embodiment describe in detail the actions in steps 105, 207 and 306.

[0067] Figure 4It is a flow chart of a method for augmenting a reality scene according to an embodiment of the present disclosure.

[0068] At block 401 , a plurality of voxel grids of the surface are acquired.

[0069] At block 402 , a plurality of boundary vertices of the plurality of voxel meshes are obtained.

[0070] In the many embodiments of the present disclosure, the active 3D volume data can be processed to quickly generate the multiple voxel grids. For example, when performing fast meshing, the updated active 3D volume data is processed by the Marching Cubes algorithm to quickly generate multiple triangular meshes, or multiple voxel grids such as multiple tetrahedral grids are generated by other methods, and then finite element analysis is used to achieve more accurate physical simulation, which is not limited here. Normally, the active 3D volume data is smaller than the entire scene, so the generation speed of multiple voxel grids is very fast. Then the multiple boundary vertices of the multiple voxel grids are obtained.

[0071] In step 403 , the deformation of the portion of the real-world scene is simulated using the plurality of boundary vertices as a plurality of anchor points to generate a deformation simulation result.

[0072] For example, refer to Figure 5 The generated voxel grid can be added to a regional mesh set, and the regional mesh set is input into the physical simulation engine in the current iteration process to calculate the deformation simulation result. The boundary vertices of the plurality of voxel grids can exist as a plurality of anchor points. In the physical simulation such as deformation, these boundary vertices remain unchanged. For example, referring to Figure 6 , boundary vertex 1 serves as the anchor point, the position remains unchanged, and non-boundary vertex 2 participates in the physical simulation.

[0073] In the physical simulation engine, if the current voxel grid has not changed in the previous step, the previous simulation results, the user-defined constraints (such as drag points), and the interactive external forces (such as collisions, gravity) are input. If the current voxel grid is processed and combined, that is, multiple regions have been combined, the user-defined constraints, interactive external forces, and the simulation results of the voxel grids of all the combined regions in the previous step are input as the initial values. That is, in the present disclosure, the deformation simulation results are calculated by the many constraints generated by the interactions and the external forces.

[0074] It should be noted that in the present disclosure, fast meshing is applicable to deformation simulation. In many practical applications, fast meshing can also be used for other physical simulations, such as tactile simulation. When the user touches or interacts with a virtual object through a tactile device (such as a glove), a mesh and a normal can be quickly generated, and tactile force feedback can be calculated.

[0075] Currently, 3D volume data is used to reconstruct many 3D scenes and create many triangular meshes in real time for 3D rendering. In detail, a selective mesh surface can be used for regeneration, the environment is represented by multiple subsets, and a mesh is regenerated only when the distance function of the subset changes by more than a certain threshold, so as to reduce the overhead of generating the mesh for each frame. For 3D rendering during the 3D reconstruction process, a selective update of each frame of the mesh is generated to speed up the mesh regeneration.

[0076] However, reference Figure 5 , the present disclosure is used for fast simulation of the deformed object, and fast mesh generation is used to simulate the region of interest (ROI) instead of the changing region in the 3D reconstruction process. In detail, in the present disclosure, the ROI generated by the interaction between the user's input / many virtual objects and the TSDF is extracted, that is, the ROI of the interaction is quickly obtained based on the TSDF. The ROI is regarded as a new simulation area to be deformed and combined with the current area set. The updated area set is quickly meshed, and the voxel grid is used for deformation simulation. The deformation simulation result is calculated through the many constraints generated by the interaction and the external force. Therefore, the entire simulation process can save a lot of time for the regeneration of the voxel grid. The assumption of small deformation makes the result of local simulation close to the effect of the entire object and scene simulation, which is suitable for fast simulation of small deformation using the TSDF data structure.

[0077] The method for an augmented reality scene of an embodiment of the present disclosure can save a lot of time for voxel grid regeneration by performing fast meshing in the updated region set and using voxel grid for deformation simulation.

[0078] In order to implement the above embodiments, the present disclosure also provides a device for an augmented reality scene.

[0079] Figure 7 is a block diagram of a device for an augmented reality scene according to an embodiment of the present disclosure. Figure 7As illustrated, the apparatus includes: a first acquisition module 110 , a second acquisition module 120 , an updating module 130 , a construction module 140 , and a simulating module 150 .

[0080] The first acquisition module 110 is configured to acquire 3D volume data for representing a real-world scene. The second acquisition module 120 is configured to acquire a current active 3D volume data set from the 3D volume data in response to an activation action on the 3D volume data. The update module 130 is configured to update multiple active 3D volume data sets based on the current active 3D volume data set. The multiple active 3D volume data sets are acquired from the 3D volume data. The construction module 140 is configured to construct a surface based on the multiple updated active 3D volume data sets. The simulation module 150 is configured to simulate a deformation of a portion of the real-world scene based on the surface, the portion of the real-world scene corresponding to the multiple updated active 3D volume data sets.

[0081] In one embodiment, the activation action includes: interaction of an input device with one or more objects in the real-world scene, or interaction of a virtual object existing in the real-world scene or input into the real-world scene with one or more objects in the real-world scene.

[0082] In one embodiment, the updating module 130 is configured to: determine whether the current active 3D volume data set overlaps with each active 3D volume data set in the multiple active 3D volume data sets; in response to the current active 3D volume data set overlapping with one or more active 3D volume data sets in the multiple active 3D volume data sets, combine the current active 3D volume data set with the one or more active 3D volume data sets to form a new active 3D volume data set in the multiple active 3D volume data sets; and in response to the current active 3D volume data set not overlapping with each active 3D volume data set in the multiple active 3D volume data sets, insert the current active 3D volume data set into the multiple active 3D volume data sets.

[0083] In one embodiment, the construction module 140 is configured to execute a marching cubes algorithm on the plurality of updated active 3D volume data sets to construct the surface.

[0084] In one embodiment, the construction module 140 is configured to: generate a bounding box that encloses a plurality of voxels in each of the plurality of updated active 3D volume data sets; and perform a marching cubes algorithm on the plurality of voxels within the bounding box to construct the surface.

[0085] In one embodiment, the construction module 140 is configured to: generate a bounding box that encloses multiple voxels in each of the multiple updated active 3D volume data sets; obtain a material property of an object corresponding to each of the multiple updated active 3D volume data sets in the real-world scene; obtain an elastic coefficient of the object based on the material property; enlarge the bounding box based on the elastic coefficient; and execute a marching cube algorithm on the multiple voxels within the enlarged bounding box to construct the surface.

[0086] In one embodiment, the simulation module 150 is configured to: obtain multiple voxel grids of the surface; obtain multiple boundary vertices of the multiple voxel grids; and simulate the deformation of the portion of the real-world scene using the multiple boundary vertices as multiple anchor points to generate a deformation simulation result.

[0087] like Figure 8 As illustrated, the apparatus includes: a first acquisition module 110 , a second acquisition module 120 , an update module 130 , a construction module 140 , a simulation module 150 , and a coloring and rendering module 160 .

[0088] The shading and rendering 160 is configured to shade and render the deformation simulation result, and map the shaded and rendered deformation simulation result to a 3D model reconstructed based on the 3D volume data, for reconstructing the real world scene.

[0089] It should be noted that the description of the many method embodiments is also applicable to the many device embodiments, and the implementation principles are similar, which will not be repeated here.

[0090] In order to implement the above embodiments, the present disclosure further provides an electronic device, comprising a memory, a processor and a computer program, wherein the computer program is stored in the memory and can be operated on the processor. When the program is executed by the processor, the method according to the above embodiments is implemented.

[0091] In order to implement the above embodiments, the present disclosure further provides a computer-readable storage medium on which a computer-readable storage instruction is stored. The computer-readable storage instruction is configured to cause a computer to implement the method according to the above embodiments.

[0092] References throughout the specification to "an embodiment," "some embodiments," "an example," "a specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment are included in at least one embodiment or example of the present disclosure. The above phrases appearing throughout this specification do not necessarily refer to the same embodiment or example of the present disclosure. In addition, the many specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples. In addition, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined by those skilled in the art without contradicting each other.

[0093] Any process or method described in a flowchart or otherwise described herein may be understood to include one or more modules, segments or code portions of executable instructions for implementing specific logical functions or steps in the process, and the scope of a preferred embodiment of the present disclosure includes other implementations, which should be understood by those skilled in the art.

[0094] The logic and / or steps described in other ways or shown in the flowchart herein, such as a specific executable instruction sequence list for implementing the logical function, can be specifically implemented in any computer-readable medium for use in an instruction execution system, device or equipment (such as a computer-based system, including a processor or other system capable of obtaining the instructions from the instruction execution system, device and equipment and executing the instructions), or used in combination with the instruction execution system, device and equipment. For the purpose of this specification, "the computer readable medium" can be any device suitable for containing, storing, communicating, propagating or transmitting a program for use by the instruction execution system, device or equipment or used in combination with the instruction execution system, device or equipment. More specific examples of the computer-readable medium include, but are not limited to: an electronic connection with one or more wires (an electronic device), a portable computer housing (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). Furthermore, the computer-readable medium may even be a paper or other suitable medium on which the program can be printed, because, for example, the paper or other suitable medium may be optically scanned and then edited, decrypted or processed in other suitable ways if necessary, so as to obtain the program in an electronic manner, which may then be stored in the computer memory.

[0095] It should be understood that various parts of the present disclosure may be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, a plurality of steps or methods may be implemented by software or firmware stored in the memory and executed by an appropriate instruction execution system. For example, if implemented by the hardware, in another embodiment, these steps or methods may be implemented by one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function of a data signal; a dedicated integrated circuit having an appropriate combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0096] Those skilled in the art will appreciate that all or part of the steps performed by the method in the above embodiments can be completed by related hardware instructed by a program. The program can be stored in a computer-readable storage medium. When the program is executed, a combination of one or more steps of the method in the above embodiments can be completed.

[0097] In addition, the various functional units in the embodiments of the present disclosure may be integrated into a processing module, or may exist as separate entities, or two or more units may be integrated into a module. The integrated module as described above may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as a separate product, the integrated module may also be stored in a computer-readable storage medium.

[0098] The storage medium as described above may be a read-only memory, a disk or a CD, etc. Although many illustrative embodiments have been shown and described, it should be understood by those skilled in the art that the above embodiments cannot be construed as limiting the present disclosure, and that changes, substitutions and modifications may be made to the many embodiments without departing from the scope of the present disclosure.

Claims

1. A computer-implemented method, characterized in that: The method comprises: Acquiring three-dimensional volume data representing a real-world scene; In response to an activation action on the three-dimensional volume data, obtaining a current active three-dimensional volume data set from the three-dimensional volume data; updating a plurality of active three-dimensional volume data sets in a union or insertion manner based on the current active three-dimensional volume data set, the plurality of active three-dimensional volume data sets being acquired from the three-dimensional volume data; constructing a surface based on the plurality of updated active three-dimensional volume data sets; and A deformation of a portion of the real-world scene is simulated based on the surface, the portion of the real-world scene corresponding to the plurality of updated active three-dimensional volumetric data sets.

2. The method according to claim 1, characterized in that: The activation action includes: an interaction between an input device and one or more objects in the real-world scene, or an interaction between a virtual object and one or more objects in the real-world scene, wherein the virtual object exists in the real-world scene or is input into the real-world scene.

3. The method according to claim 1, characterized in that: Updating the plurality of active three-dimensional volume data sets based on the current active three-dimensional volume data set comprises: determining whether the current active three-dimensional volume data set overlaps with each of the multiple active three-dimensional volume data sets; In response to the current active 3D volume data set overlapping with one or more active 3D volume data sets in the multiple active 3D volume data sets, merging the current active 3D volume data set with the one or more active 3D volume data sets to form a new active 3D volume data set in the multiple active 3D volume data sets; and In response to the current active 3D volume data set not overlapping with each of the plurality of active 3D volume data sets, the current active 3D volume data set is inserted into the plurality of active 3D volume data sets.

4. The method according to claim 1, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: A marching cubes algorithm is performed on the plurality of updated active three-dimensional volume data sets to construct the surface.

5. The method according to claim 1, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: generating a bounding box that encompasses a plurality of voxels of each of the plurality of updated active three-dimensional volume data sets; and A marching cubes algorithm is performed on a plurality of voxels within the bounding box to construct the surface.

6. The method according to claim 1, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: generating a bounding box that encloses a plurality of voxels of each of the plurality of updated active three-dimensional volume data sets; Obtaining a material property of an object corresponding to each of the plurality of updated active three-dimensional volume data sets in the real-world scene; Acquire an elastic coefficient of the object based on the material property; enlarging the bounding box based on the elastic coefficient; and A marching cubes algorithm is performed on a plurality of voxels within the enlarged bounding box to construct the surface.

7. The method according to claim 1, characterized in that: Simulating the deformation of the portion of the real-world scene based on the surface comprises: obtaining a plurality of voxel grids of the surface; Obtaining a plurality of boundary vertices of the plurality of voxel grids; and The deformation of the portion of the real-world scene is simulated using the plurality of boundary vertices as a plurality of anchor points to generate a deformation simulation result.

8. The method according to claim 7, characterized in that: The method further comprises: The deformation simulation result is colored and rendered, and the colored and rendered deformation simulation result is mapped to a three-dimensional model reconstructed based on the three-dimensional volume data for reconstructing the real world scene.

9. The method according to claim 1, characterized in that: The three-dimensional volume data includes a set of voxels in a signed distance function or a truncated signed distance function.

10. The method according to claim 1, characterized in that: The plurality of active three-dimensional volume data sets are empty.

11. An electronic device, characterized in that: The electronic device comprises: a memory; a processor; and a computer program stored on said memory and operable on said processor, The program is configured to be executed by the processor to: Acquiring three-dimensional volume data representing a real-world scene; In response to an activation action on the three-dimensional volume data, obtaining a current active three-dimensional volume data set from the three-dimensional volume data; updating a plurality of active three-dimensional volume data sets in a union or insertion manner based on the current active three-dimensional volume data set, the plurality of active three-dimensional volume data sets being acquired from the three-dimensional volume data; constructing a surface based on the plurality of updated active three-dimensional volume data sets; and A deformation of a portion of the real-world scene is simulated based on the surface, the portion of the real-world scene corresponding to the plurality of updated active three-dimensional volumetric data sets.

12. The device according to claim 11, characterized in that: Updating the plurality of active three-dimensional volume data sets based on the current active three-dimensional volume data set comprises: determining whether the current active three-dimensional volume data set overlaps with each of the multiple active three-dimensional volume data sets; In response to the current active 3D volume data set overlapping with one or more active 3D volume data sets in the multiple active 3D volume data sets, merging the current active 3D volume data set with the one or more active 3D volume data sets to form a new active 3D volume data set in the multiple active 3D volume data sets; and In response to the current active 3D volume data set not overlapping with each of the plurality of active 3D volume data sets, the current active 3D volume data set is inserted into the plurality of active 3D volume data sets.

13. The device according to claim 11, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: A marching cubes algorithm is performed on the plurality of updated active three-dimensional volume data sets to construct the surface.

14. The device according to claim 11, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: generating a bounding box that encompasses a plurality of voxels of each of the plurality of updated active three-dimensional volume data sets; and A marching cubes algorithm is performed on a plurality of voxels within the bounding box to construct the surface.

15. The device according to claim 11, characterized in that: Constructing the surface based on the plurality of updated active three-dimensional volume data sets comprises: generating a bounding box that encloses a plurality of voxels of each of the plurality of updated active three-dimensional volume data sets; Obtaining a material property of an object corresponding to each of the plurality of updated active three-dimensional volume data sets in the real-world scene; Acquire an elastic coefficient of the object based on the material property; enlarging the bounding box based on the elastic coefficient; and A marching cubes algorithm is performed on a plurality of voxels within the enlarged bounding box to construct the surface.

16. The device according to claim 11, characterized in that: Simulating the deformation of the portion of the real-world scene based on the surface comprises: obtaining a plurality of voxel grids of the surface; Obtaining a plurality of boundary vertices of the plurality of voxel grids; and The deformation of the portion of the real-world scene is simulated using the plurality of boundary vertices as a plurality of anchor points to generate a deformation simulation result.

17. The device according to claim 16, characterized in that: The program is configured to be executed by the processor to further: The deformation simulation result is colored and rendered, and the colored and rendered deformation simulation result is mapped to a three-dimensional model reconstructed based on the three-dimensional volume data for reconstructing the real world scene.

18. The device according to claim 11, characterized in that: The three-dimensional volume data includes a set of voxels in a signed distance function or a truncated signed distance function.

19. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: The program is configured to be executed by a processor to perform a simulation method for an augmented reality scene, the method comprising: Acquiring three-dimensional volume data representing a real-world scene; In response to an activation action on the three-dimensional volume data, obtaining a current active three-dimensional volume data set from the three-dimensional volume data; updating a plurality of active three-dimensional volume data sets in a union or insertion manner based on the current active three-dimensional volume data set, the plurality of active three-dimensional volume data sets being acquired from the three-dimensional volume data; constructing a surface based on the plurality of updated active three-dimensional volume data sets; and A deformation of a portion of the real-world scene is simulated based on the surface, the portion of the real-world scene corresponding to the plurality of updated active three-dimensional volumetric data sets.

20. The non-transitory computer-readable storage medium of claim 19, wherein: Updating the plurality of active three-dimensional volume data sets based on the current active three-dimensional volume data set comprises: determining whether the current active three-dimensional volume data set overlaps with each of the multiple active three-dimensional volume data sets; In response to the current active 3D volume data set overlapping with one or more active 3D volume data sets in the multiple active 3D volume data sets, merging the current active 3D volume data set with the one or more active 3D volume data sets to form a new active 3D volume data set in the multiple active 3D volume data sets; and In response to the current active 3D volume data set not overlapping with each of the plurality of active 3D volume data sets, the current active 3D volume data set is inserted into the plurality of active 3D volume data sets.

Images