A method and system for spatial three-dimensional modeling using mobile devices

Abstract

The application relates to the technical field of three-dimensional modeling, and provides a method and system for three-dimensional modeling of space by using a mobile device, which comprises the following steps: obtaining a space scene to be modeled, wherein the space scene to be modeled at least comprises a simulated human body which has an interaction relationship with an environmental object; photographing the space scene to be modeled based on the mobile device to obtain a scene depth image; extracting a key frame and a camera pose of the space scene to be modeled based on the scene depth image; constructing three-dimensional space point cloud data based on the key frame and the camera pose; collecting the space scene to be modeled according to the camera pose to obtain a scene color image, performing semantic segmentation on the scene color image, and obtaining an image semantic label; and projecting and mapping the image semantic label to the three-dimensional space point cloud data to obtain three-dimensional space point cloud data with the image semantic label. The point cloud data with semantic label information and containing human body information can be automatically generated, and the working cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method and system for spatial 3D modeling using mobile devices. Background Technology

[0002] With the groundbreaking development of embodied intelligence and mixed reality technologies, the concept of intelligent space has become a research hotspot in the field of human-computer interaction. In intelligent space, physical entities are endowed with digital attributes and interaction logic, and dynamic interactions between intelligent agents and between humans and machines are realized through computer vision technology. The core requirement of this type of interaction is to accurately identify the boundaries between objects and the human body to construct a three-dimensional spatial model, and to extract semantic information of functional parts to support the logical reasoning of the interaction.

[0003] According to relevant technologies, point cloud data with semantically labeled information is required for training in the process of constructing a 3D spatial model. However, the semantic label information of current point cloud datasets often needs to be manually annotated, a time-consuming and costly process, especially in large-scale and complex scenes where the efficiency and accuracy of manual intervention are difficult to guarantee. Furthermore, current spatial 3D modeling methods cannot generate scene point cloud data containing human information. Therefore, when dealing with applications such as embodied intelligence and mixed reality, it is impossible to directly identify human body categories, thus hindering the realization of better human-computer interaction.

[0004] Therefore, finding a 3D modeling method that can automatically generate point cloud data with semantic labeling information and human body information has become a current research hotspot. Summary of the Invention

[0005] This invention provides a method and system for spatial 3D modeling using mobile devices, which enables the automatic generation of point cloud data with semantic label information and human body information, reducing working costs and increasing the information content of the obtained point cloud data.

[0006] This invention provides a method for spatial 3D modeling using a mobile device. The method includes: acquiring a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects; capturing images of the spatial scene to be modeled using a mobile device to obtain a scene depth image; extracting keyframes of the spatial scene to be modeled and camera poses corresponding to the keyframes based on the scene depth image; constructing 3D spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; acquiring images of the spatial scene to be modeled according to the camera poses to obtain a scene color image corresponding to the scene depth image, and performing semantic segmentation on the scene color image to obtain image semantic labels; and projecting and mapping the image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels.

[0007] According to a method for spatial 3D modeling using a mobile device provided by the present invention, the image semantic labels include image semantic labels corresponding to each pixel in the scene color image; the step of semantically segmenting the scene color image to obtain image semantic labels is implemented in the following manner: calling a pre-trained semantic segmentation model, wherein the semantic segmentation model is pre-trained based on a sample dataset, and the semantic segmentation model generates image semantic labels of preset categories; the sample dataset is a sample dataset about indoor application scene information with simulated human bodies; the scene color image is input into the semantic segmentation model to obtain the image semantic labels corresponding to each pixel in the scene color image output by the semantic segmentation model.

[0008] According to a method for spatial three-dimensional modeling using a mobile device provided by the present invention, after projecting and mapping the image semantic tags onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic tags, the method further includes: extracting the color values ​​of each pixel in the scene color image; determining the color information of each pixel in the scene color image based on the color values ​​of each pixel in the scene color image; and projecting and mapping the color information of each pixel in the scene color image onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic tags and / or color information.

[0009] According to a method for spatial 3D modeling using a mobile device provided by the present invention, the spatial scene to be modeled is obtained by: interacting with the simulated human body and the environmental objects according to the following constraints to obtain the spatial scene to be modeled, wherein the constraints include any one or more of the following constraints: support constraints, wherein the support constraints are used to characterize the support interaction relationship under the support action of the environmental objects on the simulated human body; collision constraints, wherein the collision constraints are used to characterize the collision interaction relationship under the collision between the environmental objects and the simulated human body; strong interaction constraints, wherein the strong interaction constraints are used to characterize the interaction relationship where the interaction points between the environmental objects and the simulated human body exceed a preset point.

[0010] According to the present invention, a method for spatial 3D modeling using a mobile device is provided, wherein the step of extracting keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes based on the scene depth image specifically includes: extracting keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes based on the scene depth image using real-time localization and mapping technology.

[0011] According to a method for spatial 3D modeling using a mobile device provided by the present invention, after extracting the keyframes of the spatial scene to be modeled, the method further includes: traversing all keyframes, calculating the similarity between any two keyframes; if the similarity between keyframes exceeds a similarity threshold, removing keyframes that exceed the similarity threshold; and using the set of keyframes formed after removing keyframes that exceed the similarity threshold as the keyframes of the spatial scene to be modeled.

[0012] According to a method for spatial 3D modeling using a mobile device provided by the present invention, after extracting the keyframes of the spatial scene to be modeled, the method further includes: determining the current frame of the currently tracked spatial scene to be modeled; determining the co-viewing degree between the current frame and the keyframe; and if the co-viewing degree is less than a co-viewing degree threshold, using the current frame as a keyframe.

[0013] According to the present invention, a method for spatial three-dimensional modeling using a mobile device, wherein the construction of three-dimensional spatial point cloud data based on the keyframe and the camera pose corresponding to the keyframe specifically includes: obtaining the depth image of the keyframe and the homogeneous coordinates of the two-dimensional points of the keyframe based on the keyframe; determining the rotation matrix based on the camera pose corresponding to the keyframe; and constructing three-dimensional spatial point cloud data based on the homogeneous coordinates of the two-dimensional points of the keyframe, the rotation matrix, and the depth image of the keyframe.

[0014] According to a method for spatial three-dimensional modeling using a mobile device provided by the present invention, after constructing three-dimensional spatial point cloud data, the method further includes: initializing target three-dimensional spatial point cloud data, wherein each point cloud in the target three-dimensional spatial point cloud data corresponds to each point cloud in the three-dimensional spatial point cloud data, and the point clouds in the target three-dimensional spatial point cloud data are uniformly distributed; determining a rotation and translation matrix based on the corresponding distance between each point cloud in the target three-dimensional spatial point cloud data and each point cloud in the three-dimensional spatial point cloud data; and rotating and translating the target three-dimensional spatial point cloud data based on the rotation and translation matrix to obtain rotated and translated target three-dimensional spatial point cloud data.

[0015] If the target three-dimensional spatial point cloud data after rotation and translation meets the preset conditions, the target three-dimensional spatial point cloud data after rotation and translation will be used as the three-dimensional spatial point cloud data.

[0016] If the target 3D spatial point cloud data after rotation and translation does not meet the preset conditions, the step of initializing the target 3D spatial point cloud data is repeated until the target 3D spatial point cloud data after rotation and translation is obtained, until the target 3D spatial point cloud data after rotation and translation meets the preset conditions, and the target 3D spatial point cloud data after rotation and translation is used as the 3D spatial point cloud data.

[0017] This invention also provides a system for spatial 3D modeling using a mobile device. The system includes: an acquisition module for acquiring a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects; an imaging module for capturing images of the spatial scene to be modeled using a mobile device to obtain a scene depth image; an extraction module for extracting keyframes of the spatial scene to be modeled and camera poses corresponding to the keyframes based on the scene depth image; a construction module for constructing 3D spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; a segmentation module for acquiring images of the spatial scene to be modeled according to the camera poses to obtain a scene color image corresponding to the scene depth image, and performing semantic segmentation on the scene color image to obtain image semantic labels; and a generation module for projecting and mapping the image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for spatial three-dimensional modeling using a mobile device as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for spatial three-dimensional modeling using a mobile device as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for spatial three-dimensional modeling using a mobile device as described above.

[0021] This invention provides a method and system for spatial 3D modeling using a mobile device. The method involves acquiring a spatial scene to be modeled, wherein the scene includes at least a simulated human body interacting with environmental objects; capturing images of the scene using the mobile device to obtain a scene depth image; extracting keyframes and corresponding camera poses from the scene depth image; constructing 3D spatial point cloud data based on the keyframes and their corresponding camera poses; acquiring scene color images corresponding to the scene depth image according to the camera poses; performing semantic segmentation on the scene color images to obtain image semantic labels; and projecting these image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels. By projecting and mapping image semantic labels onto 3D spatial point cloud data, spatial point cloud data containing fine-grained semantic information can be obtained. This enables the automatic generation of point cloud data with semantic labels and human body information, reducing workload and increasing the information content of the obtained point cloud data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0024] Figure 2 This is the second flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0025] Figure 3 This is the third flowchart of a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0026] Figure 4 This is the fourth flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0027] Figure 5 This is the fifth flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0028] Figure 6 This is a schematic diagram of the structure of a system for spatial three-dimensional modeling using a mobile device, provided by the present invention.

[0029] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] This invention provides a method for spatial 3D modeling using mobile devices. After placing a simulated human figure in the spatial scene to be modeled, the scene is captured from all angles using a mobile device equipped with a depth camera or binocular cameras. Keyframes of the spatial point cloud and camera pose are extracted from the acquired depth images, and point cloud registration is performed to obtain 3D spatial point cloud data. Semantic information is then obtained using a large-model-based 2D image semantic segmentation method, completing the construction of semantic spatial point cloud data containing the virtual human figure. This method automatically generates point cloud data with semantic labels, reducing workload and improving the accuracy of the obtained point cloud data. Furthermore, since the simulated human figure is placed in the spatial scene to be modeled, scene point cloud data containing the human figure can be generated, accurately identifying the boundaries between objects and the human figure, facilitating subsequent 3D spatial model construction. Semantic information of functional parts is extracted to support interactive logic reasoning.

[0032] Figure 1 This is one of the flowcharts illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0033] The following will combine Figure 1 The process of a method for spatial three-dimensional modeling using a mobile device provided by the present invention will be described.

[0034] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, a method for spatial 3D modeling using mobile devices may include steps 110 to 160, and each step will be described below.

[0035] In step 110, the spatial scene to be modeled is obtained, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects.

[0036] In one embodiment, a spatial scene to be modeled can be acquired. The 3D point cloud data obtained by modeling the spatial scene can be used for training large models in subsequent application scenarios such as embodied intelligence and mixed reality. Since application scenarios such as embodied intelligence and mixed reality require simulated human interaction, by configuring a simulated human in the spatial scene to be modeled, scene point cloud data containing the human can be generated, laying the foundation for accurate training of large models in subsequent application scenarios such as embodied intelligence and mixed reality.

[0037] In step 120, the scene to be modeled is captured by a mobile device to obtain a scene depth image.

[0038] In one embodiment, the depth image of the scene to be modeled can be obtained by taking pictures of the spatial scene on a mobile device, such as a mobile device equipped with a depth camera or a binocular camera.

[0039] During the capture of depth images (corresponding to scene depth images), if the mobile device is equipped with a stereo camera, the MobiDepth system can be used for depth estimation. This system uses iterative field-of-view cropping technology to match the focal lengths of the two cameras, ensuring an equivalent field of view between the stereo color images. Then, it uses heterogeneous camera synchronization technology to synchronize the frame streams of the two cameras, reducing inter-frame displacement of target objects in dynamic scenes. Finally, a stereo matching algorithm is used to achieve fast and accurate depth mapping, thereby obtaining the depth information of each pixel in the color image and acquiring the required depth image. In this embodiment, the specific method of acquiring scene depth images is not limited.

[0040] In step 130, key frames of the spatial scene to be modeled and the camera poses corresponding to the key frames are extracted based on the scene depth image.

[0041] In step 140, three-dimensional spatial point cloud data is constructed based on key frames and the camera poses corresponding to the key frames.

[0042] In another exemplary embodiment of the present invention, the key frames of the spatial scene to be modeled and the corresponding camera poses of the key frames are extracted based on the scene depth image, which can be achieved in the following way:

[0043] Based on scene depth images, keyframes of the spatial scene to be modeled and the corresponding camera poses are extracted using real-time localization and mapping techniques.

[0044] In one embodiment, keyframes of the spatial scene to be modeled and the corresponding camera poses can be extracted based on the scene depth image using Simultaneous Localization and Mapping (SLAM) technology. Then, three-dimensional spatial point cloud data can be constructed based on the keyframes and the corresponding camera poses.

[0045] In one embodiment, keyframes of the spatial scene to be modeled and the corresponding camera poses can be extracted from the scene depth image using the ORB-SLAM2 algorithm. Further, based on the keyframes and their corresponding camera poses, 3D spatial point cloud data is constructed. Here, keyframes can be considered as representative, information-rich, and reasonably distributed frames.

[0046] In another embodiment, during the construction of 3D spatial point cloud data, constant-speed global scene tracking can be used to track and capture the scene as a whole, generating keyframes and corresponding camera poses (corresponding to the keyframes and corresponding camera poses needed for the initial scene estimation), while simultaneously constructing a coarse-grained global map. This process mainly involves the creation of map points and the insertion of keyframes. In application, depth points in the initially captured depth map can be mapped to 3D space through depth information to form the scene's point cloud. The constructed point cloud data can also be used to update the co-view (i.e., to count the degree of co-view between different keyframes and organize it into an undirected graph data structure. The degree of co-view refers to the overlap ratio of map points that can be seen from the keyframe) and construct a spanning tree (treating all keyframes as graph nodes, iteratively selecting frames with lower co-view from the existing keyframes while ensuring that map points can be covered, thereby obtaining the minimum set of keyframes), thus obtaining the set of keyframes needed to cover the scene.

[0047] During the local map tracking phase, the camera pose can be finely optimized, and detailed scene capture can be performed. The decision to set the current depth image as a new keyframe can be made based on the number of frames processed since the last global localization, the state of local mapping, the number of feature points tracked in the current frame, and a comparison with reference keyframes (e.g., currently determined keyframes). Once determined, new map points can be generated through feature matching and triangulation, and all matching points can be retrieved and optimized using a common view to improve the accuracy of the camera pose, thereby ensuring the continuity and precision of the map. This process is akin to building a detailed 3D map while ensuring the camera's positioning within the map is as accurate as possible. Based on this, 3D spatial point cloud data can be generated.

[0048] In step 150, the scene to be modeled is acquired according to the camera pose to obtain a scene color image corresponding to the scene depth image, and the scene color image is semantically segmented to obtain image semantic labels.

[0049] In step 160, the image semantic labels are projected and mapped onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels.

[0050] In another embodiment, the camera pose determined above can be used to further acquire color images based on the same camera pose, and semantic segmentation can be performed on the scene color images to obtain the semantic category label corresponding to each pixel, i.e., to obtain image semantic labels. Given that a spatial correspondence between two-dimensional image pixels and three-dimensional points has been established, this relationship can be directly used to project the semantically labeled pixels back to their corresponding three-dimensional positions, thereby generating semantic point clouds. That is, the image semantic labels are projected and mapped onto three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels. By projecting and mapping image semantic labels onto three-dimensional spatial point cloud data, spatial point cloud data containing fine-grained semantic information can be obtained. This enables the automatic generation of point cloud data with semantic labels and human information, reducing workload and increasing the information content of the obtained point cloud data.

[0051] This invention provides a method for spatial 3D modeling using a mobile device. The method involves acquiring a spatial scene to be modeled, wherein the scene includes at least a simulated human body interacting with environmental objects; capturing images of the scene using the mobile device to obtain a scene depth image; extracting keyframes and corresponding camera poses from the scene depth image; constructing 3D spatial point cloud data based on the keyframes and their corresponding camera poses; acquiring scene color images corresponding to the scene depth image according to the camera poses; performing semantic segmentation on the scene color images to obtain image semantic labels; and projecting these image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels. By projecting and mapping image semantic labels onto 3D spatial point cloud data, spatial point cloud data containing fine-grained semantic information can be obtained. This enables the automatic generation of point cloud data with semantic labels and human body information, reducing workload and increasing the information content of the obtained point cloud data.

[0052] In an exemplary embodiment of the present invention, the image semantic label may include the image semantic label corresponding to each pixel in the scene color image, continuing from the preceding text. Figure 1The above embodiment will be used as an example to illustrate how semantic segmentation of a scene color image can be performed to obtain image semantic labels (corresponding to step 150). This can be achieved in the following way:

[0053] A pre-trained semantic segmentation model is invoked, wherein the semantic segmentation model is pre-trained based on a sample dataset, and the semantic segmentation model generates image semantic labels of a preset category; the sample dataset is a sample dataset about indoor application scenarios with simulated human bodies.

[0054] The scene color image is input into the semantic segmentation model to obtain the image semantic labels corresponding to each pixel in the scene color image output by the semantic segmentation model.

[0055] In one embodiment, a pre-trained semantic segmentation model can be invoked, and the scene color image can be input into the semantic segmentation model to obtain the image semantic labels corresponding to each pixel in the scene color image output by the semantic segmentation model.

[0056] It should be noted that the semantic segmentation model is pre-trained on a sample dataset and generates image semantic labels of preset categories. The sample dataset contains information about indoor application scenarios featuring simulated human bodies. In one example, the semantic segmentation model can generate a wide variety of semantic labels. However, to improve the practicality and efficiency of the semantic segmentation model, the original semantic labels output by the model can be selectively simplified and filtered. In application, only typical object categories highly relevant to the indoor application scenario can be retained, corresponding to the preset categories of image semantic labels (e.g., bed, books, ceiling, chair, floor, furniture, pictures or paintings, sofa, table, television, wall, window, bookshelf, cabinet, curtains, desk or office desk, door, wardrobe, refrigerator, sink, bathtub, trash can, monitor, shower, toilet, night light, whiteboard, mirror, closet, microwave oven, bedside table, sink, bar, washing machine, oven, dining chair, TV cabinet, other items). These categories cover common static entities such as furniture and building structures in the daily environment. Meanwhile, considering that the acquired point cloud data includes data about simulated human body models, the human body category can be retained as an independent dynamic category. That is, during the pre-training of the semantic segmentation model, the corresponding sample training dataset is a sample dataset containing information about indoor application scenarios with simulated human bodies. In this embodiment, the semantic labels output by the semantic segmentation model are simplified in the aforementioned manner, effectively reducing redundant information in the model output while ensuring comprehensive coverage of the core semantic content.

[0057] Figure 2 This is the second flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0058] The following will combine Figure 2 The process of a method for spatial three-dimensional modeling using a mobile device provided by the present invention will be described.

[0059] In an exemplary embodiment of the present invention, combined with Figure 2 As can be seen, after projecting and mapping the image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels, the method of using mobile devices for spatial 3D modeling can also include steps 210 to 230, which will be described in detail below:

[0060] In step 210, the color values ​​of each pixel in the scene color image are extracted.

[0061] In step 220, the color information of each pixel in the scene color image is determined based on the color value of each pixel in the scene color image;

[0062] In step 230, the color information of each pixel in the scene color image is projected and mapped to the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels and / or color information.

[0063] In one embodiment, after obtaining pixel-level semantic labels, each two-dimensional pixel with a semantic label can be mapped to a corresponding three-dimensional spatial location by combining the established back projection relationship, thereby giving the corresponding three-dimensional point a semantic attribute and realizing the conversion from image semantics to three-dimensional point cloud semantics, that is, obtaining three-dimensional spatial point cloud data with image semantic labels.

[0064] Furthermore, to enhance the visualization effect and expressive power of the 3D point cloud for subsequent analysis, the color values ​​of each pixel in the scene's color image can be extracted, and the color information of each pixel in the scene's color image can be determined based on these values. In application, the color information of each pixel in the scene's color image can be projected onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels and / or color information. Through this embodiment, the obtained 3D spatial point cloud data not only contains geometric position information and semantic labels but also possesses realistic color information, further enhancing the detail richness and semantic expressive power of the 3D reconstruction results, providing a solid data foundation for subsequent applications such as 3D scene understanding, human-computer interaction modeling, and virtual reality.

[0065] In yet another exemplary embodiment of the present invention, the previously described embodiment will be used as an example for explanation. The spatial scene to be modeled can be obtained in the following manner:

[0066] The simulated human body interacts with environmental objects according to the following constraints to obtain the spatial scene to be modeled, wherein the constraints include any one or more of the following constraints:

[0067] Support constraints, which characterize the support interaction relationship between environmental objects and the simulated human body;

[0068] Collision constraints, which are used to characterize the collision interaction relationship between environmental objects and the simulated human body;

[0069] Strong interaction constraints are used to characterize the interaction relationship between environmental objects and simulated human bodies when the interaction points exceed preset points.

[0070] In one embodiment, a sufficient number of simulated human figures can be purchased based on the size of the scene, and the body shapes of the figures need to vary to some extent. During placement, the interaction between the human figures and objects such as chairs, sofas, beds, and tables needs to be considered, and the posture of the human figures should be adjusted to ensure that the interaction is physically reasonable and visually natural. In addition, simulated human figures placed in the same scene will not be duplicated, but a certain number of duplicate human figures with different actions and interaction states will appear in different scenes.

[0071] In yet another embodiment, the placement of the simulated human body requires that one or more of three constraints be met:

[0072] Support constraints refer to the support that the human body model (corresponding to a simulated human body) needs from other objects when it is in different behavioral positions. Specifically, when the human body model is in a "lying down" position, most of its body parts need to be in contact with a flat surface; when the human body model is in a "standing" position, its feet need the support of a flat surface; when the human body model is in a "sitting" position, its buttocks need the support of environmental objects, while its back and feet can make contact as needed. Support constraints characterize the support interaction relationship between the environmental objects and the simulated human body, ensuring that the placement of the simulated human body conforms to physical laws.

[0073] Collision constraints ensure that the limbs and other parts of the human model are naturally extended, only deforming upon collision with other objects. For example, when the human model is lying flat, the arms should not be excessively bent, but when the human model touches a table, the arms should be bent to fit against the table while avoiding objects on the table. Collision constraints characterize the collision interactions between environmental objects and the simulated human body, ensuring that the placement of the simulated human body conforms to physical laws.

[0074] Strong interaction constraints are used to enhance the interaction between the human model and its environment. When placing the human model, it's necessary to introduce as much interaction as possible between the human and the environment. For example, simply placing the human model on an open surface to "stand" yields too little data and is not conducive to subsequent machine learning tasks. Therefore, the human model needs to be placed around objects such as tables and chairs, and its arms and other parts can collide with these objects to enhance the interaction. Strong interaction constraints characterize the interaction relationships between environmental objects and the simulated human model where the number of interaction points exceeds a preset limit; this ensures visual quality and the richness of any point cloud data that may be acquired later.

[0075] Figure 3 This is the third flowchart of a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0076] The following will combine Figure 3 The process of another method for spatial three-dimensional modeling using a mobile device provided by the present invention will be described.

[0077] In an exemplary embodiment of the present invention, combined with Figure 3 As can be seen, the method of using mobile devices for spatial 3D modeling can include steps 310 to 330, and each step will be described below.

[0078] In step 310, all keyframes are traversed, and for any two keyframes, the similarity between the keyframes is calculated.

[0079] In step 320, if the similarity between keyframes exceeds a similarity threshold, the keyframes exceeding the similarity threshold are removed.

[0080] In step 330, the set of keyframes formed after removing keyframes that exceed the similarity threshold is used as the keyframes of the spatial scene to be modeled.

[0081] In one embodiment, the keyframes can also be optimized. After extracting the keyframes of the spatial scene to be modeled, further filtering and optimization are needed to ensure that the extracted keyframes (corresponding keyframes) can completely represent the scene.

[0082] During application, because ORB-SLAM2 keyframes depend on local map reconstruction, the final keyframe set still contains some keyframes with high repetition. In this embodiment, the similarity between any two keyframes can be calculated. Furthermore, if the similarity between keyframes exceeds a similarity threshold, any one keyframe can be removed, and the resulting keyframe set after removing that keyframe can be used as the keyframes for the spatial scene to be modeled. In this embodiment, only any two keyframes from a large set are used as examples. In actual application, all keyframes need to be traversed to remove keyframes with high similarity, thereby optimizing and simplifying the keyframe set.

[0083] In one embodiment, the similarity between keyframes can be measured by the Peak Signal-to-Noise Ratio (PSNR). During application, keyframes with a similarity exceeding 80% (corresponding to a similarity threshold) can be removed. The PSNR can be calculated using the following formulas (1)-(2):

[0084] (1)

[0085] (2)

[0086] Here, MSE represents the mean squared error, which calculates the mean squared error of pixel values ​​between two keyframe color images I and K. Since a color image has three color channels, the mean squared error is calculated for each of the three channels to obtain the final result. Here, m and n represent the length and width of the image, in pixels; PSNR is the desired peak signal-to-noise ratio, calculated using the maximum pixel value MAX in image I. I Calculated with MSE, The position coordinates of the color image I representing the keyframe; The coordinates of the position of the color image K in the keyframe.

[0087] It should be noted that this embodiment does not limit the specific method for calculating the similarity between two keyframes, which can be adjusted according to the actual situation.

[0088] In another embodiment, to obtain a complete scene from keyframes, a loop closure detection and correction algorithm can be used. The acquired keyframes are input into a loop closure detection program to detect and close loops in the motion trajectory, i.e., to detect whether the camera has returned to a previously traversed position. This process optimizes the consistency of the entire SLAM system and reduces accumulated errors by detecting loop closures, i.e., detecting a good match between the current frame and a previous frame. During loop closure correction, appropriate keyframes need to be supplemented based on the loop closure status of the current keyframe; that is, missing frames are selected from all captured images to ensure loop closure is achieved.

[0089] Figure 4 This is the fourth flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0090] The following will combine Figure 4 The process of another method for spatial three-dimensional modeling using a mobile device provided by the present invention will be described.

[0091] In an exemplary embodiment of the present invention, combined with Figure 4 As can be seen, after extracting the keyframes of the spatial scene to be modeled, the method of performing spatial 3D modeling using a mobile device can also include steps 410 to 430, which will be described in detail below:

[0092] In step 410, the current frame of the spatial scene to be modeled is determined.

[0093] In step 420, the co-viewing degree between the current frame and the keyframe is determined.

[0094] In step 430, if the degree of co-visibility is less than the degree of co-visibility threshold, the current frame is used as a key frame.

[0095] In one embodiment, the current frame of the spatial scene to be modeled can be determined, and the degree of co-view between the current frame and the key frame can be judged.

[0096] When the degree of shared view is less than a shared view threshold, the current frame is designated as a keyframe, i.e., a new keyframe. The corresponding camera position and pose are then defined as the camera pose, thereby optimizing the camera pose. The degree of shared view measures the extent to which two or more keyframes (or camera viewpoints) observe the same set of 3D spatial points (map points). It directly reflects the size of the scene overlap area between different viewpoints and is a crucial basis for the system to understand the scene structure and the optimization process.

[0097] During local map tracking, camera pose can be further optimized in detail, enabling fine-grained capture of local scenes. During tracking, factors such as the number of frames processed since the last global localization, the state of local mapping, the number of feature points tracked in the current frame, and comparison with reference keyframes are considered. For example, if the co-view level is less than a threshold, the current frame is used as the keyframe to determine whether to set the current depth image as the new keyframe. After acquiring the new keyframe, feature matching is performed between this keyframe and existing keyframes to determine the location of the new keyframe and generate new map points. In this embodiment, the accuracy of camera pose is improved by further optimizing all matches.

[0098] In yet another exemplary embodiment of the present invention, the preceding text continues... Figure 1 The above embodiment will be used as an example for illustration. The construction of 3D spatial point cloud data (corresponding to step 140) based on keyframes and the corresponding camera poses can be achieved in the following way:

[0099] Based on the keyframes, the depth image of the keyframes and the homogeneous coordinates of the two-dimensional points of the keyframes are obtained.

[0100] Determine the rotation matrix based on the camera pose corresponding to the keyframe;

[0101] Based on the homogeneous coordinates and rotation matrix of the two-dimensional points in the keyframes, as well as the depth image of the keyframes, three-dimensional spatial point cloud data is constructed.

[0102] In one embodiment, the two-dimensional points of the depth image can be back-projected into three dimensions based on the depth data in the acquired keyframes (corresponding keyframes) and the corresponding camera poses, and all the acquired three-dimensional points can be integrated into a point set containing color information, which is the three-dimensional point cloud data of the scene.

[0103] In the process of projecting two-dimensional points into three dimensions, it is necessary to use camera pose parameters for back projection, which can be achieved using the following formula (3):

[0104] (3)

[0105] in, Represents the projected 3D coordinates (corresponding to 3D spatial point cloud data); Represents the homogeneous coordinates of a 2D point (corresponding to the homogeneous coordinates of a 2D point in a keyframe); This represents the rotation matrix, which is calculated from the rotation parameters in the camera extrinsic parameters, where the camera extrinsic parameters are determined by the camera pose. It represents the depth of a two-dimensional point in three-dimensional space (which can be read from the depth image of a keyframe); Indicates camera intrinsic parameters; This represents the camera's movement matrix. Wherein, Expressed using formula (4):

[0106] (4)

[0107] in, and This indicates the camera's focal length on the x and y axes. and These are the coordinates of the center of the camera aperture.

[0108] Figure 5 This is the fifth flowchart illustrating a method for spatial three-dimensional modeling using a mobile device provided by the present invention.

[0109] The following will combine Figure 5 Another process for spatial 3D modeling using mobile devices is described.

[0110] In an exemplary embodiment of the present invention, combined with Figure 5 As can be seen, after constructing the 3D spatial point cloud data, the method for spatial 3D modeling using mobile devices can also include the following steps 510 to 550, which will be described in detail below:

[0111] In step 510, the target three-dimensional spatial point cloud data is initialized, wherein each point cloud in the target three-dimensional spatial point cloud data corresponds to each point cloud in the three-dimensional spatial point cloud data, and the point clouds in the target three-dimensional spatial point cloud data are evenly distributed.

[0112] In one embodiment, to obtain more accurate point cloud data, the Intermediate Point Proximity (ICP) algorithm can be used. A target point cloud (corresponding to target 3D spatial point cloud data) is initialized, that is, uniform sampling is performed within a 3D region at a certain resolution to obtain a regular 3D point set. It can be understood that each point cloud in the target 3D spatial point cloud data corresponds to each point cloud in the 3D spatial point cloud data, and the point clouds in the target 3D spatial point cloud data are uniformly distributed. The correspondence between each point cloud in the target 3D spatial point cloud data and each point cloud in the 3D spatial point cloud data means that every point cloud existing in the 3D spatial point cloud data can be correspondingly found in the target 3D spatial point cloud data.

[0113] In step 520, the rotation and translation matrix is ​​determined based on the corresponding distances between each point cloud in the target 3D spatial point cloud data and each point cloud in the 3D spatial point cloud data.

[0114] In step 530, the target three-dimensional spatial point cloud data is rotated and translated based on the rotation and translation matrix to obtain the target three-dimensional spatial point cloud data after rotation and translation.

[0115] In another embodiment, a rotation and translation matrix RT can be constructed by calculating the distance between corresponding points in the source point cloud (corresponding to 3D spatial point cloud data) and the target point cloud (corresponding to target 3D spatial point cloud data). Furthermore, based on the rotation and translation matrix, the target 3D spatial point cloud data is rotated and translated to obtain the rotated and translated target 3D spatial point cloud data.

[0116] In step 540, if the target three-dimensional spatial point cloud data after rotation and translation meets the preset conditions, the target three-dimensional spatial point cloud data after rotation and translation is used as the three-dimensional spatial point cloud data.

[0117] In step 550, if the target 3D spatial point cloud data after rotation and translation does not meet the preset conditions, the step of initializing the target 3D spatial point cloud data is repeated until the target 3D spatial point cloud data after rotation and translation is obtained, until the target 3D spatial point cloud data after rotation and translation meets the preset conditions, and the target 3D spatial point cloud data after rotation and translation is used as 3D spatial point cloud data.

[0118] In another embodiment, the mean squared error after transformation can be calculated. If the mean squared error meets a threshold condition, the algorithm terminates. That is, if the target 3D spatial point cloud data after rotation and translation meets the preset conditions, the target 3D spatial point cloud data after rotation and translation is used as the 3D spatial point cloud data. It can be understood that the distribution of points in the 3D spatial point cloud data obtained in this scenario is relatively uniform.

[0119] In another embodiment, if the mean squared error meets the threshold condition, the iteration continues until the error meets the threshold condition or the iteration count terminates. That is, if the target 3D spatial point cloud data after rotation and translation does not meet the preset conditions, the step of initializing the target 3D spatial point cloud data (corresponding to step 510) can be repeated until the step of obtaining the target 3D spatial point cloud data after rotation and translation (corresponding to step 530) is obtained, until the obtained target 3D spatial point cloud data after rotation and translation meets the preset conditions, and the target 3D spatial point cloud data after rotation and translation is used as the 3D spatial point cloud data. Through this embodiment, the distribution of each point cloud in the obtained 3D spatial point cloud data is relatively uniform, and the 3D spatial point cloud data is relatively accurate.

[0120] According to relevant technologies, there is currently no research on scene modeling that includes simulated human bodies in real-world scenarios. However, in practical applications, such as robotics and embodied intelligence, machine learning is needed to recognize human bodies and perform corresponding interactions. Therefore, corresponding data is required to train the machine learning model. The 3D spatial point cloud data obtained based on this invention can effectively reduce the difference between synthetic data and real data.

[0121] As described above, the method for spatial 3D modeling using mobile devices provided by this invention mainly includes three stages: placement of a simulated human body, acquisition of scene point clouds, and construction of semantic tags. In the process of placing the simulated human body model, this invention uses three constraints to ensure that the constructed virtual human data conforms to physical laws and has a natural visual effect. In the process of acquiring scene point clouds, this invention introduces ORB-SLAM2 onto mobile devices and designs a 3D scene point cloud data construction scheme based on binocular images. In the process of constructing semantic tags, this invention uses large model technology to construct semantic tags for the scene's color images, and then utilizes the mapping relationship between scene images and scene point clouds to achieve fully automatic acquisition of point cloud semantic tags, eliminating the need for manual annotation.

[0122] The following describes a system for spatial three-dimensional modeling using a mobile device, which is provided by the present invention. The system for spatial three-dimensional modeling using a mobile device described below and the method for spatial three-dimensional modeling using a mobile device described above can be referred to and correspond to each other.

[0123] Figure 6 This is a schematic diagram of the structure of a system for spatial three-dimensional modeling using a mobile device, provided by the present invention.

[0124] The following will combine Figure 6 The structure of a system for spatial three-dimensional modeling using a mobile device, provided by the present invention, will be described.

[0125] In an exemplary embodiment of the present invention, combined with Figure 6 As can be seen, a system for spatial 3D modeling using a mobile device may include an acquisition module 610, an image capture module 620, an extraction module 630, a construction module 640, a segmentation module 650, and a generation module 660. Each module will be described in detail below.

[0126] The acquisition module 610 can be configured to acquire a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects;

[0127] The shooting module 620 can be configured to shoot the spatial scene to be modeled based on a mobile device to obtain a scene depth image;

[0128] The extraction module 630 can be configured to extract key frames of the spatial scene to be modeled and the camera poses corresponding to the key frames based on the scene depth image.

[0129] The construction module 640 can be configured to construct three-dimensional spatial point cloud data based on the key frame and the camera pose corresponding to the key frame;

[0130] The segmentation module 650 can be configured to acquire the spatial scene to be modeled according to the camera pose, obtain a scene color image corresponding to the scene depth image, and perform semantic segmentation on the scene color image to obtain image semantic labels.

[0131] The generation module 660 can be configured to project and map the image semantic labels onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels.

[0132] In an exemplary embodiment of the present invention, the image semantic label includes the image semantic label corresponding to each pixel in the scene color image; the segmentation module 650 can perform semantic segmentation on the scene color image to obtain the image semantic label in the following manner:

[0133] A pre-trained semantic segmentation model is invoked, wherein the semantic segmentation model is pre-trained based on a sample dataset and generates image semantic labels of a preset category; the sample dataset is a sample dataset about indoor application scenarios with simulated human bodies.

[0134] The scene color image is input into the semantic segmentation model to obtain the image semantic labels corresponding to each pixel in the scene color image output by the semantic segmentation model.

[0135] In an exemplary embodiment of the present invention, the generation module 660 may further be configured to:

[0136] The color values ​​of each pixel in the scene color image are extracted;

[0137] Based on the color values ​​of each pixel in the scene color image, determine the color information of each pixel in the scene color image;

[0138] The color information of each pixel in the scene color image is projected and mapped onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels and / or color information.

[0139] In an exemplary embodiment of the present invention, the acquisition module 610 may acquire the spatial scene to be modeled in the following manner:

[0140] The simulated human body interacts with the environmental objects according to the following constraints to obtain the spatial scene to be modeled, wherein the constraints include any one or more of the following constraints:

[0141] Support constraints, wherein the support constraints are used to characterize the support interaction relationship under the support action of the environmental objects on the simulated human body;

[0142] Collision constraints, wherein the collision constraints are used to characterize the collision interaction relationship between the environmental object and the simulated human body;

[0143] Strong interaction constraints, wherein the strong interaction constraints are used to characterize the interaction relationship between the environmental object and the simulated human body where the interaction points exceed preset points.

[0144] In an exemplary embodiment of the present invention, the extraction module 630 can extract keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes based on the scene depth image in the following manner:

[0145] Based on the scene depth image, keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes are extracted using real-time localization and mapping technology.

[0146] In an exemplary embodiment of the present invention, the extraction module 630 may further be configured to:

[0147] Iterate through all keyframes, and for any two keyframes, calculate the similarity between them.

[0148] If the similarity between keyframes exceeds a similarity threshold, remove the keyframes that exceed the similarity threshold.

[0149] The set of keyframes formed after removing keyframes that exceed the similarity threshold is used as the keyframes of the spatial scene to be modeled.

[0150] In an exemplary embodiment of the present invention, the extraction module 630 may further be configured to:

[0151] Determine the current frame of the spatial scene to be modeled that is currently being tracked;

[0152] Determine the degree of co-view between the current frame and the keyframe;

[0153] If the degree of shared visibility is less than the degree of shared visibility threshold, the current frame is designated as a keyframe.

[0154] In an exemplary embodiment of the present invention, the construction module 640 can construct three-dimensional spatial point cloud data based on the keyframe and the camera pose corresponding to the keyframe in the following manner:

[0155] Based on the keyframe, a depth image of the keyframe and two-dimensional homogeneous coordinates of the keyframe points are obtained.

[0156] Determine the rotation matrix based on the camera pose corresponding to the keyframe;

[0157] Based on the homogeneous coordinates of the two-dimensional points in the keyframe, the rotation matrix, and the depth image of the keyframe, three-dimensional spatial point cloud data is constructed.

[0158] In an exemplary embodiment of the present invention, the construction module 640 may further be configured to:

[0159] Initialize target three-dimensional spatial point cloud data, wherein each point cloud in the target three-dimensional spatial point cloud data corresponds to each point cloud in the three-dimensional spatial point cloud data, and each point cloud in the target three-dimensional spatial point cloud data is evenly distributed;

[0160] Based on the corresponding distances between each point cloud in the target three-dimensional spatial point cloud data and each point cloud in the three-dimensional spatial point cloud data, the rotation and translation matrix is ​​determined.

[0161] Based on the rotation and translation matrix, the target three-dimensional spatial point cloud data is rotated and translated to obtain the target three-dimensional spatial point cloud data after rotation and translation.

[0162] If the target three-dimensional spatial point cloud data after rotation and translation meets the preset conditions, the target three-dimensional spatial point cloud data after rotation and translation will be used as the three-dimensional spatial point cloud data.

[0163] If the target 3D spatial point cloud data after rotation and translation does not meet the preset conditions, the step of initializing the target 3D spatial point cloud data is repeated until the target 3D spatial point cloud data after rotation and translation is obtained, until the target 3D spatial point cloud data after rotation and translation meets the preset conditions, and the target 3D spatial point cloud data after rotation and translation is used as the 3D spatial point cloud data.

[0164] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a method for spatial 3D modeling using a mobile device. The method includes: acquiring a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body interacting with environmental objects; capturing images of the spatial scene to be modeled using the mobile device to obtain a scene depth image; extracting keyframes of the spatial scene to be modeled and camera poses corresponding to the keyframes based on the scene depth image; constructing 3D spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; acquiring images of the spatial scene to be modeled according to the camera poses to obtain a scene color image corresponding to the scene depth image, and performing semantic segmentation on the scene color image to obtain image semantic labels; and projecting and mapping the image semantic labels onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels.

[0165] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for spatial three-dimensional modeling using a mobile device provided by the above methods. The method includes: acquiring a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects; taking a picture of the spatial scene to be modeled using a mobile device to obtain a scene depth image; extracting keyframes of the spatial scene to be modeled and camera poses corresponding to the keyframes based on the scene depth image; constructing three-dimensional spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; acquiring the spatial scene to be modeled according to the camera poses to obtain a scene color image corresponding to the scene depth image, and performing semantic segmentation on the scene color image to obtain image semantic labels; projecting and mapping the image semantic labels onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels.

[0167] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for spatial three-dimensional modeling using a mobile device provided by the above methods. The method includes: acquiring a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body interacting with environmental objects; capturing images of the spatial scene to be modeled using a mobile device to obtain a scene depth image; extracting keyframes of the spatial scene to be modeled and camera poses corresponding to the keyframes based on the scene depth image; constructing three-dimensional spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; acquiring images of the spatial scene to be modeled according to the camera poses to obtain a scene color image corresponding to the scene depth image, and performing semantic segmentation on the scene color image to obtain image semantic labels; projecting and mapping the image semantic labels onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for spatial three-dimensional modeling using mobile devices, characterized in that, The method includes: Obtain a spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects; The scene depth image is obtained by taking a picture of the spatial scene to be modeled using a mobile device; Based on the scene depth image, key frames of the spatial scene to be modeled and the camera poses corresponding to the key frames are extracted. Based on the keyframes and the camera poses corresponding to the keyframes, construct three-dimensional spatial point cloud data; The scene to be modeled is acquired according to the camera pose to obtain a scene color image corresponding to the scene depth image, and the scene color image is semantically segmented to obtain image semantic labels. The image semantic labels are projected and mapped onto the 3D spatial point cloud data to obtain 3D spatial point cloud data with image semantic labels. The spatial scene to be modeled is obtained in the following way: The simulated human body interacts with the environmental objects according to the following constraints to obtain the spatial scene to be modeled, wherein the constraints include any one or more of the following constraints: Support constraints, wherein the support constraints are used to characterize the support interaction relationship under the support action of the environmental objects on the simulated human body; Collision constraints, wherein the collision constraints are used to characterize the collision interaction relationship between the environmental object and the simulated human body; Strong interaction constraints, wherein the strong interaction constraints are used to characterize the interaction relationship between the environmental object and the simulated human body where the interaction points exceed preset points.

2. The method for spatial three-dimensional modeling using a mobile device according to claim 1, characterized in that, The image semantic label includes the image semantic label corresponding to each pixel in the scene color image; the semantic segmentation of the scene color image to obtain the image semantic label is implemented in the following way: A pre-trained semantic segmentation model is invoked, wherein the semantic segmentation model is pre-trained based on a sample dataset, and the semantic segmentation model generates image semantic labels of a preset category; the sample dataset is a sample dataset about indoor application scenarios with simulated human bodies; The scene color image is input into the semantic segmentation model to obtain the image semantic labels corresponding to each pixel in the scene color image output by the semantic segmentation model.

3. A method for spatial three-dimensional modeling using a mobile device according to claim 1 or 2, characterized in that, After projecting and mapping the image semantic labels onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels, the method further includes: The color values ​​of each pixel in the scene color image are extracted; Based on the color values ​​of each pixel in the scene color image, determine the color information of each pixel in the scene color image; The color information of each pixel in the scene color image is projected and mapped onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels and / or color information.

4. The method for spatial three-dimensional modeling using a mobile device according to claim 1, characterized in that, The step of extracting keyframes of the spatial scene to be modeled and the corresponding camera poses based on the scene depth image specifically includes: Based on the scene depth image, keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes are extracted using real-time localization and mapping technology.

5. A method for spatial three-dimensional modeling using a mobile device according to claim 1 or 4, characterized in that, After extracting the keyframes of the spatial scene to be modeled, the method further includes: Iterate through all keyframes, and for any two keyframes, calculate the similarity between them. If the similarity between keyframes exceeds a similarity threshold, remove the keyframes that exceed the similarity threshold. The set of keyframes formed after removing keyframes that exceed the similarity threshold is used as the keyframes of the spatial scene to be modeled.

6. A method for spatial three-dimensional modeling using a mobile device according to claim 1 or 4, characterized in that, After extracting the keyframes of the spatial scene to be modeled, the method further includes: Determine the current frame of the spatial scene to be modeled that is currently being tracked; Determine the degree of co-view between the current frame and the keyframe; If the degree of shared visibility is less than the degree of shared visibility threshold, the current frame is designated as a keyframe.

7. A method for spatial three-dimensional modeling using a mobile device according to claim 1 or 4, characterized in that, The construction of 3D spatial point cloud data based on the keyframes and the corresponding camera poses specifically includes: Based on the keyframe, a depth image of the keyframe and two-dimensional homogeneous coordinates of the keyframe points are obtained. Determine the rotation matrix based on the camera pose corresponding to the keyframe; Based on the homogeneous coordinates of the two-dimensional points in the keyframe, the rotation matrix, and the depth image of the keyframe, three-dimensional spatial point cloud data is constructed.

8. A method for spatial three-dimensional modeling using a mobile device according to claim 7, characterized in that, After constructing the three-dimensional spatial point cloud data, the method further includes: Initialize target three-dimensional spatial point cloud data, wherein each point cloud in the target three-dimensional spatial point cloud data corresponds to each point cloud in the three-dimensional spatial point cloud data, and each point cloud in the target three-dimensional spatial point cloud data is evenly distributed; Based on the corresponding distances between each point cloud in the target three-dimensional spatial point cloud data and each point cloud in the three-dimensional spatial point cloud data, the rotation and translation matrix is ​​determined. Based on the rotation and translation matrix, the target three-dimensional spatial point cloud data is rotated and translated to obtain the target three-dimensional spatial point cloud data after rotation and translation. If the target three-dimensional spatial point cloud data after rotation and translation meets the preset conditions, the target three-dimensional spatial point cloud data after rotation and translation will be used as the three-dimensional spatial point cloud data. If the target 3D spatial point cloud data after rotation and translation does not meet the preset conditions, the step of initializing the target 3D spatial point cloud data is repeated until the target 3D spatial point cloud data after rotation and translation is obtained, until the target 3D spatial point cloud data after rotation and translation meets the preset conditions, and the target 3D spatial point cloud data after rotation and translation is used as the 3D spatial point cloud data.

9. A system for spatial three-dimensional modeling using mobile devices, characterized in that, The system is used to implement the method for spatial three-dimensional modeling using a mobile device as described in any one of claims 1 to 8, the system comprising: The acquisition module is used to acquire the spatial scene to be modeled, wherein the spatial scene to be modeled includes at least a simulated human body that interacts with environmental objects; The camera module is used to capture images of the spatial scene to be modeled using a mobile device to obtain a scene depth image; The extraction module is used to extract keyframes of the spatial scene to be modeled and the camera poses corresponding to the keyframes based on the scene depth image. A construction module is used to construct three-dimensional spatial point cloud data based on the keyframes and the camera poses corresponding to the keyframes; The segmentation module is used to acquire the spatial scene to be modeled according to the camera pose, obtain a scene color image corresponding to the scene depth image, and perform semantic segmentation on the scene color image to obtain image semantic labels. The generation module is used to project and map the image semantic labels onto the three-dimensional spatial point cloud data to obtain three-dimensional spatial point cloud data with image semantic labels.

Images