Semantic map construction method, service equipment, terminal equipment and storage medium

By generating and semantic segmentation of the three-dimensional grid model on the service device side, the problem of low accuracy of three-dimensional semantic maps in AR devices is solved, the accurate display of virtual objects in the physical environment is realized, and the resource consumption of terminal devices is reduced .

CN119013702APending Publication Date: 2024-11-22GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280093416.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In real-time positioning and scene reconstruction, the three-dimensional semantic map has low accuracy, and it is impossible to effectively display virtual objects on specific physical objects. The computing power and memory of the terminal device are not enough to handle large-scale environmental reconstruction in real time. and semantic segmentation.

Method used

By obtaining the collected data of the wearable device on the terminal device side, processing and sending image and attitude data to the service device, the service device generates a three-dimensional grid model and performs semantic segmentation to achieve more complete scene reconstruction and virtual objects in the physical environment. Displays to reduce memory usage and power consumption of terminal devices.

Benefits of technology

Improves the accuracy of semantic maps, better display of virtual objects on specific physical objects, and reduces memory usage and power consumption of terminal devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013702A_ABST
    Figure CN119013702A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a semantic map construction method, service equipment, terminal equipment and a storage medium, the method is applied to the service equipment, and the method comprises the following steps: receiving first image data and attitude data sent by the terminal equipment; generating a three-dimensional grid model according to the first image data and the attitude data; performing semantic segmentation on the three-dimensional grid model to obtain a target semantic map; wherein the target semantic map is used for displaying the virtual object in the physical environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to a semantic map construction method, service device, terminal device, and storage medium. Background Art

[0002] In recent years, with the development of augmented reality (AR) devices, AR technology has become ubiquitous. AR technology is a new technology that "seamlessly" integrates real-world information and virtual-world information, and can apply virtual information to the real world, which is then perceived by human senses, allowing people to feel the realism of "being there".

[0003] In related technologies, most AR devices have SLAM (simultaneous localization and mapping) technology, which allows users to generate maps or grids and locate users. However, although there are solutions to generate three-dimensional (3D) semantic information, the existing technical solutions are not comprehensive, resulting in low accuracy of the entire reconstructed semantic map, and unable to better realize the display of virtual objects on specific physical objects. Summary of the invention

[0004] The present application provides a semantic map construction method, service equipment, terminal equipment and storage medium.

[0005] The technical solution of this application can be implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a semantic map construction method, which is applied to a service device, and the method includes:

[0007] Receiving first image data and posture data sent by a terminal device;

[0008] generating a three-dimensional mesh model according to the first image data and the posture data;

[0009] The three-dimensional mesh model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment.

[0010] In a second aspect, an embodiment of the present application provides a semantic map construction method, which is applied to a terminal device, and the method includes:

[0011] Acquire collected data of the wearable device; wherein the collected data includes first image data and second image data;

[0012] Processing the second image data to generate posture data;

[0013] The first image data and the posture data are sent to the service device so that the service device constructs a target semantic map.

[0014] In a third aspect, an embodiment of the present application provides a service device, including a first receiving unit, a modeling unit and a segmentation unit; wherein:

[0015] A first receiving unit, configured to receive first image data and posture data sent by a terminal device;

[0016] A modeling unit configured to generate a three-dimensional mesh model according to the first image data and the posture data;

[0017] The segmentation unit is configured to perform semantic segmentation on the three-dimensional grid model to obtain a target semantic map; wherein the target semantic map is used to realize the display of the virtual object in the physical environment.

[0018] In a fourth aspect, an embodiment of the present application provides a service device, including a first memory and a first processor; wherein:

[0019] A first memory, for storing a computer program that can be run on the first processor;

[0020] The first processor is configured to execute the method according to the first aspect when running a computer program.

[0021] In a fifth aspect, an embodiment of the present application provides a terminal device, including an acquisition unit, a data processing unit and a second sending unit; wherein:

[0022] An acquisition unit, configured to acquire acquisition data of the wearable device; wherein the acquisition data includes first image data and second image data;

[0023] a data processing unit configured to process the second image data to generate posture data;

[0024] The second sending unit is configured to send the first image data and the posture data to the service device, so that the service device constructs a target semantic map.

[0025] In a sixth aspect, an embodiment of the present application provides a terminal device, including a second memory and a second processor; wherein:

[0026] A second memory for storing a computer program that can be run on a second processor;

[0027] The second processor is used to execute the method described in the second aspect when running the computer program.

[0028] In a seventh aspect, an embodiment of the present application provides a computer storage medium storing a computer program, which implements the method of the first aspect when executed by a first processor, or implements the method of the second aspect when executed by a second processor.

[0029] The embodiment of the present application provides a semantic map construction method, service device, terminal device and storage medium. On the terminal device side, the collected data of the wearable device is obtained; wherein the collected data includes first image data and second image data; the second image data is processed to generate posture data; the first image data and posture data are sent to the service device so that the service device constructs a target semantic map. On the service device side, the first image data and posture data sent by the terminal device are received; a three-dimensional mesh model is generated according to the first image data and posture data; the three-dimensional mesh model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment. In this way, the posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be realized, the accuracy of the entire semantic map is improved, and the display of virtual objects on specific physical objects can be better realized; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the composition of a visual enhancement system provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of a semantic map construction method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of an AR glasses provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a moving cube provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the architecture of semantic and instance segmentation provided in an embodiment of the present application;

[0035] Figure 6 A flowchart of another semantic map construction method provided in an embodiment of the present application;

[0036] Figure 7 An application schematic diagram of an AR application provided in an embodiment of the present application;

[0037] Figure 8A detailed flowchart of a semantic map construction method provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of the composition structure of a service device provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of a specific hardware structure of a service device provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0041] Fig.12 A schematic diagram of the specific hardware structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific order for the objects. It is understandable that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0045] In recent years, with the development of AR devices (such as Magic Leap One, HoloLens, etc.) and development kits (such as ARcore, ARkit, etc.), AR technology has become ubiquitous. Most AR devices and development kits have SLAM technology, allowing users to generate maps or grids and locate users. However, there is currently no dedicated system architecture to solve the problem of real-time positioning / scene reconstruction / 3D semantic segmentation directly using AR devices and service devices.

[0046] In the related art, although there are solutions for generating 3D semantic information, such as using an Internet depth camera (RGB-D Intel RealSense) to obtain real-time 3D semantic information and display this information on Magic Leap One. This solution requires RealSense to be connected to the service device through a physical cable, resulting in data collection being limited by the length of the cable. Then each color image and depth image (Colorimage includes red, green, blue band information and depth image, RGBD) frame from RealSense containing red, green and blue band information is sent to the service device. Here, semantic segmentation technology is applied to the color image (Colorimage includes red, green, blue band information, RGB image) containing red, green and blue band information, and the corresponding depth (point cloud) is marked with a semantic class; finally, the semantic points are sent back to Magic Leap One to display the semantic information. However, this solution does not combine 6-DOF posture data to reconstruct the entire semantic map.

[0047] It is understandable that AR is a technology that combines the virtual world and the real world, allowing people to interact with the virtual world in real time. The key technologies of AR are SLAM and real-world reconstruction. The 6-DOF posture and depth images generated by SLAM can achieve more complete scene reconstruction. In order to perceive and interact with virtual objects in the real world more intelligently, scene understanding in the 3D reconstructed world requires semantic information, and RGBD data from AR devices can be used for 3D world perception.

[0048] In an embodiment of the present application, the computing power and memory size of the AR device (for example, a terminal device with AR glasses) are relatively low compared to the service device. In order to achieve large-scale environmental reconstruction and real-time semantic segmentation, the AR device may not be able to process all data in real time. Therefore, it is necessary to perform systematic real-time positioning / scene reconstruction / 3D semantic segmentation between the AR device and the service device. A communication connection can be established between the AR device and the service device through the Transmission Control Protocol (TCP). The user can access real-time 6-DOF posture data and RGBD data from the AR device, and send these data to the server for scene reconstruction and 3D semantic segmentation. In this way, the embodiment of the present application enables the user to interact with the physical environment in a more intelligent way after 3D semantic segmentation, such as displaying virtual objects on specific physical objects.

[0049] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0050] See also Figure 1 , which shows a schematic diagram of the composition of a visual enhancement system provided by an embodiment of the present application. Figure 1 As shown, the visual enhancement system 10 may include a wearable device 110, a terminal device 120, and a service device 130. The wearable device 110 and the terminal device 120 are connected to each other through a physical cable, and the terminal device 120 and the service device 130 are connected to each other through a TCP protocol.

[0051] It should be noted that the wearable device 110 may specifically refer to a monocular or binocular head-mounted display (HMD), such as AR glasses. Figure 1 In the embodiment, the wearable device 110 may include one or more display modules placed in a position area close to a single eye or both eyes of the user. Among them, through the display module of the wearable device 110, the display content therein can be presented in front of the user, and the display content can fill or partially fill the user's field of vision. It should also be noted that the display module may refer to one or more organic light-emitting diode (OLED) modules, liquid crystal display (LCD) modules, laser display modules, etc.

[0052] In some embodiments, the wearable device 110 may further include one or more sensors and one or more cameras. For example, the wearable device 110 may include one or more sensors such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, a proximity sensor, and a depth camera to obtain the collected data.

[0053] It should also be noted that the terminal device 120 can be implemented in various forms. For example, the terminal device described in the embodiments of the present application may include, for example, a smart phone, a tablet computer, a notebook computer, a laptop computer, a PDA, a smart watch, etc. In addition, the service device 130 may be a cloud server, a network server, etc. The service device 130 can be regarded as a computer that manages computing resources and can provide computing or application services for the terminal device 120. Moreover, the service device 130 has high-speed CPU computing power, long-term reliable operation, strong data throughput, and better scalability.

[0054] based on Figure 1 The application scenario shown is shown in Figure 2, which shows a flow chart of a semantic map construction method provided by an embodiment of the present application. Figure 2 As shown, the method may include:

[0055] S201: The terminal device obtains collected data of the wearable device; wherein the collected data includes first image data and second image data.

[0056] It should be noted that the embodiment of the present application is applied to a visual enhancement system including a wearable device, a terminal device and a service device. In the system, the wearable device and the terminal device are connected by a physical cable to achieve information exchange; the terminal device and the service device are connected wirelessly by a TCP protocol to achieve information exchange.

[0057] It should also be noted that for the terminal device, the terminal device first needs to obtain the collected data of the wearable device, where the collected data may include first image data and second image data. The first image data is used for scene reconstruction and semantic segmentation, and the second image data is used for posture data estimation.

[0058] S202: The terminal device processes the second image data to generate posture data.

[0059] It should be noted that the terminal device is provided with a SLAM system, and the SLAM system can use the second image data to estimate the camera posture. In the embodiment of the present application, the second image data may include fisheye image data and inertial sensor data. Therefore, in some embodiments, the processing of the second image data to generate posture data may include:

[0060] The SLAM system is used to perform posture calculation on the fisheye image data and inertial sensor data to generate posture data.

[0061] It should be noted that the SLAM system can use fisheye image data and inertial sensor data to generate reliable posture data on the terminal device. Specifically, fisheye image data and inertial sensor data can be used to generate six degrees of freedom (DOF) posture data. Among them, the six degrees of freedom can include the freedom of movement along the three rectangular coordinate axes of x, y, and z and the freedom of rotation around these three coordinate axes, so that the position information can be fully determined, and then it can be better used for scene reconstruction and semantic segmentation.

[0062] In the embodiment of the present application, the first image data may refer to RGBD image data, which may specifically include depth image data and RGB image data, which are used for scene reconstruction and semantic segmentation; the second image data may include fisheye image data and inertial sensor data, which are used for posture data estimation. In addition, the inertial sensor may also be referred to as an inertial measurement unit (IMU), that is, the inertial sensor data may also be referred to as IMU data.

[0063] Specifically, in some embodiments, for S201, obtaining the collected data of the wearable device may include:

[0064] Acquire the depth image data and the corresponding timestamp information through the first thread;

[0065] Obtain RGB image data and corresponding timestamp information through the second thread;

[0066] Obtain fisheye image data and corresponding timestamp information through a third thread;

[0067] Obtain inertial sensor data and corresponding timestamp information through the fourth thread;

[0068] Among them, the timestamp information is used to measure whether the depth image data, RGB image data, fisheye image data and inertial sensor data are time synchronized.

[0069] It should be noted that for wearable devices, Figure 3 The AR glasses shown in the figure can collect data through the AR glasses, so that depth image data, RGB image data, fisheye image data and IMU data can be collected. Here, the depth image data is obtained by the depth camera, the RGB image data is obtained by the RGB camera, the fisheye image data is obtained by the wide-angle fisheye camera, and the IMU data is obtained by the inertial measurement unit.

[0070] It should also be noted that the depth image data and RGB image data can be used for scene reconstruction and semantic segmentation, the fisheye image data and IMU data can be used for camera pose estimation, and then all the data can be sent to the terminal device via a physical cable for further processing.

[0071] It should also be noted that the SLAM system uses fisheye image data and inertial sensor data to generate reliable posture data on the terminal device. Here, since the fisheye image has a larger field of view than the RGB image, more reliable posture data can be obtained in this way; in addition, compared to using only visual SLAM, the camera posture estimation using visual inertial SLAM in the embodiment of the present application has better robustness and accuracy.

[0072] In this way, the terminal device can obtain depth image data, RGB image data, fisheye image data and inertial sensor data and their corresponding timestamp information from the wearable device through different threads. The timestamp information here is used to obtain synchronization data. Since the sending time of each data in the AR glasses may be asynchronous, it is necessary to use the timestamp information to select data at similar time points to synchronize the data.

[0073] Furthermore, in some embodiments, the method may also include: performing time synchronization processing on the depth image data, the RGB image data and the posture data through a time synchronization program.

[0074] In a specific embodiment, the time synchronization process may include:

[0075] Determine first timestamp information corresponding to the posture data;

[0076] Selecting the depth image data synchronized with the first timestamp information from the depth image data and the corresponding timestamp information;

[0077] Selecting RGB image data synchronized with the first timestamp information from the RGB image data and the corresponding timestamp information;

[0078] The first image data is determined according to the selected depth image data and the selected RGB image data; wherein the first image data is synchronized with the time of the posture data.

[0079] It should be noted that in an embodiment of the present application, posture data and first timestamp information can be obtained based on fisheye image data, inertial sensor data and corresponding timestamp information; then, time-synchronized depth image data and RGB image data can be selected based on the first timestamp information; at this time, the first image data and posture data obtained are time synchronized.

[0080] It should also be noted that since the collected data / data sets come from different hardware and different threads, the timestamps of these data / data sets will be different and asynchronous. In order to better perform scene reconstruction and semantic segmentation, data synchronization is necessary. Therefore, a time synchronization program can be set in the terminal device so that the time synchronization program runs on the terminal device. Only synchronized data is allowed to be sent to the service device.

[0081] S203: The terminal device sends the first image data and the posture data to the service device.

[0082] In the embodiment of the present application, the time synchronization between the first image data and the posture data is performed, and the terminal device will send the data to the service device at this time, so that the service device can construct a target semantic map.

[0083] In addition, the terminal device and the service device are wirelessly connected, and specifically, a TCP connection may be used to implement information exchange. Therefore, in some embodiments, sending the first image data and the posture data to the service device may include:

[0084] Establish a TCP connection with the service device;

[0085] Based on the TCP connection, the first image data and the posture data are sent to the service device.

[0086] It should be noted that a TCP connection needs to be established between the terminal device and the service device, and then the terminal device can send data to the service device or receive data sent by the service device based on the TCP connection. In this way, the first image data and the posture data are allowed to be sent to the service device only when they are synchronized data. In this way, data can be sent efficiently and the bandwidth of TCP communication can be reduced.

[0087] S204: The service device generates a three-dimensional mesh model according to the first image data and the posture data.

[0088] It should be noted that, for the service device, the service device needs to receive the first image data and posture data sent by the terminal device, which may specifically include: establishing a TCP connection with the terminal device; and receiving the first image data and posture data sent by the terminal device based on the TCP connection.

[0089] That is to say, after establishing a TCP connection between the service device and the terminal device, the service device can receive the first image data and posture data sent by the terminal device. Here, the first image data may include depth image data and RGB image data, and the time of the depth image data, RGB image data and posture data is synchronized, thereby reducing the bandwidth of TCP communication.

[0090] In some embodiments, for S204, generating a three-dimensional mesh model according to the first image data and the posture data may include:

[0091] Determine, based on the first image data and the posture data, local point clouds corresponding to different observation points of the depth camera;

[0092] Perform fusion calculation on the local point clouds corresponding to different observation points to determine the fusion value of at least one voxel in the three-dimensional space;

[0093] A three-dimensional mesh model is constructed based on the fused value of at least one voxel.

[0094] Furthermore, in some embodiments, for a three-dimensional space including at least one voxel, the method may further include:

[0095] Create geometry;

[0096] The geometric body is voxelized to obtain a three-dimensional space including at least one voxel.

[0097] It should be noted that the geometric body may specifically refer to a rectangular bounding box, which may completely surround the object to be reconstructed; and then the rectangular bounding box is voxelized, so as to obtain a three-dimensional space including at least one voxel.

[0098] It should also be noted that after obtaining the three-dimensional space, the fusing calculation of the local point clouds corresponding to different observation points to determine the fusion value of at least one voxel in the three-dimensional space may include: based on the three-dimensional space, using a truncated signed distance function (TSDF) algorithm to fusing the local point clouds corresponding to different observation points to obtain a TSDF value of at least one voxel;

[0099] Accordingly, constructing the three-dimensional mesh model based on the fusion value of at least one voxel may include: constructing the three-dimensional mesh model according to the TSDF value of at least one voxel.

[0100] That is to say, in the embodiment of the present application, the local point clouds corresponding to different observation points can be fused into the TSDF algorithm model to obtain the TSDF value of at least one voxel in the three-dimensional space; then, according to the TSDF value of at least one voxel in the three-dimensional space, a three-dimensional grid model is constructed. Specifically, by voxelizing the cuboid bounding box, a three-dimensional space including at least one voxel is obtained; then, based on the three-dimensional space, when the local point clouds corresponding to different observation points are fused into the truncated signed distance function TSDF model, the TSDF value of at least one voxel in the three-dimensional space can be calculated using the TSDF algorithm.

[0101] Furthermore, in some embodiments, the method may further include: if the TSDF value is a positive value, determining that the voxel is located between the depth camera and the surface of the object; if the TSDF value is a negative value, determining that the voxel is located outside the depth camera and the surface of the object.

[0102] That is to say, in an embodiment of the present application, after obtaining RGB image data, depth image data and posture data, a three-dimensional network model is obtained by fusion of the TSDF algorithm model. Specifically, by observing at different positions with a depth camera, a local point cloud can be obtained; in order to obtain a three-dimensional network model of the environment, it is necessary to fuse the local point clouds observed by the depth camera from different angles. By extracting surface information from dense point clouds through this fusion method, information redundancy can be greatly reduced, a more accurate surface model (i.e., a three-dimensional network model) can be obtained, and the influence of noise can be reduced. The TSDF model is an effective method for obtaining a reconstructed surface. Specifically, the distance from the voxel center (i.e., the center coordinate of the voxel) to the nearest surface can be represented by a signed distance function TSDF. For a certain voxel in space, if the TSDF value is positive, then the voxel center is between the depth camera and the surface of the object; if the TSDF value is negative, then the voxel center is located outside the camera and the surface of the object. It should be noted that the zero point of the TSDF function represents the surface of the object.

[0103] In a specific embodiment, the using of the TSDF algorithm to perform fusion calculation on the local point clouds corresponding to the different observation points to obtain the TSDF value of at least one voxel may include:

[0104] Based on the first voxel, determine the first TSDF value and the first weight value obtained at the current observation point, and obtain the fused TSDF value and the fused weight value obtained after fusion of the previous observation point;

[0105] A TSDF algorithm is used to perform weighted average calculation according to the fused TSDF value, the fused weight value, the first TSDF value and the first weight value, so as to obtain a second TSDF value and a second weight value corresponding to the first voxel;

[0106] The fused TSDF value and the fused weight value are updated according to the second TSDF value and the second weight value to achieve the fusion of the local point cloud corresponding to the current observation point into the TSDF model; wherein the first voxel is any one of at least one voxel in the three-dimensional space.

[0107] It should be noted that in the camera coordinate system, the light beam from the depth camera passes through the voxel at a specific position in the space. The distance between the center of the voxel and the surface of the object can be approximated as the distance from the center of the voxel to the observation point along the direction of the light. Then, the distance from the voxel to the light observation point is the TSDF value of the voxel. If the voxel is between the depth camera and the observation point, the TSDF value is positive, otherwise the TSDF value is negative. In this way, the depth maps observed at different positions are fused. When different lights pass through the same voxel, the TSDF value is updated using the weighted average method, as follows,

[0108]

[0109] Among them, α represents the first weight value obtained at the current observation point, which can also be called confidence; d(v) represents the first TSDF value obtained at the current observation point, that is, the observation value, that is, the distance from the voxel center to the object surface calculated according to the depth observed by the depth camera; D i (v) represents the TSDF value of the i-th point, that is, the fused TSDF value; W i (v) represents the weight value of the i-th point, that is, the fusion weight value.

[0110] That is to say, for each voxel, in addition to storing a distance function D(v), it also retains a continuously accumulated weight value to superimpose the confidence ɑ of each measurement. Among them, ɑ is a constant, which can usually be taken as 1. In addition, the updated TSDF value here mainly refers to the geometric information of the voxel; for the color information of the voxel, the fused RGB color can also be obtained by the weighted average method.

[0111] In the mapping thread, when new RGB image data and depth image data arrive, the fusion process of the TSDF model continues. After obtaining the TSDF model, we need to extract the surface information of the object from the TSDF model. Among them, the surface of the object can usually be represented by many interconnected triangles. Extract the triangles of the object surface from the TSDF model by marching cubes. The specific steps are as follows:

[0112] See also Figure 4 , which shows a schematic diagram of a moving cube provided in an embodiment of the present application. Figure 4 As shown, the cube provided in (a) can be composed of 8 adjacent voxel centers in space. In the TSDF model, the stored value of each voxel center is the distance from the point to the nearest plane. If the object surface passes through the cube, the TSDF values ​​stored in the cube vertices on both sides of the object surface must be different. By calculating the product of the TSDF values ​​of the two endpoints on each side of the cube, it can be determined whether there are different signs, thereby determining whether there is a surface passing through. If a surface passes through, the TSDF value of the cube vertex can be used to determine the position of the surface passing through the edge of the cube. After obtaining multiple surface positions, the triangles can be connected to form a complete object surface, see for details. Figure 4 As shown in (b) in the figure, triangles can be extracted from the vertices on the surface of the object.

[0113] In short, in the embodiment of the present application, the TSDF function is a common method for calculating the surface of an object in 3D reconstruction. TSDF can be used to construct spatial voxels by calculating the TSDF value of each voxel and then using the previously mentioned Marching Cube method to extract the surface of the object.

[0114] For the TSDF model, a large space (called Volume) is used as the three-dimensional model to be established. This space can completely include the object model to be reconstructed. The Volume is composed of many small voxels (Voxels, i.e. small cubes). Each Voxel corresponds to a point in the space. This point mainly involves two parameters: the distance value from the voxel to the nearest surface and the weight value when the voxel is updated.

[0115] First, a rectangular bounding box needs to be established that can completely surround the object to be reconstructed; then the rectangular bounding box is voxelized, that is, the rectangular bounding box is divided into n equal parts, and the size of the voxel depends on the number of bounding boxes and divided voxels; then each voxel is converted into a three-dimensional position point in the world coordinate system.

[0116] Secondly, traverse all voxels, take the three-dimensional position point v of a voxel in the world coordinate system as an example, and fuse the local point clouds observed at different positions according to the above formulas (1) and (2) until the final output result can accurately reconstruct the three-dimensional mesh model. In addition, it should be noted that the characteristic of the TSDF model is that the calculation is very simple, there is no complex calculation, and the details of the generated mesh are well maintained and the accuracy is also good.

[0117] S205: The service device performs semantic segmentation on the three-dimensional mesh model to obtain a target semantic map.

[0118] It should be noted that after reconstructing the three-dimensional network model, the service device can continue to perform semantic segmentation on the three-dimensional mesh model, so as to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in the physical environment.

[0119] In some embodiments, performing semantic segmentation on the three-dimensional mesh model to obtain a target semantic map may include:

[0120] Inputting the three-dimensional mesh model into the neural network structure;

[0121] In the neural network structure, semantic and instance segmentation is performed using point-by-point feature learning to obtain the target semantic map.

[0122] In a specific embodiment, performing semantic and instance segmentation by point-by-point feature learning to obtain a target semantic map may include:

[0123] Perform 3D sparse convolution operations on the 3D mesh model to determine semantic information as well as feature embedding information, spatial embedding information, and occupancy information;

[0124] Using image segmentation algorithm to group supervoxels of the three-dimensional mesh model to obtain supervoxel information;

[0125] Perform covariance estimation based on feature embedding information and spatial embedding information to obtain target embedding information;

[0126] Perform clustering operations based on target embedding information, occupancy information, and supervoxel information to determine instance information;

[0127] According to the semantic information and instance information, the target semantic map is obtained.

[0128] It should be noted that a real-time semantic segmentation architecture is executed on the service device to process the reconstructed three-dimensional mesh model. Here, the neural network structure can be a 3D UNet structure, but it can also be other neural network structures, and the embodiment of the present application does not make any limitation.

[0129] It should also be noted that object detection not only needs to provide the category of the object in the image, but also the location of the object (Bounding box). Semantic segmentation needs to predict the label of which category each pixel of the input image belongs to. On the basis of semantic segmentation, instance segmentation also needs to distinguish different individuals of the same category. For example, after semantic segmentation, three people can be determined to have labels that belong to people, but each person can be regarded as an instance. Here, after obtaining the target semantic map, point cloud labeling can also be performed to distinguish different instances.

[0130] For example, see Figure 5 , which shows a schematic diagram of the architecture of semantic and instance segmentation provided by an embodiment of the present application. Here, for the input point cloud, RGB features are used as input, and a 3D UNet structure is used to perform point-by-point feature learning. The learned features are decoded into various representations through a fully connected layer, which can be used for 3D instance segmentation. Figure 5As shown in the figure, with 3D geometric information as input, the 3D UNet structure is used for point-by-point feature learning, and different representations of each input voxel can be obtained, including: semantic information, feature embedding information, spatial embedding information and occupancy information; among them, the purpose of semantic information is to assign category labels, the purpose of feature embedding information and spatial embedding information is to fuse feature and spatial information, and the purpose of occupancy information is to indicate how likely it is that there is an object in the input voxel; then covariance estimation is performed on the feature embedding information and the spatial embedding information, which aims to learn an embedding vector that takes into account feature and spatial embedding for instance segmentation; after obtaining the combination of the two, it is also necessary to weight the occupancy information obtained in advance, so that the weight value can be determined (using w i,j In other words, w i,j The larger the value, the greater the possibility that i and j belong to the same instance. In addition, it is necessary to perform image segmentation and clustering on the three-dimensional geometric information to obtain supervoxel grouping. Then, an initial graph can be defined based on the supervoxel. Here, the embedding feature of the supervoxel is obtained by averaging the sum of the feature embedding and spatial embedding of other voxels in the supervoxel. The initial graph can be represented by (V, E, W). Among them, the embedding feature of the supervoxel ∈V, the edge between supervoxels i and j ∈E, and the weight value w i,j ∈W; then perform graph operations on the initial graph to obtain the final graph; in this process, for each edge in E, select w i,j The largest edge, that is, it means that i and j are most likely to be the same instance. Set a threshold T. If the weight value w i,j >T, then the two nodes can be merged, and the graph can be updated after the merger. The update is continued until no edge weight value is greater than T, at which time the final result can be output.

[0131] That is to say, Figure 5In this paper, 3D geometric information is taken as input to generate instance-level semantic information prediction point by point. Considering that 3D metric space provides more reliable 3D scene perception than 2D image-based projection observations, 3D occupancy information can be used to represent the number of voxels occupied by each instance. Such occupancy information represents the inherent and basic properties of each 3D instance. The occupancy information is encoded into the traditional 3D instance segmentation pipeline. In the occupancy perception process, both the learning and clustering stages fully exploit the characteristics of occupancy information. In addition, the learning stage can also take the color 3D scene as input and use the spatial sparse convolution method to extract the mixture vector of each voxel, which not only learns the classical embedding (including spatial embedding and feature embedding), but also produces an occupancy information that implies the volume at the object level. In order to make full use of the semantic information and geometric information, the feature and spatial embeddings are explicitly supervised under different objectives and further combined by estimating the covariance of the distance between the feature and spatial embeddings. For the clustering stage, an image-based segmentation algorithm is used to group supervoxels of the 3D input point cloud based on geometric and appearance constraints; then, in order to merge supervoxels with similar feature embeddings into the same instance, an adaptive threshold is used to evaluate the similarity between embedding distance and occupancy size. Clustering can help correctly cluster hard samples through reliable comparison of predicted occupancy and clustered occupancy, and can eliminate false positives that identify partial instances as independent instances.

[0132] It should also be noted that TSDF can only provide a three-dimensional mesh model. The above content describes how to use the three-dimensional mesh model generated by TSDF to obtain semantic information and instance information. The neural network structure here can be divided into two parts: semantic information and instance information. Among them, semantic information is such as tables, chairs, etc., and instance information is two objects, etc.; after semantic and instance segmentation, information about two chairs and a table can be obtained. In addition, occupancy information is also provided here, which is defined as the number of voxels occupied by each instance; on this basis, an occupancy-aware three-dimensional instance segmentation scheme is proposed. The occupancy information indicates how likely it is that there is a real object inside a voxel. Figure 5 The architecture shown not only uses spatial and feature embeddings, but also considers occupancy information, so that instances can be segmented more accurately and the target semantic map can be reconstructed.

[0133] Furthermore, after obtaining the target semantic map, potential AR applications of the semantic information can also be used, such as moving virtual objects to corresponding locations of interest to the user and displaying them. Figure 6 As shown, after S205, the method may further include:

[0134] S601: The service device obtains target coordinate information of an object of interest to the user.

[0135] S602: The service device sends the target coordinate information to the terminal device.

[0136] S603: The terminal device generates a rendering image according to the target coordinate information and the virtual object and sends the rendering image to the wearable device, so that the virtual object is displayed at a position corresponding to the target coordinate information through the wearable device.

[0137] It should be noted that, for the service device, the service device can obtain the target coordinate information of the object of interest to the user from the target semantic map; and then send the target coordinate information to the terminal device so that the terminal device generates and displays a rendered image containing the virtual object.

[0138] It should also be noted that, for the terminal device, the terminal device can receive the target coordinate information sent by the service device based on the TCP connection; generate a rendered image based on the target coordinate information and the virtual object; and then send the rendered image to the wearable device so that the virtual object is displayed at the corresponding position of the target coordinate information through the wearable device.

[0139] It is understandable that the obtained instance and semantic information can be used for AR applications (e.g., the interaction between virtual objects and the physical environment). For example, the embodiment of the present application can use semantic and instance information to automatically identify the coordinate information of the table and the wall, and then move the virtual object to the corresponding position through a simple command to realize the display of the virtual object in the physical environment. Figure 7 , which shows an application schematic diagram of an AR application provided in an embodiment of the present application. Figure 7 As shown, first, the visual enhancement system collects and processes data, and then constructs a three-dimensional network model based on these data to obtain a three-dimensional mesh model, as shown in (a); secondly, the three-dimensional network model is semantically and instance-segmented to determine the semantic information and instance information of the three-dimensional mesh model, as shown in (b); thirdly, after obtaining the semantic information and instance information of the model, the model is interacted with, specifically, the target coordinate information of the object of interest to the user is determined, and then a rendered image containing the virtual object is generated based on the target coordinate information and virtual objects (e.g., oil paintings and displays, etc.); for example, the table and wall of the object of interest to the user, the coordinate information of the table and wall is determined by detecting the table and wall, and then the oil painting is hung on the wall and the display is placed on the table, and (c) shows the interaction result with the model. In addition, it should be noted that virtual objects such as oil paintings and displays are simulated. For example, some software will provide some virtual objects, and virtual objects can be displayed in a real environment in human-computer interaction.

[0140] In this way, after the terminal device generates the rendered image, it can send the rendered image to the AR glasses. Specifically, if there is semantic and instance information, the virtual object can be displayed on the physical object of interest through a monitor or AR glasses connected to the service device. The target coordinate information of the physical object of interest is sent to the terminal device. A rendered image of the virtual object displayed on the target coordinate information is generated on the terminal device, and the generated rendered image is sent to the AR glasses, so that the user can visualize the virtual objects in the physical environment on the AR glasses.

[0141] This embodiment provides a method for constructing a semantic map, in which, on the terminal device side, the collected data of the wearable device is obtained; wherein the collected data includes first image data and second image data; the second image data is processed to generate posture data; the first image data and posture data are sent to the service device; then, on the service device side, the first image data and posture data sent by the terminal device are received; a three-dimensional mesh model is generated according to the first image data and posture data; the three-dimensional mesh model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment. In this way, the posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be realized, the accuracy of the entire semantic map is improved, and the display of virtual objects on specific physical objects can be better realized; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced.

[0142] Based on the same inventive concept as the above embodiments, see Figure 8 , which shows a detailed flow chart of a semantic map construction method provided by an embodiment of the present application. Figure 8 As shown, the wearable device takes AR glasses as an example, and the detailed process may include:

[0143] S801: The AR glasses collect RGBD image data, fisheye image data, and IMU data.

[0144] S802: The terminal device obtains RGBD image data, fisheye image data and IMU data, and generates posture data using the SLAM system based on the fisheye image data and the IMU data.

[0145] S803: The service device obtains RGBD image data and posture data, and generates a three-dimensional mesh model.

[0146] S804: The service device performs semantic segmentation on the three-dimensional mesh model to obtain a target semantic map.

[0147] S805: The service device displays the interaction result on the monitor according to the interaction between the target semantic map and the virtual object.

[0148] S806: The terminal device obtains coordinate information of the object of interest, and generates a rendering image of the virtual object located on the coordinate information.

[0149] S807: The AR glasses display the rendered image.

[0150] It should be noted that the execution subject of steps S801 and S807 is the AR glasses, the execution subject of steps S802 and S803 is the terminal device, and the execution subject of steps S803, S804 and S805 is the service device.

[0151] It should also be noted that, in an embodiment of the present application, the AR glasses and the terminal device are connected by a physical cable, and the RGBD image data, fisheye image data, and IMU data collected by the AR glasses can be sent to the terminal device, where the RGBD image data may include RGB image data and depth image data; in addition, the rendered image generated by the terminal device can also be sent to the AR glasses.

[0152] It should also be noted that in the embodiment of the present application, the terminal device and the service device are wirelessly connected via the TCP protocol, and RGBD image data and posture data can be sent to the service device; in addition, the coordinate information of the object of interest to the user determined by the service device can also be sent to the terminal device.

[0153] That is to say, in an embodiment of the present application, the terminal device obtains RGB image data, depth image data, fisheye image data, IMU data and their timestamps from the AR glasses through different threads respectively. The posture data is generated by the SLAM system, which uses fisheye image data and IMU data to obtain reliable 6-DOF posture data. Then, the synchronized RGB image data, depth image data and posture data are sent to the service device via a TCP connection. The service device reconstructs the three-dimensional mesh model based on the obtained posture data and RGBD data. After reconstructing the three-dimensional mesh model, the service device uses information such as RGB and mesh to run effective semantic segmentation.

[0154] Here, semantic information can be used for human-computer interaction in different applications (such as displaying virtual objects on a table or wall, etc.). This technical solution integrates data collection, real-time 3D reconstruction, real-time semantics and instance segmentation of AR glasses into an overall system. After collecting data through the network, AR glasses data can be reconstructed online to achieve real-time semantics and instance segmentation.

[0155] In short, in an embodiment of the present application, the system architecture can provide a large-scale real-time semantic model using a terminal device with AR glasses. TCP communication efficiently sends synchronous data to the service device. A semantic three-dimensional model is established in real time based on the visual inertial SLAM system and the RGBD data on the service device. In this way, a terminal device with AR glasses can be used to capture a real scene model containing semantic information in real time. In addition, when TCP communication is used to transmit relevant data, scene reconstruction is not limited to a small range. Through powerful service equipment, not only can semantic segmentation be achieved in real time and the scale of three-dimensional reconstruction be expanded; but the reconstructed semantic three-dimensional model can also be applied to fields such as virtual object display and semantic building model reconstruction.

[0156] In addition, the embodiments of the present application can also reconstruct directly on the terminal device, and then send the reconstruction results to the service device, so that the service device can perform semantic segmentation; however, the transmission of the three-dimensional model will be much slower than the transmission of camera data. Semantic segmentation can be run offline on the service device; in this way, the user needs to wait for the semantic segmentation results of the scene, and the scene cannot grow dynamically. In addition, the entire semantic three-dimensional model can also be sent back to the terminal device for different types of applications; however, the transmission speed of the three-dimensional model is slow. In addition, if power consumption, time cost, and memory usage are not considered, the ideal method is to process everything on the terminal device rather than on the service device; however, the terminal device will not have such large computing resources, which will cause time delays and exhaust memory and battery.

[0157] The present embodiment provides a semantic map construction method. The specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that according to the technical solution of the aforementioned embodiment, posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be achieved, the accuracy of the entire semantic map is improved, and the display of virtual objects on specific physical objects can be better realized; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced.

[0158] Based on the same inventive concept as the above embodiments, see Fig. 9 , which shows a schematic diagram of the composition structure of a service device 90 provided in an embodiment of the present application. Fig. 9 As shown, the service device 90 may include: a first receiving unit 901, a modeling unit 902 and a segmentation unit 903; wherein,

[0159] A first receiving unit 901 is configured to receive first image data and gesture data sent by a terminal device;

[0160] A modeling unit 902 is configured to generate a three-dimensional mesh model according to the first image data and the posture data;

[0161] The segmentation unit 903 is configured to perform semantic segmentation on the three-dimensional mesh model to obtain a target semantic map; wherein the target semantic map is used to realize the display of the virtual object in the physical environment.

[0162] In some embodiments, the first receiving unit 901 is specifically configured to establish a transmission control protocol TCP connection with the terminal device; and based on the TCP connection, receive the first image data and posture data sent by the terminal device.

[0163] In some embodiments, see Fig. 9 , the service device 90 may further include a fusion unit 904, configured to determine, based on the first image data and the posture data, local point clouds corresponding to different observation points of the depth camera; and perform fusion calculation on the local point clouds corresponding to different observation points to determine a fusion value of at least one voxel in the three-dimensional space;

[0164] The modeling unit 902 is specifically configured to construct a three-dimensional mesh model based on the fusion value of at least one voxel.

[0165] In some embodiments, see Fig. 9 , the service device 90 may further include a voxelization unit 905 configured to establish a geometric body; and voxelize the geometric body to obtain a three-dimensional space including at least one voxel;

[0166] Accordingly, the fusion unit 904 is further configured to use a truncated signed distance function TSDF algorithm to perform fusion calculation on the local point clouds corresponding to different observation points to obtain a TSDF value of at least one voxel;

[0167] The modeling unit 902 is further configured to construct a three-dimensional mesh model according to the TSDF value of at least one voxel.

[0168] In some embodiments, the fusion unit 904 is specifically configured to determine the first TSDF value and the first weight value obtained at the current observation point based on the first voxel, and obtain the fused TSDF value and the fused weight value obtained after the fusion of the previous observation point; and perform weighted average calculation using the TSDF algorithm based on the fused TSDF value, the fused weight value, the first TSDF value and the first weight value to obtain the second TSDF value and the second weight value corresponding to the first voxel; and update the fused TSDF value and the fused weight value according to the second TSDF value and the second weight value to achieve the fusion of the local point cloud corresponding to the current observation point into the TSDF model; wherein the first voxel is any one of at least one voxel in the three-dimensional space.

[0169] In some embodiments, the modeling unit 902 is specifically configured to determine that the voxel is located between the depth camera and the object surface if the TSDF value is positive; and determine that the voxel is located outside the depth camera and the object surface if the TSDF value is negative.

[0170] In some embodiments, the first image data includes depth image data and RGB image data; wherein the depth image data, the RGB image data, and the posture data are time synchronized.

[0171] In some embodiments, the segmentation unit 903 is further configured to input the three-dimensional mesh model into the neural network structure; and in the neural network structure, perform semantic and instance segmentation using point-by-point feature learning to obtain a target semantic map.

[0172] In some embodiments, the segmentation unit 903 is specifically configured to perform a three-dimensional sparse convolution operation on the three-dimensional mesh model to determine semantic information as well as feature embedding information, spatial embedding information and occupancy information; and use an image segmentation algorithm to perform supervoxel grouping on the three-dimensional mesh model to obtain supervoxel information; and perform covariance estimation based on the feature embedding information and the spatial embedding information to obtain target embedding information; and perform clustering operations based on the target embedding information, occupancy information and supervoxel information to determine instance information; and obtain a target semantic map based on the semantic information and instance information.

[0173] In some embodiments, see Fig. 9 The service device 90 may further include a first sending unit 906, configured to obtain target coordinate information of an object of interest to the user from the target semantic map; and send the target coordinate information to the terminal device so that the terminal device generates and displays a rendered image containing a virtual object.

[0174] It is understandable that in the embodiments of the present application, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course, it may be a module, or it may be non-modular. Moreover, the components in the present embodiment may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional module.

[0175] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0176] Therefore, an embodiment of the present application provides a computer storage medium, which is applied to a service device 90, and the computer storage medium stores a computer program, and when the computer program is executed by a first processor, the method described in any one of the above embodiments is implemented.

[0177] Based on the composition of the service device 90 and the computer storage medium, see Fig.10 , which shows a schematic diagram of the specific hardware structure of the service device 90 provided in the embodiment of the present application. Fig.10 As shown, it may include: a first communication interface 1001, a first memory 1002 and a first processor 1003; each component is coupled together through a first bus system 1004. It is understandable that the first bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the first bus system 1004 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 In the figure, various buses are marked as the first bus system 1004. Among them,

[0178] The first communication interface 1001 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0179] A first memory 1002, used to store a computer program that can be run on the first processor 1003;

[0180] The first processor 1003 is configured to, when running the computer program, execute:

[0181] Receiving first image data and posture data sent by a terminal device;

[0182] generating a three-dimensional mesh model according to the first image data and the posture data;

[0183] The three-dimensional mesh model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment.

[0184] It can be understood that the first memory 1002 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The first memory 1002 of the system and method described in the present application is intended to include but is not limited to these and any other suitable types of memory.

[0185] The first processor 1003 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the first processor 1003. The above-mentioned first processor 1003 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 1002, and the first processor 1003 reads the information in the first memory 1002 and completes the steps of the above method in combination with its hardware.

[0186] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units or combinations thereof for performing functions described in the present application. For software implementation, the technology described in the present application can be implemented by a module (such as a process, function, etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in a processor or outside a processor.

[0187] Optionally, as another embodiment, the first processor 1003 is further configured to execute the method described in any one of the aforementioned embodiments when running the computer program.

[0188] This embodiment provides a service device, which may include a first receiving unit, a modeling unit, and a segmentation unit. In this way, the posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be achieved, the accuracy of the entire semantic map is improved, and the virtual object can be better displayed on a specific physical object; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced.

[0189] Based on the same inventive concept as the above embodiments, see Fig.11 , which shows a schematic diagram of the structure of a terminal device 110 provided in an embodiment of the present application. Fig.11 As shown, the terminal device 110 may include: an acquisition unit 1101, a data processing unit 1102 and a second sending unit 1103; wherein,

[0190] The acquisition unit 1101 is configured to acquire the collected data of the wearable device; wherein the collected data includes the first image data and the second image data;

[0191] The data processing unit 1102 is configured to process the second image data to generate posture data;

[0192] The second sending unit 1103 is configured to send the first image data and the posture data to the service device, so that the service device constructs a target semantic map.

[0193] In some embodiments, the first image data includes depth image data and RGB image data, and the second image data includes fisheye image data and inertial sensor data; accordingly, the acquisition unit 1101 is specifically configured to acquire depth image data and corresponding timestamp information through a first thread; and acquire RGB image data and corresponding timestamp information through a second thread; and acquire fisheye image data and corresponding timestamp information through a third thread; and acquire inertial sensor data and corresponding timestamp information through a fourth thread; wherein the timestamp information is used to measure whether the depth image data, RGB image data, fisheye image data and inertial sensor data are time synchronized.

[0194] In some embodiments, the data processing unit 1102 is specifically configured to perform posture calculation on the fisheye image data and the inertial sensor data using a real-time positioning and mapping SLAM system to generate posture data.

[0195] In some embodiments, see Fig.11The terminal device 110 may also include a synchronization unit 1104, configured to determine the first timestamp information corresponding to the posture data; and select the depth image data synchronized with the first timestamp information from the depth image data and the corresponding timestamp information; and select the RGB image data synchronized with the first timestamp information from the RGB image data and the corresponding timestamp information; and determine the first image data based on the selected depth image data and the selected RGB image data; wherein the first image data is synchronized with the time of the posture data.

[0196] In some embodiments, the second sending unit 1103 is specifically configured to establish a TCP connection with the service device; and send the first image data and the posture data to the service device based on the TCP connection.

[0197] In some embodiments, see Fig.11 , the terminal device 110 may further include a second receiving unit 1105, configured to receive target coordinate information sent by the service device based on a TCP connection;

[0198] The data processing unit 1102 is further configured to generate a rendered image according to the target coordinate information and the virtual object; and send the rendered image to the wearable device so that the virtual object is displayed at a position corresponding to the target coordinate information through the wearable device.

[0199] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0200] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer storage medium, which is applied to the terminal device 110, and the computer storage medium stores a computer program. When the computer program is executed by the second processor, the method described in any one of the above embodiments is implemented.

[0201] Based on the above-mentioned components of the terminal device 110 and the computer storage medium, see Fig.12 , which shows a schematic diagram of the specific hardware structure of the terminal device 110 provided in the embodiment of the present application. Fig.12As shown, it may include: a second communication interface 1201, a second memory 1202, and a second processor 1203; each component is coupled together via a second bus system 1204. It is understood that the second bus system 1204 is used to realize the connection and communication between these components. In addition to the data bus, the second bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.12 In the figure, various buses are labeled as the second bus system 1204. Among them,

[0202] The second communication interface 1201 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0203] The second memory 1202 is used to store a computer program that can be run on the second processor 1203;

[0204] The second processor 1203 is configured to, when running the computer program, execute:

[0205] Acquire collected data of the wearable device; wherein the collected data includes first image data and second image data;

[0206] Processing the second image data to generate posture data;

[0207] The first image data and the posture data are sent to a service device, so that the service device constructs a target semantic map.

[0208] Optionally, as another embodiment, the second processor 1203 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0209] It can be understood that the hardware functions of the second memory 1202 and the first memory 1002 are similar, and the hardware functions of the second processor 1203 and the first processor 1003 are similar; they will not be described in detail here.

[0210] This embodiment provides a terminal device, which may include a data processing unit and a second sending unit. In this way, the posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be achieved, the accuracy of the entire semantic map is improved, and the virtual object can be better displayed on a specific physical object; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced.

[0211] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0212] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0213] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0214] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0215] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0217] Industrial Applicability

[0218] In an embodiment of the present application, on the terminal device side, the collected data of the wearable device is obtained; wherein the collected data includes first image data and second image data; the second image data is processed to generate posture data; and the first image data and posture data are sent to the service device so that the service device constructs a target semantic map. On the service device side, the first image data and posture data sent by the terminal device are received; a three-dimensional mesh model is generated based on the first image data and posture data; the three-dimensional mesh model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment. In this way, the posture data is also combined in the construction of the target semantic map, so that a more complete scene reconstruction can be achieved, the accuracy of the entire semantic map is improved, and the display of virtual objects on specific physical objects can be better realized; in addition, since the scene reconstruction and semantic segmentation are performed in the service device, the memory usage and power consumption of the terminal device can also be reduced.

Claims

1. A semantic map construction method, applied to a service device, comprising: Receiving first image data and posture data sent by a terminal device; generating a three-dimensional mesh model according to the first image data and the posture data; The three-dimensional grid model is semantically segmented to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment.

2. The method according to claim 1, wherein: The step of generating a three-dimensional mesh model according to the first image data and the posture data comprises: Determine, based on the first image data and the posture data, local point clouds corresponding to different observation points of the depth camera; Performing fusion calculation on the local point clouds corresponding to the different observation points to determine a fusion value of at least one voxel in the three-dimensional space; The three-dimensional mesh model is constructed based on the fused value of the at least one voxel.

3. The method according to claim 2, wherein: The method further comprises: Create geometry; voxelize the geometric body to obtain the three-dimensional space including at least one voxel; Accordingly, the performing fusion calculation on the local point clouds corresponding to the different observation points to determine the fusion value of at least one voxel in the three-dimensional space includes: Based on the three-dimensional space, using a truncated signed distance function (TSDF) algorithm to perform fusion calculation on the local point clouds corresponding to the different observation points, to obtain a TSDF value of the at least one voxel; The constructing the three-dimensional mesh model based on the fusion value of the at least one voxel includes: constructing the three-dimensional mesh model according to the TSDF value of the at least one voxel.

4. The method according to claim 3, wherein: The using the TSDF algorithm to perform fusion calculation on the local point clouds corresponding to the different observation points to obtain the TSDF value of the at least one voxel includes: Based on the first voxel, determine the first TSDF value and the first weight value obtained at the current observation point, and obtain the fused TSDF value and the fused weight value obtained after fusion of the previous observation point; Performing a weighted average calculation using a TSDF algorithm according to the fused TSDF value, the fused weight value, the first TSDF value, and the first weight value to obtain a second TSDF value and a second weight value corresponding to the first voxel; The fused TSDF value and the fused weight value are updated according to the second TSDF value and the second weight value to achieve the fusion of the local point cloud corresponding to the current observation point into the TSDF model; wherein the first voxel is any one of at least one voxel in the three-dimensional space.

5. The method according to claim 3, wherein: The step of constructing the three-dimensional mesh model according to the TSDF value of the at least one voxel comprises: If the TSDF value is positive, it is determined that the voxel is located between the depth camera and the surface of the object; If the TSDF value is negative, it is determined that the voxel is located outside between the depth camera and the object surface.

6. The method according to claim 1, wherein: The first image data includes depth image data and RGB image data; wherein the depth image data, the RGB image data and the posture data are time synchronized.

7. The method according to claim 1, wherein: The performing semantic segmentation on the three-dimensional grid model to obtain a target semantic map includes: Inputting the three-dimensional mesh model into a neural network structure; In the neural network structure, semantic and instance segmentation is performed using point-by-point feature learning to obtain the target semantic map.

8. The method according to claim 7, wherein: The method of performing semantic and instance segmentation by point-by-point feature learning to obtain the target semantic map includes: Performing a three-dimensional sparse convolution operation on the three-dimensional grid model to determine semantic information as well as feature embedding information, spatial embedding information and occupancy information; Using an image segmentation algorithm to perform supervoxel grouping on the three-dimensional grid model to obtain supervoxel information; Performing covariance estimation based on the feature embedding information and the spatial embedding information to obtain target embedding information; Performing a clustering operation according to the target embedding information, the occupancy information and the supervoxel information to determine instance information; The target semantic map is obtained according to the semantic information and the instance information.

9. The method according to any one of claims 1 to 8, wherein: After obtaining the target semantic map, the method further includes: Obtaining target coordinate information of an object of interest to the user from the target semantic map; The target coordinate information is sent to the terminal device, so that the terminal device generates and displays a rendered image including the virtual object.

10. A semantic map construction method, applied to a terminal device, the method comprising: Acquire collected data of the wearable device; wherein the collected data includes first image data and second image data; Processing the second image data to generate posture data; The first image data and the posture data are sent to a service device, so that the service device constructs a target semantic map.

11. The method according to claim 10, wherein: The first image data includes depth image data and RGB image data, and the second image data includes fisheye image data and inertial sensor data; Accordingly, the acquisition of collected data from the wearable device includes: Acquire the depth image data and corresponding timestamp information through a first thread; Acquire the RGB image data and corresponding timestamp information through a second thread; Acquire the fisheye image data and corresponding timestamp information through a third thread; Acquiring the inertial sensor data and corresponding timestamp information through a fourth thread; The timestamp information is used to measure whether the depth image data, the RGB image data, the fisheye image data and the inertial sensor data are time synchronized.

12. The method according to claim 11, wherein: The processing of the second image data to generate posture data includes: The real-time positioning and mapping SLAM system is used to perform posture calculation on the fisheye image data and the inertial sensor data to generate the posture data.

13. The method according to claim 12, wherein: Before sending the first image data and the posture data to a service device, the method further includes: Determine first timestamp information corresponding to the posture data; Selecting, from the depth image data and the corresponding timestamp information, depth image data synchronized with the first timestamp information; Selecting RGB image data synchronized with the first timestamp information from the RGB image data and the corresponding timestamp information; The first image data is determined according to the selected depth image data and the selected RGB image data; wherein the first image data is synchronized with the time of the posture data.

14. The method according to any one of claims 10 to 13, wherein: The method further comprises: Receiving target coordinate information sent by the service device; Generate a rendering image according to the target coordinate information and the virtual object; The rendered image is sent to the wearable device, so that the virtual object is displayed at a position corresponding to the target coordinate information through the wearable device.

15. A service device, comprising a first receiving unit, a modeling unit and a segmentation unit; wherein: The first receiving unit is configured to receive first image data and posture data sent by the terminal device; The modeling unit is configured to generate a three-dimensional mesh model according to the first image data and the posture data; The segmentation unit is configured to perform semantic segmentation on the three-dimensional grid model to obtain a target semantic map; wherein the target semantic map is used to realize the display of virtual objects in a physical environment.

16. A service device, comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 9 when running the computer program.

17. A terminal device, comprising an acquisition unit, a data processing unit and a second sending unit; wherein: The acquisition unit is configured to acquire acquisition data of the wearable device; wherein the acquisition data includes first image data and second image data; The data processing unit is configured to process the second image data to generate posture data; The second sending unit is configured to send the first image data and the posture data to a service device, so that the service device constructs a target semantic map.

18. A terminal device, comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 10 to 14 when running the computer program.

19. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, which implements the method according to any one of claims 1 to 9 when executed by a first processor, or implements the method according to any one of claims 10 to 14 when executed by a second processor.