Color three-dimensional map display method and device

Abstract

The present disclosure provides a color 3D map display method and apparatus, which includes the steps of acquiring initial color 3D map information of a space in which a robot exists, the space in which the robot exists representing a space corresponding to an area detectable by a robot sensor, performing color 3D map interpolation based on the initial color 3D map information to estimate and interpolate structural information and color information of areas not detected by the robot sensor, and displaying the interpolated color 3D map obtained by the interpolation process.

Description

Related Applications

[0001] This application claims priority to Chinese Patent Application No. 202211325855.2, filed on October 27, 2022, the entire disclosure of which is incorporated herein by reference as part of the present disclosure. [Technical Field]

[0002] The present disclosure relates to the technical field of robots, and in particular to a color three-dimensional map display method and device. [Background technology]

[0003] With the development of computer technology and artificial intelligence technology, various robots equipped with intelligent systems have appeared, such as sweeping robots, mopping robots, vacuum cleaner robots, and weeding robots, which can move automatically within an area and perform tasks such as sweeping and cleaning without user operation. The robots use sensors to obtain information corresponding to the work area, such as RGB images and point clouds, draw a color three-dimensional map of the area where the robot is located, and provide feedback of the drawn map to the user, allowing the user to grasp the map information of the area where the robot is located.

[0004] Currently, due to factors such as limitations on the robot's viewpoint or access area, the color 3D map constructed by the robot and fed back to the user is partially missing. In such a partially missing map, it is difficult for the user to easily identify obstacles or specific objects in the robot's location area. As a result, the user cannot easily and intuitively grasp the robot's working environment, which reduces the display effect of the color 3D map and degrades the user's experience using the robot. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure provide a color 3D map display method and device to solve the problem existing in the prior art that a complete 3D map of a robot working environment cannot be displayed, which reduces the display effect of the color 3D map and the user experience.

[0006] In a first aspect of an embodiment of the present disclosure, a color 3D map display method is provided, including the steps of acquiring initial color 3D map information of a space in which a robot exists, where the space in which the robot exists represents a space corresponding to an area detectable by a robot sensor; performing color 3D map completion based on the initial color 3D map information to estimate and complete structural information and color information of areas not detected by the robot sensor; and displaying the completed color 3D map obtained by the completion process.

[0007] In a second aspect of an embodiment of the present disclosure, a color 3D map display device includes an acquisition module configured to acquire initial color 3D map information of a space in which a robot exists, the space in which the robot exists representing a space corresponding to an area detectable by a robot sensor; an acquisition module configured to perform color 3D map completion based on the initial color 3D map information and estimate and complete structural information and color information of areas not detected by the robot sensor; and a display module configured to display the completed color 3D map obtained by the completion process.

[0008] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a processor and a memory, the memory being adapted to store a computer program, the computer program being adapted to perform the method of any of the above embodiments when executed by the processor.

[0009] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored therein, the computer program being operable, when executed by a processor, to implement the method of any of the above embodiments.

[0010] The at least one technical solution adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0011] By obtaining initial color 3D map information of the space in which the robot exists, where the space in which the robot exists represents the space corresponding to the area detectable by the robot sensor, performing color 3D map completion based on the initial color 3D map information, estimating and completing structural information and color information of areas not detected by the robot sensor, and displaying the completed color 3D map obtained by the completion process, the present disclosure can completely display a color 3D map corresponding to the robot's working environment, allowing the user to intuitively feel the robot's working environment, improving the display effect of the color 3D map, and improving the user's experience of using the robot. [Brief explanation of the drawings]

[0012] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following will briefly describe the accompanying drawings that need to be used in the description of the embodiments or prior art. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these accompanying drawings without any creative work. [Figure 1] 1 is a flowchart of a color 3D map display method provided by an embodiment of the present disclosure. [Figure 2] FIG. 1 is a 3D schematic diagram of an initial color three-dimensional map after reconstruction provided by an embodiment of the present disclosure. [Figure 3] FIG. 1 is a 3D schematic diagram of a color three-dimensional map after completion provided by an embodiment of the present disclosure. [Figure 4] FIG. 1 is a 3D schematic diagram of a color three-dimensional map after segmentation and coloring provided by an embodiment of the present disclosure. [Figure 5] 1 is a structural schematic diagram of a color 3D map display device provided by an embodiment of the present disclosure; [Figure 6]1 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure; DETAILED DESCRIPTION OF THE INVENTION

[0013] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system architectures, techniques, etc., to provide a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced in other embodiments without these specific details. In other embodiments, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present disclosure with unnecessary detail.

[0014] As described in the background art, with the development of computer technology and artificial intelligence technology, various robots equipped with intelligent systems have appeared, such as sweeping robots, mopping robots, vacuum cleaner robots, and weeding robots, which can automatically move around an area and perform tasks such as sweeping and cleaning without user operation. Robots are usually equipped with sensors such as RGB cameras, depth cameras, and laser radars, and during the work process, the robot obtains information corresponding to the work area, such as RGB images and point clouds, through the sensors, draws a color three-dimensional map of the area where the robot is located, and feeds the drawn map back to the user, allowing the user to grasp the map information of the area where the robot is located.

[0015] In related art, due to factors such as a limited robot viewpoint or limited access area, the color 3D map constructed by the robot and fed back to the user is partially missing. Such partially missing maps make it difficult for the user to identify obstacles or specific objects in the robot's location, resulting in the user being unable to intuitively grasp the robot's working environment, reducing the display effectiveness of the color 3D map and degrading the user's experience using the robot. At the same time, color 3D maps constructed by existing robots lack annotations for object categories, making it impossible to highlight objects of interest to the user and preventing the user from more directly identifying objects.

[0016] In view of the problems present in the related art, there is an urgent need to provide a color 3D map display method that can complement a color 3D map constructed by a robot with structure and color, and can arbitrarily annotate category information for objects in the completed color 3D map.

[0017] In view of this, an embodiment of the present disclosure provides a color 3D map display method, which obtains initial color 3D map information of the space where a robot exists, and uses a color 3D map completion algorithm to estimate structural information and color information of areas not detected by the robot sensor according to a predetermined accuracy, thereby obtaining a completed color 3D map, and rendering and displaying the completed color 3D map on a terminal, allowing the user to intuitively feel the robot's working environment, thereby improving the display effect of the color 3D map and the user's experience of using the robot.

[0018] The robots in the embodiments of the present disclosure include, but are not limited to, automatic cleaning robots such as sweeping robots, mopping robots, and sweeping-mopping robots. The work scenes of the robots in the embodiments of the present disclosure are not limited to automatic cleaning scenes, and work scenes corresponding to other types of robots are also applicable to the present disclosure. The space in which the robots in the embodiments of the present disclosure exist may be an indoor space, an outdoor space, or any other space (rooftop space). The types, work scenes, and location spaces of the robots described above do not constitute limitations on the technical solutions of the present disclosure.

[0019] Hereinafter, embodiments of the color 3D map display method provided by the present disclosure will be described in detail in conjunction with the accompanying drawings and specific examples.

[0020] 1 is a flowchart of a color 3D map display method provided by an embodiment of the present disclosure. The color 3D map display method of FIG. 1 can be executed by a computer program installed on a robot or a terminal. As shown in FIG. 1, the color 3D map display method specifically includes the following steps: S101, obtaining initial color three-dimensional map information of a space where a robot exists, the space where the robot exists representing a space corresponding to an area detectable by a robot sensor; S102, performing color 3D map completion based on the initial color 3D map information to estimate and complete the structure information and color information of the area not detected by the robot sensor; S103: The interpolated color three-dimensional map obtained by the interpolation process is displayed.

[0021] Specifically, the robot of the embodiment of the present disclosure may be a sweeping robot, a mopping robot, or a sweeping and mopping cleaning robot, and a possible application scenario of the embodiment of the present disclosure is that when the robot sweeps or mops within a working area, it uses sensors such as an RGB camera, a depth camera (optional), and a laser radar (optional) mounted on the robot to scan information within the working area, reconstructs a color 3D map of the space where the robot exists based on the acquired information, complements the reconstructed color 3D map, and displays the complemented color 3D map. This application scenario is merely an optional scenario of the present disclosure and does not constitute a limitation of the technical solution of the present disclosure.

[0022] Furthermore, before completing a color 3D map of the robot's location area, initial color 3D map information of the space where the robot exists must first be obtained, and the initial color 3D map information must be used as input to a color 3D map completion algorithm model or a construction completion and integration algorithm model, and the initial color 3D map information must be processed using the pre-trained model to obtain a completed color 3D map. The specific content and acquisition method of the initial color 3D map information will be described in detail below, along with specific examples.

[0023] In some embodiments, the initial color three-dimensional map information includes an initial color three-dimensional map of the space in which the robot resides and / or information necessary to reconstruct the initial color three-dimensional map of the space in which the robot resides.

[0024] Specifically, the initial color three-dimensional map information of the embodiments of the present disclosure includes at least one of an initial color three-dimensional map of the space in which the robot exists, i.e., an initial color three-dimensional map of the space in which the robot exists, and information required to reconstruct the initial color three-dimensional map of the space in which the robot exists, i.e., relevant information for constructing the initial color three-dimensional map.

[0025] Since different initial color 3D map information requires different color 3D map completion algorithms, the color 3D map completion processes for the above two different initial color 3D map information will be described below.

[0026] In some embodiments, obtaining information necessary to reconstruct an initial color three-dimensional map of the space in which the robot resides comprises: Acquire RGB images taken by the robot using an RGB camera and acquire position and orientation information corresponding to the RGB camera. Alternatively, the robot acquires an RGB image captured using an RGB camera, and acquires position and orientation information corresponding to the RGB camera, and a depth image captured using a depth camera and position and orientation information corresponding to the depth camera; Alternatively, the robot may acquire an RGB image captured using an RGB camera, and acquire position and orientation information corresponding to the RGB camera, as well as point cloud information measured using a radar and position and orientation information corresponding to the radar.

[0027] Specifically, the information for reconstructing an initial color 3D map of the space in which the robot exists includes at least position and orientation information corresponding to an RGB image and an RGB camera, and in addition, position and orientation information corresponding to a depth image and a depth camera may be selectively acquired, and position and orientation information corresponding to point cloud information and a radar may also be selectively acquired.

[0028] In some embodiments, performing color 3D map completion based on information necessary to reconstruct an initial color 3D map of the space in which the robot resides comprises: The method includes processing information required to reconstruct an initial color three-dimensional map of the space in which the robot exists using a preset first completion algorithm model, estimating and completing structural information and color information of areas not detected by the robot sensor, rendering the estimated and completed three-dimensional scene, and obtaining the completed color three-dimensional map.

[0029] Specifically, the first completion algorithm model in the embodiments of the present application is a prediction model obtained by training a model based on the construction completion integration algorithm. In one specific embodiment, the construction completion integration algorithm may adopt the NERF-VAE method, using a pre-trained NERF-VAE neural network model to input the RGB image and position and orientation information corresponding to the RGB camera captured by the robot into the pre-trained NERF-VAE neural network model, and then render the scene.

[0030] Here, the NeRF-VAE neural network model is a 3D scene generation model that combines neural radiation fields (NeRF) and geometric structures derived from microscopic rendering. The NeRF-VAE model has the following advantages: its distributed estimation eliminates the need to perform expensive optimization from scratch for each new scene; it can reconstruct unseen scenes from fewer input views by learning shared information across multiple scenes; it generalizes better than existing convolutional generative models for view synthesis (e.g., GQN) when evaluating out-of-distribution camera views; and NeRF-VAE is the only distributed NeRF variant that uses a compact scene representation in the form of latent variables, allowing it to cope with input uncertainty.

[0031] In some embodiments, obtaining an initial color three-dimensional map of the space in which the robot resides comprises: Acquiring an RGB image sequence consisting of RGB images taken by the robot using an RGB camera, and determining position and orientation information corresponding to the RGB camera mounted on the robot; Reconstructing an initial color 3D map of the space in which the robot exists based on the RGB image sequence and the position and orientation information.

[0032] Specifically, in an embodiment of the present disclosure, when acquiring an initial color 3D map of the space in which the robot exists, an RGB image sequence consisting of RGB images taken by an RGB camera while the robot is moving is acquired, and a real-time localization and map reconstruction algorithm (SLAM algorithm) is executed to determine the camera position and orientation corresponding to the RGB camera on the robot, and an initial color 3D map of the space in which the robot exists is reconstructed based on the RGB image sequence and the position and orientation information of the RGB camera.

[0033] Simultaneous Localization and Mapping (SLAM) is a real-time localization and map-building algorithm primarily used to solve localization and map-building problems for robots moving in unknown environments. Using the SLAM algorithm, a 3D map of the world can be constructed in real time, while simultaneously tracking the position and orientation of the robot's camera. In addition to using the SLAM algorithm, maps can also be reconstructed using the traditional structure-from-motion (SFM) algorithm. The SFM algorithm is an offline algorithm for 3D reconstruction based on a series of sequential photographs. The SFM algorithm can recover the 3D structure of an object from its motion (a collection of photographs taken at different times). The main processes of the SFM algorithm include feature point extraction and matching, decomposition of the basis matrix F and eigenmatrix E to obtain R and T, and triangulation to obtain a sparse point cloud.

[0034] Alternatively, in addition to using the above SLAM algorithm and SFM algorithm, a pre-trained neural network (e.g., Atlas) can be used to reconstruct the initial color 3D map. The present disclosure does not improve the flow of the above algorithm and the structure of the neural network itself, so the implementation process of the above algorithm will not be described in detail.

[0035] In some embodiments, obtaining an initial color three-dimensional map of the space in which the robot resides comprises: The method includes acquiring an RGB image sequence consisting of RGB images captured by the robot using an RGB camera, determining position and orientation information corresponding to the RGB camera mounted on the robot, acquiring at least one of a depth image, point cloud information, accelerometer-collected information, and gyroscope-collected information, and reconstructing an initial color three-dimensional map of the space in which the robot exists.

[0036] Specifically, in another embodiment, in addition to acquiring an RGB image sequence consisting of RGB images taken by an RGB camera while the robot is moving and position and orientation information of the RGB camera, at least one of a depth image, point cloud information, accelerometer collected information, and gyroscope collected information is acquired, where the depth image is an image taken using a depth camera mounted on the robot, the point cloud information is information obtained by scanning using a laser radar, and the accelerometer collected information and gyroscope collected information are information collected by an accelerometer and gyroscope (IMU) on the robot.

[0037] Furthermore, if the acquired information also includes depth images captured by a depth camera, the initial color 3D map can be reconstructed for any ambient lighting using the Kinect Fusion algorithm, which is a reconstruction and summation algorithm based on an RGB-D camera (i.e., a stereo color camera, which can reconstruct color stereo data space in real time).The Kinect Fusion algorithm employs a point cloud fusion method based on the TSDF (truncated signed distance function) model to construct a point cloud model with fewer redundant points.

[0038] The Kinect Fusion algorithm matches, localizes, and fuses the depth data collected by the Kinect to reconstruct a 3D scene corresponding to the initial color 3D map. The algorithm process mainly consists of the following parts: (1) depth data processing: converting the original depth data into a 3D point cloud and obtaining the 3D coordinates and normal vectors of the vertices in the point cloud; (2) camera tracking: performing ICP matching of the 3D point cloud of the current frame with the predicted 3D point cloud generated from the existing model to calculate the current frame camera position and orientation; (3) point cloud fusion: fusing the 3D point cloud of the current frame with the existing model using the TSDF point cloud fusion algorithm according to the current camera position and orientation; and (4) scene rendering: using ray tracking to predict the environment point cloud observed by the current camera according to the existing model and the current camera position and orientation.

[0039] Alternatively, in addition to the Kinect Fusion algorithm, a radial neural fields algorithm (NeRF algorithm) may be used, which obtains a 3D scene corresponding to an initial color three-dimensional map at different 2D viewpoints by directly rendering the 3D scene at different 2D viewpoints. Since the present disclosure does not improve upon the processes of the Kinect Fusion algorithm and the NeRF algorithm, the processes for implementing them will not be described herein. It should be understood that any algorithm that enables three-dimensional map reconstruction is applicable to the present disclosure.

[0040] It should be noted that the color 3D map completion operation in the embodiments of the present disclosure may also involve acquiring environmental information (e.g., RGB images, depth images, point cloud information, etc.) within the detectable area while the robot is walking during the robot's work process, reconstructing an initial color 3D map in real time using the environmental information within the detectable area, and performing completion based on the reconstructed initial color 3D map. In this case, the inputs to the map reconstruction algorithm are the RGB image sequence, camera position and orientation information, and any other information (depth images, point cloud information, accelerometer information, gyroscope information, etc.) acquired in real time by the robot during its movement process. That is, in the scene of real-time map completion of the robot, the reconstruction and completion of the initial color 3D map of the robot are also performed based on the information acquired in real time.

[0041] In addition to the above scenarios, the color 3D map completion operation of the embodiments of the present disclosure may also be performed after the robot has completed its task to reconstruct and complete an initial color 3D map based on the RGB image sequence, camera position and orientation information, and optional other information (depth image, point cloud information, accelerometer information, gyroscope information, etc.) acquired during the robot's historical motion process (e.g., the previous task execution process). In this case, the input to the map reconstruction algorithm is information acquired during the historical motion process, rather than information acquired in real time. In this case, the robot can complete all color 3D maps at once, i.e., achieve global completion of the 3D map. The 3D map completion operation in real time can be performed only based on the information collected by the robot according to the area to be completed.

[0042] FIG. 2 is a 3D schematic diagram of an initial color three-dimensional map after reconstruction provided by an embodiment of the present disclosure. As shown in FIG. 2, the initial color three-dimensional map after reconstruction includes the following contents: In some embodiments, the completed color 3D map includes structural and color information corresponding to the initial color 3D map, as well as structural and color information in areas not detected by the robotic sensors.

[0043] Specifically, after reconstructing an initial color 3D map of the space in which the robot exists, the initial color 3D map includes only some or all of the structural information within the area that the robot sensor can detect and some or all of the color information corresponding to this structural information, but does not include all of the structural information within the area that the robot sensor cannot detect and all of the color information corresponding to this structural information.

[0044] In some embodiments, the step of performing color 3D map completion based on the initial color 3D map of the space in which the robot resides comprises: The method includes processing the reconstructed initial color 3D map using a preset second interpolation algorithm model to interpolate and estimate structural information and color information of areas not detected by the robot sensor, thereby obtaining an interpolated color 3D map.

[0045] Specifically, in an embodiment of the present disclosure, after reconstructing an initial color 3D map, the reconstructed initial color 3D map is input into a preset color 3D map completion algorithm model (i.e., a second completion algorithm model), and the color 3D map completion algorithm model is used to complete the 3D map. In practice, the color 3D map completion algorithm can be considered as a set of instructions that takes the initial color 3D map as input information (the input may not be unique), processes the input information, and outputs a completed color 3D map within a limited time. The completion operation principle of the color 3D map completion algorithm and the completed color 3D map will be described in detail below, in conjunction with the accompanying drawings and specific embodiments. FIG. 3 is a 3D schematic diagram of the completed color 3D map provided by an embodiment of the present disclosure. As shown in FIG. 3, the completion of the color 3D map includes the following: In one specific embodiment, the second completion algorithm model in the embodiment of the present disclosure is a prediction model obtained by training a model based on a color 3D map completion algorithm. In one specific embodiment, the color 3D map completion algorithm model may employ an SG-NN neural network for structural completion and then employ a Texture Fields algorithm for coloring, or may directly employ the SPSG algorithm for completion. The basic principles of the SPSG algorithm will be explained below using the SPSG algorithm to complete a color 3D map as an example.

[0046] The basic working principle of the SPSG algorithm for color 3D map completion is to use an instruction set to estimate geometric and color information for areas not detected by the robot sensor with a certain degree of accuracy, resulting in a completed color 3D map. Here, the SPSG algorithm can learn to estimate geometric and color information for unobserved scenes in a self-supervised manner, thereby generating a high-quality color 3D scene model based on the observation of RGB-D scans. This self-supervised method restores both geometry and color by correlating incomplete RGB-D scans with more complete versions. The SPSG algorithm does not rely on 3D reconstruction to inform 3D geometry and color reconstruction. Instead, it proposes manipulating adversarial and perceptual losses in 2D rendering to achieve high-resolution, high-quality color reconstruction of a scene. This method utilizes high-resolution, self-consistent signals from a single original RGB-D frame, allowing the 2D signals to be directly used to inform the generation of the 3D scene, thereby achieving high-quality color reconstruction of the 3D scene.

[0047] In some other embodiments, the SPSG algorithm provided by the above embodiments is used to complement the reconstructed initial color 3D map to obtain a complemented color 3D map. The embodiments of the present disclosure also provide another implementation for complementing the initial color 3D map, which uses a Scan2CAD algorithm to identify objects, a Bundle Fusion algorithm to calculate the camera position and orientation corresponding to the RGBD picture, and a Texture Fields algorithm to predict the RGBD picture, thereby obtaining color information of all objects in the RGBD picture, thereby completing the initial color 3D map. The complete process of this implementation for complementing the initial color 3D map will be described in detail below in conjunction with specific embodiments, and may specifically include the following: (1) First, obtain RGBD information for reconstructing an initial color 3D map, and reconstruct an RGBD picture corresponding to the initial color 3D map based on the RGBD information; (2) Using the algorithm Scan2CAD or Scene2CAD, identify the object category, position, orientation, and size information corresponding to the objects present in the RGBD picture, and find the corresponding model for each object from a pre-defined 3D CAD model library; (3) Using the CAD model of each object found, arrange each object on the empty 3D map according to its position and orientation information (including its position and orientation), forming a colorless 3D map; (4) Using a bundle fusion algorithm to calculate the camera position and orientation corresponding to the RGBD picture, and using a hierarchical bracketed box (BVHTree) and an occlusion culling algorithm (e.g., an occlusion culling algorithm) according to the camera position and orientation corresponding to the RGBD picture and the identified object to obtain an RGBD picture of at least one object captured from the RGBD picture in (1) above; (5) Using the Texture Fields algorithm to predict each identified object and its corresponding picture, obtain color information for each object, and then use the color information to color the object; In this way, a color three-dimensional map after interpolation is obtained.

[0048] The principle of the initial color 3D map completion method corresponding to (1) to (5) above is that since there are no gaps in the CAD model in the color 3D map, for a color 3D map with gaps, the CAD model in the color 3D map is structurally completed, and at the same time, the Texture Fields algorithm is used to complete the color of the CAD model, thereby obtaining a completed color 3D map.

[0049] In reality, the completed color 3D map includes a prediction of the color 3D map of the area that the robot cannot detect, i.e., in addition to the structural information of part or all of the initial color 3D map and the color information corresponding to the structural information, the completed color 3D map also includes the structural information corresponding to the 3D map in the area that the robot's sensor cannot detect, which is estimated with a certain accuracy by the color 3D map completion algorithm, and the color information corresponding to the structural information.

[0050] In some embodiments, the method further includes dividing the elements in the completed color three-dimensional map into regions, where different regions after division include one object, one type of object, or multiple types of objects, marking categories for the objects in the different regions, obtaining object category information in the completed color three-dimensional map, coloring the elements in the different regions, and obtaining colors corresponding to the objects in the different regions in the completed color three-dimensional map.

[0051] Specifically, after obtaining the interpolated color 3D map using the algorithm and processing process of the above embodiment, the elements (e.g., points, surfaces, voxels, slice elements, etc.) on the interpolated color 3D map are divided into regions using a segmentation algorithm, and the elements in different regions after segmentation are colored. The adopted region segmentation algorithm and the content of the segmented and colored color 3D map will be described below in conjunction with the accompanying drawings and specific embodiments. Figure 4 is a 3D schematic diagram of the segmented and colored color 3D map provided by the embodiment of the present disclosure. As shown in Figure 4, the region segmentation algorithm and the segmented and colored color 3D map include the following: The interpolated color 3D map is divided into regions using a region segmentation algorithm, and different regions in the interpolated color 3D map contain, with a certain accuracy, only one object, one type of object, multiple pre-specified types of objects, or other types of objects other than the multiple pre-specified types of objects. In practice, the different regions after segmentation may be connected or disconnected.

[0052] In a specific embodiment, the region segmentation algorithm of the present disclosure may be Mix3D, BPNet, or the like. Preferably, the present disclosure employs the Mix3D algorithm to segment elements in the interpolated color 3D map into regions. Mix3D is a data augmentation technology for segmenting large-scale 3D scenes, balancing global texture and local geometry. By applying Mix3D to nearest point- or voxel-based 3D models, segmentation performance can be continuously improved in large-scale indoor and outdoor 3D datasets. After using Mix3D to segment the interpolated color 3D map into regions, categories are automatically marked for objects in the interpolated color 3D map, and category information corresponding to each object is output.

[0053] Furthermore, by coloring the elements in different regions after segmentation, the objects in the different regions take on corresponding colors. For example, in the color 3D map after segmentation and coloring shown in Figure 4, the "sofa" in the living room is pink, the "dining table" is blue, and the "bed" in the bedroom is green. By labeling objects in different regions with categories and presenting them in different colors, the user can intuitively sense the object information in the robot's workspace and conveniently identify objects more directly.

[0054] In some embodiments, the method further includes placing a three-dimensional model of the robot on the completed color three-dimensional map based on the position and posture information of the robot, and displaying the three-dimensional model of the robot in proportion to the completed color three-dimensional map.

[0055] Specifically, in addition to segmenting and coloring the completed color 3D map, a 3D model of the robot is placed on the completed color 3D map based on the position and posture information of the robot, maintaining the corresponding position and posture, and displayed in proportion to the completed color 3D map. For example, in one specific embodiment, taking a mobile phone as the display device, a 3D scene object is first created using an application program installed on the mobile phone, and then the vertex and surface patch information of the pre-designed robot 3D model is added to the 3D scene object, and the position and posture of the robot 3D model are set, thereby realizing the insertion of the robot 3D model into the scene.

[0056] In some embodiments, the embodiments of the present disclosure further include modifying and displaying the interpolated color 3D map by adopting one or a combination of the following modification methods, wherein the modification methods include: manually setting the color of the completed color 3D map or automatically setting the color of the completed color 3D map based on a preset color scheme; generating a decal using a preset pattern or a custom pattern, and displaying the decal on the completed color 3D map; adding a passable area, an impassable area, or a virtual wall for the robot to the completed color three-dimensional map, and displaying the passable area, the impassable area, or the virtual wall on the completed color three-dimensional map; setting a color scheme for the completed color 3D map that can be automatically changed according to time according to a preset color scheme corresponding to different times, and automatically changing the color of all or part of the completed color 3D map according to the color scheme; A camera trajectory is set on the interpolated color 3D map, or a motion trajectory of the robot is adopted, and the interpolated color 3D map is rendered based on the camera trajectory or the position and posture of the camera on the motion trajectory and an internal reference, and two-dimensional images are obtained that are rendered according to the trajectory, and the two-dimensional images are sequentially reproduced.

[0057] Specifically, in the embodiments of the present disclosure, after obtaining the interpolated color 3D map, the interpolated color 3D map can also be modified and displayed. The above are some modification methods provided by the embodiments of the present disclosure. In practice, one of the above modification methods can be adopted to modify the interpolated color 3D map, or a combination of the above modification methods can be adopted to modify the interpolated color 3D map. The specific contents of the above modification methods will be described in detail below, and it should be noted that the specific modification methods do not constitute limitations on the technical solutions of the present disclosure.

[0058] First correction method: The user can set the color of the completed color 3D map. The user can set the same color for the entire map, or different colors for different areas. Furthermore, the user can select a pre-set color scheme stored inside the robot and automatically set the desired color for the completed color 3D map based on the color scheme.

[0059] Second modification method: The user can select a picture previously stored inside the robot or a picture added by the user to form a decal, and paste it onto the completed color 3D map according to the robot's presets or the user's preferences. In other words, the user can select a pre-set pattern or a custom pattern and project the pattern as a decal onto the completed color 3D map.

[0060] Third modification method: The user adds area controls such as robot passable areas, impassable areas, or virtual walls to the completed color 3D map, and then the areas or virtual walls added by the user are displayed on the completed color 3D map, where the virtual walls are virtual walls that prevent the robot from entering a certain area.

[0061] Fourth Modification Method: The user can set certain conditions on the completed color 3D map to automatically change the color scheme. For example, the user can set color schemes corresponding to days, weeks, months, quarters, or seasons, or display a color scheme corresponding to a local festival in the location where the robot is used.

[0062] Fifth modification method: The user can set the category areas in the completed color 3D map to change color according to an automatically changing color scheme, for example, plants in the map will display colors according to the season, such as green for summer and yellow for autumn.

[0063] Sixth Modification Method: The user sets a camera trajectory within the interpolated color 3D map or adopts the robot's existing motion trajectory, then obtains the camera position and orientation on the trajectory and the camera internal reference according to the motion trajectory, renders the interpolated color 3D map, obtains 2D pictures rendered according to the trajectory, and sequentially displays and plays the 2D pictures on a display device. In practice, the rendering method may be implemented by rasterization, ray tracing, Neural Radiation Field (NeRF), etc.

[0064] In some embodiments, the method further comprises displaying the completed color three-dimensional map in two dimensions according to pre-defined cutouts or user custom cutouts.

[0065] Specifically, in the embodiments of the present disclosure, the complemented color 3D map may also be displayed in two dimensions, and the cutout of the two-dimensional display may be displayed according to the cutout preset by the robot or according to the cutout set by the user. For example, the vertical axis of the complemented color 3D map may be directly set to z=0.1 m, and the intersection plane with the 3D map at z=0.1 m may be the cutout of the two-dimensional display.

[0066] According to the technical solution provided by the embodiments of the present disclosure, the embodiments of the present disclosure obtain initial color 3D map information of the space where the robot exists, and use a construction and completion integration algorithm model or a color 3D map completion algorithm model to estimate structural information and color information of areas not detected by the robot sensor with a predetermined accuracy to obtain a completed color 3D map. The completed color 3D map is rendered and displayed on the terminal, allowing the user to intuitively feel the robot's working environment, improving the display effect of the color 3D map, and improving the user's experience of using the robot.

[0067] The following are apparatus embodiments of the present disclosure that can be used to perform the method embodiments of the present disclosure: For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0068] 5 is a structural schematic diagram of a color 3D map display device provided by an embodiment of the present disclosure. As shown in FIG. 5, the color 3D map display device includes: The acquisition module 501 is configured to acquire initial color three-dimensional map information of a space where the robot exists, where the space where the robot exists represents a space corresponding to an area detectable by the robot sensor; a completion module 502 configured to perform color 3D map completion based on the initial color 3D map information, and to estimate and complete structural information and color information of areas not detected by the robot sensor; The display module 503 is configured to display the interpolated color three-dimensional map obtained by the interpolation process.

[0069] In some embodiments, the acquisition module 501 in FIG. 5 acquires an RGB image captured by a robot using an RGB camera and acquires position and orientation information corresponding to the RGB camera; alternatively, the acquisition module 501 acquires an RGB image captured by a robot using an RGB camera and acquires position and orientation information corresponding to the RGB camera, and a depth image captured by a depth camera and position and orientation information corresponding to the depth camera; alternatively, the acquisition module 501 acquires an RGB image captured by a robot using an RGB camera and acquires position and orientation information corresponding to the RGB camera, and point cloud information measured by a radar and position and orientation information corresponding to the radar.

[0070] In some embodiments, the completion module 502 of FIG. 5 processes information required to reconstruct an initial color 3D map of the space in which the robot exists using a preset first completion algorithm model, estimates and completes structural information and color information of areas not detected by the robot sensor, renders the estimated and completed 3D scene, and obtains the completed color 3D map.

[0071] In some embodiments, the acquisition module 501 in FIG. 5 acquires an RGB image sequence consisting of RGB images captured by a robot using an RGB camera, determines position and orientation information corresponding to the RGB camera mounted on the robot, and reconstructs an initial color 3D map of the space in which the robot exists based on the RGB image sequence and the position and orientation information.

[0072] In some embodiments, the acquisition module 501 in FIG. 5 acquires an RGB image sequence consisting of RGB images captured by a robot using an RGB camera, determines position and orientation information corresponding to the RGB camera mounted on the robot, acquires at least one of a depth image, point cloud information, accelerometer-collected information, and gyroscope-collected information, and reconstructs an initial color three-dimensional map of the space in which the robot exists.

[0073] In some embodiments, the completion module 502 of FIG. 5 processes the reconstructed initial color 3D map using a preset second completion algorithm model to complete and estimate structural information and color information of areas not detected by the robot sensor, thereby obtaining a completed color 3D map.

[0074] In some embodiments, the display module 503 of FIG. 5 divides the elements in the completed color 3D map into regions, where different regions after division include one object, one type of object, or multiple types of objects, marks the objects in the different regions with categories, obtains object category information in the completed color 3D map, colors the elements in the different regions, and obtains colors corresponding to the objects in the different regions in the completed color 3D map.

[0075] In some embodiments, the display module 503 of FIG. 5 places the 3D model of the robot on the completed color 3D map based on the position and posture information of the robot, and displays the 3D model of the robot in proportion to the completed color 3D map.

[0076] In some embodiments, the display module 503 of FIG. 5 may use one or more combinations of the following correction methods to correct and display the interpolated color 3D map, including: Manually set the color of the completed color 3D map or automatically set the color of the completed color 3D map based on a pre-defined color scheme; Generate decals using pre-defined or custom patterns, and display the decals on a complete color 3D map. Adding a passable area, an impassable area, or a virtual wall to the completed color three-dimensional map, and displaying the passable area, the impassable area, or the virtual wall on the completed color three-dimensional map; According to a preset color scheme corresponding to different times, a color scheme that can be automatically changed according to time is set for the completed color 3D map, and the color of the entire area or a partial area in the completed color 3D map automatically changes according to the color scheme; A camera trajectory is set on the interpolated color 3D map or a motion trajectory of the robot is adopted, and the interpolated color 3D map is rendered based on the position and posture of the camera on the camera trajectory or the motion trajectory and the internal reference, and two-dimensional images are rendered according to the trajectory, and the two-dimensional images are sequentially reproduced.

[0077] In some embodiments, the display module 503 of FIG. 5 displays the completed color 3D map in two dimensions according to pre-defined cutouts or user-customized cutouts.

[0078] It should be understood that the magnitude of the serial numbers of each step in the above embodiments does not indicate the order of execution, and the order of execution of each process should be determined by its function and inherent logic, without constituting a limitation on the implementation process of the embodiments of the present disclosure.

[0079] 6 is a structural schematic diagram of an electronic device 6 provided by an embodiment of the present disclosure. As shown in FIG. 6, the electronic device 6 of this embodiment includes a processor 601 and a memory 602. The memory 602 is used to store a computer program 603 executable on the processor 601. When the processor 601 executes the computer program 603, it realizes the steps in each of the above method embodiments. Or, when the processor 601 executes the computer program 603, it realizes the functions of each module / unit in each of the above device embodiments.

[0080] Illustratively, the computer program 603 is divided into one or more modules / units, which are stored in the memory 602 and executed by the processor 601 to implement the present disclosure. The one or more modules / units may be a series of computer program instruction segments that implement a specific function, and the instruction segments are used to describe the execution process of the computer program 603 in the electronic device 6.

[0081] The electronic device 6 may be an electronic device such as a desktop computer, a laptop, a PDA, a cloud server, etc. The electronic device 6 includes, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that FIG. 6 is merely an example of the electronic device 6 and does not constitute a limitation of the electronic device 6, which may include more or fewer components than those shown, or a combination of certain components, or different components; for example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0082] The processor 601 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0083] The memory 602 may be an internal storage unit of the electronic device 6, such as a hard disk or internal memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., installed in the electronic device 6. Furthermore, the memory 602 may include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device. The memory 602 is used to temporarily store data that has been output or is to be output.

[0084] For convenience and brevity, only the division of each functional unit or module is described. It should be understood by those skilled in the art that in actual applications, the above-described functional allocations may be performed by different functional units or modules as needed, i.e., the internal structure of the device may be divided into different functional units or modules to achieve all or part of the above-described functions. The functional units and modules in the embodiments may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units. Furthermore, the specific names of the functional units and modules are used only to facilitate differentiation and do not limit the scope of protection of the present disclosure. For the specific operating processes of the units and modules in the above system, please refer to the corresponding processes in the method embodiments and will not be repeated here.

[0085] In the above embodiments, the description of each embodiment has its own focus, and the parts not detailed or described in one embodiment may be referred to the relevant descriptions of other embodiments.

[0086] Those skilled in the art can recognize that the units and algorithm steps of various examples described in conjunction with the embodiments claimed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered outside the scope of the present disclosure.

[0087] In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus / computer apparatus and method may be implemented in other ways. For example, the above-described device / computer apparatus embodiments are merely schematic, and for example, the division into modules or units is merely a logical division of function. In actual implementation, the division may be performed in another way, multiple units or components may be combined or integrated into another system, and some functions may be ignored or not implemented. In other respects, the illustrated or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections via electrical, mechanical, or some other interfaces, devices, or units.

[0088] The units illustrated as separate components may or may not be physically separated, and the components illustrated as units may or may not be physical units, i.e., they may be located in a single location or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment according to actual needs.

[0089] Furthermore, the functional units in various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or a software functional unit.

[0090] When implemented in the form of a software functional unit and sold or used as a separate product, the integrated module / unit may be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described exemplary methods implemented by the present disclosure may be stored in a computer-readable storage medium and may also be achieved by instructing associated hardware with a computer program that, when executed by a processor, performs the steps of the various exemplary methods described above. The computer program may be composed of computer program code, which may be in the form of source code, object code, an executable file, or any intermediate form. The computer-readable medium may include any entity or device capable of transmitting computer program code, such as a recording medium, a USB flash drive, a removable hard drive, a magnetic disk, a compact disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, an electrical communication signal, and a software distribution medium. It should be noted that the content included in a computer-readable medium may be appropriately increased or decreased in accordance with the requirements of the laws and patent practices of any jurisdiction; for example, in some jurisdictions, the laws and patent practices of computer-readable medium do not include electrical carrier wave signals and telecommunications signals.

[0091] The above examples are only used for the purpose of illustrating the technical solutions of the present disclosure, and are not intended to limit the same. The present disclosure has been described in detail with reference to the above examples. However, it should be understood that those skilled in the art can still make modifications to the technical solutions described in each of the above examples, or equivalently replace some technical features. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each of the embodiments of the present disclosure, and all shall be included in the protection scope of the present disclosure.

Claims

1. obtaining initial color three-dimensional map information of a space in which the robot exists, the space in which the robot exists representing a space corresponding to an area detectable by a robot sensor; performing color 3D map completion based on the initial color 3D map information to estimate and complete structural information and color information of areas not detected by the robot sensor; and displaying the interpolated color three-dimensional map obtained by the interpolation processing.

2. The method of claim 1 , wherein the initial color three-dimensional map information includes an initial color three-dimensional map of a space in which the robot exists and / or information necessary to reconstruct an initial color three-dimensional map of a space in which the robot exists.

3. The step of obtaining information necessary to reconstruct an initial color three-dimensional map of the space in which the robot exists comprises: acquiring an RGB image captured by the robot using an RGB camera, and acquiring position and orientation information corresponding to the RGB camera; Alternatively, the robot acquires an RGB image taken by using an RGB camera, acquires position and orientation information corresponding to the RGB camera, and acquires a depth image taken by using a depth camera and position and orientation information corresponding to the depth camera; Alternatively, the method according to claim 2 includes acquiring RGB images taken by the robot using an RGB camera, acquiring position and orientation information corresponding to the RGB camera, and acquiring point cloud information measured using a radar and position and orientation information corresponding to the radar.

4. performing color 3D map completion based on information necessary to reconstruct an initial color 3D map of the space in which the robot exists, 4. The method of claim 3, comprising: utilizing a preset first interpolation algorithm model to process information required to reconstruct an initial color three-dimensional map of a space in which the robot exists; estimating and interpolating structural information and color information of areas not detected by the robot sensor; rendering the estimated and interpolated three-dimensional scene; and obtaining an interpolated color three-dimensional map.

5. The step of obtaining an initial color three-dimensional map of a space in which the robot exists includes: acquiring an RGB image sequence consisting of RGB images taken by the robot using an RGB camera, and determining position and orientation information corresponding to the RGB camera attached to the robot; and reconstructing an initial color three-dimensional map of a space in which the robot exists based on the sequence of RGB images and the position and orientation information.

6. The step of obtaining an initial color three-dimensional map of a space in which the robot exists includes:

3. The method of claim 2, comprising acquiring an RGB image sequence consisting of RGB images taken by the robot using an RGB camera, determining position and orientation information corresponding to the RGB camera attached to the robot, acquiring at least one of a depth image, point cloud information, accelerometer-collected information, and gyroscope-collected information, and reconstructing an initial color three-dimensional map of a space in which the robot exists.

7. The step of performing color 3D map completion based on the initial color 3D map of the space in which the robot exists includes:

7. The method according to claim 5 or 6, further comprising processing the reconstructed initial color 3D map using a second predetermined interpolation algorithm model to interpolate and estimate structural information and color information of areas not detected by the robot sensor, thereby obtaining the interpolated color 3D map.

8. 8. The method of claim 1, wherein the completed color 3D map includes structural and color information corresponding to the initial color 3D map and structural and color information in areas not detected by the robot sensor.

9. The method comprises:

9. The method according to claim 1, further comprising the steps of: dividing elements in the completed color three-dimensional map into regions, each of which includes one object, one type of object, or multiple types of objects; marking categories of the objects in the different regions; obtaining object category information in the completed color three-dimensional map; coloring elements in the different regions; and obtaining colors corresponding to the objects in the different regions in the completed color three-dimensional map.

10. The method comprises: The method according to any one of claims 1 to 9, further comprising the step of placing a 3D model of the robot on the completed color 3D map based on position and posture information of the robot, and displaying the 3D model of the robot and the completed color 3D map in equal proportions.

11. The method further includes modifying and displaying the interpolated color three-dimensional map using one or a combination of the following modification methods, wherein the modification includes: manually setting the color of the completed color 3D map or automatically setting the color of the completed color 3D map based on a preset color scheme; generating a decal using a preset pattern or a custom pattern, and displaying the decal on the completed color 3D map; adding a passable area, an impassable area, or a virtual wall for the robot to the completed color three-dimensional map, and displaying the passable area, the impassable area, or the virtual wall on the completed color three-dimensional map; setting a color scheme for the completed color 3D map that can be automatically changed according to time according to a preset color scheme corresponding to different times, and automatically changing the color of all or part of the completed color 3D map according to the color scheme; The method according to any one of claims 1 to 10, comprising: setting a camera trajectory on the interpolated color three-dimensional map or adopting a motion trajectory of the robot; rendering the interpolated color three-dimensional map based on the camera trajectory or a position and orientation of a camera on the motion trajectory and an internal reference; obtaining two-dimensional images rendered according to the trajectory; and sequentially playing back the two-dimensional images.

12. The method comprises: The method according to any one of claims 1 to 11, further comprising the step of displaying the completed color three-dimensional map in two dimensions according to pre-set cutouts or user custom cutouts.

13. an acquisition module configured to acquire initial color three-dimensional map information of a space in which the robot exists, the space in which the robot exists representing a space corresponding to an area detectable by the robot sensor; a completion module configured to perform color 3D map completion based on the initial color 3D map information, and to estimate and complete structural information and color information of areas not detected by the robot sensor; a display module configured to display the interpolated color 3D map obtained by the interpolation processing.

14. The apparatus of claim 13 , wherein the initial color three-dimensional map information includes an initial color three-dimensional map of a space in which the robot exists and / or information necessary to reconstruct an initial color three-dimensional map of a space in which the robot exists.

15. The acquisition module acquires information necessary to reconstruct an initial color three-dimensional map of a space in which the robot exists, acquiring an RGB image captured by the robot using an RGB camera, and acquiring position and orientation information corresponding to the RGB camera; Alternatively, the robot acquires an RGB image taken by using an RGB camera, acquires position and orientation information corresponding to the RGB camera, and acquires a depth image taken by using a depth camera and position and orientation information corresponding to the depth camera; Alternatively, the device according to claim 14 includes acquiring RGB images taken by the robot using an RGB camera, acquiring position and orientation information corresponding to the RGB camera, and acquiring point cloud information measured using a radar and position and orientation information corresponding to the radar.

16. The step of performing color 3D map completion by the completion module based on information necessary to reconstruct an initial color 3D map of the space in which the robot exists, 16. The apparatus of claim 15, further comprising: processing information required to reconstruct an initial color three-dimensional map of a space in which the robot exists using a preset first interpolation algorithm model; estimating and interpolating structural information and color information of areas not detected by the robot sensor; rendering the estimated and interpolated three-dimensional scene; and obtaining an interpolated color three-dimensional map.

17. The step of the acquisition module acquiring an initial color three-dimensional map of a space in which the robot exists includes: acquiring an RGB image sequence consisting of RGB images taken by the robot using an RGB camera, and determining position and orientation information corresponding to the RGB camera attached to the robot; and reconstructing an initial color three-dimensional map of a space in which the robot exists based on the sequence of RGB images and the position and orientation information.

18. The step of the acquisition module acquiring an initial color three-dimensional map of a space in which the robot exists includes:

15. The apparatus of claim 14, further comprising: acquiring an RGB image sequence consisting of RGB images taken by the robot using an RGB camera; determining position and orientation information corresponding to the RGB camera attached to the robot; acquiring at least one of a depth image, point cloud information, accelerometer-collected information, and gyroscope-collected information; and reconstructing an initial color three-dimensional map of a space in which the robot exists.

19. The step of the completion module performing color 3D map completion based on the initial color 3D map of the space where the robot exists, The apparatus according to claim 17 or 18, further comprising: processing the reconstructed initial color 3D map using a preset second interpolation algorithm model to interpolate and estimate structural information and color information of areas not detected by the robot sensor, thereby obtaining the interpolated color 3D map.

20. The apparatus of any one of claims 13 to 19, wherein the completed color 3D map includes structural information and color information corresponding to the initial color 3D map and structural information and color information in areas not detected by the robot sensor.

21. 21. The device of claim 13, wherein the display module divides elements in the interpolated color three-dimensional map into regions, each of which includes one object, one type of object, or multiple types of objects, marks categories of the objects in the different regions, obtains object category information in the interpolated color three-dimensional map, colors elements in the different regions, and obtains colors corresponding to the objects in the different regions in the interpolated color three-dimensional map.

22. The device according to any one of claims 13 to 21, wherein the display module places a 3D model of the robot on the completed color 3D map based on position and posture information of the robot, and displays the 3D model of the robot and the completed color 3D map in equal proportions.

23. The display module uses one or a combination of the following correction methods to correct and display the interpolated color 3D map, and the correction method includes: manually setting the color of the completed color 3D map or automatically setting the color of the completed color 3D map based on a preset color scheme; generating a decal using a preset pattern or a custom pattern, and displaying the decal on the completed color 3D map; adding a passable area, an impassable area, or a virtual wall for the robot to the completed color three-dimensional map, and displaying the passable area, the impassable area, or the virtual wall on the completed color three-dimensional map; setting a color scheme for the completed color 3D map that can be automatically changed according to time according to a preset color scheme corresponding to different times, and automatically changing the color of all or part of the completed color 3D map according to the color scheme; The apparatus according to any one of claims 13 to 22, further comprising: setting a camera trajectory on the interpolated color three-dimensional map or adopting a motion trajectory of the robot; rendering the interpolated color three-dimensional map based on the camera trajectory or a position and posture of a camera on the motion trajectory and an internal reference; obtaining two-dimensional images rendered according to the trajectory; and sequentially playing back the two-dimensional images.

24. The device according to any one of claims 13 to 23, wherein the display module displays the completed color three-dimensional map in two dimensions according to pre-set cutouts or user-custom cutouts.

25. An electronic device comprising a processor and a memory, the memory being used to store a computer program which, when executed by the processor, performs the method of any one of claims 1 to 12.

26. A computer-readable storage medium having stored thereon a computer program that, when executed by a processor, performs the method of any one of claims 1 to 12.

Images