Map construction method and device, robot, storage medium and computer program product

Abstract

This application provides a map construction method, apparatus, robot, storage medium, and computer program product; wherein the method includes: a first robot collecting first video data and corresponding inertial measurement data of a first scene; the first scene representing the inspection scene of the first robot; processing the first video data and inertial measurement data based on a first SLAM algorithm to obtain a first point cloud; and calling a first semantic segmentation model to process image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate a first detection target in the image frame; the first detection target representing the inspection target of the first robot in the first scene; fusing the first point cloud and the first semantic information to obtain a second point cloud, thereby constructing a first map of the first scene based on the second point cloud.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a map building method, apparatus, robot, storage medium, and computer program product. Background Technology

[0002] In related technologies, maps are built for target scenes based on the Simultaneous Localization and Mapping (SLAM) algorithm. However, the constructed maps have poor descriptive ability for the target scenes. Summary of the Invention

[0003] To address the related technical issues, embodiments of this application provide a map building method, apparatus, robot, storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a map construction method applied to a first robot, the method comprising: Collect first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot; Based on the first SLAM algorithm, the first video data and the inertial measurement data are processed to obtain a first point cloud; and the first semantic segmentation model is called to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate the first detection target in the image frame; the first detection target represents the inspection target of the first robot in the first scene; The first point cloud and the first semantic information are fused to obtain a second point cloud, and a first map of the first scene is constructed based on the second point cloud.

[0005] In the above scheme, when processing the first video data and the inertial measurement data based on the first SLAM algorithm, the method includes: Feature points are extracted from the image frames in the first video data to obtain one or more feature points corresponding to the image frames; Based on the second semantic information, the first feature point is removed from the one or more feature points; the second semantic information is obtained by processing the image frames in the first video data by calling the first semantic segmentation model, and the second semantic information is used to indicate the second detection target in the image frame; the second detection target represents a dynamic target in the first scene, and the second detection target is different from the first detection target; the first feature point represents the feature point corresponding to the second detection target in the image frame; Based on the one or more feature points after removing the first feature point, the pose of the first robot is estimated, and the first point cloud is determined based on the obtained pose estimation result.

[0006] In the above scheme, the step of removing the first feature point from the one or more feature points based on the second semantic information includes: Based on the second semantic information, a first image mask is determined; the first image mask is used to indicate the image region where the second detected target is located in the image frame; Based on the first image mask, a first feature point is removed from the one or more feature points; the first feature point represents a feature point within the image region indicated by the first image mask.

[0007] In the above scheme, when processing the first video data and the inertial measurement data based on the first SLAM algorithm, the method includes: Based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame, a set of matching point pairs is constructed; each matching point pair in the set is used to indicate a feature point in the first image frame and a feature point in the second image frame, and each matching point pair is used to indicate that the two feature points contained in the matching point pair correspond to the same region in the first scene; The matching point pair set is processed sequentially by calling the Grid-based Motion Statistics (GMS) algorithm, the Locality Preserving Matching (LMP) algorithm, and the Bayesian Adaptive Consensus Sampling (BANSAC) algorithm to remove the first matching point pair from the matching point pair set; the first matching point pair indicates that the two feature points correspond to different regions in the first scene; Based on the set of matching point pairs after removing the first matching point pair, pose estimation is performed on the first robot to determine the first point cloud based on the obtained pose estimation result.

[0008] In the above scheme, the fusion processing of the first point cloud and the first semantic information includes: Based on the first semantic information, the first detected target is marked in the first point cloud to obtain the second point cloud.

[0009] The method in the above scheme further includes: Synchronize the first thread, the second thread, and the third thread; among them, The first thread is used to call the first semantic segmentation model, the second thread is used to process based on the first SLAM algorithm, and the third thread is used to fuse the first point cloud with the first semantic information.

[0010] In the above scheme, the first robot represents the terminal in the cloud-edge-device architecture.

[0011] This application also provides a map building apparatus for use with a first robot, comprising: The acquisition unit is used to acquire first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot. The calling unit is used to process the first video data and the inertial measurement data based on the first SLAM algorithm to obtain a first point cloud; and to call the first semantic segmentation model to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate the first detection target in the image frame; the first detection target represents the inspection target of the first robot in the first scene; The fusion unit is used to fuse the first point cloud and the first semantic information to obtain a second point cloud, so as to construct a first map of the first scene based on the second point cloud.

[0012] This application also provides a first robot, including: a processor and a memory for storing a computer program capable of running on the processor. Wherein, when the processor is used to run the computer program, it executes the steps of any of the aforementioned methods.

[0013] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the aforementioned methods.

[0015] In this embodiment, a first robot collects first video data and corresponding inertial measurement data of a first scene, where the first scene represents the inspection scene of the first robot. Then, based on a first SLAM algorithm, the first video data and inertial measurement data are processed to obtain a first point cloud. A first semantic segmentation model is then called to process image frames in the first video data to obtain first semantic information. Here, the first semantic information is used to indicate a first detection target in the image frame, and the first detection target represents the inspection target of the first robot in the first scene. Afterwards, the first point cloud and the first semantic information are fused to obtain a second point cloud, which is then used to construct a first map of the first scene. In the above scheme, the first robot also calls a semantic segmentation model to obtain semantic information during the generation of the first point cloud based on the first SLAM algorithm. This allows the first robot to understand the scene in a manner close to human cognition during map construction, enhancing its perception and understanding of the scene. Based on this, the map constructed by fusing the first point cloud and semantic information can provide a more accurate and intuitive description of the scene, enhancing the map's ability to describe the target scene compared to related technologies. Attached Figure Description

[0016] Figure 1 A schematic diagram illustrating the implementation process of a map construction method provided in this application embodiment; Figure 2 A schematic diagram of the architecture of an inspection system provided for an application embodiment of this application; Figure 3 A schematic diagram of a robot architecture provided for an application embodiment of this application; Figure 4 A schematic diagram of a map construction processing architecture provided for an application embodiment of this application; Figure 5 A schematic diagram illustrating the construction of a set of matching point pairs provided for an application embodiment of this application; Figure 6 This is a schematic diagram of the structure of a map building device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware composition of a robot provided in an embodiment of this application. Detailed Implementation

[0017] In related technologies, SLAM algorithms are used to build maps for target scenes, such as maps for robot inspection scenarios. However, SLAM algorithms have limited map-building performance under varying lighting conditions or dynamic scenes, and their ability to perceive and understand complex environments is weak. Therefore, the constructed maps may deviate from the actual scene and are difficult to directly apply to practical business. For example, a map can only determine the location of storm and sewage pipes in a target scene, but cannot determine whether there are defects such as silt blockages in the pipes. Consequently, it is impossible to perform navigation or path planning tasks to locate defects in storm and sewage pipes in business operations. Therefore, the maps constructed in related technologies have poor descriptive capabilities for the target scene.

[0018] Based on this, in this embodiment, the first robot collects first video data and corresponding inertial measurement data of a first scene, wherein the first scene represents the inspection scene of the first robot; then, based on the first SLAM algorithm, the first video data and inertial measurement data are processed to obtain a first point cloud, and the first semantic segmentation model is called to process the image frames in the first video data to obtain first semantic information; here, the first semantic information is used to indicate the first detection target in the image frame, and the first detection target represents the inspection target of the first robot in the first scene; then, the first point cloud and the first semantic information are fused to obtain a second point cloud, and a first map of the first scene is constructed based on the second point cloud. In the above scheme, the first robot also calls the semantic segmentation model to obtain semantic information during the process of generating the first point cloud based on the first SLAM algorithm. In this way, the first robot can understand the scene in a way close to human cognition during the map construction process, which enhances the first robot's perception and understanding of the scene. On this basis, the map constructed by fusing the first point cloud and semantic information can describe the scene more accurately and intuitively, which enhances the map's ability to describe the target scene compared with related technologies.

[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0020] This application provides a map building method applied to a first robot. In practical applications, the first robot can be used to perform inspection tasks. The first robot can execute the map building method provided in this application while performing the inspection tasks to build a map of the inspection scene.

[0021] See Figure 1 The method includes: Step 101: Collect the first video data and corresponding inertial measurement data of the first scene.

[0022] The first scenario represents the inspection scenario of the first robot.

[0023] In practical applications, the first scenario can be understood as the scenario that the first robot needs to inspect when performing its inspection task; the scenario can also be called the environment. The first scenario can include one or more of the following: indoor, outdoor, and scenarios where the Global Positioning System (GPS) signal is unavailable.

[0024] For example, the first robot can be used to inspect stormwater and sewage pipes to detect defects in the pipes; in this case, the first scenario is the stormwater and sewage pipes. The stormwater and sewage pipes may be partially indoors and partially outdoors, and GPS signals may be unavailable within the stormwater and sewage pipes.

[0025] In practical applications, the first robot can move within a first scene without a map of that scene being known, based on a localization-related algorithm. For example, the localization-related algorithm may include a SLAM algorithm.

[0026] During the movement of the first robot in the first scene, it can collect the first video data and corresponding inertial measurement data of the first scene.

[0027] In practical applications, the first video data can be understood as the video data collected by the first robot for the first scene.

[0028] The first video data may contain multiple image frames; an image frame can be understood as a frame in the video, represented in the form of an image. The first video data may contain color information and depth information. For example, the first video data may be represented as RGB-D video data, and each image frame in the first video data may contain a color image and a depth image.

[0029] For example, the first robot may be equipped with a red-green-blue-depth (RGB-D) camera, and the first robot may acquire first video data based on the RGB-D camera.

[0030] In practical applications, inertial measurement data can be understood as data reflecting the motion state of the first robot during its movement. For example, inertial measurement data may include one or more of the following: angular velocity, acceleration, and magnetic field strength.

[0031] For example, the first robot may be equipped with an inertial measurement unit (IMU), which may include sensing devices such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer. The first robot can collect inertial measurement data based on the inertial measurement unit.

[0032] Step 102: Based on the first SLAM algorithm, process the first video data and inertial measurement data to obtain the first point cloud; and call the first semantic segmentation model to process the image frames in the first video data to obtain the first semantic information.

[0033] The first semantic information is used to indicate the first detection target in the image frame; the first detection target represents the inspection target of the first robot in the first scene.

[0034] In practical applications, a point cloud is a collection of multiple data points, each of which can be used to indicate a three-dimensional coordinate. Based on point clouds, corresponding maps can be constructed. For example, maps can be built using open-source tools, such as Octomap, an efficient probabilistic 3D mapping framework based on octrees, to process point clouds. Point clouds can also be understood as a form of map representation, and can be called point cloud maps or 3D point clouds.

[0035] In practical applications, the SLAM algorithm can be used by robots to localize in unknown environments using data collected by their own sensors and gradually create a map of the scene. During the processing based on the SLAM algorithm, a visual processing framework implemented using the SLAM algorithm can be used. For example, a visual processing framework implemented using the SLAM algorithm can include the ORB-SLAM3 framework.

[0036] In practical applications, the processing flow of the SLAM algorithm can mainly include the following steps: Step 1: Data collection.

[0037] In practical applications, video data and inertial measurement data can be collected.

[0038] Step 2: Tracking.

[0039] In practical applications, each image frame in the acquired video data can be tracked sequentially to estimate the robot's pose and determine the corresponding map points in real time.

[0040] During the processing of each image frame, feature points can be extracted from that frame. These feature points can be used to indicate representative locations within the image frame for map construction; for example, a feature point may contain the coordinates of that location within the image frame and a corresponding feature description. A specific location in an image frame can correspond to a specific location in the scene.

[0041] Based on the acquired feature points, during the processing of each image frame, feature matching can be performed between the image frame and the corresponding reference frame based on the feature points. Then, the robot's real-time pose can be estimated based on the feature matching results and inertial measurement data.

[0042] The reference frame corresponding to an image frame can be understood as the image frame used for reference during feature matching. The reference frame corresponding to an image frame can be the previous image frame or the previous keyframe.

[0043] A keyframe can be understood as a representative image frame in video data used for map construction. The feature points corresponding to the keyframe can be used to determine points in the point cloud map. Whether to use an image frame as a keyframe can be determined based on set keyframe selection criteria. For example, the set keyframe selection criteria may include: the frame interval between the image frame and the previous keyframe has exceeded a threshold.

[0044] In practical applications, during feature matching between an image frame and its corresponding reference frame, it can be determined whether the feature points corresponding to the image frame and the feature points corresponding to the reference frame match. Then, a set of matching point pairs is constructed based on the matching feature points. Each matching point pair in the set includes: a feature point corresponding to the image frame and a feature point corresponding to the reference frame, and these two feature points match.

[0045] In practical applications, map points can be used to indicate specific locations in a scene, equivalent to data points in a point cloud map. For example, a map point can contain: the three-dimensional coordinates of the specific location in the scene and the corresponding observation information. The observation information can be used to indicate in which keyframes the location corresponding to the map point was observed.

[0046] Map points can be determined based on feature points. For example, based on set map point selection criteria, it can be determined whether the location corresponding to a feature point in the scene should be used as a map point indicator, thereby determining the map point; the set map point selection criteria can include, for example, whether the feature point is observed in multiple keyframes.

[0047] Step 3: Local mapping.

[0048] In practical applications, local mapping can mainly include the selection of local keyframes and local bundle adjustment (BA).

[0049] For the selection of local keyframes, the co-view relationship between keyframes can be used to select local keyframes. Local keyframes can be understood as representative keyframes that are only related to the current image frame and are used for local optimization. Local keyframes can be used to construct and maintain local maps, and combined with inertial measurement data to jointly optimize the robot's pose. Multiple map points corresponding to local areas in the scene can be considered as forming a local map.

[0050] For local BA, the associated map points can be optimized based on the current keyframe and the corresponding local keyframe, thereby improving the accuracy of the map points in describing the scene.

[0051] Step 4: Loop closure detection and map fusion.

[0052] In practical applications, closed-loop detection can be used to determine whether the robot has moved to a previously reached position. If a closed loop is detected, closed-loop correction can be performed to eliminate accumulated errors in the map building process. When the newly determined local map is related to previously determined local maps, map fusion can be performed to gradually merge them into a global map specific to the scene.

[0053] For example, if a shared area is detected between the current keyframe and the active map, it can be considered a loop closure, and loop closure correction can be performed. If a shared area is detected between the current keyframe and the inactive map, a map fusion operation can be performed, and the fused map can be used as the current active map. If both a loop closure and the need for map fusion are detected simultaneously, the loop closure can be ignored, and the map fusion operation can be performed. The active map can be understood as a local map that is currently being built. The inactive map can be understood as a local map that has been built historically and is not updated in real time.

[0054] Step 5: All BAs.

[0055] In practical applications, BA (Balanced Array) can be used to optimize all keyframes and their map points in a constructed map, thereby obtaining an optimized global map. This global map can be viewed as a point cloud generated based on the SLAM algorithm.

[0056] In practical applications, each step in the processing flow of the SLAM algorithm described above can be performed in real time during the robot's movement. For example, data can be collected during the robot's movement, and processing such as tracking and local mapping can be performed based on the collected data. At the same time, the robot continues to move and collect data for subsequent processing until the map is completed.

[0057] In practical applications, the implementation details of the processing flow corresponding to the above SLAM algorithm can be understood by referring to relevant technologies.

[0058] In this embodiment, the first video data and inertial measurement data are processed based on the first SLAM algorithm to obtain the first point cloud. In practical applications, the processing flow corresponding to the first SLAM algorithm in this embodiment can be the same as the processing flow corresponding to the SLAM algorithm described above. That is, the first video data and inertial measurement data can be processed according to the algorithm implementation method in related technologies. Specifically, the first video data and inertial measurement data can be collected based on step 1, and the collected data can be processed into the first point cloud based on steps 2 to 5.

[0059] The first SLAM algorithm in this application embodiment can also be improved based on the processing flow corresponding to the above-described SLAM algorithm. For example, steps can be added or the processing method of some steps can be changed. Then, the first video data and inertial measurement data can be processed according to the improved processing flow, thereby improving the performance of map construction, such as improving the accuracy of the constructed map. To facilitate the distinction between related technologies and the SLAM algorithm in this application embodiment, the SLAM algorithm corresponding to the related technologies will be referred to as the second SLAM algorithm below.

[0060] In practical applications, semantic segmentation models can be used to perform pixel-level recognition on input images, thereby identifying the specific semantic type of each pixel in the image. For example, identifying whether the pixel belongs to a person, vehicle, or road. In other words, semantic segmentation models can be used to identify targets in an image that belong to a specific semantic type, such as identifying people, vehicles, or roads in the image. Semantic type can also be understood as semantic label.

[0061] For example, a semantic segmentation model may include a lightweight real-time semantic segmentation (PP-Liteseg) model.

[0062] Here, the first semantic segmentation model is invoked to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate the first detected target in the image frame. In practical applications, the first semantic segmentation model can be used to identify the first detected target in the image frame.

[0063] In practical applications, the first detection target can be understood as the target that the first robot needs to detect during the process of performing the inspection task in the first scenario.

[0064] For example, in the case of a sewage pipeline in the first scenario, the first detection target may include defects in the sewage pipeline. For example, the first detection target may include one or more of the following: cracks, deformation, corrosion, misalignment, unevenness, disconnection, detachment of interface material, concealed branch pipe connections, foreign object penetration, leakage, deposition, scaling, and scum.

[0065] Here, in the process of generating the first point cloud based on the first SLAM algorithm, the first robot also calls the semantic segmentation model to obtain semantic information. In this way, the first robot can understand the scene in a way that is close to human cognition during the map construction process, which enhances the first robot's perception and understanding of the scene, thereby improving the accuracy of the constructed map.

[0066] Step 103: The first point cloud and the first semantic information are fused to obtain the second point cloud, and the first map of the first scene is constructed based on the second point cloud.

[0067] For example, the second point cloud can be processed using open-source tools such as Octomap to obtain the first map.

[0068] Here, the second point cloud is obtained by fusing the first point cloud with the first semantic information. Therefore, the second point cloud can contain more semantic information than the first point cloud. For example, based on the first point cloud, it is only possible to determine the geographical distribution of the first scene in the real world, while based on the second point cloud, it is possible not only to determine the geographical distribution of the first scene, but also the geographical location of the first detected target in the first scene. On this basis, the first map constructed from the second point cloud can provide a more accurate and intuitive description of the first scene, thus enabling direct application to practical business.

[0069] For example, in the case where the first scenario is a sewage pipe and the first detection target is a defect in the sewage pipe, the first map can not only indicate the geographical distribution of the sewage pipe, but also indicate the location of the defect in the sewage pipe. For example, the defect type is marked by highlighting at the location of the defect. Based on this, the defect location in the sewage pipe can be intuitively located and navigated to according to the first map, which is equivalent to being directly applied to the sewage pipe inspection business.

[0070] In practical applications, steps 101, 102, and 103 can be performed simultaneously. For example, during the execution of step 101, step 102 can be executed concurrently to process the data collected in step 101, and step 103 can be executed to fuse the local point cloud map of the first point cloud generated in step 102 with the corresponding first semantic information. Simultaneously, step 101 continues to be executed to collect more data, and steps 102 and 103 are then executed based on the collected data. This achieves real-time map construction, thereby improving the accuracy and efficiency of map construction.

[0071] In practical applications, the operation of fusing the first point cloud and the first semantic information in the embodiments of this application can be regarded as an improvement on the processing flow corresponding to the second SLAM algorithm. Specifically, the corresponding first semantic information is fused on the local map constructed by the processing flow, and / or the corresponding first semantic information is fused on the global map constructed by the processing flow.

[0072] In this embodiment, during the process of generating the first point cloud based on the first SLAM algorithm, the first robot also calls a semantic segmentation model to obtain semantic information. In this way, the first robot can understand the scene in a way that is close to human cognition during the map construction process, which enhances the first robot's perception and understanding of the scene. On this basis, the map constructed by fusing the first point cloud and semantic information can describe the scene more accurately and intuitively. Compared with related technologies, it enhances the map's ability to describe the target scene.

[0073] The following section provides a further explanation of the map construction method.

[0074] In one embodiment, when processing the first video data and inertial measurement data based on the first SLAM algorithm, the map construction method provided in this application includes: Feature points are extracted from the image frames in the first video data to obtain one or more feature points corresponding to the image frames; Based on the second semantic information, the first feature point is removed from one or more feature points; the second semantic information is obtained by processing the image frames in the first video data by calling the first semantic segmentation model, and the second semantic information is used to indicate the second detection target in the image frame; the second detection target represents a dynamic target in the first scene, and the second detection target is different from the first detection target; the first feature point represents the feature point corresponding to the second detection target in the image frame; Based on one or more feature points after removing the first feature point, pose estimation is performed on the first robot to determine the first point cloud based on the obtained pose estimation result.

[0075] In practical applications, feature points can be extracted from the color image in the image frame to obtain one or more feature points corresponding to the image frame.

[0076] Feature points can be used to indicate representative locations in an image frame for building a map. A feature point can contain: the coordinates of the indicated location in the image frame and the corresponding feature description, which can also be called the feature point descriptor.

[0077] For example, feature points can represent Oriented FAST and Rotated BRIEF (ORB) feature points, and the corresponding extraction process can be implemented based on the following parameters: nfeatures=2500, scaleFactor=1.2, edgeThreshold=8, and fastThreshold=12; where nfeatures can be used to indicate the total number of feature points to be extracted, scaleFactor can be used to indicate the scaling factor between adjacent layers of the image pyramid during the extraction process, edgeThreshold can be used to indicate the minimum pixel distance of the feature point from the edge of the image frame, and fastThreshold can be used to represent the threshold used to determine whether the feature point is a corner point.

[0078] In practical applications, the first semantic segmentation model can also be used to identify a second detected target in an image frame. The second detected target can be understood as a dynamically changing target in the first scene that does not belong to the inspection target of the first robot.

[0079] For example, in the case of a sewage pipeline in the first scenario, the second detection target may include one or more of the following: fluid in the pipeline, suspended solids, organisms, and operating equipment that the first robot can carry.

[0080] In related technologies, map construction is based on the assumption of static scenes, that is, the scene that needs to be mapped is understood as a static scene. However, for scenes with dynamic changes, this map construction method will be affected by the dynamic changes, thereby reducing the accuracy of the constructed map.

[0081] In this embodiment, dynamic targets are detected by a semantic segmentation model, and then the feature points corresponding to the dynamic targets are removed from the feature points corresponding to the image frames. This reduces the interference of dynamic targets on the map construction process and improves the accuracy of map construction.

[0082] The process of removing the first feature point in this embodiment can be regarded as an improvement on the processing flow of the second SLAM algorithm. Specifically, based on the feature points extracted in step 2 of the processing flow, the first feature point corresponding to the dynamic target is removed.

[0083] In one embodiment, based on second semantic information, removing a first feature point from one or more feature points includes: Based on the second semantic information, a first image mask is determined; the first image mask is used to indicate the image region where the second detected target is located in the image frame. Based on the first image mask, a first feature point is removed from one or more feature points; the first feature point represents a feature point within the image region indicated by the first image mask.

[0084] In practical applications, by overlaying the first image mask onto the corresponding image frame, the image region where the second detection target is located in the image frame can be determined.

[0085] Here, the first feature point represents the feature point within the image region indicated by the first image mask. In other words, the process of removing the first feature point involves removing the feature points within the image region indicated by the first image mask, or more specifically, removing the feature points corresponding to the image content occupied by the second detection target. Thus, the interference of the second detection target on the map construction process can be accurately removed using the image mask, further improving the accuracy of map construction.

[0086] In one embodiment, when processing the first video data and inertial measurement data based on the first SLAM algorithm, the map construction method provided in this application includes: Based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame, a set of matching point pairs is constructed; each matching point pair in the set is used to indicate a feature point in the first image frame and a feature point in the second image frame, and each matching point pair is used to indicate that the two feature points contained in the matching point pair correspond to the same region in the first scene; The GMS algorithm, LMP algorithm, and BANSAC algorithm are called sequentially to process the set of matching point pairs in order to remove the first matching point pair in the set; the first matching point pair indicates that the two feature points correspond to different regions in the first scene. Based on the set of matching point pairs after removing the first matching point pair, the pose of the first robot is estimated, and the first point cloud is determined based on the obtained pose estimation result.

[0087] Here, the set of matching point pairs is constructed based on one or more feature points corresponding to the image frame. In practical applications, the set of matching point pairs can be constructed based on one or more feature points after removing the first feature point.

[0088] In practical applications, when the first image frame represents an image frame preceding the second image frame, the first image frame can be understood as the reference frame corresponding to the second image frame; when the second image frame represents an image frame preceding the first image frame, the second image frame can be understood as the reference frame corresponding to the first image frame.

[0089] In practical applications, each matching point pair in the matching point pair set indicates two feature points that can be understood as two matching feature points. The matching point pair set can also be called the matching point set.

[0090] For each feature point descriptor in the first image frame, the Hamming distance between this descriptor and the descriptors of each feature point in the second image frame can be calculated. Then, the feature point with the smallest Hamming distance among the feature points in the second image frame is determined as the matching feature point with that feature point in the first image frame, resulting in a matching point pair. Based on this, a set of matching point pairs can be constructed.

[0091] Here, the GMS algorithm, LMP algorithm, and BANSAC algorithm are called in sequence to process the set of matching point pairs in order to remove the first matching point pair from the set.

[0092] In practical applications, the first matching point pair can be understood as an incorrect matching point pair. That is, although the prior processing assumes that the two feature points indicated by the matching point pair match, the two feature points do not actually match.

[0093] Here, to facilitate the distinction between the sets of matching points obtained based on Hamming distance, GMS algorithm, LMP algorithm and BANSAC algorithm, the sets of matching points obtained based on these processing methods are respectively referred to as the first set of matching points, the second set of matching points, the third set of matching points and the fourth set of matching points.

[0094] In practical applications, a first set of matching point pairs can be constructed based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame. Then, the first set of matching point pairs can be processed using the GMS algorithm to remove first-matching point pairs, resulting in a second set of matching point pairs. Next, the second set of matching point pairs can be processed using the LMP algorithm to remove first-matching point pairs, resulting in a third set of matching point pairs. Finally, the third set of matching point pairs can be processed using the BANSAC algorithm to remove first-matching point pairs, resulting in a fourth set of matching point pairs. Finally, the pose of the first robot can be estimated based on the fourth set of matching point pairs, and the resulting pose estimation can be used to determine the first point cloud.

[0095] In practical applications, the GMS algorithm can quickly eliminate erroneous matching point pairs by dividing the image into a grid. For example, the size of the image grid can be 20×20 pixels, that is, 20 pixels long and 20 pixels wide.

[0096] The LPM algorithm can filter the set of matching point pairs based on local neighborhood constraints, thus obtaining a more accurate set of matching point pairs. The LPM algorithm relies on an image grid during processing; the image grid used by the LPM algorithm can be the same as that used by the GMS algorithm. That is, the LPM algorithm can further filter the set of matching point pairs obtained from the GMS algorithm based on local neighborhood constraints within the image grid divided by the GMS algorithm.

[0097] The BANSAC algorithm can be seen as an improvement on the guided sampling algorithm (RANSAC) based on Bayesian networks. It can be used to quickly remove outliers from the set of matching point pairs while ensuring accuracy.

[0098] In related technologies, Hamming distance or a single algorithm is typically used to construct a set of matching point pairs. However, in this embodiment, considering the processing characteristics of the GMS, LMP, and BANSAC algorithms, the GMS, LMP, and BANSAC algorithms are called sequentially to process the set of matching point pairs. This combination of algorithms improves the accuracy of the final determined set of matching point pairs, thereby improving the accuracy of map construction.

[0099] The processing of removing the first feature point pair in this embodiment can be regarded as an improvement on the processing flow of the second SLAM algorithm. Specifically, based on the set of matching point pairs extracted in step 2 of the processing flow, erroneous matching point pairs are effectively removed according to a specific algorithm combination.

[0100] In one embodiment, the fusion processing of the first point cloud and the first semantic information includes: Based on the first semantic information, the first detection target is marked in the first point cloud to obtain the second point cloud.

[0101] In practical applications, for data points in the first point cloud that belong to the first detection target, corresponding semantic labels can be identified.

[0102] For example, in the case where the first scenario is a sewage pipe and the first detection target is a defect in the sewage pipe, data points in the first point cloud that belong to sewage pipe defects can be highlighted and / or have text labels added to indicate that the data point is the first detection target.

[0103] In this embodiment of the application, the second point cloud is marked with the first detection target, or in other words, the second point cloud contains semantic information. Based on this, the first map constructed from the second point cloud can provide a more accurate and intuitive description of the first scene, and thus can be directly applied to actual business.

[0104] In one embodiment, the map construction method provided in this application further includes: Synchronize the first thread, the second thread, and the third thread; among them, The first thread is used to call the first semantic segmentation model, the second thread is used to process based on the first SLAM algorithm, and the third thread is used to fuse the first point cloud with the first semantic information.

[0105] In practical applications, the first thread, the second thread, and the third thread can run simultaneously. The first thread can also be called the object detection thread, the second thread can be called the semantic mapping thread, and the third thread can be called the SLAM thread.

[0106] For example, during the map construction process based on the first SLAM algorithm, the first semantic segmentation model can be synchronously invoked to process the image frames already processed by the first SLAM algorithm. Furthermore, the local point cloud map generated by the first SLAM algorithm regarding the first point cloud is fused with the corresponding first semantic information. Simultaneously, unprocessed image frames, such as newly acquired image frames, continue to be processed based on the first SLAM algorithm, and on this basis, the first semantic segmentation model is invoked and fusion processing is performed. In this way, real-time map construction is achieved, thereby improving the accuracy and efficiency of map construction.

[0107] In practical applications, the image frame processing rates of the first, second, and third threads can be set to be the same to ensure synchronization among the three threads, thereby guaranteeing the stability of real-time map construction.

[0108] In one embodiment, the first robot represents a terminal in a cloud-edge-device architecture.

[0109] In practical applications, the constructed first map can be used for actual business operations, and the system architecture corresponding to the business system can be a cloud-edge-device architecture. For example, the business system may include an inspection system.

[0110] In practical applications, the cloud-edge-device architecture can also be called a cloud-edge-device collaborative architecture. By integrating the technological advantages of cloud computing and edge computing, the cloud-edge-device architecture can effectively reduce the data transmission requirements from the edge to the cloud, significantly improve data processing efficiency and business response speed, and form an innovative computing paradigm that combines centralized and distributed computing.

[0111] The cloud-edge-device architecture mainly includes: cloud servers, edge nodes, and terminals. Cloud servers can also be called cloud computing centers, edge nodes can also be called edge computing nodes, and terminals can also be called terminal devices or edge devices.

[0112] In related technologies, cloud servers are used for centralized data processing and storage; edge nodes are used for rapid local data computation and decision-making; and terminals are used to collect data and transmit it to edge nodes or cloud servers for computational processing. For map building, terminals typically collect large amounts of data from sensors and then upload it to edge nodes or cloud servers for processing. However, the transmission of large amounts of data increases the demand for network bandwidth and may also affect the real-time performance of data processing due to network latency. With the improvement of computing power of embedded artificial intelligence (AI) devices on the terminal side, the computing resources on the terminal side in the cloud-edge-device architecture of related technologies are not fully utilized, resulting in uneven resource allocation and reduced overall computing efficiency.

[0113] In this embodiment, the map is constructed by a first robot, which represents the terminal in the cloud-edge-device architecture. In other words, the terminal is not only used for data collection, but also for performing calculation-related processing on the collected data in the map construction. Compared with related technologies, this reduces data transmission requirements and processing latency, and improves the utilization rate of edge computing resources.

[0114] In practical applications, the terminal can also upload the constructed first map to a cloud server or edge node to further enhance the map's ability to assist business operations. For example, an edge node can determine the defect level of a sewage pipe based on the first map.

[0115] The present application will be further described in detail below with reference to application examples.

[0116] This application provides an inspection system through its application embodiments. See [link to relevant documentation]. Figure 2 The inspection system is built on a cloud-edge-device architecture and mainly includes: terminal devices, edge nodes and cloud servers.

[0117] In practical applications, the terminal device can be a robot, which can be customized according to specific business needs. The robot can be used for collecting and sensing multi-source heterogeneous information in complex scenarios, as well as for further map construction.

[0118] As one implementation method, see Figure 3 The robot's components may include, but are not limited to: a core control module, a power propulsion module, a communication module, an environmental perception module, a combined positioning module, an optical detection module, and a user operation module. Functions that can be implemented may include: precise positioning, environmental monitoring, online identification, and cloud storage.

[0119] As one implementation approach, the core control module can be implemented using an embedded AI module. For example, an embedded module carrying deep learning computing power can be used, and a robot operating system can be used for robot business logic control and inter-module instruction communication.

[0120] The propulsion module can be designed according to specific application scenarios. The robot's power system can offer power supply options such as batteries and cables, while the propulsion system can offer structures such as wheeled chassis, tracked chassis, and double-helix rollers.

[0121] The environmental perception module can be designed according to specific application scenarios. It can include sensors for robot status monitoring, such as temperature and humidity sensors, current sensors, and Hall effect sensors. It can also include sensors for environmental detection, such as temperature and humidity sensors, gas sensors, sound sensors, and ultrasonic sensors.

[0122] The combined positioning module can employ a visual positioning module, whose camera unit can be a binocular camera or an RGB-D camera, and the module can include an inertial measurement unit (IMU). Alternatively, instead of using a visual positioning module with both a camera and an IMU, a separate camera and IMU can be used for combined positioning. In this case, multi-sensor calibration between the camera and the IMU can be performed first.

[0123] The optical detection module can be implemented using an optical camera. The camera can be equipped with a pan-tilt unit to enable all-round monitoring of the robot. The camera can have zoom capability, and the video resolution can reach at least 1280*720 pixels at 30 frames per second (fps), that is, 1280*720p / 30fps or higher.

[0124] The user operation module can be an industrial-grade rugged laptop or rugged handheld tablet, which can be equipped with a robot operation host computer for remotely operating the robot to perform tasks. The user operation module can be equipped with a module that complies with a specific communication protocol, such as a 5G (5th Generation Mobile Communication Technology) module, to enable communication with the robot and to collect data and upload it to the user's data center.

[0125] The communication function module can be implemented by combining 5G modules, narrowband Internet of Things (NB-IoT), switches, and cables. For example, the 5G module on the robot body can be used to communicate with the user operation module, the NB-IoT module can be used to communicate with the environmental perception module, the switch can be used for internal communication within the robot, and cables can be used for communication in environments with severe signal shielding or interference.

[0126] In practical applications, edge nodes can serve as user data centers, receiving data transmitted from terminal devices and performing compute migration. Edge nodes can use Software Defined Networking (SDN) technology as a global controller for compute migration. That is, by employing an SDN controller, the edge network status is monitored and aggregated, and migration task scheduling schemes are determined. This enables policy configuration, network topology management, monitoring and alarming, and status statistics for edge nodes and end-side devices, thereby reducing the computational pressure on the edge network and lowering data transmission latency.

[0127] As one implementation, the SDN controller used in the edge node can communicate with the end-side device through the southbound interface, using a pre-configured first protocol for policy configuration and device management, and a second protocol for network topology discovery, monitoring and alarming, and status statistics. For example, the first protocol may include OpenFlow, NXAPI, or SSH, and the second protocol may include SNMP or Telemetry.

[0128] In terms of resource coordination, the SDN controller and related systems, i.e., the SDN system, can automatically distribute configurations, establish multiple network links, and optimize traffic allocation through shortest path algorithms to ensure that data is transmitted on the optimal link. When link congestion occurs, the SDN system can detect it and automatically adjust traffic to less congested links to maintain efficient network operation.

[0129] In terms of security collaboration, SDN systems can manage security components in edge nodes and end-side devices, detecting and blocking malicious traffic in real time to prevent its spread. Security components can include, but are not limited to, firewalls and security groups, thereby ensuring the security and stability of the network environment. For example, cloud network access control lists (ACLs) and security groups can be used to improve network security, allowing users to define custom network segments and Internet Protocol (IP) addresses.

[0130] In terms of service collaboration, SDN systems can provide unified service orchestration capabilities, offering customized network services for edge nodes and end-side devices to meet service needs in different scenarios.

[0131] In practical applications, cloud servers can be used for offline training of deep learning models, as well as for storing relevant inspection data uploaded by edge nodes.

[0132] As one implementation method, cloud servers can build isolated and private virtual network environments based on virtual private cloud technology, and use cloud dedicated lines to connect to the virtual private cloud (VPC) in the cloud, building an independent network space for users. In this way, users can have their own dedicated communication links, completely isolated from the business data of other users, and avoid data leakage.

[0133] As one implementation method, cloud servers can connect VPCs to user data centers via cloud private lines or Virtual Private Network (VPN) tunnels, thereby enabling users to flexibly deploy hybrid clouds.

[0134] As one implementation method, cloud servers can utilize cloud-based computing-accelerated graphics processing units (GPUs) to configure deep learning environments. The GPU can be a specific GPU card with 32 gigabytes (GB) of dedicated memory per card, supporting the PCIe 3.0 interface. The central processing unit (CPU) can be a specific processor with a clock speed of 2.6 GHz, and a CPU-to-memory ratio of 1:8. The maximum internal network bandwidth can be set to 30 gigabits per second (Gbit / s).

[0135] As one implementation approach, in a cloud server, the deep learning dataset used to train the deep learning model can be categorized and labeled according to the user scenario. Annotation tools such as Labelme can be used. For the deep learning framework, PyTorch can be chosen; for software libraries such as CUDA, version 10.1 can be selected; and for cross-platform computer vision libraries such as OpenCV, version 4.2 can be used.

[0136] During the training phase of training the semantic segmentation model, the number of iterations (iters) can be set to 160,000; the optimizer can use the stochastic gradient descent algorithm; the loss function can use the OhemCrossEntropyLoss for online hard sample mining; and the learning strategy can use the polynomial lDecay learning rate iteration strategy, with the corresponding learning rate parameter set to 0.0025.

[0137] In practical applications, robots serving as terminal devices can be equipped with intelligent detection algorithms. For example, semantic segmentation models such as the PP-LiteSeg model can be built, as well as visual SLAM localization algorithm frameworks such as ORB-SLAM3, thereby improving the robot's ability to understand target semantic information and acquire scene map information.

[0138] See Figure 4 The robot can build a map based on the onboard intelligent detection algorithm. The map building method can be understood by referring to the map building method provided in the embodiments of this application.

[0139] The process of a robot building a map can mainly include the following steps: Step 1: System initialization.

[0140] In practical applications, the ORB-SLAM3 framework can be initialized and its parameters set to adapt to dynamic environments; and the trained PP-LiteSeg model can be loaded to ensure that it can process the input images in real time.

[0141] In practical applications, robots can perform data acquisition. For example, color and depth images can be acquired using an RGB-D camera, and combined with data measured by a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer in the inertial measurement unit, as input to an intelligent detection algorithm.

[0142] Step 2: Tracking.

[0143] In practical applications, ORB feature points can be extracted from color images, and initial pose estimation of the feature points can be performed with the assistance of inertial measurement unit data.

[0144] For the extracted feature points, a dynamic target mask can be obtained based on a semantic segmentation algorithm, and then the dynamic feature points can be removed using the dynamic target mask. The dynamic target mask can be understood as a mask used to indicate dynamic targets, where the dynamic target is equivalent to the second detection target in this application embodiment, and the dynamic feature points are equivalent to the first feature points in this application embodiment.

[0145] In the application embodiments of this application, the method for constructing the matching point pair set during the tracking process can be improved compared to the method for constructing the matching point pair set in the ORB-SLAM3 framework in related technologies. See [link to relevant documentation]. Figure 5 The improved processing flow can mainly include the following processing steps: (1) Calculate the descriptors of ORB feature points and construct a basic matching point set by comparing their similarity through Hamming distance. The basic matching point set can be equivalent to the first matching point pair set in the embodiments of this application.

[0146] (2) Based on the GMS algorithm, the image is divided into grids to quickly remove incorrect matching pairs and construct a coarse matching point set. The coarse matching point set can be equated to the second matching point pair set in the embodiments of this application. The processing based on the GMS algorithm can be understood as image grid-based filtering.

[0147] (3) Using the LPM algorithm, the exact matching point set is further filtered based on local neighborhood constraints in the mesh constructed by the GMS algorithm. The exact matching point set can be equivalent to the third set of matching point pairs in the embodiments of this application. The processing based on the LPM algorithm can be understood as filtering based on neighborhood constraints.

[0148] (4) The BANSAC algorithm is used to quickly remove outliers while ensuring accuracy. The set of matching points obtained based on the BANSAC algorithm can be equated to the fourth set of matching point pairs in the embodiments of this application.

[0149] Step 3: Semantic segmentation.

[0150] In practical applications, color images can be input into the PP-LiteSeg model for real-time semantic segmentation, and accelerated using a relevant inference framework. Based on the segmentation results of the PP-LiteSeg model, target objects in the scene can be identified and their semantic categories can be labeled. The segmentation results can be equivalent to the first semantic information in this application embodiment, and the target objects can be equivalent to the first detection objects in this application embodiment.

[0151] In practical applications, by fusing the semantic segmentation results of the PP-LiteSeg model with 3D point clouds, a 3D semantic point cloud with semantic labels can be formed.

[0152] Step 4: Local mapping.

[0153] In practical applications, local keyframes can be selected based on the co-view relationship between keyframes for the construction and maintenance of local maps, and pose can be jointly optimized by combining visual inertial information.

[0154] Step 5: Loop closure detection and map fusion.

[0155] In practical applications, closed-loop detection can be used to determine whether the robot has moved to a previously reached position. If a closed loop is detected, closed-loop correction can be performed to eliminate accumulated errors in the map building process. When the newly determined local map is related to previously determined local maps, map fusion can be performed to gradually merge them into a global map specific to the scene.

[0156] For example, if a common area is detected between the current keyframe and the active map, it can be considered a loop closure, and loop closure correction can be performed. If a common area is detected between the current keyframe and the inactive map, a map fusion operation is performed, and the fused map is used as the current active map. If both a loop closure and the need for map fusion are detected simultaneously, the loop closure can be ignored, and map fusion can be performed. The active map can be understood as a local map that is being built.

[0157] Step 6: Global BA.

[0158] In practical applications, BA can be used to optimize all keyframes and corresponding map points in the map.

[0159] Step 7: Global map construction and update.

[0160] In practical applications, map and semantic information can be integrated into a global map to form a complete environment model; then, a map can be built based on open-source tools such as Octomap and updated in real time to adapt to environmental changes.

[0161] In practical applications, semantic information can be fused with a local map, or a global map can be fused with semantic information. This fusion process ensures that the final generated global map contains semantic information.

[0162] In practical applications, during the execution of the above map building process, thread synchronization can be used to ensure synchronization between the target detection thread, semantic mapping thread, and SLAM thread, thereby achieving real-time data processing.

[0163] The object detection thread can be equivalent to the first thread in the embodiments of this application. The SLAM thread can be equivalent to the second thread in the embodiments of this application, and the SLAM thread may include... Figure 4 The semantic mapping thread includes a tracking thread, a local mapping thread, and a loop closure detection thread. The semantic mapping thread can be considered equivalent to the third thread in the embodiments of this application.

[0164] In this application embodiment, the real-time semantic segmentation model PP-Liteseg is integrated into the visual SLAM framework ORB-SLAM3. This enables the robot to perform synchronous localization based on sensor data and semantic mapping of the surrounding scene in unknown environments. The robot can understand the scene in a way that is close to human cognition during the map construction process, which enhances the robot's perception and understanding of the scene. On this basis, the map constructed by fusing 3D point cloud and semantic information can provide a more accurate and intuitive description of the scene. Compared with related technologies, this enhances the map's ability to describe the target scene.

[0165] Furthermore, in the application embodiments of this application, feature points corresponding to dynamic targets are removed by semantic information, which reduces the interference of dynamic targets on the map construction process. Compared with the map construction method based on static scene assumptions in related technologies, the accuracy of map construction is improved.

[0166] Furthermore, in the application embodiments of this application, by sequentially calling the GMS algorithm, LMP algorithm, and BANSAC algorithm during the construction of the matching point pair set, erroneous matching point pairs are eliminated, thereby improving the feature matching accuracy while ensuring real-time processing, and thus improving the accuracy of map construction.

[0167] Furthermore, the robot in this application embodiment represents a terminal in a cloud-edge-device architecture. This means the terminal is not only used for data collection but also for computational processing of the collected data during map construction. Compared to related technologies, this reduces data transmission requirements and processing latency while improving the utilization of edge computing resources. Moreover, the robot's core control module uses a high-performance embedded AI device, ensuring stable target detection and recognition tasks on the edge, thereby further improving operational stability and reducing data transmission requirements and processing latency.

[0168] In practical applications, the solution presented in this application can be applied to various indoor environmental monitoring fields, as well as inspection and automation fields. Exemplary applications include, but are not limited to, industrial automation, smart warehousing, smart homes, and security monitoring. In the field of industrial automation, it can improve production efficiency and safety. For example, by reducing manual inspection costs and improving the monitoring capabilities of the production process, it is expected to save enterprises significant operating costs while simultaneously improving production efficiency and product quality. In the field of smart warehousing, it can optimize inventory management and logistics scheduling. In the field of smart homes, it can enhance the intelligence level of the living environment. In the field of security monitoring, it can enhance the real-time performance and accuracy of monitoring.

[0169] Based on the embodiments described above, this application also provides a map building device applied to a first robot, see [link to documentation]. Figure 6 The device includes: The acquisition unit 61 is used to acquire first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot. The calling unit 62 is used to process the first video data and the inertial measurement data based on the first SLAM algorithm to obtain a first point cloud; and to call the first semantic segmentation model to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate the first detection target in the image frame; the first detection target represents the inspection target of the first robot in the first scene; The fusion unit 63 is used to fuse the first point cloud and the first semantic information to obtain a second point cloud, so as to construct a first map of the first scene based on the second point cloud.

[0170] In one embodiment, when the calling unit 62 processes the first video data and the inertial measurement data based on the first SLAM algorithm, it is used to: Feature points are extracted from the image frames in the first video data to obtain one or more feature points corresponding to the image frames; Based on the second semantic information, the first feature point is removed from the one or more feature points; the second semantic information is obtained by processing the image frames in the first video data by calling the first semantic segmentation model, and the second semantic information is used to indicate the second detection target in the image frame; the second detection target represents a dynamic target in the first scene, and the second detection target is different from the first detection target; the first feature point represents the feature point corresponding to the second detection target in the image frame; Based on the one or more feature points after removing the first feature point, the pose of the first robot is estimated, and the first point cloud is determined based on the obtained pose estimation result.

[0171] In one embodiment, the calling unit 62, based on second semantic information, removes a first feature point from the one or more feature points, including: Based on the second semantic information, a first image mask is determined; the first image mask is used to indicate the image region where the second detected target is located in the image frame; Based on the first image mask, a first feature point is removed from the one or more feature points; the first feature point represents a feature point within the image region indicated by the first image mask.

[0172] In one embodiment, when the calling unit 62 processes the first video data and the inertial measurement data based on the first SLAM algorithm, it is used to: Based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame, a set of matching point pairs is constructed; each matching point pair in the set is used to indicate a feature point in the first image frame and a feature point in the second image frame, and each matching point pair is used to indicate that the two feature points contained in the matching point pair correspond to the same region in the first scene; The GMS algorithm, LMP algorithm, and BANSAC algorithm are sequentially called to process the set of matching point pairs to remove the first matching point pair in the set of matching point pairs; the two feature points indicated by the first matching point pair correspond to different regions in the first scene; Based on the set of matching point pairs after removing the first matching point pair, pose estimation is performed on the first robot to determine the first point cloud based on the obtained pose estimation result.

[0173] In one embodiment, the fusion unit 63 performs fusion processing on the first point cloud and the first semantic information, including: Based on the first semantic information, the first detected target is marked in the first point cloud to obtain the second point cloud.

[0174] In one embodiment, the map building apparatus further includes a synchronization unit, the synchronization unit being used for: Synchronize the first thread, the second thread, and the third thread; among them, The first thread is used to call the first semantic segmentation model, the second thread is used to process based on the first SLAM algorithm, and the third thread is used to fuse the first point cloud with the first semantic information.

[0175] In one embodiment, the first robot represents a terminal in a cloud-edge-device architecture.

[0176] In practical applications, the acquisition unit 61, the calling unit 62, the fusion unit 63, and the synchronization unit can be implemented by the processor in the map building device.

[0177] It should be noted that the map building apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the map building apparatus and map building method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0178] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides a first robot, referring to... Figure 7 The first robot includes: Communication interface 1 enables information exchange with other devices; Processor 2 is connected to communication interface 1 to enable information interaction with other devices and, when running a computer program, executes the methods provided by one or more technical solutions in the above embodiments. The computer program is stored in memory 3.

[0179] Specifically, the processor 2 is used to collect first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot; Based on a first SLAM algorithm, the first video data and the inertial measurement data are processed to obtain a first point cloud; and a first semantic segmentation model is invoked to process image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate a first detected target in the image frame; the first detected target represents the inspection target of the first robot in the first scene; and... The first point cloud and the first semantic information are fused to obtain a second point cloud, and a first map of the first scene is constructed based on the second point cloud.

[0180] In one embodiment, when the processor 2 processes the first video data and the inertial measurement data based on the first SLAM algorithm, it is used to: Feature points are extracted from the image frames in the first video data to obtain one or more feature points corresponding to the image frames; Based on the second semantic information, the first feature point is removed from the one or more feature points; the second semantic information is obtained by processing the image frames in the first video data by calling the first semantic segmentation model, and the second semantic information is used to indicate the second detection target in the image frame; the second detection target represents a dynamic target in the first scene, and the second detection target is different from the first detection target; the first feature point represents the feature point corresponding to the second detection target in the image frame; Based on the one or more feature points after removing the first feature point, the pose of the first robot is estimated, and the first point cloud is determined based on the obtained pose estimation result.

[0181] In one embodiment, the processor 2, based on second semantic information, removes a first feature point from the one or more feature points, including: Based on the second semantic information, a first image mask is determined; the first image mask is used to indicate the image region where the second detected target is located in the image frame; Based on the first image mask, a first feature point is removed from the one or more feature points; the first feature point represents a feature point within the image region indicated by the first image mask.

[0182] In one embodiment, when the processor 2 processes the first video data and the inertial measurement data based on the first SLAM algorithm, it is used to: Based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame, a set of matching point pairs is constructed; each matching point pair in the set is used to indicate a feature point in the first image frame and a feature point in the second image frame, and each matching point pair is used to indicate that the two feature points contained in the matching point pair correspond to the same region in the first scene; The GMS algorithm, LMP algorithm, and BANSAC algorithm are sequentially called to process the set of matching point pairs to remove the first matching point pair in the set of matching point pairs; the two feature points indicated by the first matching point pair correspond to different regions in the first scene; Based on the set of matching point pairs after removing the first matching point pair, pose estimation is performed on the first robot to determine the first point cloud based on the obtained pose estimation result.

[0183] In one embodiment, the processor 2 performs a fusion process on the first point cloud and the first semantic information, including: Based on the first semantic information, the first detected target is marked in the first point cloud to obtain the second point cloud.

[0184] In one embodiment, the processor 2 is further configured to: Synchronize the first thread, the second thread, and the third thread; among them, The first thread is used to call the first semantic segmentation model, the second thread is used to process based on the first SLAM algorithm, and the third thread is used to fuse the first point cloud with the first semantic information.

[0185] In one embodiment, the first robot represents a terminal in a cloud-edge-device architecture.

[0186] It should be noted that the specific processing procedure of communication interface 1 can be understood by referring to the above method.

[0187] Of course, in practical applications, the various components of the first robot are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general will label all buses as Bus System 4.

[0188] In this embodiment, memory 3 is used to store various types of data to support operations in the first robot. Examples of such data include any computer programs used for operations on the first robot.

[0189] The methods disclosed in the embodiments of this application can be applied to the processor 2, or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 2 or by instructions in the form of software. The processor 2 mentioned above may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 3. The processor 2 reads the information in the memory 3 and combines its hardware to complete the steps of the aforementioned method.

[0190] In an exemplary embodiment, the first robot may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0191] It is understood that the memory 3 in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0192] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 that stores a computer program, which can be executed by the processor 2 of the first robot to complete the steps described in the aforementioned map construction method.

[0193] Computer-readable storage media can be FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0194] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by the processor 2 of a first robot to complete the steps described in the aforementioned map construction method.

[0195] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0196] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document refers to any combination of at least two of any one or more elements from a set of A, B, and C. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set of A, B, and C. Additionally, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expressions, such as one or more, at least one, or at least one of, etc.

[0197] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0198] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A map construction method, characterized in that, Applied to a first robot, the method includes: Collect first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot; Based on the first Simultaneous Localization and Mapping (SLAM) algorithm, the first video data and the inertial measurement data are processed to obtain a first point cloud; and a first semantic segmentation model is invoked to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate a first detected target in the image frame; the first detected target represents the inspection target of the first robot in the first scene; The first point cloud and the first semantic information are fused to obtain a second point cloud, and a first map of the first scene is constructed based on the second point cloud.

2. The method according to claim 1, characterized in that, When processing the first video data and the inertial measurement data based on the first SLAM algorithm, the method includes: Feature points are extracted from the image frames in the first video data to obtain one or more feature points corresponding to the image frames; Based on the second semantic information, the first feature point is removed from the one or more feature points; the second semantic information is obtained by processing the image frames in the first video data by calling the first semantic segmentation model, and the second semantic information is used to indicate the second detection target in the image frame; the second detection target represents a dynamic target in the first scene, and the second detection target is different from the first detection target; the first feature point represents the feature point corresponding to the second detection target in the image frame; Based on the one or more feature points after removing the first feature point, the pose of the first robot is estimated, and the first point cloud is determined based on the obtained pose estimation result.

3. The method according to claim 2, characterized in that, The step of removing the first feature point from one or more feature points based on the second semantic information includes: Based on the second semantic information, a first image mask is determined; the first image mask is used to indicate the image region where the second detected target is located in the image frame; Based on the first image mask, a first feature point is removed from the one or more feature points; the first feature point represents a feature point within the image region indicated by the first image mask.

4. The method according to claim 1, characterized in that, When processing the first video data and the inertial measurement data based on the first SLAM algorithm, the method includes: Based on the Hamming distance between the descriptors of feature points corresponding to the first image frame and the descriptors of feature points corresponding to the second image frame, a set of matching point pairs is constructed; each matching point pair in the set is used to indicate a feature point in the first image frame and a feature point in the second image frame, and each matching point pair is used to indicate that the two feature points contained in the matching point pair correspond to the same region in the first scene; The set of matching point pairs is processed sequentially by the grid-based motion statistics (GMS) algorithm, the local matching-preserving (LMP) algorithm, and the Bayesian adaptive consistent sampling (BANSAC) algorithm to remove the first matching point pair from the set of matching point pairs; the two feature points indicated by the first matching point pair correspond to different regions in the first scene. Based on the set of matching point pairs after removing the first matching point pair, pose estimation is performed on the first robot to determine the first point cloud based on the obtained pose estimation result.

5. The method according to claim 1, characterized in that, The process of fusing the first point cloud with the first semantic information includes: Based on the first semantic information, the first detected target is marked in the first point cloud to obtain the second point cloud.

6. The method according to claim 1, characterized in that, The method further includes: Synchronize the first thread, the second thread, and the third thread; among them, The first thread is used to call the first semantic segmentation model, the second thread is used to process based on the first SLAM algorithm, and the third thread is used to fuse the first point cloud with the first semantic information.

7. The method according to claim 1, characterized in that, The first robot represents the terminal in the cloud-edge-device architecture.

8. A map building device, characterized in that, Applied to the first robot, including: The acquisition unit is used to acquire first video data and corresponding inertial measurement data of the first scene; the first scene represents the inspection scene of the first robot. The calling unit is used to process the first video data and the inertial measurement data based on the first SLAM algorithm to obtain a first point cloud; and to call the first semantic segmentation model to process the image frames in the first video data to obtain first semantic information; the first semantic information is used to indicate the first detection target in the image frame; the first detection target represents the inspection target of the first robot in the first scene; The fusion unit is used to fuse the first point cloud and the first semantic information to obtain a second point cloud, so as to construct a first map of the first scene based on the second point cloud.

9. A first robot, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images