An electronic map updating method, computer device and storage medium

Abstract

The application discloses an electronic map updating method, a computer device and a storage medium, and can avoid comprehensive semantic division on an original point cloud data set covering the whole scene, can significantly reduce the calculation overhead, and is beneficial to maintaining good real-time performance when performing electronic map updating in a complex and dynamically changing construction site and the like; through semantic labeling and optimization of a region of interest on the original point cloud data set, the obtained prior semantic map has accurate semantic labeling information, can quickly determine a region that may need to be updated in the prior semantic map according to semantic information of a real-time image when updating according to the real-time image, reduces the possibility of globally searching the prior semantic map, can significantly reduce the calculation overhead, improves the speed and real-time performance of the electronic map updating, and improves the navigation efficiency and use safety of an execution subject in the scene. The application is widely applied to the technical field of electronic maps.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic map, and particularly relates to an electronic map updating method, a computer device and a storage medium. BACKGROUND

[0002] SLAM (Simultaneous Localization And Mapping) is a technology that can simultaneously realize localization and map construction. SLAM can realize self-localization and navigation of an autonomous driving vehicle by moving from an unknown position in an unknown environment, performing self-localization according to a position offset and map data during the movement, and constructing an incremental map on the basis of the self-localization, thereby being applied to aspects such as construction site inspection, autonomous driving, scene surveying and virtual reality.

[0003] The key technology of SLAM is to rely on a mobile object such as an autonomous driving vehicle carrying a sensor to realize the construction of a surrounding map while moving. However, the current implementation of the SLAM technology requires a large amount of computing resources, and the real-time performance of the system is often affected, especially when processing high-density point cloud data, and there are still significant challenges between ensuring real-time performance and accuracy. SUMMARY

[0004] In view of the technical problems such as poor real-time performance of the current SLAM technology, the present application aims to provide an electronic map updating method, a computer device and a storage medium.

[0005] In one aspect, an embodiment of the present application includes an electronic map updating method, which comprises the following steps:

[0006] In a first time period, original point cloud data set and image information of a scene are acquired;

[0007] According to the image information, semantic labeling and optimization of a region of interest are performed on the original point cloud data set to obtain a prior semantic map;

[0008] In a second time period, real-time images of the scene are acquired; the second time period is a time period after the first time period;

[0009] According to the real-time images, the prior semantic map is updated.

[0010] Further, the step of performing semantic labeling and optimization of a region of interest on the original point cloud data set according to the image information to obtain a prior semantic map comprises:

[0011] Target detection is performed on the original point cloud data set to determine at least one region of interest;

[0012] semantically label a point cloud subset in the region of interest, so as to determine a corresponding local semantic space in the original point cloud dataset; the local semantic space comprises the point cloud subset in the region of interest and a semantic label obtained by semantic labeling.

[0013] Further, the semantic labeling and optimization of the region of interest on the original point cloud dataset according to the image information to obtain the prior semantic map further comprises:

[0014] According to the semantic category, the local semantic space with the corresponding semantic label is cut out from the original point cloud dataset to obtain the prior semantic map.

[0015] Further, the updating of the prior semantic map according to the real-time image comprises:

[0016] Registering the prior semantic map;

[0017] Determining a target semantic;

[0018] According to the target semantic, detecting the real-time image and the registered prior semantic map;

[0019] When no part corresponding to the target semantic is detected from the real-time image, and no point cloud corresponding to the target semantic is detected from the prior semantic map, the prior semantic map remains unchanged;

[0020] When no part corresponding to the target semantic is detected from the real-time image, and point cloud corresponding to the target semantic is detected from the prior semantic map, the detected point cloud in the prior semantic map is deleted.

[0021] Further, the updating of the prior semantic map according to the real-time point cloud further comprises:

[0022] In a second time period, real-time point cloud of a scene is obtained;

[0023] According to the target semantic, detecting the real-time point cloud and the prior semantic map;

[0024] When point cloud corresponding to the target semantic is detected from the real-time point cloud, and no point cloud corresponding to the target semantic is detected from the prior semantic map, a 3D detection box is generated according to the point cloud detected from the real-time point cloud, and the prior semantic map is updated according to the 3D detection box;

[0025] When a point cloud corresponding to the target semantic is detected from the real-time point cloud and a point cloud corresponding to the target semantic is detected from the prior semantic map, a position change of the point cloud detected from the real-time point cloud and the point cloud detected from the prior semantic map is detected, when the position change is detected, a 3D detection box is generated according to the point cloud detected from the real-time point cloud, the prior semantic map is updated according to the 3D detection box, when the position change is not detected, the prior semantic map is kept unchanged.

[0026] Further, the detection of the real-time point cloud and the prior semantic map according to the target semantic comprises:

[0027] performing three-dimensional target detection on the prior semantic map according to the target semantic to determine a bounding volume;

[0028] performing target detection on the real-time image according to the target semantic to determine a target region;

[0029] projecting the target region to the real-time point cloud;

[0030] clustering the points projected in the real-time point cloud to obtain at least one point cloud cluster;

[0031] screening each point cloud cluster;

[0032] using the screened point cloud cluster as the detected point cloud from the real-time point cloud.

[0033] Further, the screening of each point cloud cluster comprises:

[0034] respectively acquiring a confidence of each point cloud cluster;

[0035] screening the point cloud cluster with a confidence higher than a confidence threshold.

[0036] Further, the acquisition of the confidence of each point cloud cluster comprises:

[0037] for any point cloud cluster, according to the formula

[0038]

[0039] performing calculation, wherein C represents the confidence of the point cloud cluster, N points is the number of points in the point cloud cluster, d mean is the average distance between the point cloud cluster and the center of the first detection component, V is the volume of the point cloud cluster, and a and β are weight parameters.

[0040] In another aspect, the embodiments of the present application further include a computer device comprising a memory and a processor, the memory being configured to store at least one program, and the processor being configured to load the at least one program to execute the electronic map updating method in the embodiments.

[0041] In another aspect, the embodiments of the present application further include a computer readable storage medium having stored therein a program executable by a processor, the program executable by the processor when executed by the processor being configured to execute the electronic map updating method in the embodiments.

[0042] The electronic map updating method in the embodiments can avoid comprehensive semantic division on the original point cloud data set covering the entire scene, so as to significantly reduce the computational overhead and facilitate maintaining good real-time performance when performing electronic map updating in complex and dynamically changing construction sites and the like; through semantic labeling and optimization of the region of interest of the original point cloud data set, the obtained prior semantic map has accurate semantic labeling information, which can quickly determine the region in the prior semantic map that may need to be updated according to the semantic information of the real-time image when updating according to the real-time image, thereby reducing the possibility of global search on the prior semantic map and significantly reducing the computational overhead, improving the speed and real-time performance of electronic map updating, and improving the navigation efficiency and use safety of the execution subject such as a robot or a car in the scene. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 a schematic diagram of the steps of the electronic map updating method in the embodiments;

[0044] Figure 2 a schematic diagram of the system to which the electronic map updating method in the embodiments can be applied;

[0045] Figure 3 a schematic diagram of the principle of the electronic map updating method in the embodiments. DETAILED DESCRIPTION

[0046] TERMS EXPLANATION:

[0047] 1. Laser SLAM (Simultaneous Localization and Mapping)

[0048] Definition: Laser SLAM can be divided into 2D-SLAM and 3D-SLAM according to the dimension. Laser radar can provide high-precision distance data, so that laser SLAM has high accuracy and stability in positioning and mapping in complex environments.

[0049] 2. Dynamic SLAM

[0050] Definition: Dynamic SLAM refers to the technology that allows robots to perceive, recognize, and track dynamic objects such as pedestrians, vehicles, etc. in real-time in an environment with dynamic objects, and update and optimize the map accordingly.

[0051] 3. Semantic Information

[0052] Definition: Refers to information that has meaning or function in the environment, such as objects, roads, traffic rules, etc. This information can be used to explain the meaning and role of these objects in the real world. For example, a set of information x, y, z, r, g, b, label describing a point cloud, x, y, z represents the coordinates of the point cloud, r, g, b represents the color of the point cloud in the RGB channel, and label is semantic information used to represent what object this point cloud belongs to, etc.

[0053] 4. Prior Semantic Map

[0054] Definition: Semantic map is a map representation that combines geographic information and semantic information. Prior semantic map refers to a map containing semantic information obtained by annotating the target area with a 3D target detection model when building the initial global map.

[0055] Purpose: As a reference and reference for subsequent map updating and path planning of SLAM system.

[0056] 5. Semantic Segmentation

[0057] Definition: Semantic segmentation is a technique in computer vision that aims to classify each pixel in an image or video into one of the predefined categories, such as roads, vehicles, pedestrians, etc.

[0058] Purpose: In the SLAM system, semantic segmentation can be used to distinguish dynamic objects from static environment, thereby improving the robustness and accuracy of SLAM algorithm in dynamic environment.

[0059] Current SLAM technology usually adopts a global reconstruction strategy when updating its electronic map. This process not only leads to excessive consumption of computing resources, but also significantly increases communication burden, thereby negatively affecting overall running efficiency, resulting in low overall running efficiency and further reducing the real-time performance of electronic map updating.

[0060] Based on the above principles, in the present embodiment, an electronic map updating method is provided. Referring to Figure 1 , the electronic map updating method includes the following steps:

[0061] S1. Obtain the original point cloud data set and image information of the scene in the first time period;

[0062] S2. Based on the image information, perform semantic annotation and optimization of the region of interest in the original point cloud dataset to obtain a priori semantic map;

[0063] S3. During the second time period, acquire real-time images of the scene;

[0064] S4. Update the prior semantic map based on the real-time images.

[0065] The electronic map update method in this embodiment can be applied to... Figure 2 The system shown can specifically be a car or a robot. In this embodiment, a robot transporting materials at a construction site is used as an example. Figure 2 The system will be illustrated with examples.

[0066] Reference Figure 2 The system includes a control module, a first detection component, and a second detection component. In this embodiment, the first detection component may specifically be a lidar, and the second detection component may specifically be a camera that captures images using visible light. The control module can invoke either the first or second detection component to detect the external environment, where the data detected by the first detection component is point cloud data, and the data detected by the second detection component is a visible light image. Specifically, the control module can execute the various steps in the electronic map update method.

[0067] The principle of the electronic map update method in this embodiment is as follows: Figure 3 As shown.

[0068] In step S1, during the first time period T1, the control module calls the first detection component, namely the LiDAR, to detect the scene where the robot is located, namely the construction site, thereby obtaining the original point cloud dataset P = {p1, p2, ..., p n}, where p i This represents the data at the i-th spatial location obtained by the first detection component through laser detection; n such data points form a point cloud.

[0069] In step S1, the control module calls the second detection component, namely the camera, to take pictures of the scene where the robot is located, namely the construction site, within the first time period T1, thereby obtaining image information.

[0070] In step S2, the control module performs semantic annotation and optimization of the regions of interest in the original point cloud dataset P based on the image information to obtain a priori semantic map P. semanteme Prior semantic map P semanteme It is also in the form of point cloud data, some of which has been semantically annotated, so it contains both point cloud information and semantic information.

[0071] In step S3, during the second time period T2 following the first time period T1, the control module can invoke the second detection component to detect the scene, thereby obtaining a real-time image. realtime The control module can also detect real-time point cloud P realtime Simultaneously, during the second time period T2, the first detection component, specifically the lidar, is invoked to detect the scene, thereby obtaining real-time point cloud P. realtime .

[0072] The control module then executes step S4, based on the image. realtime Determine the prior semantic map P semanteme The relevant part of the map P indicates whether the target has changed (e.g., the target object existed in the first time period T1, but no longer exists in the second time period T2; or the target object did not exist in the first time period T1, but exists in the second time period T2). If it has changed, then the prior semantic map P is updated. semanteme Modify the relevant parts of the map; if no changes occur, then retain the prior semantic map P. semanteme The prior semantic map P remains unchanged, thus ensuring that the prior semantic map P remains unchanged. semanteme The points at each location in the map represent the existence of objects, which are consistent with the latest detected results in the second time period T2, thus updating the prior semantic map.

[0073] In this embodiment, the principle of executing steps S1-S4 is as follows: the prior semantic map is obtained by semantically annotating and optimizing the regions of interest in the original point cloud dataset. This avoids the need for comprehensive semantic division of the original point cloud dataset covering the entire scene, thereby significantly reducing computational overhead and facilitating good real-time performance when updating electronic maps in complex and dynamically changing scenarios such as construction sites. By semantically annotating and optimizing the regions of interest in the original point cloud dataset, the resulting prior semantic map contains accurate semantic annotation information. When updating based on real-time images, it can quickly determine the regions in the prior semantic map that may need to be updated based on the semantic information of the real-time images, thereby reducing the possibility of needing to perform a global search on the prior semantic map. This also significantly reduces computational overhead, improves the speed and real-time performance of electronic map updates, and enhances the navigation efficiency and safety of robots or vehicles in the scene.

[0074] In this embodiment, when performing step S2, which involves semantic annotation and optimization of the region of interest in the original point cloud dataset based on image information to obtain a priori semantic map, the following steps can be performed:

[0075] S201. Target detection is performed on the original point cloud dataset to determine at least one region of interest;

[0076] S202. The point cloud subset in the region of interest is semantically labeled, thereby determining the corresponding local semantic space in the original point cloud dataset;

[0077] S203. According to the semantic category, the local semantic space with the corresponding semantic label is cut from the original point cloud dataset, and a prior semantic map is obtained.

[0078] In step S201, the control module can run a 3D target detection model (such as PointRCNN, etc.) to perform target detection on the original point cloud dataset P = {p1, P2, …, p n} to determine at least one region of interest, each region of interest having a corresponding bounding box, for example, the jth region of interest has a bounding box B j , and the point cloud in the original point cloud dataset P in the bounding box B j forms a point cloud subset P B = {p i ∈P|p i ∈B j}.

[0079] In step S202, the control module can automatically perform semantic recognition on the point cloud subset P B , or manually input the semantics of the point cloud of the point cloud subset P B , thereby performing semantic labeling on the point cloud subset P B , so that the point cloud subset P B obtains a corresponding semantic label l i . The semantic label l i represents the object category (such as a stool, a table, etc.) and other information represented by the point cloud in the point cloud subset P B . The point cloud subset P B and the corresponding semantic label l i form a local semantic space S B , i.e., S B = {(p i , l i )|p i ∈P B}.

[0080] In step S203, the control module can define a screening function f(d i , l j ), which defines that if the semantic category d i corresponds to the semantic label l i (which can be the same, for example, the semantic category d i and the semantic label L iIf the content is all "stool", then f(d) i ,l j If the semantic category d = 1, then... i With semantic tags i Incompatible (e.g., different content, such as semantic category d) i The content is "stool", semantic tag l i If the content is "table", then f(d) i ,l j = 0. Iterate through all i and j, and find the semantic labels l that make the filtering function f equal to 1. i Filter out the semantic tags. i The corresponding local semantic space S B The local semantic space is cut out from the original point cloud dataset P and labeled as follows:

[0081] In this embodiment, for each segmented local semantic space A certain extended region buffer can be set for it, in the local semantic space Based on this, the buffer area is extended outward to obtain... That is In this embodiment, This indicates an expansion operation; the buffer can be 0, meaning it does not affect the local semantic space. The process is then expanded. The entire segmented local semantic space and its expanded regions constitute the prior semantic map.

[0082] In this embodiment, before executing step S4 (specifically, before executing steps S1-S4, or after executing S1-S3 and before executing step S4), the following steps can be used to construct a nonlinear optimization problem based on constraints such as the relative position and orientation between the first detection component (LiDAR) and the second detection component (camera), solve for the external calibration parameters, and thus register the first detection component (LiDAR) and the second detection component (camera):

[0083] (1) The coordinate system of the point detected by the first detection component (LiDAR) transformed to that of the second detection component (camera) satisfies:

[0084]

[0085] in Let i be the point detected by the first detection component (LiDAR). To be The point obtained by transforming into the coordinate system of the second detection component (camera) is (R, T), which is the transformation matrix from the first detection component (laser radar) to the second detection component (camera) ;

[0086] (2) For The point represented by that point can be directly photographed by the second detection component (camera) to obtain the corresponding pixel point which is still a point in three-dimensional space. The second detection component (camera) can be calibrated for internal parameters, or the internal camera matrix K of the second detection component (camera) can be obtained from the product parameters of the second detection component (camera), wherein the internal camera matrix K can be represented in the form of

[0087]

[0088]

[0089] (3) Using the internal camera matrix K, project onto the image plane to satisfy

[0090]

[0091] The normalized can be obtained Specifically, the can be represented in the form of

[0092]

[0093] wherein u, v and w represent the components of , T represents the transpose of the matrix, and is calculated according to the formula

[0094]

[0095]

[0096] (4) After calculating , the following formula is used for calculation:

[0097]

[0098] wherein represents calculated according to (R, T) before optimization represents the re-projection error between the first detection component and the second detection component, and N represents the number of points detected by the first detection component (laser radar) and the second detection component (camera) ;

[0099] ​(5) Using the non-linear least squares method Levenberg-Marquardt optimization algorithm, (R, T) is gradually optimized. In the first step of optimization, an initial value R0 can be estimated for R, and an initial value T0 can be estimated for T.

[0100] The matrix T and the external calibration parameter R obtained by optimizing and solving according to the steps (1)-(5) above can be used to determine the transformation matrix [R, T], which can convert the coordinates of the point detected by the first detection component (laser radar) into the coordinates of the same point detected by the second detection component (camera); similarly, [R, T] -1 can be obtained according to the transformation matrix [R, T] -1 , which can convert the coordinates of the point detected by the second detection component (camera) into the coordinates of the same point detected by the first detection component (laser radar).

[0101] In this embodiment, when performing step S4, that is, updating the prior semantic map according to the real-time image, the following steps can be performed:

[0102] S401. Registering the prior semantic map;

[0103] S402. Determining the target semantics;

[0104] S403. Detecting the real-time image and the registered prior semantic map according to the target semantics;

[0105] S404. When no part corresponding to the target semantics is detected from the real-time image, and no point cloud corresponding to the target semantics is detected from the prior semantic map, keeping the prior semantic map unchanged;

[0106] S405. When no part corresponding to the target semantics is detected from the real-time image, and a point cloud corresponding to the target semantics is detected from the prior semantic map, deleting the detected point cloud in the prior semantic map;

[0107] S406. Detecting the real-time point cloud and the prior semantic map according to the target semantics;

[0108] S407. When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and no point cloud corresponding to the target semantics is detected from the prior semantic map, generating a 3D detection box according to the detected point cloud from the real-time point cloud, and updating the prior semantic map according to the 3D detection box;

[0109] S408. When a point cloud corresponding to the target semantics is detected from the real-time point cloud, and a point cloud corresponding to the target semantics is detected from the prior semantic map, detecting the position change of the detected point cloud from the real-time point cloud and the detected point cloud from the prior semantic map.

[0110] S409. When a position change is detected, generate a 3D detection box based on the point cloud detected from the real-time point cloud, and update the prior semantic map based on the 3D detection box;

[0111] S410. When no location change is detected, keep the prior semantic map unchanged.

[0112] In S401, a rigid transformation matrix T can be used. * The prior semantic map obtained in step S2 is registered. In this embodiment, the rigid transformation matrix T can be determined through the following steps. * Specific value:

[0113] (A) Using a rigid transformation matrix T * The local semantic space segmented after the expansion of execution steps S201-S203 Perform registration;

[0114] (B) Using the expanded, segmented local semantic space The optimization objective is to minimize the matching error between the registered and unregistered point cloud data. An optimization equation is then established; specifically, the optimization equation is as follows:

[0115]

[0116] Among them, T * ·p k Indicates the use of a rigid transformation matrix T * The segmented local semantic space point p in k Perform registration, |T * ·p k -p k | 2 Represents the segmented local semantic space point p in k The registered numerical value T * ·p k Compared with the value p before registration k Matching error;

[0117] (C) Solve the optimization equation in step (B), for the rigid transformation matrix T. * Optimize the solution.

[0118] By performing steps (A)-(C), the rigid transformation matrix T can be determined. * The specific value. Using the rigid transformation matrix T. * Register the points in the prior semantic map.

[0119] In step S402, the control module can use the original point cloud dataset P obtained in step S1 and the target object, such as a "stool", existing in the image information image to obtain the semantics of the target object, thereby obtaining the target semantics.

[0120] In step S403, the control module detects the real-time image and the prior semantic map respectively with the detection target being "whether there is a point cloud (or part of the image) corresponding to the target semantics (for example, a stool)".

[0121] Through the execution of step S403, the possible detection results include:

[0122] ① No part corresponding to the target semantics is detected from the real-time image, and no point cloud corresponding to the target semantics is detected from the prior semantic map.

[0123] ② No part corresponding to the target semantics is detected from the real-time image, and a point cloud corresponding to the target semantics is detected from the prior semantic map.

[0124] If the detection result of step S403 is ①, for example, the real-time image image realtime detected in the second time period T2 does not contain a part with the target semantics of "stool", and the prior semantic map P semanteme detected in the first time period T1 does not contain a point cloud with the target semantics of "stool", it indicates that in the scene where the robot is located, there is no target object of "stool" from the first time period T1 to the second time period T2, then step S404 is executed, it is considered that the scene does not change, and the prior semantic map P semanteme is kept unchanged.

[0125] If the detection result of step S403 is ②, for example, the real-time image image realtime detected in the second time period T2 does not contain a part with the target semantics of "stool", but the prior semantic map P semanteme detected in the first time period T1 contains a point cloud with the target semantics of "stool", it indicates that in the scene where the robot is located, there is a target object of "stool" in the first time period T1, but there is no target object of "stool" in the second time period T2, then step S405 is executed, it is considered that the target object of "stool" in the scene is removed, and the detected point cloud in the prior semantic map P semanteme is deleted.

[0126] In this embodiment, steps S403-S405 are steps of roughly detecting and updating the prior semantic map, and the data processing amount required by steps S404-S405 is relatively small, so that the fast updating of the prior semantic map can be realized.

[0127] In this embodiment, by performing step S403, the possible detection result can still be "detecting a part corresponding to the target semantics from the real-time image", at this time, step S406 can be performed.

[0128] In this embodiment, before performing step S406, the non-ground points and ground points in the points in the real-time point cloud P realtime are further screened, and only the non-ground points in the real-time point cloud P realtime are processed in subsequent steps S406-S410.

[0129] Specifically, the RANSAC algorithm combined with the region growing algorithm can be used to fit the points in the real-time point cloud P realtime , so as to determine the points in the real-time point cloud P realtime corresponding to the ground plane, obtain the equation of the ground plane, if the coordinates of a point in the real-time point cloud P realtime satisfy the equation of the ground plane, it is determined that the point is on the ground plane, that is, the point belongs to the ground point; if the coordinates of a point in the real-time point cloud P realtime do not satisfy the equation of the ground plane, it is determined that the point is not on the ground plane, that is, the point belongs to the non-ground point. The ground points can be deleted from the real-time point cloud P realtime before performing step S406, only the non-ground points are retained. In this embodiment, the real-time point cloud retaining only the non-ground points can still be marked as P realtime .

[0130] Since the ground points have a relatively low possibility of containing effective semantic information, by deleting the ground points in the real-time point cloud P realtime , the data processing amount required for map updating can be reduced without affecting the effect of map updating.

[0131] In step S406, similar to step S403, the control module takes "whether there is a point cloud corresponding to the target semantics (for example, a stool)" as the detection target, and respectively detects the real-time point cloud and the prior semantic map.

[0132] By performing step S406, the possible detection results include:

[0133] ③Detecting a point cloud corresponding to the target semantics from the real-time point cloud, and not detecting a point cloud corresponding to the target semantics from the prior semantic map;

[0134] ④Detecting a point cloud corresponding to the target semantics from the real-time point cloud, and detecting a point cloud corresponding to the target semantics from the prior semantic map.

[0135] If the detection result of step S406 is ③, for example, the real-time point cloud P realtime detected in the second time period T2 contains point clouds belonging to the target object of "stool", while the prior semantic map P semanteme detected in the first time period T1 does not contain point clouds belonging to the target object of "stool", it indicates that the scene where the robot is located has no target object of "stool" in the first time period T1, and the target object of "stool" appears in the second time period T2, then step S407 is executed, considering that the target object of "stool" appears in the scene, and the control module generates a 3D detection box according to the point cloud detected from the real-time point cloud P realtime , that is, the point cloud corresponding to the target object of "stool", and updates the prior semantic map P semanteme according to the 3D detection box. For example, the control module can map the 3D detection box to the prior semantic map P semanteme , and label the mapped part of the prior semantic map P semanteme with semantic information such as "stool" to indicate the target object of "stool" and its position in the prior semantic map P semanteme .

[0136] If the detection result of step S406 is ④, for example, the real-time point cloud P realtime detected in the second time period T2 contains point clouds belonging to the target object of "stool", while the prior semantic map P semanteme detected in the first time period T1 also contains point clouds belonging to the target object of "stool", it indicates that the target object of "stool" has existed in the scene where the robot is located from the first time period T1 to the second time period T2, then step S408 is executed to determine whether the position of the point cloud belonging to the target object of "stool" detected from the real-time point cloud P realtime has changed compared with the point cloud belonging to the target object of "stool" detected from the prior semantic map P semanteme .

[0137] Specifically, in step S408, the point cloud belonging to the target object of "stool" detected from the prior semantic map P semanteme is detected using a 3D target recognition algorithm such as PointRCNN, and the detected points form an oriented bounding box (OBB), and the point cloud in the oriented bounding box OBB is the point cloud belonging to the target object of "stool"; on the other hand, a target detection algorithm such as YOLOv5 can be used to recognize target objects such as stools and tables from the real-time image image realtime , and these target objects in the real-time image image realtimeThe regions in the real-time image image -1 are target regions. In this embodiment, each target region has a bounding box. Then, the target regions identified in the real-time image image realtime are projected into the coordinate system in which the real-time point cloud P realtime is located, so that some points in the real-time point cloud P realtime will be projected. Then, the control module can run a clustering algorithm such as DBSCAN to cluster the part of the real-time point cloud P realtime that is projected, thereby obtaining multiple point cloud clusters. The point cloud clusters that are clustered into the same class correspond to the same target object (for example, both belong to the points detected by the first detection component, i.e., the laser radar, for the target object “stool”), and the point cloud clusters that are clustered into different classes correspond to different target objects. In this embodiment, for any point cloud cluster formed by clustering the real-time point cloud P realtime , the confidence C of the point cloud cluster can be calculated according to the formula

[0138]

[0139] . In the formula, N points is the number of points in the point cloud cluster, d mean is the average distance between all points in the point cloud cluster and the center of the first detection component (laser radar), and V is the volume of the point cloud cluster. α and β are weight parameters that can balance the influence of the point density and the volume of the point cloud cluster on the confidence.

[0140] After the confidence of each point cloud cluster is calculated, a fixed confidence threshold can be set. The confidence higher than the confidence threshold is determined as high confidence, and the confidence lower than or equal to the confidence threshold is determined as low confidence. The point cloud clusters with high confidence are selected to form the point cloud belonging to the target object “stool”. In this way, the entire real-time point cloud P realtime is simplified to the point cloud belonging to the target object “stool” in the real-time point cloud P realtime , and other parts of the real-time point cloud P realtime do not need to be processed, which is beneficial to improve the data processing efficiency.

[0141] In step S408, the control module can compare the point cloud belonging to the target object “stool” obtained by clustering the real-time point cloud P realtime with the oriented bounding box (OBB) in the prior semantic map P semanteme . The change is evaluated by judging whether a point q in the point cloud belonging to the target object “stool” obtained by clustering the real-time point cloud P realtime is located in the OBB. The points q located in the OBB are counted, and the counted points q account for the real-time point cloud P realtimeThe proportion of the number of all points of the point cloud belonging to the target object "stool" obtained by clustering. The proportion is compared with a threshold, and there are two cases:

[0142] If the proportion is greater than the threshold, it is determined that the target object "stool" has a position change, and step S409 is executed. The control module updates the prior semantic map P realtime according to the 3D detection box generated by the point cloud obtained by clustering (i.e., the point cloud corresponding to the target object "stool"). semanteme For example, the control module can map the 3D detection box to the prior semantic map P semanteme , and mark the mapped part of the prior semantic map P semanteme with semantic information such as "stool" to indicate the existence of the target object "stool" and its position in the prior semantic map P semanteme , and delete the detected point cloud in the prior semantic map P semanteme to indicate that the target object "stool" no longer exists in the original position.

[0143] If the proportion is less than or equal to the threshold, it is determined that the target object "stool" has no position change, and step S410 is executed. It is considered that the scene has not changed, and the prior semantic map P semanteme is maintained unchanged.

[0144] Steps S406-S410 are steps of detailed detection and update of the prior semantic map. Through the division of rough detection and update (steps S403-S405) and detailed detection and update (steps S406-S410), it can be realized that only rough detection and update steps need to be performed in a specific case, for example, only part of the prior semantic map P semanteme is deleted or maintained unchanged, so that detailed detection and update steps do not need to be performed, reducing the amount of data processing required. Moreover, the execution speed of the rough detection and update steps is generally faster than that of the detailed detection and update steps, so the processing speed can be improved.

[0145] In this embodiment, steps S403-S410 dynamically update the prior semantic map P semanteme through the "add, delete, and modify" strategy, which can guarantee the accuracy and timeliness of the prior semantic map P semanteme .

[0146] In this embodiment, after updating the prior semantic map P semanteme , the prior semantic map P semantemeThe target objects in the image are divided into three categories: static objects, semi-static objects, and dynamic objects, and corresponding processing measures and safety measures in path planning are formulated. For example, static objects, semi-static objects, and dynamic objects can be managed hierarchically, and objects of corresponding levels are selected for path planning according to tasks of different danger levels, so as to improve the safety and efficiency of navigation.

[0147] In the electronic map updating method in this embodiment, the semantic annotation process is limited to the target region of interest when the map is semantically labeled, avoiding full-coverage semantic annotation processing of the entire scene, thereby significantly reducing the computational complexity. By comparing the target objects appearing in the real-time point cloud with the information in the prior semantic map, the current position of the robot is located, and the semantic information is combined with the image recognition result to optimize the computational efficiency and accuracy of map updating by narrowing the search space and targeted area screening. The effects of the electronic map updating method in this embodiment include:

[0148] High efficiency: The semantic map construction method based on 3D target detection greatly reduces the amount of point cloud data processed, significantly reducing the computational burden. At the same time, the semantic-driven multi-stage map dynamic updating strategy effectively improves the efficiency of the system, especially in handling dynamic changing building environments.

[0149] Higher safety: The system classifies and manages objects in the map based on object recognition results, and selects objects of corresponding levels for path planning according to tasks of different danger levels.

[0150] Significant reduction in resource consumption and improvement in real-time performance: The system only updates the changed part, and after obtaining the prior semantic map, the semantic-guided point cloud efficient segmentation based on target detection and the semantic-driven multi-stage map dynamic updating strategy are combined to update the map in real time. Through this method, the computational overhead is significantly reduced, and the real-time performance of the system is improved.

[0151] The steps S1-S4 in this embodiment are executed in real time. For example, the material transportation robot in the construction site scene can execute steps S1-S2 to obtain the prior semantic map in the first time period T1 (such as the period when the daily project is about to start), and the prior semantic map can be used as the latest global map in the daily scene. In the second time period T2, for example, when the robot is transporting materials, high real-time performance is required, and steps S3-S4 can be executed to combine the semantic-guided point cloud efficient segmentation based on target detection and the semantic-driven multi-stage map dynamic updating strategy, thereby updating the map in real time based on the prior semantic map, significantly reducing the computational overhead of the robot, improving the real-time performance of the robot, and effectively improving the navigation and work efficiency of the robot in the construction scene.

[0152] The electronic map updating method in the embodiments can be implemented by writing a computer program for executing the electronic map updating method, writing the computer program into a computer device or a storage medium, and executing the electronic map updating method when the computer program is read out and run, thereby achieving the same technical effects as the electronic map updating method in the embodiments.

[0153] It should be noted that, unless otherwise specified, when a certain feature is referred to as being "fixed", "connected" to another feature, it can be directly fixed, connected to the other feature, or indirectly fixed, connected to the other feature. In addition, the up, down, left, right and the like used in the disclosure are only relative to the relative positional relationship of the components of the disclosure in the drawings. The singular forms "a" and "the" used in the disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used in the embodiments have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments are only used to describe the specific embodiments, and are not intended to limit the present application. The term "and / or" used in the embodiments includes any combination of one or more related listed items.

[0154] It should be understood that although the terms first, second, third, etc. can be used in the present disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one type of element from another type of element. For example, without departing from the scope of the present disclosure, a first element can also be referred to as a second element, and similarly, a second element can also be referred to as a first element. The use of any and all examples or exemplary language (e.g., "for example", "as such", etc.) provided in the present embodiments is intended merely to better illustrate the present embodiments and should not be construed as imposing a limitation on the scope of the present application unless otherwise required.

[0155] It should be recognized that embodiments of the present application can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods can be implemented in a computer program configured with a non-transitory computer readable storage medium, in which the storage medium so configured causes a computer to operate in a specific and predefined manner according to the methods described in the specific embodiments and the accompanying drawings. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed special-purpose integrated circuit for this purpose.

[0156] Further, the operations of the processes described in this embodiment can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described in this embodiment (or variations and / or combinations thereof) can be implemented under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications), by hardware, or combinations thereof. The computer programs include a plurality of instructions that are executable by one or more processors.

[0157] Further, the methods can be implemented in any suitable type of computing platform operatively coupled to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present invention can be implemented in machine readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. Further, the machine readable code, or portions thereof, can be transmitted over wired or wireless networks. The present invention encompasses these and other different types of non-transitory computer readable storage media when the instructions or programs implementing the above steps are included in conjunction with a microprocessor or other data processor. The present invention also encompasses the computer itself when programmed in accordance with the methods and techniques of the present invention.

[0158] The computer programs are capable of applying to input data to perform the functions of the present embodiment, thereby transforming the input data to generate output data that is stored to non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In preferred embodiments of the present invention, the transformed data represents a physical and tangible object, including a particular visual depiction of the physical and tangible object produced on a display.

[0159] The above merely preferred embodiments of the present invention, and the present invention is not limited to only the above-described embodiments, as long as the same means achieve the technical effects of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. The technical solutions and / or embodiments of the present invention can have various modifications and changes within the scope of protection.

Claims

1. An electronic map updating method characterized by comprising: The electronic map updating method comprises: in a first time period, obtaining an original point cloud data set and image information of a scene; according to the image information, performing semantic labeling and optimization of a region of interest on the original point cloud data set to obtain a prior semantic map; in a second time period, obtaining real-time images of the scene; the second time period is a time period after the first time period; updating the prior semantic map according to the real-time images; the updating of the prior semantic map according to the real-time images comprises: registering the prior semantic map; determining a target semantic; detecting the real-time images and the registered prior semantic map according to the target semantic; when no part corresponding to the target semantic is detected from the real-time images and no point cloud corresponding to the target semantic is detected from the prior semantic map, keeping the prior semantic map unchanged; when no part corresponding to the target semantic is detected from the real-time images and a point cloud corresponding to the target semantic is detected from the prior semantic map, deleting the detected point cloud in the prior semantic map; in the second time period, obtaining real-time point clouds of the scene; performing three-dimensional target detection on the prior semantic map according to the target semantic to determine a bounding box; performing target detection on the real-time images according to the target semantic to determine a target region; projecting the target region to the real-time point clouds; clustering the projected points in the real-time point clouds to obtain at least one point cloud cluster; respectively obtaining a confidence of each point cloud cluster; screening out the point cloud clusters with a confidence higher than a confidence threshold; taking the screened point cloud clusters as the detected point clouds from the real-time point clouds; when a point cloud corresponding to the target semantic is detected from the real-time point clouds and no point cloud corresponding to the target semantic is detected from the prior semantic map, generating a 3D detection box according to the detected point cloud from the real-time point clouds and updating the prior semantic map according to the 3D detection box; when a point cloud corresponding to the target semantic is detected from the real-time point clouds and a point cloud corresponding to the target semantic is detected from the prior semantic map, detecting a position change of the detected point cloud from the real-time point clouds and the detected point cloud from the prior semantic map, when the position change is detected, generating a 3D detection box according to the detected point cloud from the real-time point clouds and updating the prior semantic map according to the 3D detection box, and when the position change is not detected, keeping the prior semantic map unchanged.

2. The electronic map updating method according to claim 1, characterized by, the updating of the prior semantic map according to the real-time images comprises: registering the prior semantic map; determining a target semantic; detecting the real-time images and the registered prior semantic map according to the target semantic; when no part corresponding to the target semantic is detected from the real-time images and no point cloud corresponding to the target semantic is detected from the prior semantic map, keeping the prior semantic map unchanged; when no part corresponding to the target semantic is detected from the real-time images and a point cloud corresponding to the target semantic is detected from the prior semantic map, deleting the detected point cloud in the prior semantic map; in the second time period, obtaining real-time point clouds of the scene; performing three-dimensional target detection on the prior semantic map according to the target semantic to determine a bounding box; performing target detection on the real-time images according to the target semantic to determine a target region; projecting the target region to the real-time point clouds; clustering the projected points in the real-time point clouds to obtain at least one point cloud cluster; respectively obtaining a confidence of each point cloud cluster; screening out the point cloud clusters with a confidence higher than a confidence threshold; taking the screened point cloud clusters as the detected point clouds from the real-time point clouds; when a point cloud corresponding to the target semantic is detected from the real-time point clouds and no point cloud corresponding to the target semantic is detected from the prior semantic map, generating a 3D detection box according to the detected point cloud from the real-time point clouds and updating the prior semantic map according to the 3D detection box; when a point cloud corresponding to the target semantic is detected from the real-time point clouds and a point cloud corresponding to the target semantic is detected from the prior semantic map, detecting a position change of the detected point cloud from the real-time point clouds and the detected point cloud from the prior semantic map, when the position change is detected, generating a 3D detection box according to the detected point cloud from the real-time point clouds and updating the prior semantic map according to the 3D detection box, and when the position change is not detected, keeping the prior semantic map unchanged.

3. The electronic map updating method according to claim 2, characterized by, The semantic labeling and optimization of the region of interest on the original point cloud dataset according to the image information to obtain a prior semantic map further comprises: According to the semantic category, the local semantic space with the corresponding semantic label is cut from the original point cloud dataset to obtain the prior semantic map.

4. The electronic map updating method according to claim 1, characterized by, The confidence of each point cloud cluster is obtained respectively, comprising: For any point cloud cluster, the confidence is obtained according to the formula a calculation is performed, wherein denotes a confidence of the point cloud cluster, is the number of points within the point cloud cluster, is the average distance of the point cloud cluster to the center of a first detection component used for obtaining the raw point cloud data set, is the volume of the point cloud cluster, and is a weight parameter.

5. A computer apparatus, characterized in that, The electronic map updating method comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load the at least one program to execute the electronic map updating method according to any one of claims 1-4.

6. A computer readable storage medium having stored therein a program which is executable by a processor, characterized in that, The program executable by the processor is used to execute the electronic map updating method according to any one of claims 1-4 when executed by the processor.

Images